crabapples
/
learn-spider
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

debug(spider): 修改爬虫调试配置并更新安装脚本 

- 将浏览器启动模式从无头模式改为有头模式以便调试
- 在预安装脚本中添加虚拟环境激活命令
- 重新排列安装脚本中的依赖安装顺序
This commit is contained in:
mshe 2026-05-27 15:41:50 +08:00
parent 3c31ad240d
commit 3fcc2d3a36
2 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
pre_install.sh
View File

@ -1,5 +1,6 @@
apt install -y python3.12-dev python3.12-venv apt install -y python3.12-dev python3.12-venv
python3 -m venv .venv --system-site-packages python3 -m venv .venv --system-site-packages
source .venv/bin/activate
.venv/bin/python3.12 -m pip install -r requirements.txt .venv/bin/python3.12 -m pip install -r requirements.txt
playwright install playwright install
playwright install-deps playwright install-deps

2
spider/china_net.py
View File

@ -4,7 +4,7 @@ from bs4 import BeautifulSoup
def crawl_vue_app(): def crawl_vue_app():
with sync_playwright() as p: with sync_playwright() as p:
# 启动浏览器（headless=False 可以看到浏览器窗口） # 启动浏览器（headless=False 可以看到浏览器窗口）
browser = p.chromium.launch(headless=True) browser = p.chromium.launch(headless=False)
page = browser.new_page() page = browser.new_page()
# 访问页面 # 访问页面
page.goto('https://caigou.chinatelecom.com.cn/') page.goto('https://caigou.chinatelecom.com.cn/')