liuyuqi-dellpc
20788980df
0
|
1 year ago | |
|---|---|---|
| bin | 2 years ago | |
| conf | 2 years ago | |
| crawl_mrdx | 1 year ago | |
| data | 5 years ago | |
| screenshot | 2 years ago | |
| shell | 5 years ago | |
| test | 5 years ago | |
| utils | 5 years ago | |
| .gitignore | 2 years ago | |
| LICENSE | 2 years ago | |
| README.md | 1 year ago | |
| main.py | 1 year ago | |
| requirements.txt | 5 years ago |
README.md
新闻电讯爬虫
初步完成，单线程，文明爬虫（每次爬虫1-3s休息）。
cd my_project_dir
virtualenv -p /opt/python/bin/python3 venv
source venv/bin/activate
pip install -r requirements.txt
python main.py --start 20210525 --end 20210525
截图
目前下载到 ./data/20130822/07.pdf ，2275天的资讯日报，总共16G。