liuyuqi-dellpc
24df2046f9
Merge branch 'master' of https://git.yoqi.me/lyq/crawl_mrdx
|
7 months ago | |
|---|---|---|
| .vscode | 8 months ago | |
| bin | 1 year ago | |
| conf | 8 months ago | |
| crawl_mrdx | 7 months ago | |
| data | 4 years ago | |
| screenshot | 1 year ago | |
| shell | 8 months ago | |
| test | 4 years ago | |
| utils | 4 years ago | |
| .dockerignore | 8 months ago | |
| .gitignore | 8 months ago | |
| 506.png | 7 months ago | |
| Dockerfile | 8 months ago | |
| LICENSE | 1 year ago | |
| README.md | 7 months ago | |
| docker-compose.debug.yml | 8 months ago | |
| docker-compose.yml | 8 months ago | |
| main.py | 7 months ago | |
| main.spec | 8 months ago | |
| main.ui | 7 months ago | |
| requirements.txt | 8 months ago | |
| resources.qrc | 7 months ago |
README.md
新闻电讯爬虫
初步完成，单线程，文明爬虫（每次爬虫1-3s休息）。
cd my_project_dir
virtualenv -p /opt/python/3.8.5/bin/python3 .venv
source .venv/bin/activate
pip install -r requirements.txt
# method 1
python main.py --start 20230822 --end 20230823
# method 2，先配置 conf/config.json
python main.py
Ubuntu 打包：
pip install pyinstaller
pyinstaller -F -c main.py
docker 打包：
docker build -t jianboy/crawl_mrdx:v1.0.5 .
docker run -it --rm -v /data/crawl-mrdx:/app jianboy/crawl_mrdx:v1.0.5
截图
目前下载到 ./data/20130822/07.pdf ，2275天的资讯日报，总共16G。
History
2015年开始
python main.py --start 20210822 --end 20220822
python main.py --start 20220822 --end 20230418
之后不生成pdf版本了
License
Licensed under the Apache 2.0 © liuyuqi.gov@msn.cn