在linux下安装并运行scrapyd
程序员文章站
2022-03-25 22:02:51
系统:centos7.4 安装scrapyd:pip isntall scrapyd 因为我腾讯云上是python2与python3并存的 所以我执行的命令是:pip3 isntall scrapyd 安装后新建一个配置文件: sudo mkdir /etc/scrapyd sudo vim /et ......
系统:centos7.4
安装scrapyd:pip isntall scrapyd
因为我腾讯云上是python2与python3并存的 所以我执行的命令是:pip3 isntall scrapyd
安装后新建一个配置文件:
sudo mkdir /etc/scrapyd
sudo vim /etc/scrapyd/scrapyd.conf
写入如下内容:(给内容在https://scrapyd.readthedocs.io/en/stable/config.html可找到)
[scrapyd] eggs_dir = eggs logs_dir = logs items_dir = jobs_to_keep = 5 dbs_dir = dbs max_proc = 0 max_proc_per_cpu = 10 finished_to_keep = 100 poll_interval = 5.0 bind_address = 0.0.0.0 http_port = 6800 debug = off runner = scrapyd.runner application = scrapyd.app.application launcher = scrapyd.launcher.launcher webroot = scrapyd.website.root [services] schedule.json = scrapyd.webservice.schedule cancel.json = scrapyd.webservice.cancel addversion.json = scrapyd.webservice.addversion listprojects.json = scrapyd.webservice.listprojects listversions.json = scrapyd.webservice.listversions listspiders.json = scrapyd.webservice.listspiders delproject.json = scrapyd.webservice.deleteproject delversion.json = scrapyd.webservice.deleteversion listjobs.json = scrapyd.webservice.listjobs daemonstatus.json = scrapyd.webservice.daemonstatus
主要更改bind_address=0.0.0.0
创建文件后执行命令启动scrapyd: (scrapyd > /dev/null &) 当想要记录输出日志时: (scrapyd > /root/scrapyd.log &)
坑1:当我执行完命令后报错,说是找不到命令:
那是因为我系统上python2与3并存,所以找不到,这时应该做软连接:
我的python3路径: /usr/local/python3
制作软连接: ln -s /usr/local/python3/bin/scrapy /usr/bin/scrapy
昨晚软连接后,执行上边命令,又报错:
坑2:
这个好像是那个配置文件的最后一行有问题,具体原因不大清楚,我将最后一行删除,再次重新执行,scrapyd就跑起来了
推荐阅读