ubuntu18.04 搭建scrapy环境（连环踩坑+解决办法）

---恢复内容开始---

预期需求：

打算搭建scrapy环境，基于python3.x的

 

环境描述：

ubuntu18.04自带了python3.6，打算在虚拟环境vlenv中跑scrapy，装好虚拟环境后，直接pip3 install scrapy命令走起，然后，跑错了

@ubuntu:~$ pip3 install scrapy
collecting scrapy
downloading https://files.pythonhosted.org/packages/29/4b/585e8e111ffb01466c59281f34febb13ad1a95d7fb3919fd57c33fc732a5/scrapy-1.7.3-py2.py3-none-any.whl (234kb)
100% |████████████████████████████████| 235kb 23kb/s
collecting pyopenssl (from scrapy)
downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyopenssl-19.0.0-py2.py3-none-any.whl (53kb)
100% |████████████████████████████████| 61kb 18kb/s
collecting twisted>=13.1.0; python_version != "3.4" (from scrapy)

。。。

downloading https://files.pythonhosted.org/packages/be/70/e5ea8afd6d08a4b99ebfc77bd1845248d56cfcf43d11f9dc324b9580a35c/pyasn1_modules-0.2.6-py2.py3-none-any.whl (95kb)
100% |████████████████████████████████| 102kb 29kb/s
collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyopenssl->scrapy)
could not find a version that satisfies the requirement cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyopenssl->scrapy) (from versions: )
no matching distribution found for cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyopenssl->scrapy)

 

安装失败！似乎python版本不对？

@ubuntu:~$ which python
/usr/bin/python
@ubuntu:~$ which python3
/usr/bin/python3
@ubuntu:~$ which python3.6
/usr/bin/python3.6

上面是环境中的python版本，去scrapy官网查了一下，好像也有提到支持版本2.7和3.4的，

然后开始百度安装python3.4的办法。按照下面的文章的方法安装成功了。感谢这个作者！

下载源码直接编译确实靠谱！哈哈

 

下面是装完的python3.4的路径

@ubuntu:~/python-3.4.10$ which python3.4
/usr/local/bin/python3.4

 

安装完又用update-alternatives来控制python的版本，可以自定义python默认使用版本。这个方法也是上面链接里提供的，确实好用！

@ubuntu:~/python-3.4.10$ sudo update-alternatives --config python
有 3 个候选项可用于替换 python (提供 /usr/bin/python)。

选择 路径 优先级 状态
------------------------------------------------------------
* 0 /usr/local/bin/python3.4 3 自动模式
1 /usr/bin/python2.7 1 手动模式
2 /usr/bin/python3.6 2 手动模式
3 /usr/local/bin/python3.4 3 手动模式

要维持当前值[*]请按<回车键>，或者键入选择的编号：
leiting@ubuntu:~/python-3.4.10$
leiting@ubuntu:~/python-3.4.10$
leiting@ubuntu:~/python-3.4.10$ python
python 3.4.10 (default, aug 20 2019, 21:31:05)
[gcc 7.4.0] on linux
type "help", "copyright", "credits" or "license" for more information.
>>>
@ubuntu:~/python-3.4.10$ python3
python 3.6.8 (default, jan 14 2019, 11:02:34)
[gcc 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
type "help", "copyright", "credits" or "license" for more information.
>>>
@ubuntu:~/python-3.4.10$ python2
python 2.7.15+ (default, nov 27 2018, 23:36:35)
[gcc 7.3.0] on linux2
type "help", "copyright", "credits" or "license" for more information.
>>>
@ubuntu:~/python-3.4.10$ ipython3
python 3.6.8 (default, jan 14 2019, 11:02:34)
type "copyright", "credits" or "license" for more information.

ipython 5.5.0 -- an enhanced interactive python.
? -> introduction and overview of ipython's features.
%quickref -> quick reference.
help -> python's own help system.
object? -> details about 'object', use 'object??' for extra details.

in [1]:
do you really want to exit ([y]/n)? y
@ubuntu:~/python-3.4.10$

 

创建虚拟环境，并指定python3.4版本，pip安装scrapy，报错...提示pip版本低，虚拟环境升级pip没成功，又把系统环境的pip升级了。从9.0升级到19.1，虚拟环境还是pip安装不了scrapy，有点崩溃了

@ubuntu:~/python-3.4.10$ pip3 list|grep pip
deprecation: the default format will switch to columns in the future. you can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)

@ubuntu:~/python-3.4.10$ pip3 install pip -u
collecting pip
cache entry deserialization failed, entry ignored
downloading https://files.pythonhosted.org/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl (1.4mb)
100% |████████████████████████████████| 1.4mb 48kb/s
installing collected packages: pip
successfully installed pip-19.2.2
leiting@ubuntu:~/python-3.4.10$

 

升级完pip，好像pip3就命令就没了，全是pip，原因不明

 

升级完pip，创建新虚拟环境，还是pip安装不起scrapy，烦~后面想，还是不在虚拟环境安装scrapy，不然每个环境都要按照一次scrapy，还是麻烦，打算还是在默认环境安装scrapy，每个虚拟环境创建时，都带上默认环境的包，就可以了，命令用下面这个--system-site-packages

 

继续默认环境装scrapy，还是pip install scrapy，走起，然后报错了

@ubuntu:~/python-3.4.10$ pip install scrapy
collecting scrapy

。。。

error: pyopenssl 19.0.0 has requirement cryptography>=2.3, but you'll have cryptography 2.1.4 which is incompatible.
installing collected packages: cssselect, pyasn1, attrs, pyasn1-modules, service-identity, queuelib, pyopenssl, incremental, automat, zope.interface, constantly, pyhamcrest, hyperlink, twisted, pydispatcher, w3lib, parsel, scrapy
error: could not install packages due to an environmenterror: [errno 13] 权限不够: '/usr/local/lib/python3.6/dist-packages/cssselect'
consider using the `--user` option or check the permissions.

 

真的快崩溃了！！！

百度找了一圈办法，用下面的命令，问题解决！

pip install --user scrapy

加上--user参数

 

@ubuntu:~/python-3.4.10$ scrapy
scrapy 1.7.3 - no active project

usage:
scrapy <command> [options] [args]

available commands:
bench run quick benchmark test
fetch fetch a url using the scrapy downloader
genspider generate new spider using pre-defined templates
runspider run a self-contained spider (without creating a project)
settings get settings values
shell interactive scraping console
startproject create new project
version print scrapy version
view open url in browser, as seen by scrapy

[ more ] more commands available when run from project directory

use "scrapy <command> -h" to see more info about a command

真不容易~~~~

 

总结：

1、用源码安装python各个版本，用update-alternatives来控制python的版本，可以自定义python默认使用版本

2、pip3 install pip -u，这个-u，意思把pip包升级到最新版本

3、pip install报错提示consider using the `--user` option or check the permissions.，就加一个--user参数，有奇效！