Scrapy中使用fake_useragent

pip install fake-useragent

middlewares.py:

from fake_useragent import UserAgent
ua = UserAgent()
from fake_useragent import UserAgent
class RandomUAMiddleware(object):
    def process_request(self,request,sipder):
        ua = UserAgent().random
        request.headers.setdefault('User-Agent',ua)

settints.py:

DOWNLOADER_MIDDLEWARES = {
'sina.middlewares.RandomUAMiddleware':1
   # 'sina.middlewares.SinaDownloaderMiddleware': 543,
}

更多访问地址：https://github.com/hellysmile/fake-useragent