Scrapy-redis分布式爬虫爬取豆瓣电影详情页

平时爬虫一般都使用scrapy框架，通常都是在一台机器上跑，爬取速度也不能达到预期效果，数据量小，而且很容易就会被封禁ip或者账号，这时候可以使用代理ip或者登录方式爬，然而代理ip很多时候都很鸡肋，除非使用付费版ip，但是和真实ip差别很大。这时候便有了scrapy-redis分布式爬虫框架，它基于scrapy改造，把scrapy的调度器（scheduler）换成了scrapy-redis的调度器，可以轻松达到目的，利用多台服务器来爬取数据，而且还可以自动去重，效率高。爬取的数据默认保存在redis缓存中，速度很快。

scrapy工作原理：

scrapy-redis工作原理：

中间的就是调度器

豆瓣电影简易分布式爬虫

我这里直接使用start_urls的方式，数据存入到mysql中

class doubanspider(redisspider):
    name = 'douban'
    redis_key = 'douban:start_urls'
    allowed_domains = ['douban.com']


    def start_requests(self):
        urls = get_urls()
        for url in urls:
            yield scrapy.request(url=url, callback=self.parse)


    def parse(self, response):
        # item_loader = movieitemloader(item=movieitem, response=response)
        #
        # item_loader.add_xpath('title', '')
        item = movieitem()
        print(response.url)
        item['movieid'] = int(response.url.split('subject/')[1].replace('/', ''))
        item['title'] = response.xpath('//h1/span/text()').extract()[0]
        item['year'] = response.xpath('//h1/span/text()').extract()[1].split('(')[1].split(')')[0] or '2019'
        item['url'] = response.url
        item['cover'] = response.xpath('//a[@class="nbgnbg"]/img/@src').extract()[0]
        try:
            item['director'] = response.xpath('//a[@rel="v:directedby"]/text()').extract()[0] or '无'
        except exception:
            item['director'] = '暂无'
        item['major'] = '/'.join(response.xpath('//a[@rel="v:starring"]/text()').extract())
        item['category'] = ','.join(response.xpath('//span[@property="v:genre"]/text()').extract())


        item['time'] = ','.join(response.xpath('//span[@property="v:initialreleasedate"]/text()').extract())
        try:
            item['duration'] = response.xpath('//span[@property="v:runtime"]/text()').extract()[0]
        except exception:
            item['duration'] = '暂无'

        item['score'] = response.xpath('//strong[@property="v:average"]/text()').extract()[0]
        item['comment_nums'] = response.xpath('//span[@property="v:votes"]/text()').extract()[0] or 0
        item['desc'] = response.xpath('//span[@property="v:summary"]/text()').extract()[0].strip()

        actor_list = response.xpath('//ul[@class="celebrities-list from-subject __oneline"]/li/a/@title').extract()
        actor_img_list = response.xpath('//ul[@class="celebrities-list from-subject __oneline"]/li/a/div/@style').extract()
        actor_img_list = [i.split('url(')[1].replace(')', '') for i in actor_img_list]

        item['actor_name_list'] = '----'.join(actor_list)
        item['actor_img_list'] = '----'.join(actor_img_list)


        yield item

settings.py文件

bot_name = 'moviespider'

spider_modules = ['moviespider.spiders']
newspider_module = 'moviespider.spiders'

# redis_host = '127.0.0.1'
# redis_port = 6379
redis_url = 'redis://username:[e-mail]:6379'


# obey robots.txt rules
robotstxt_obey = false
scheduler = "scrapy_redis.scheduler.scheduler"

# ensure all spiders share same duplicates filter through redis.
dupefilter_class = "scrapy_redis.dupefilter.rfpdupefilter"
item_pipelines = {
    'scrapy_redis.pipelines.redispipeline': 300,
    'moviespider.pipelines.mysqlpipeline': 200,
}

这里只是为了多台服务器一起爬取，没有手动在redis中推入起始的url

此时将爬虫项目上传到其他服务器上，一起开始

效果如下：