欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

Scrapy中如何向Spider传入参数

程序员文章站 2022-05-07 12:44:15
...
# -*- coding: utf-8 -*-
import scrapy

class TiebaSpider(scrapy.Spider):
    name = 'tieba'  # 贴吧爬虫
    allowed_domains = ['tieba.baidu.com']  # 允许爬取的范围
    start_urls = []  # 爬虫起始地址

    # 命令格式: scrapy crawl tieba -a tiebaName=放置奇兵 -a pn=250
    def __init__(self, tiebaName=None, pn=None, *args, **kwargs):
        print('< 贴吧名称 >: ' + tiebaName)
        super(eval(self.__class__.__name__), self).__init__(*args, **kwargs)
        self.start_urls = ['https://tieba.baidu.com/f?kw=%s&ie=utf-8&pn=%s' % (tiebaName,pn)]

    def parse(self, response):
        print(response.request.url) # 结果:https://tieba.baidu.com/f?kw=%E6%94%BE%E7%BD%AE%E5%A5%87%E5%85%B5&ie=utf-8&pn=250

参考文章:

https://blog.csdn.net/c0411034/article/details/81750028 

https://blog.csdn.net/qq_24760381/article/details/80361400 

https://blog.csdn.net/qq_38282706/article/details/80991196