Scrapy案例02-腾讯招聘信息爬取
程序员文章站
2023-10-11 21:52:40
[TOC] 1. 目标 目标:https://hr.tencent.com/position.php?&start=0 a 爬取所有的职位信息信息 职位名 职位url 职位类型 职位人数 工作地点 发布时间 2. 网站结构分析 3. 编写爬虫程序 3.1. 配置需要爬取的目标变量 3.2. 写爬虫文 ......
目录
1. 目标
目标:https://hr.tencent.com/position.php?&start=0#a
爬取所有的职位信息信息
- 职位名
- 职位url
- 职位类型
- 职位人数
- 工作地点
- 发布时间
2. 网站结构分析
3. 编写爬虫程序
3.1. 配置需要爬取的目标变量
class tecentjobitem(scrapy.item): # define the fields for your item here like: positionname = scrapy.field() positionlink = scrapy.field() positiontype = scrapy.field() peoplenum = scrapy.field() worklocation = scrapy.field() publishtime = scrapy.field()
3.2. 写爬虫文件scrapy
# -*- coding: utf-8 -*- import scrapy from tecentjob.items import tecentjobitem class tencentspider(scrapy.spider): name = 'tencent' allowed_domains = ['tencent.com'] url = 'https://hr.tencent.com/position.php?&start=' offset = 0 start_urls = [url + str(offset)] def parse(self, response): for each in response.xpath("//tr[@class = 'even'] | //tr[@class = 'odd']"): # 初始化模型对象 item = tecentjobitem() item['positionname'] = each.xpath("./td[1]/a/text()").extract()[0] item['positionlink'] = each.xpath("./td[1]/a/@href").extract()[0] item['positiontype'] = each.xpath("./td[2]/text()").extract()[0] item['peoplenum'] = each.xpath("./td[3]/text()").extract()[0] item['worklocation'] = each.xpath("./td[4]/text()").extract()[0] item['publishtime'] = each.xpath("./td[5]/text()").extract()[0] yield item if self.offset < 100: self.offset += 10 # 将请求重写发送给调度器入队列、出队列、交给下载器下载 # 拼接新的rurl,并回调parse函数处理response # yield scrapy.request(url, callback = self.parse) yield scrapy.request(self.url + str(self.offset), callback=self.parse)
3.3. 编写yield需要的管道文件
import json class tecentjobpipeline(object): def __init__(self): self.filename = open("tencent.json", 'wb') def process_item(self, item, spider): text = json.dumps(dict(item),ensure_ascii=false) + "\n" self.filename.write(text.encode('utf-8')) return item def close_spider(self, spider): self.filename.close()
3.4. setting中配置请求抱头信息
default_request_headers = { 'user-agent': 'mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/71.0.3578.98 safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'accept-language': 'en', }
4. 最后结果
上一篇: oracle查看当前用户信息
下一篇: 王峻是什么人?后周大臣王峻生平简介