Python爬虫实战--爬取天气数据保存
程序员文章站
2022-05-02 16:59:26
...
个人总结的爬虫(爬取数据)的简单步骤:
1、获取待爬取网页的html信息
2、解析爬取的html信息,得到相关的数据
3、保存数据
# coding : UTF-8
import requests
import csv
import random
import time
import socket
import http.client
from bs4 import BeautifulSoup
def get_content(url, data=None):
header = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Connection': 'keep-alive',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70'
}
timeout = random.choice(range(80, 180))
while True:
try:
rep = requests.get(url, headers=header, timeout=timeout)
rep.encoding = 'utf-8'
break
# 异常处理
except socket.timeout as e:
print('3:', e)
time.sleep(random.choice(range(8, 15)))
except socket.error as e:
print('4:', e)
time.sleep(random.choice(range(20, 60)))
except http.client.BadStatusLine as e:
print('5:', e)
time.sleep(random.choice(range(30, 80)))
except http.client.IncompleteRead as e:
print('6:', e)
time.sleep(random.choice(range(5, 15)))
return rep.text
def get_data(html_text):
final = []
bs = BeautifulSoup(html_text, "html.parser")
body = bs.body
data = body.find('div', {'class': 'c7d'})
ul = data.find('ul')
li = ul.find_all('li')
for day in li:
temp = []
date = day.find('h1').string
temp.append(date)
inf = day.find_all('p')
temp.append(inf[0].string, )
if inf[1].find('span') is None:
temperature_highest = None
else:
temperature_highest = inf[1].find('span').string
temperature_highest = temperature_highest.replace('℃', '')
temperature_lowest = inf[1].find('i').string
temperature_lowest = temperature_lowest.replace('℃', '')
temp.append(temperature_highest)
temp.append(temperature_lowest)
wind = inf[2].find('i').string #获取风的级数
temp.append(wind)
final.append(temp)
return final
def write_data(data, name):
file_name = name
with open(file_name, 'a', errors='ignore', newline='') as f:
f_csv = csv.writer(f)
f_csv.writerows(data)
if __name__ == '__main__':
url = 'http://www.weather.com.cn/weather/101220802.shtml'
html = get_content(url)
result = get_data(html)
write_data(result, 'weather.csv')
上一篇: 猫眼电影爬取
推荐阅读
-
python爬虫实现爬取同一个网站的多页数据的实例讲解
-
Python爬虫实战:爬取腾讯视频的评论
-
Python爬虫——实战三:爬取苏宁易购的商品价格
-
Python3实现的爬虫爬取数据并存入mysql数据库操作示例
-
Python爬取数据保存为Json格式的代码示例
-
Python爬虫练习:爬取赶集网数据信息
-
python中urllib2与BeautifulSoup爬取数据保存MongoDB
-
Python爬虫之自动爬取某车之家各车销售数据
-
ASP.NET网络爬虫小研究 HtmlAgilityPack基础,爬取数据保存在数据库中再显示再自己的网页中
-
神箭手云爬虫-爬取携程【国际】航班/机票信息-利用python解析返回的json文件将信息存储进Mysql数据库