[python爬虫之路day7]：实战之中国天气网全国城市天气情况爬取
通过今天的学习，我们将中国天气网的所有城市天气信息按照最低温度的排序爬取出来，并将排名前10的城市可视化。
通过本次学习又温习了以下：
1.sort函数，可以排序，但是数据必须是整型数据，
2.pyecharts的Bar库，可以进行绘制表格。
代码如下：
import requests
from bs4 import BeautifulSoup
from pyecharts import Bar
ALL_DATA = []
def parse_pade(url):
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36"}
response = requests.get(url, headers=headers)
text = response.content.decode('utf-8')
soup = BeautifulSoup(text, "html5lib")
conMidtab = soup.find('div', class_="conMidtab")
tables = conMidtab.find_all("table")
for table in tables:
trs = table.find_all("tr")[2:]
for index, tr in enumerate(trs):
tds = tr.find_all("td")
city_td = tds[-8]
weather_td = tds[-7]
city = list(city_td.stripped_strings)[0]
weather = list(weather_td.stripped_strings)[0]
low_td = tds[-2]
low = list(low_td.stripped_strings)[0]
high_td = tds[-5]
high = list(high_td.stripped_strings)[0]
ALL_DATA.append({"city": city, "weather": weather, "low_feel": int(low), "high_feel": high})
def main():
urls = ["http://www.weather.com.cn/textFC/hb.shtml#", "http://www.weather.com.cn/textFC/db.shtml#",
"http://www.weather.com.cn/textFC/hd.shtml#", "http://www.weather.com.cn/textFC/hz.shtml#",
"http://www.weather.com.cn/textFC/hn.shtml#", "http://www.weather.com.cn/textFC/xb.shtml#",
"http://www.weather.com.cn/textFC/xn.shtml#", "http://www.weather.com.cn/textFC/gat.shtml#"]
for url in urls:
parse_pade(url)
# 按最低气温排序
ALL_DATA.sort(key=lambda data: data['low_feel'])
print(ALL_DATA)
# 前十城市可视化
chart = Bar("中国最低气温排行榜")
data = ALL_DATA[0:10]
cities = list(map(lambda x: x['city'], data))
low_feels = list(map(lambda x: x['low_feel'], data))
chart.add(" ", cities, low_feels)
chart.render("weather.html")
if __name__ == '__main__':
main()
运行结果：
C:\python38\python.exe “C:/python38/new project/2rd/day7.py”
[{‘city’: ‘大兴安岭’, ‘weather’: ‘晴’, ‘low_feel’: -27, ‘high_feel’: ‘-6’}, {‘city’: ‘呼伦贝尔’, ‘weather’: ‘晴’, ‘low_feel’: -24, ‘high_feel’: ‘-11’}, {‘city’: ‘阿勒泰’, ‘weather’: ‘小雪’, ‘low_feel’: -23, ‘high_feel’: ‘-5’}, {‘city’: ‘北屯’, ‘weather’: ‘小雪’, ‘low_feel’: -23, ‘high_feel’: ‘-5’}, {‘city’: ‘双河’, ‘weather’: ‘小雪’, ‘low_feel’: -23, ‘high_feel’: ‘-5’}, {‘city’: ‘佳木斯’, ‘weather’: ‘多云’, ‘low_feel’: -22, ‘high_feel’: ‘-2’}, {‘city’: ‘黑河’, ‘weather’: ‘多云’, ‘low_feel’: -22, ‘high_feel’: ‘-8’}, {‘city’: ‘伊春’, ‘weather’: ‘晴’, ‘low_feel’: -22, ‘high_feel’: ‘-4’}, {‘city’: ‘齐齐哈尔’, ‘weather’: ‘晴’, ‘low_feel’: -21, ‘high_feel’: ‘-5’}, {‘city’: ‘锡林郭勒’, ‘weather’: ‘晴’, ‘low_feel’: -20, ‘high_feel’: ‘-7’}, {‘city’: ‘鹤岗’, ‘weather’: ‘多云’, ‘low_feel’: -20, ‘high_feel’: ‘-3’}, {‘city’: ‘白山’, ‘weather’: ‘晴’, ‘low_feel’: -19, ‘high_feel’: ‘-1’}, {‘city’: ‘哈尔滨’, ‘weather’: ‘晴’, ‘low_feel’: -18, ‘high_feel’: ‘-3’}, 删除部分结果**{‘city’: ‘七台河’, ‘weather’: ‘多云’, ‘low_feel’: -18, ‘high_feel’: ‘-4’}, {‘city’: ‘海北’, ‘weather’: ‘多云’, ‘low_feel’: -18, ‘high_feel’: ‘3’}, {‘city’: ‘巴彦淖尔’, ‘weather’: ‘晴’, ‘low_feel’: -6, ‘high_feel’: ‘5’}, {‘city’: ‘葫芦岛’, ‘weather’: ‘多云’, ‘low_feel’: -6, ‘high_feel’: ‘5’}, {‘city’: ‘延安’, ‘weather’: ‘晴’, ‘low_feel’: -6, ‘high_feel’: ‘10’}, {‘city’: ‘榆林’, ‘weather’: ‘晴’, ‘low_feel’: -6, ‘high_feel’: ‘7’}, {‘city’: ‘荣昌’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘14’}, {‘city’: ‘铜梁’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘14’}, {‘city’: ‘璧山’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘14’}, {‘city’: ‘丰都’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘13’}, {‘city’: ‘武隆’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘12’}, {‘city’: ‘綦江’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘12’}, {‘city’: ‘遵义’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘13’}, {‘city’: ‘曲靖’, ‘weather’: ‘多云’, ‘low_feel’: 6, ‘high_feel’: ‘18’}, {‘city’: ‘楚雄’, ‘weather’: ‘晴’, ‘low_feel’: 6, ‘high_feel’: ‘19’}, {‘city’: ‘上海’, ‘weather’: ‘多云’, ‘low_feel’: 7, ‘high_feel’: ‘11’}, {‘city’: ‘黄浦’, ‘weather’: ‘多云’, ‘low_feel’: 7, ‘high_feel’: ‘11’},{‘city’: ‘贵港’, ‘weather’: ‘阴’, ‘low_feel’: 13, ‘high_feel’: ‘19’}, {‘city’: ‘防城港’, ‘weather’: ‘阴’, ‘low_feel’: 13, ‘high_feel’: ‘18’}, {‘city’: ‘深圳’, ‘weather’: ‘晴’, ‘low_feel’: 13, ‘high_feel’: ‘20’}, {‘city’: ‘珠海’, ‘weather’: ‘晴’, ‘low_feel’: 13, ‘high_feel’: ‘18’}, {‘city’: ‘西双版纳’, ‘weather’: ‘晴’, ‘low_feel’: 13, ‘high_feel’: ‘26’}, {‘city’: ‘香港’, ‘weather’: ‘多云’, ‘low_feel’: 13, ‘high_feel’: ‘19’}, {‘city’: ‘澳门’, ‘weather’: ‘晴’, ‘low_feel’: 13, ‘high_feel’: ‘18’}, {‘city’: ‘钦州’, ‘weather’: ‘阴’, ‘low_feel’: 14, ‘high_feel’: ‘20’}, {‘city’: ‘儋州’, ‘weather’: ‘多云’, ‘low_feel’: 14, ‘high_feel’: ‘23’}, {‘city’: ‘白沙’, ‘weather’: ‘多云’, ‘low_feel’: 14, ‘high_feel’: ‘23’}, {‘city’: ‘五指山’, ‘weather’: ‘多云’, ‘low_feel’: 14, ‘high_feel’: ‘22’}, {‘city’: ‘台北’, ‘weather’: ‘多云’, ‘low_feel’: 14, ‘high_feel’: ‘19’}, {‘city’: ‘北海’, ‘weather’: ‘多云’, ‘low_feel’: 15, ‘high_feel’: ‘20’}, {‘city’: ‘茂名’, ‘weather’: ‘多云’, ‘low_feel’: 15, ‘high_feel’: ‘23’}, {‘city’: ‘澄迈’, ‘weather’: ‘多云’, ‘low_feel’: 15, ‘high_feel’: ‘22’}, {‘city’: ‘海口’, ‘weather’: ‘多云’, ‘low_feel’: 17, ‘high_feel’: ‘20’}, {‘city’: ‘昌江’, ‘weather’: ‘多云’, ‘low_feel’: 17, ‘high_feel’: ‘24’}, {‘city’: ‘琼海’, ‘weather’: ‘多云’, ‘low_feel’: 17, ‘high_feel’: ‘22’}, {‘city’: ‘保亭’, ‘weather’: ‘多云’, ‘low_feel’: 17, ‘high_feel’: ‘25’}, {‘city’: ‘万宁’, ‘weather’: ‘小雨’, ‘low_feel’: 17, ‘high_feel’: ‘23’}, {‘city’: ‘*’, ‘weather’: ‘多云’, ‘low_feel’: 17, ‘high_feel’: ‘21’}, {‘city’: ‘三亚’, ‘weather’: ‘多云’, ‘low_feel’: 18, ‘high_feel’: ‘25’}, {‘city’: ‘东方’, ‘weather’: ‘多云’, ‘low_feel’: 18, ‘high_feel’: ‘24’}, {‘city’: ‘文昌’, ‘weather’: ‘多云’, ‘low_feel’: 18, ‘high_feel’: ‘22’}, {‘city’: ‘陵水’, ‘weather’: ‘多云’, ‘low_feel’: 18, ‘high_feel’: ‘23’}, {‘city’: ‘西沙’, ‘weather’: ‘多云’, ‘low_feel’: 22, ‘high_feel’: ‘27’}, {‘city’: ‘中沙’, ‘weather’: ‘小雨’, ‘low_feel’: 24, ‘high_feel’: ‘29’}, {‘city’: ‘南沙’, ‘weather’: ‘小雨’, ‘low_feel’: 25, ‘high_feel’: ‘29’}]
Process finished with exit code 0
可视化结果如下：
本人将持续更新自己的python爬虫之路，欢迎小伙伴关注,大家共同进步。