用python爬取网站的数据

导入库

需要的库： requests、 bs4 、 fake_useragent
pip方法安装
requests.get 爬取源代码
BeautifulSoup 整理代码
UserAgent 访问设置

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent

输入要爬取的网站

url = 'https://movie.douban.com/chart'

访问设置

ua = UserAgent()
print(ua.chrome)
headers = {'User-Agent': ua.chrome}

爬取——整理——缩小爬取范围

request = requests.get(url, headers = headers)
soup = BeautifulSoup(request.text)
soup = soup.find(id = 'wrapper').find(id = 'content').find(class_ ="indent").find_all(style = 'font-size:13px;')

定义列表——for循环有选择性输出列表（遍历？）

a = [0,0,0,0,0,0,0,0,0,0]
for i in range(9):
    print( soup[i].contents)
    a[i] = soup[i].contents
print(a)

结果展示

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent

url = 'https://movie.douban.com/chart'
ua = UserAgent()
print(ua.chrome)
headers = {'User-Agent': ua.chrome}
request = requests.get(url, headers = headers)
soup = BeautifulSoup(request.text)
soup = soup.find(id = 'wrapper').find(id = 'content').find(class_ ="indent").find_all(style = 'font-size:13px;')
#print(soup[0].contents)

a = [0,0,0,0,0,0,0,0,0,0]
for i in range(9):
    print( soup[i].contents)
    a[i] = soup[i].contents
print(a)

运行结果如下：

显示Response 418的解决方法

<Response [418]>

r = requests.get（url,headers=header）