fake-useragent 的使用
程序员文章站
2022-03-02 23:06:19
...
fake-useragent文档:
https://fake-useragent.readthedocs.io/en/latest/
在爬虫中进行request请求,很多时候,都需要添加请求头,不然服务器会认为是非法的请求,从而拒绝你的访问。,在添加请求头中最常用的就是添加user-agent来讲本次请求伪装成浏览器。
User Agent 中文名为用户代理,简称 UA,它是一个特殊字符串头,使得服务器能够识别客户使用的操作系统及版本、CPU 类型、浏览器及版本、浏览器渲染引擎、浏览器语言、浏览器插件等。
那么个人怎么搞这个user-agent呢,没错?八仙过海各显神通,但一般都是用手来解决个人问题!
url = 'https://www.zhihu.com/question/315387406/answer/812734512'
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
}
但,自从有了fake_useragent。。。。
from fake_useragent import UserAgent
# 实例化 user-agent 对象
ua = UserAgent()
url = 'https://www.zhihu.com/question/315387406/answer/812734512'
headers = {"user-agent": ua.chrome} # 指定浏览器 user-agent
# 或者可以这样写
# headers = {"user-agent": UserAgent().random} # 一步到位,随机生成一个 user-agent
response = requests.get(url=url, headers=headers)
print(response.status_code) # 200
install
pip install fake_useragent
update
pip install -U fake-useragent
查看版本
import fake_useragent
print(fake_useragent.VERSION)
生成指定浏览器的user-agent
import fake_useragent
# 实例化 user-agent 对象
ua = fake_useragent.UserAgent()
# ua.ie
print(ua.ie) # Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; chromeframe/13.0.782.215)
# ua.msie
print(ua['Internet Explorer']) # Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)
# ua.opera
print(ua.opera) # Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01
# ua.chrome
print(ua.chrome) # Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.16 Safari/537.36
# ua.google
print(ua['google chrome']) # Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36
# ua.firefox
print(ua.firefox) # Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:16.0.1) Gecko/20121011 Firefox/21.0.1
# ua.ff
print(ua.ff) # Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/29.0
# ua.safari
print(ua.safari) # Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
随机生成user-agent
import fake_useragent
# 实例化 user-agent 对象
ua = fake_useragent.UserAgent()
print(ua.random) # Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36
print(ua.random) # Mozilla/5.0 (compatible; MSIE 10.0; Macintosh; Intel Mac OS X 10_7_3; Trident/6.0)
print(ua.random) # Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36
每次都能随机生成一个UA表示,大大增强了爬虫的真实性。
上一篇: userAgent