欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识！
您现在的位置是： 首页  >  IT编程

Python使用request包请求网页乱码解决方法

程序员文章站 2024-01-26 20:49:40
使用requests请求网页时，返回的页面信息有时是乱码，如下代码 返回的部分内容； 解决方法和思路过程； 代码； 控制台输出（部分）； done。 ......

 使用requests请求网页时，返回的页面信息有时是乱码，如下代码

headers = {
    'user-agent': 'mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome/65.0.3325.181 safari/537.36'
}
def get_all(url,key):
    params = {
        'keyword':key,
        'enc':'utf-8'
    }
    response = requests.get(url=url,params=params,headers=headers)

    with open('jd.html','w',encoding='utf-8') as f:
        f.write(response.text)



if __name__ == '__main__':
    key = input('输入搜索内容：')
    url = 'https://search.jd.com/search?'
    get_all(url,key)

返回的部分内容；

<div class="p-name p-name-type-2">
            <a target="_blank" title="极地传说短袖t恤男夏季韩版潮流短袖男士半袖t恤圆领休闲修身大码五分袖潮牌青年学生t恤衣服男装 430黄色 m" href="//item.jd.com/51029271063.html" onclick="searchlog(1,51029271063,8,1,'','flagsclk=1077936264')">
                <em>极地传说短袖t恤<font class="skcolor_ljg">男</font>夏季韩版潮流短袖男士半袖t恤圆领休闲修身大码五分袖潮牌青年学生t恤衣服<font class="skcolor_ljg">男装</font> 430黄色 m</em>
                <i class="promo-words" id="j_ad_51029271063"></i>
            </a>
        </div>

解决方法和思路过程；

代码；

def get_all(url,key):
    params = {
        'keyword':key,
        'enc':'utf-8'
    }
    response = requests.get(url=url,params=params,headers=headers)
    # 打印出所请求页面返回的编码方式
    print(response.encoding)
    # response.apparent_encoding是通过内容分析出的编码方式，这里是urf-8
    print(response.apparent_encoding)
    # 转码
    content = response.text.encode(response.encoding).decode(response.apparent_encoding)
    print(content)
    with open('jd.html','w',encoding='utf-8') as f:
        f.write(content)



if __name__ == '__main__':
    key = input('输入搜索内容：')
    url = 'https://search.jd.com/search?'
    get_all(url,key)

控制台输出（部分）；

e:\anaconda\python.exe e:/练习/最后阶段/0808/jd.py
输入搜索内容：男装
iso-8859-1
utf-8
<!doctype html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="renderer" content="webkit">
<meta http-equiv="cache-control" content="max-age=300" />
<link rel="dns-prefetch" href="//search.jd.com" /><link rel="dns-prefetch" href="//item.jd.com" /><link rel="dns-prefetch" href="//list.jd.com" /><link rel="dns-prefetch" href="//p.3.cn" /><link rel="dns-prefetch" href="//misc.360buyimg.com" /><link rel="dns-prefetch" href="//nfa.jd.com" /><link rel="dns-prefetch" href="//d.jd.com" /><link rel="dns-prefetch" href="//img12.360buyimg.com" /><link rel="dns-prefetch" href="//img13.360buyimg.com" /><link rel="dns-prefetch" href="//static.360buyimg.com" /><link rel="dns-prefetch" href="//csc.jd.com" /><link rel="dns-prefetch" href="//mercury.jd.com" /><link rel="dns-prefetch" href="//x.jd.com" /><link rel="dns-prefetch" href="//wl.jd.com" /><title>男装 - 商品搜索 - 京东</title><meta name="keywords" content="男装,京东男装" /><meta name="description" content="在京东找到了260867件男装的类似商品，其中包含了“男装”等类型的男装的商品。" /><script>
window.loadfa_tojson_data={query:'%e7%94%b7%e8%a3%85'};
window.jdpts={};jdpts._st=new date().gettime();window.pageconfig={
    closejpg : 1,
    compatible: false,
    searchtype: 0,
    jdfversion: '2.0.0',
    floatnav: 1,
    price_pdos_off: 0,
    actname: '',
    psource: 'search_pc',
    queryparam: {
        c1: 0,
        c2: 1342,
        c3: 0,
        brand: '',
        price: '',
        keyword: '男装',
        page: '1'
    }
};
window.searchunit={
    resizeonebox: function(g,f,j){var g=parseint(g),i=typeof f,h=typeof j;if(!isnan(g)){if("string"==i&&f!=""&&g>0){$("#j_oneboxframe_"+f).css("height",g+10);h=="function"&&j()}else{if(i=="undefined"||i=="function"){$("#virtualwareiframe").css("height",g>0?g+10:0);i=="function"&&f()}}}},
    resizeshopbox: function(e,d){var f=0;switch(e){case 1:case 2:f=145;break;case 3:f=75;break;case 4:f=80;break;default:break}f&&$("#shopboxiframe").css("height",f).show();typeof(d)=="string"&&(new image().src=d)},
    coupon: {}};
window.query_keyword='男装';
window.real_keyword='男装';
</script>
<link type="text/css" rel="stylesheet" href="//misc.360buyimg.com/??jdf/1.0.0/unit/ui-base/5.0.0/ui-base.css,jdf/1.0.0/unit/shortcut/5.0.0/shortcut.css,jdf/1.0.0/unit/global-header/5.0.0/global-header.css,jdf/1.0.0/unit/myjd/5.0.0/myjd.css,jdf/1.0.0/unit/nav/5.0.0/nav.css,jdf/1.0.0/unit/shoppingcart/5.0.0/shoppingcart.css,jdf/1.0.0/unit/global-footer/5.0.0/global-footer.css,jdf/1.0.0/unit/service/5.0.0/service.css,jdf/1.0.0/unit/global-header-photo/5.0.0/global-header-photo.css,jdf/1.0.0/ui/area/1.0.0/area.css" />
<link type="text/css" rel="stylesheet" href="//misc.360buyimg.com/product/search/1.0.7/css/search.css" />
<script type="text/javascript" src="//misc.360buyimg.com/??jdf/1.0.0/unit/base/5.0.0/base.js,jdf/lib/jquery-1.6.4.js,product/module/es5-shim.js"></script>
<script>
window.search = {
    cid: 1349,
    ui_ver: '1.0.7',
    c_category: 1342,
    p_category: 0,
    enable_adv: 1,
    enable_prom_adwords: 1,
    enable_prom_flag: 1,
    enable_price: 1,
    enable_stock: 2,
    enable_yyk: 0,
    lottery_code: '',
    is_correct_hash: function(e){var a=["keyword","brand_id","activity_id","coupon_batch","ecard_id"];for(var c=0,b=a.length;c<b;c++){var d=new regexp("(^|\\?|&)"+a[c]+"=([^&]*)(\\s|&|$)");if(d.test(e)){return true}}return false},
    get_real_hash: function(){var a=window.location.hash.substr(1);if(a&&$.browser.mozilla){return location.href.substr(location.href.indexof("#")+1)}else{return a}}
};
(function(a,b){var c=b.get_real_hash();if(b.is_correct_hash(c)){a.location.href=a.location.pathname+"?"+c;return false}else{if(a.self!=a.top||$.browser.msie&&$.browser.version<=9){var f=null,e=function(){var d=$(a).width();return 1210>d?$("html").removeclass():$("html").removeclass().addclass(d>=1210&&1390>d?"resp01":"resp02"),true};e();$(a).resize(function(){cleartimeout(f),f=settimeout(e,20)})}}})(window,search);
</script>
</head>
<body>
<!--shortcut start-->
<div id="shortcut-2014">
    <div class="w">
        <ul class="fl">
            <li id="ttbar-home"><i class="iconfont">&#xe608;</i><a href="//www.jd.com/" target="_blank">京东首页</a></li>
            <li class="dorpdown" id="ttbar-mycity"></li>
        </ul>

done。

上一篇： Python3 - Bytes类型

下一篇： Python学习笔记之从文件中读取数据

推荐阅读