利用定向网络爬虫爬取豆瓣电影top250【Python】

最近在外地实习，闲来无事学了一下requests库和BeautifulSoup，掌握基本用法之后试着爬取了一下豆瓣电影top250，中间也参考了不少其他大佬的博客，所以最后写出来的代码也都大同小异吧，就当聊以自慰了。

简介

利用requests库和bs4中的BeautifulSoup，实现对豆瓣电影top250的爬取，最后将电影信息写入一个文本文件中。

代码如下

'''
@Author     : Guo Yingwei
@Date       : 2019-07-06 18:10:29
@E-mail     : [email protected]
@Description: craw douban Movies Top250 ,
              and write the information to a file. 
'''

import requests
from bs4 import BeautifulSoup
import codecs


def get_page(url):
    try:
        r = requests.get(url)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return ''
    
def parse_html(html, return_list):
    soup = BeautifulSoup(html, 'html.parser')    
    movie_list = soup.find('ol', attrs={'class': 'grid_view'}).find_all('li')
    for movie in movie_list:
        title = movie.find('span', attrs={'class': 'title'}).get_text()
        rating_num = movie.find('span', attrs={'class': 'rating_num'}).get_text()
        inq = movie.find('span', attrs={'class': 'inq'}).get_text()
        return_list.append([title, rating_num, inq])
    return return_list

def write_file(return_list):   
    with codecs.open('doubanMoviesTop250.txt','wb+', encoding='utf-8') as f:
        tplt = u'{0:<4}\t{1:{4}^16}\t{2:{4}^10}\t{3:<20}'
        f.write(tplt.format('排名', '名称', '评分', '一句话简介', chr(12288)) + '\n')
        count = 0
        for i in return_list:
            count += 1
            f.write(tplt.format(count, i[0], i[1],i[2], chr(12288)) + '\n')
                   
def main():
    start_url='https://movie.douban.com/top250'
    movie_list = []    
    for i in range(10):
        html = get_page(start_url + '?start=' + str(25*i))
        parse_html(html, movie_list)
    write_file(movie_list)    

if __name__ == '__main__':
    main()

博主也是刚刚接触Python的新手，如有错误，欢迎指正。