综合项目（1）：爬取拉勾网python全国各地职位并在省（直辖市）的层面上统计平均薪资标准进行可视化

代码

注明：后面的 key 我没有公布出来，大家可以自己去高德地图的开发 API 去申请

本项目旨在完成以下步骤：

• 爬取拉勾网所有的 python （或者其他职位）信息并按照一定格式写入文件
• 将所有的信息进行数据的清洗、筛选
• 对不同省的python职位的薪资进行平均值求算
• 将不同省份的python薪资标准通过 pyecharts 进行可视化

import time
from selenium import webdriver
from lxml import etree
import requests
import pandas
import json
from pyecharts.charts import Map
from pyecharts import options as opts

def build_chromedriver():
    driver = webdriver.Chrome(executable_path=r'C:\Users\dell\Anaconda3\Scripts\chromedriver.exe')
    return driver


def request_page(driver,url):
    driver.get(url)
    text = driver.page_source
    return text


def search_position_info(driver):
    info = 'python'
    input_div = driver.find_element_by_xpath('//input[@id="search_input"]')
    input_div.send_keys('%s'%info)
    # 点击搜索按键
    driver.find_element_by_xpath('//input[@id="search_button"]').click()
    # 因为这里并没有产生新的页面，所以不用改变 driver 指向的页面

    # 点击广告页面
    driver.find_element_by_xpath('/html/body/div[9]/div/div[2]').click()
    return driver


def attain_whole_page_infos(driver,html,page_number):
    #先点击页面
    driver.find_element_by_xpath('//span[@page="%s"]'%page_number).click()
    # 获取这一页所有职位的信息
    text = driver.page_source
    html = etree.HTML(text)
    position_info_lst = html.xpath('//h3/text()')
    position_locaiton_lst = html.xpath('//span[@class="add"]//em/text()')
    money_lst = html.xpath('//span[@class="money"]/text()')

    with open('career.txt', 'a') as f:
        # for i in zip(position_info_lst,position_locaiton_lst,money_lst):
        #     f.write(str(i)+'\n')
        for i in range(len(position_locaiton_lst)):
            f.write(str(position_info_lst[i]) + '|' + str(position_locaiton_lst[i]) + '|' + str(money_lst[i]) + '\n')

    time.sleep(1)

def attain_province_info(position):
    key = '请用自己的key（去高德地图申请一个）'
    url = 'https://restapi.amap.com/v3/geocode/geo?address=%s&key=%s' % (position, key)
    text = requests.get(url).text
    diction_data = json.loads(text)
    province = diction_data['geocodes'][0]['province']
    return province


def cut_string(postion):
    position = postion.split("·")[0]
    return position


def calculate_salary(salary):
    if '薪' in salary:
        start_point = int(salary.split('-')[0].strip('k')) * 1000
        end_point = int(salary.split('·')[0].split('-')[1].strip('k')) * 1000
        average = (start_point + end_point) / 2
        year_average = average * int(salary.split('·')[1].strip('薪'))
        return year_average
    else:
        start_point = int(salary.split('-')[0].strip('k')) * 1000
        end_point = int(salary.split('-')[1].strip('k')) * 1000
        average = (start_point + end_point) / 2
        year_average = average * 12
        return year_average



if __name__ == '__main__':

    url = 'https://www.lagou.com/'
    driver = build_chromedriver()
    text = request_page(driver,url)
    driver.find_element_by_xpath('//a[@class="tab focus"]').click()  #自动点击全国的位置标签
    driver = search_position_info(driver)   # 输入查找的职位并返回页面
    page_source = driver.page_source
    html = etree.HTML(page_source)
    number_lst = html.xpath('//span[@page]/text()')     # 再新打开的页面上获取所有页数的信息
    number_int_lst = [int(number) for number in number_lst]
    max_page = max(number_int_lst)  # 得到页数的最大值，遍历这么多次
    for i in range(1,max_page+1):
    # for i in range(1,15):
        attain_whole_page_infos(driver,html,i)      # 在每一页上都进行信息的抽取操作，并且写入文件中

    with open('career.txt', 'r') as f:
        data_lst = f.readlines()
        print(data_lst)
    data_lst = [data.strip('\n').split('|')[1:] for data in data_lst]
    print(data_lst)

    df = pandas.DataFrame(columns=['地区', '薪资'], data=data_lst[1:10])  # 只用少量数据来做测试
    df['地区'] = df['地区'].map(cut_string)  # 只保留市信息
    print(df)
    df['地区'] = df['地区'].map(attain_province_info)  # 只保留市信息
    df['薪资'] = df['薪资'].map(calculate_salary)
    print(df)
    df = df.groupby('地区').agg({'薪资':'mean'}).reset_index()
    number_lst = df['薪资'].tolist()
    place = df['地区'].tolist()
    place = [i[:-1] for i in place]        # 每个字符串的最后一个字不要，例如上海市只保留上海
    print(number_lst, place)
    map = Map()
    map.add("薪资", list(zip(place,number_lst)), maptype="china")
    map.set_global_opts(title_opts=opts.TitleOpts(title='实例'),
                        visualmap_opts=opts.VisualMapOpts(max_=max(number_lst), is_piecewise=True))

    map.render('money.html')

作者联系方式：微信 Jeffrey-qin

有问题可以联系我