利用spring boot 写一个稳定的爬虫

1、前言

这篇文章是利用spring boot 写一个稳定的爬虫，爬取的网页数据包含未执行js的网页数据、http/https接口的请求数据、和经过网页渲染的js数据（需要chorme浏览器），数据库使用mysql，程序的运行逻辑定去抓取网页数据，解析数据，存入mysql数据库中，爬取百度股市通的数据为例。

2、创建项目

使用idea开发，首先创建一个spring boot 项目，Group设置为com.crawler，Artifact设置为example，创建项目如图1所示

勾选web模块

设置项目名称为example

3、爬取的数据和存储的数据表结构

1、爬虫百度股市通 https://gupiao.baidu.com/concept/ 上面的概要数据
2、获取某一只股票的今日价格数据

https://gupiao.baidu.com/api/stocks/stocktimeline?from=pc&os_ver=1&cuid=xxx&vv=100&format=json&stock_code=sh600358&timestamp=

概要数据爬取的内容包含热点概念，驱动事件和具体的股票数据，概要数据的数据抽象成msql表如下：

CREATE TABLE `baidu_hot` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `title_line1` varchar(255) DEFAULT NULL,
  `title_line2` int(11) DEFAULT NULL,
  `title_line3` varchar(255) DEFAULT NULL,
  `title_line4` varchar(255) DEFAULT NULL,
  `dirver_thing` text,
  `hot_stock_name_1` varchar(255) DEFAULT NULL,
  `hot_stock_code_1` varchar(11) DEFAULT NULL,
  `hot_stock_price_1` double DEFAULT NULL,
  `hot_stock_increment_1` varchar(20) DEFAULT NULL,
  `hot_stock_name_2` varchar(255) DEFAULT NULL,
  `hot_stock_code_2` varchar(11) DEFAULT NULL,
  `hot_stock_price_2` double DEFAULT NULL,
  `hot_stock_increment_2` varchar(20) DEFAULT NULL,
  `hot_stock_name_3` varchar(255) DEFAULT NULL,
  `hot_stock_code_3` varchar(11) NOT NULL,
  `hot_stock_price_3` double DEFAULT NULL,
  `hot_stock_increment_3` varchar(20) DEFAULT NULL,
  `insert_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8;

百度股市的接口数据直接存为json数据，其表抽象如下

CREATE TABLE `stock` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `stock_id` varchar(30) DEFAULT NULL,
  `data` json DEFAULT NULL,
  `insert_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

数据库名字取为：
baidugushi

4、spring boot 项目配置

4.1、日志配置
日志使用logback，每天生成一个info和error级别的日志，logback配置文件如下，配置文件名称为 logback-spring.xml ，logback配置文件放到resources目录下面即可生效：

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <appender name="consoleLog" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>%d - %msg%n</pattern>
        </layout>
    </appender>
    <appender name="fileInfoLog" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>DENY</onMatch>
            <onMismatch>ACCEPT</onMismatch>
        </filter>
        <encoder>
            <pattern>%msg%n</pattern>
        </encoder>
        <!--滚动策略-->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--路径-->
            <!-- winwods -->
            <fileNamePattern>D:\logs\example.info.%d.log</fileNamePattern>
            <!--<fileNamePattern>/data/log/crawler.info.%d.log</fileNamePattern>-->
        </rollingPolicy>
    </appender>
    <appender name="fileErrorLog" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <encoder>
            <pattern>%msg%n</pattern>
        </encoder>
        <!--滚动策略-->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--路径-->
            <fileNamePattern>D:\logs\example.error.%d.log</fileNamePattern>
            <!--<fileNamePattern>/data/log/crawler.error.%d.log</fileNamePattern>-->
        </rollingPolicy>
    </appender>
    <root level="info">
        <appender-ref ref="consoleLog"/>
        <appender-ref ref="fileInfoLog"/>
        <appender-ref ref="fileErrorLog"/>
    </root>
</configuration>

4.2、mysql配置
首先在项目的example包底下创建driver,entity,map,web包，如下图所示

mysql使用mybaits去连接数据库
配置之前首先需要在maven导入的包如下：

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>6.0.6</version>
        </dependency>

        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>1.1.1</version>
        </dependency>

application.properties配置文件的内容如下：

mybatis.type-aliases-package=com.crawler.example.entity
spring.datasource.driverClassName = com.mysql.cj.jdbc.Driver
#本机调试
spring.datasource.url = jdbc:mysql://127.0.0.1:3306/baidugushi?useUnicode=true&characterEncoding=UTF-8&useSSL=false&autoReconnect=true&serverTimezone=UTC
spring.datasource.username = root
spring.datasource.password = pwd

其中mybatis.type-aliases-package 是指定mybaits的数据库中表对应类的包，用户名和密码请自行修改
4.3、tomcat 设置
项目最终发布的形式是一个war包，jar包不方便部署和管理。
修改项目的pom.xml文件，packagin修改为war

    <packaging>war</packaging>

删除自带的tomcat，和添加必要的依赖

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-tomcat</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
            <scope>provided</scope>
        </dependency>

修改完pom.xml文件还需要添加有一个启动类，其中ExampleApplication是spring boot自动生成的启动类

package com.crawler.example;

import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.support.SpringBootServletInitializer;

public class SpringBootStartApplication extends SpringBootServletInitializer {
    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
        // 注意这里要指向原先用main方法执行的Application启动类
        return builder.sources(ExampleApplication.class);
    }
}

4.4、idea项目调试设置
首先电脑上要安装了tomcat，随后打开Idea 的Run –Run… edit configurations 删除掉spring boot自带启动设置，随后新建tomcat的配置，tomcat配置如下面两张图所示，将After Launchq勾线去掉，否则项目启动时会启动浏览器。

上图主要需要配置的是要选择tomcat的位置

上图是配置启动时部署的war包

5、抓取静态页面数据

5.1、定时任务编写
爬虫是定时抓取网页数据，html页面解析使用Jsoup，jsoup 是一款Java 的HTML解析器，可直接解析某个URL地址、HTML文本内容。它提供了一套非常省力的API，可通过DOM，CSS以及类似于jQuery的操作方法来取出和操作数据。
需要增加的pom依赖为

        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.10.3</version>
        </dependency>

页面下载程序为：

package com.crawler.example.web;

import com.crawler.example.dirver.BaiDuHotProcess;
import com.crawler.example.entity.BaiduHot;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import com.crawler.example.map.BaiduHotMap;
import org.slf4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.bind.annotation.RestController;
import java.io.IOException;
import java.util.ArrayList;

//百度股市热门下载
@RestController
public class BaiduHotDown {
    public  static Logger logger;
    @Autowired
    BaiduHotMap baiduHotMap;

    @Scheduled(cron = "0/20 * * * * ? ")
    public void downBaiduHot(){
        String url = "https://gupiao.baidu.com/concept/";
        try {
            Document doc = Jsoup.connect(url).get();
            ArrayList<BaiduHot> abh = new BaiDuHotProcess().processBaiduHot(doc);
            for(BaiduHot b:abh){
                baiduHotMap.InsertBaiduHot(b);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

5.2、数据提取解析为：

package com.crawler.example.dirver;

import com.crawler.example.entity.BaiduHot;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.util.ArrayList;

public class BaiDuHotProcess {

    public ArrayList<BaiduHot> processBaiduHot(Document doc){

        ArrayList<BaiduHot> abh = new ArrayList<BaiduHot>();
        //提取数据
        Elements divsBig = doc.getElementsByClass("hot-concept clearfix");
        for(int i=0;i<divsBig.size();i++){
            BaiduHot baiduHot = new BaiduHot();
            //获得行业数据
            Elements cloumn1 = divsBig.get(i).getElementsByClass("concept-header column1");
            //获取行数数据
            baiduHot.title_line1 = cloumn1.get(0).getElementsByClass("text-ellipsis").get(0).ownText();
            //获取热搜指数
            baiduHot.title_line2 =Integer.parseInt( cloumn1.get(0).getElementsByTag("h3").get(0).getElementsByTag("span").get(0).ownText());
            //获得发布时间
            baiduHot.title_line3 = cloumn1.get(0).getElementsByTag("p").get(0).ownText();
            //获得简要内容
            baiduHot.title_line4 = cloumn1.get(0).getElementsByTag("p").get(1).ownText();
            //概述内容
            baiduHot.dirver_thing = divsBig.get(i).getElementsByClass("concept-event column3").get(0).ownText();
            //获得推荐股价
            Elements stockUl = divsBig.get(i).getElementsByClass("no-click");
            //System.out.println(stockUl.get(0).getElementsByTag("a").get(0).getElementsByTag("div").size());
            //股票1名称
            baiduHot.hot_stock_name_1 = stockUl.get(0).getElementsByTag("a").get(0).getElementsByTag("div").get(0).ownText();
            //股票1代码
            baiduHot.hot_stock_code_1 = stockUl.get(0).getElementsByTag("a").get(0).getElementsByTag("div").get(1).ownText();
            //股票1价格
            baiduHot.hot_stock_price_1 = Double.parseDouble(stockUl.get(0).getElementsByClass("column2").get(1).ownText());
            //股票1涨幅

            baiduHot.hot_stock_increment_1 = stockUl.get(0).child(2).ownText();

            //股票2名称
            baiduHot.hot_stock_name_2 = stockUl.get(1).getElementsByTag("a").get(0).getElementsByTag("div").get(0).ownText();
            //股票2代码
            baiduHot.hot_stock_code_2 = stockUl.get(1).getElementsByTag("a").get(0).getElementsByTag("div").get(1).ownText();
            //股票2价格
            baiduHot.hot_stock_price_2 = Double.parseDouble(stockUl.get(1).getElementsByClass("column2").get(1).ownText());
            //股票2涨幅
            baiduHot.hot_stock_increment_2 = stockUl.get(1).child(2).ownText();

            //股票3名称
            baiduHot.hot_stock_name_3 = stockUl.get(2).getElementsByTag("a").get(0).getElementsByTag("div").get(0).ownText();
            //股票3代码
            baiduHot.hot_stock_code_3 = stockUl.get(2).getElementsByTag("a").get(0).getElementsByTag("div").get(1).ownText();
            //股票3价格
            baiduHot.hot_stock_price_3 = Double.parseDouble(stockUl.get(2).getElementsByClass("column2").get(1).ownText());
            //股票3涨幅
            baiduHot.hot_stock_increment_3 = stockUl.get(2).child(2).ownText();

            abh.add(baiduHot);
        }
        return  abh;
    }
}

定时周期使用corn表达式实现，可以使用在线网址：http://cron.qqe2.com/，点点鼠标即可生成想要的定时周期
实体和数据库接口类如下

package com.crawler.example.map;

import com.crawler.example.entity.BaiduHot;
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Mapper;

//百度接口数据
@Mapper
public interface BaiduHotMap {
    @Insert("insert into baidu_hot(title_line1,title_line2,title_line3,title_line4,dirver_thing,hot_stock_name_1," +
            "hot_stock_code_1,hot_stock_price_1,hot_stock_increment_1,hot_stock_name_2,hot_stock_code_2,hot_stock_price_2," +
            "hot_stock_increment_2,hot_stock_name_3,hot_stock_code_3,hot_stock_price_3,hot_stock_increment_3) values(" +
            "#{title_line1},#{title_line2},#{title_line3},#{title_line4},#{dirver_thing},#{hot_stock_name_1},#{hot_stock_code_1}," +
            "#{hot_stock_price_1},#{hot_stock_increment_1},#{hot_stock_name_2},#{hot_stock_code_2},#{hot_stock_price_2},#{hot_stock_increment_2}," +
            "#{hot_stock_name_3},#{hot_stock_code_3},#{hot_stock_price_3},#{hot_stock_increment_3})")
    public void InsertBaiduHot(BaiduHot baiduHot);
}

实体类：

package com.crawler.example.entity;

public class BaiduHot implements Cloneable {

    public int id;

    public String title_line1;

    public int title_line2;

    public String title_line3;

    public String title_line4;

    public String dirver_thing;

    public String hot_stock_name_1;

    public String hot_stock_code_1;

    public double hot_stock_price_1;

    public String hot_stock_increment_1;

    public String hot_stock_name_2;

    public String hot_stock_code_2;

    public double hot_stock_price_2;

    public String hot_stock_increment_2;

    public String hot_stock_name_3;

    public String hot_stock_code_3;

    public double hot_stock_price_3;

    public String hot_stock_increment_3;
  }

启动类加上注解：@EnableScheduling
最后获得的数据如下图所示：

6、抓取http接口的数据

6.1、获取某只股票当日的数据
http下载程序为

        <dependency>
            <groupId>org.json</groupId>
            <artifactId>json</artifactId>
            <version>20160810</version>
        </dependency>

程序源码如下，启动定时器sh60035下载当日的股票数据：

package com.crawler.example.web;

import com.crawler.example.dirver.GetJson;
import com.crawler.example.entity.StockPrice;
import com.crawler.example.map.StockPriceMap;
import org.json.JSONObject;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.bind.annotation.RestController;

//查询股票的价格
@RestController
public class BaiduStockPrice {

    @Autowired
    StockPriceMap stockPriceMap;

    //下载股票曲线图
    //@Scheduled(cron = "0/20 * * * * ? ")
    public void downStockPrice(){
        //url 生成
        String url = "https://gupiao.baidu.com/api/stocks/stocktimeline?from=pc&os_ver=1&cuid=xxx&vv=100&format=json&stock_code=sh600358&timestamp=" + System.currentTimeMillis();
        //访问获得json数据
        JSONObject stock = new GetJson().getHttpJson(url,1);
        StockPrice stockPrice = new StockPrice();
        stockPrice.stock_id = "sh60035";
        stockPrice.data = stock.toString();
        //将json数据存入数据库中
        stockPriceMap.insertIntoStock(stockPrice);
    }
}

http下载并返回json

package com.crawler.example.dirver;

import org.json.JSONObject;

import javax.net.ssl.HttpsURLConnection;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;

public class GetJson {
    public JSONObject getHttpJson(String url,int comefrom){
        try {
            URL realUrl = new URL(url);
            HttpURLConnection connection = (HttpURLConnection)realUrl.openConnection();
            connection.setRequestProperty("accept", "*/*");
            connection.setRequestProperty("connection", "Keep-Alive");
            connection.setRequestProperty("user-agent","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;SV1)");
            // 建立实际的连接
            connection.connect();
            //请求成功
            if(connection.getResponseCode()==200){
                InputStream is=connection.getInputStream();
                ByteArrayOutputStream baos=new ByteArrayOutputStream();
                //10MB的缓存
                byte [] buffer=new byte[10485760];
                int len=0;
                while((len=is.read(buffer))!=-1){
                    baos.write(buffer, 0, len);
                }
                String jsonString=baos.toString();
                baos.close();
                is.close();
                //转换成json数据处理
                JSONObject jsonArray=getJsonString(jsonString,comefrom);
                return jsonArray;
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException ex) {
           ex.printStackTrace();
        }
        return null;
    }

    public JSONObject getHttpsJson(String url){
        try {
            URL realUrl = new URL(url);
            HttpsURLConnection httpsConn = (HttpsURLConnection)realUrl.openConnection();
            httpsConn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
            httpsConn.setRequestProperty("connection", "Keep-Alive");
            httpsConn.setRequestProperty("user-agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36");
            httpsConn.setRequestProperty("Accept-Charset","utf-8");
            httpsConn.setRequestProperty("contentType", "utf-8");
            httpsConn.connect();
            if(httpsConn.getResponseCode()==200){
                InputStream is = httpsConn.getInputStream();
                ByteArrayOutputStream baos=new ByteArrayOutputStream();
                //10MB的缓存
                byte [] buffer=new byte[10485760];
                int len=0;
                while((len=is.read(buffer))!=-1){
                    baos.write(buffer, 0, len);
                }
                String jsonString=baos.toString("utf-8");
                baos.close();
                is.close();
                return new JSONObject(jsonString);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return  null;
    }

    public JSONObject getJsonString(String str, int comefrom){
           JSONObject jo = null;
            if(comefrom==1){
                return new JSONObject(str);
            }else if(comefrom==2){
                int indexStart = 0;
                //字符处理
                for(int i=0;i<str.length();i++){
                    if(str.charAt(i)=='('){
                        indexStart = i;
                        break;
                    }
                }
                String strNew = "";
                //分割字符串
                for(int i=indexStart+1;i<str.length()-1;i++){
                    strNew += str.charAt(i);
                }
                return new JSONObject(strNew);
            }
           return jo;
    }
}

getHttpJson函数的后面的参数1，表示返回的是json数据，2表示http接口的数据在一个（）中的数据
6.2、数据库实体类和数据库的程序如下
数据库实体类

package com.crawler.example.entity;

import org.json.JSONObject;

public class StockPrice {

    public  String stock_id;

    public String data;

    public String getStock_id() {
        return stock_id;
    }

    public void setStock_id(String stock_id) {
        this.stock_id = stock_id;
    }

    public String getData() {
        return data;
    }

    public void setData(String data) {
        this.data = data;
    }
}

数据库接口如下：

package com.crawler.example.map;

import com.crawler.example.entity.StockPrice;
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Mapper;

@Mapper
public interface StockPriceMap {

    @Insert("insert into stock(stock_id,data) values(#{stock_id},#{data})")
    public void insertIntoStock(StockPrice stockPrice);

}

最后的运行结果为：

7、获取动态页面数据

7.1、抓取需要渲染的网页

有时网页的数据经过渲染才能有，或者网站有饭爬虫措施，那么可以借助浏览器下载网页数据，这样爬虫部署到windows server上比较方便，centos 安装chorme 比较困难，这种数据抓取方式几乎可以为所欲为0.0。
操作chorme的java插件有

cdp4j - Chrome DevTools Protocol for Java
链接地址为：
https://github.com/webfolderio/cdp4j

还是抓取百度股市热点网页数据
抓取程序为：

package com.crawler.example.web;

import com.crawler.example.dirver.BaiDuHotProcess;
import com.crawler.example.entity.BaiduHot;
import com.crawler.example.map.BaiduHotMap;
import io.webfolder.cdp.Launcher;
import io.webfolder.cdp.session.Session;
import io.webfolder.cdp.session.SessionFactory;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;

@RestController
public class BaiduHotDownChorme {

    @Autowired

    BaiduHotMap baiduHotMap;
    @Scheduled(cron = "0/20 * * * * ? ")
    public void downBaiDuHot(){
        ArrayList<String> command = new ArrayList<String>();
        //不显示google 浏览器
        command.add("--headless");
        Launcher launcher = new Launcher();
        try (SessionFactory factory = launcher.launch(command);
             Session session = factory.create()){
            session.navigate("https://gupiao.baidu.com/concept/");
            session.waitDocumentReady();
            String content = (String) session.getContent();
            //System.out.println(content);
            Document doc = Jsoup.parse(content);
            ArrayList<BaiduHot> abh = new BaiDuHotProcess().processBaiduHot(doc);
            for(BaiduHot b:abh){
                baiduHotMap.InsertBaiduHot(b);
            }
        }catch (Exception e){
            e.printStackTrace();
        }

    }
}

上面就是java 抓取网页数据的三种方式，项目源码在github上，源码地址为：
https://github.com/xiaoyangmoa/java-crawler