欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

SpringBoot中使用Jsoup爬取网站数据的方法

程序员文章站 2024-01-28 21:37:28
爬取数据导入jar包 1.8

爬取数据

导入jar包

 <properties>
    <java.version>1.8</java.version>
    <elasticsearch.version>7.6.1</elasticsearch.version>
  </properties>
 
  <dependencies>
    <dependency>
      <groupid>org.jsoup</groupid>
      <artifactid>jsoup</artifactid>
      <version>1.10.2</version>
    </dependency>
    <dependency>
      <groupid>com.alibaba</groupid>
      <artifactid>fastjson</artifactid>
      <version>1.2.62</version>
    </dependency>
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-starter-data-redis</artifactid>
    </dependency>
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-starter-data-elasticsearch</artifactid>
    </dependency>
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-starter-thymeleaf</artifactid>
    </dependency>
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-starter-web</artifactid>
    </dependency>
 
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-devtools</artifactid>
      <scope>runtime</scope>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-configuration-processor</artifactid>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupid>org.projectlombok</groupid>
      <artifactid>lombok</artifactid>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupid>org.springframework.boot</groupid>
      <artifactid>spring-boot-starter-test</artifactid>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupid>org.junit.vintage</groupid>
          <artifactid>junit-vintage-engine</artifactid>
        </exclusion>
      </exclusions>
    </dependency>
  </dependencies>

新建实体类

@data
@noargsconstructor
@allargsconstructor
public class content {
  private string title;
  private string img;
  private string price;
}

编写爬虫工具类

public class htmlparseutil {
  public static void main(string[] args) throws exception {
    new htmlparseutil().parseddjj("包").foreach(system.out::println);
  }
 
  public list<content> parseddjj(string keywords) throws exception {
    //爬取url地址
    string url = "https://search.xxxx.com/search?keyword="+keywords;
    //解析网页,30s内未爬取成功,打印错误
    document document = jsoup.parse(new url(url),30000);
    //获取每一本书籍的id
    element element = document.getelementbyid("dj_goodslist");
    //获取所有的li标签
    elements elements = element.getelementsbytag("li");
 
    arraylist<content> goodslist = new arraylist<>();
 
    //遍历li标签的内容
    for (element el : elements) {
      string img = el.getelementsbytag("img").eq(0).attr("src");
      string price = el.getelementsbyclass("p-price").eq(0).text();
      string title = el.getelementsbyclass("p-name").eq(0).text();
 
      content content = new content();
      content.settitle(title);
      content.setprice(price);
      content.setimg(img);
      goodslist.add(content);
    }
    return goodslist;
  }
}

可以看到内容、图片、价格系数爬取

SpringBoot中使用Jsoup爬取网站数据的方法

到此这篇关于springboot中使用jsoup爬取网站数据的方法的文章就介绍到这了,更多相关springboot jsoup爬取内容请搜索以前的文章或继续浏览下面的相关文章希望大家以后多多支持!