ElasticSearch 使用java代码查询

程序员文章站 2022-05-03 19:18:07

...

准备

（一）加maven依赖

  <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
            <version>3.2.9.RELEASE</version>
        </dependency>

（二）配置application.properties文件

spring.data.elasticsearch.cluster-name=my-es
spring.data.elasticsearch.cluster-nodes=ip:9300

（三）部分使用到的vo类

插入es中的vo类

@Document(indexName = "es_index_prefix_20201003", type = "es_type", shards = 1, replicas = 0)
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class ItemEsSearch {
    @Id
    private String id;

    @Field(type = FieldType.Text, analyzer = "ik_max_word")
    private String describe;


    @Field(type = FieldType.Keyword)
    private String name;

    @Field(type = FieldType.Keyword)
    private String item;

    @Field(type = FieldType.Date)
    private Date serverTime;
}

es中列名类

public class ItemEsQueryCommon {
    public final static String ITEM = "item";

    public final static String NAME = "name";

    public final static String DESCRIBE = "describe";

    public final static String SERVER_TIME = "serverTime";

    public final static String ID = "_id";
}

（四）普通分页查询

查询每页数据


   /**
     * es 索引前缀
     */
    public final static String INDEX_PREFIX = "es_index_prefix_";


    /**
     * type
     */
    private final static String INDEX_TYPE = "es_type";


    /**
     * 格式化的时间
     */
    private final static String DATE_FORMATTER = "yyyyMMdd";
    
 @Override
    public List<ItemEsSearch> findInstance(ItemEsQueryVO esQuery, PagerPlugin pagerPlugin) {
        NativeSearchQuery nativeSearchQuery = buildNativeSearchQueryWithPage(esQuery)
                .withPageable(PageRequest.of(pagerPlugin.getCurrPage() - 1, pagerPlugin.getPageSize()))
                       .build();
        Gson gson = new GsonBuilder().setDateFormat("YYYY-MM-dd'T'HH:mm:ss").create();
        return elasticsearchTemplate.query(nativeSearchQuery, response -> {
            List<ItemEsSearch> itemEsSearches = new ArrayList<>();
            SearchHits hits = response.getHits();
            Arrays.stream(hits.getHits()).forEach(h -> {
                String source = h.getSourceAsString();
                ItemEsSearch itemEsSearch = gson.fromJson(source, ItemEsSearch.class);
                itemEsSearch.setId(h.getId());
                itemEsSearches.add(itemEsSearch);
            });
            return itemEsSearches;
        });
    }

private NativeSearchQueryBuilder buildNativeSearchQueryWithPage(ItemEsQueryVO itemEsQuery) {
        NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
        List<String> indexList = this.indexList(itemEsQuery.getStartDate(), itemEsQuery.getEndDate());
        BoolQueryBuilder filterBoolBuilder = new BoolQueryBuilder();
        if (!CollectionUtils.isEmpty(itemEsQuery.getItemList())) {
            TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery(ItemEsQueryCommon.ITEM, itemEsQuery.getItemList().toArray(new String[0]));
            filterBoolBuilder.must(termQueryBuilder);
        }

        if(!StringUtils.isEmpty(itemEsQuery.getName())){
            TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery(ItemEsQueryCommon.NAME, itemEsQuery.getName());
            filterBoolBuilder.must(termQueryBuilder);
        }

        if (!StringUtils.isEmpty(itemEsQuery.getDescribe())) {
            TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery(ItemEsQueryCommon.DESCRIBE, itemEsQuery.getDescribe());
            filterBoolBuilder.must(termQueryBuilder);
        }

        if (!StringUtils.isEmpty(itemEsQuery.getId())) {
            TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery(ItemEsQueryCommon.ID, itemEsQuery.getId());
            filterBoolBuilder.must(termQueryBuilder);
        }

        FieldSortBuilder fieldSortBuilder = SortBuilders.fieldSort(ItemEsQueryCommon.SERVER_TIME).order(SortOrder.DESC);
        return nativeSearchQueryBuilder.withIndices(indexList.toArray(new String[0]))
                .withQuery(filterBoolBuilder)
                .withSort(fieldSortBuilder)
                .withTypes(INDEX_TYPE);

    }




  /**
     * 获取索引列表
     * @return 所有符合要求索引集合
     */
    private List<String> indexList(Date startDate, Date endDate) {
        List<String> indexList = new ArrayList<>();
        SimpleDateFormat dateFormat = new SimpleDateFormat(DATE_FORMATTER);
        GregorianCalendar startCalendar = new GregorianCalendar();
        startCalendar.setTime(startDate);
        GregorianCalendar endCalendar = new GregorianCalendar();
        endCalendar.setTime(endDate);
        while (startCalendar.before(endCalendar)) {
            Date time = startCalendar.getTime();
            String dateIndex = dateFormat.format(time);
            indexList.add(INDEX_PREFIX + dateIndex);
            startCalendar.add(Calendar.DATE, 1);
        }
        indexList.add(INDEX_PREFIX + dateFormat.format(endDate));
        return indexList;
    }

查询总条数：（缺少方法，在前面已经贴出）

  @Override
    public long getSearchTotalNum(ItemEsQueryVO esQuery) {
        return elasticsearchTemplate.count(buildNativeSearchQueryWithPage(esQuery).build());

    }

（五） Scroll 查询方式

private SearchResultMapper searchResultMapper = new SearchResultMapper() {


        @Override
        public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
            if (searchResponse.getHits().getHits().length <= 0) {
                return new AggregatedPageImpl<T>(Collections.EMPTY_LIST);
            }
            Gson gson = new GsonBuilder().setDateFormat("YYYY-MM-dd'T'HH:mm:ss").create();
            SearchHits hits = searchResponse.getHits();
            List<ItemEsSearch> resultList=new ArrayList<>((int)hits.totalHits);
            for (SearchHit hit : hits) {
                String source = hit.getSourceAsString();
                ItemEsSearch itemEsSearch = gson.fromJson(source, ItemEsSearch.class);
                itemEsSearch.setId(hit.getId());
                resultList.add(itemEsSearch);
            }
            return new AggregatedPageImpl<T>((List<T>) resultList, pageable, searchResponse.getHits().getTotalHits(), searchResponse.getScrollId());
        }

        @Override
        public <T> T mapSearchHit(SearchHit searchHit, Class<T> aClass) {
            return null;
        }
    };

 @Override
    public List<ItemEsSearch> findInstanceWithScroll(ItemEsQueryVO esQuery, PagerPlugin pagerPlugin) {
        List<ItemEsSearch> resultList = new ArrayList<>();
        NativeSearchQuery searchQuery = buildNativeSearchQueryWithPage(esQuery).withPageable(PageRequest.of(pagerPlugin.getCurrPage() - 1, pagerPlugin.getPageSize())).build();
        ScrolledPage<ItemEsSearch> scroll;
        if (StringUtils.isEmpty(pagerPlugin.getScrollId())) {
            scroll= elasticsearchTemplate.startScroll(SCROLL_TIMEOUT, searchQuery, null, searchResultMapper);
        }else {
            //取下一页，scrollId 在es 服务器上可能发生变化，重新生成快照continueScroll
            scroll = elasticsearchTemplate.continueScroll(pagerPlugin.getScrollId(), SCROLL_TIMEOUT, ItemEsSearch.class, searchResultMapper);
        }
        pagerPlugin.setScrollId(scroll.getScrollId());
        log.info("查询命中数：{}", scroll.getTotalElements());
        if (scroll.hasContent()) {
            resultList = scroll.getContent();
        }
        return resultList;

    }

（六）高亮查询（由普通查询添加调整而来）

 @Override
    public List<ItemEsSearch> findInstance(ItemEsQueryVO esQuery, PagerPlugin pagerPlugin) {
        NativeSearchQuery nativeSearchQuery = buildNativeSearchQueryWithPage(esQuery)
                .withPageable(PageRequest.of(pagerPlugin.getCurrPage() - 1, pagerPlugin.getPageSize()))
                        .withHighlightFields(new HighlightBuilder.Field(ItemEsQueryCommon.DESCRIBE)).build();
        Gson gson = new GsonBuilder().setDateFormat("YYYY-MM-dd'T'HH:mm:ss").create();
        return elasticsearchTemplate.query(nativeSearchQuery, response -> {
            List<ItemEsSearch> itemEsSearches = new ArrayList<>();
            SearchHits hits = response.getHits();
            Arrays.stream(hits.getHits()).forEach(h -> {
                String source = h.getSourceAsString();
                HighlightEsSearchVO itemEsSearch = gson.fromJson(source, HighlightEsSearchVO.class);
                itemEsSearch.setId(h.getId());
                Text[] fragments = h.getHighlightFields().get(ItemEsQueryCommon.DESCRIBE).getFragments();
                if (!Objects.isNull(fragments) && fragments.length > 0) {
                    itemEsSearch.setHighField(fragments[0].toString());
                }
                itemEsSearches.add(itemEsSearch);
            });
            return itemEsSearches;
        });
    }

ElasticSearch 使用java代码查询

（六）聚合分析–统计数量

 @Override
    public  Map<String, Long> getEsAggSearch(ItemEsQueryVO esQuery, PagerPlugin pagerPlugin) {
        NativeSearchQuery nativeSearchQuery = buildNativeSearchQueryWithPage(esQuery)
                .withPageable(PageRequest.of(pagerPlugin.getCurrPage() - 1, pagerPlugin.getPageSize()))
                .addAggregation(
                        AggregationBuilders.terms("groupNum")
                                .field(ItemEsQueryCommon.ITEM)
                ).build();

        Aggregations aggregations = elasticsearchTemplate.query(nativeSearchQuery,
                new ResultsExtractor<Aggregations>() {
                    @Override
                    public Aggregations extract(SearchResponse response) {
                        return response.getAggregations();
                    }
                });

        StringTerms modelTerms = (StringTerms) aggregations.asMap().get("groupNum");

        Map<String, Long> map = new HashMap<>();
        for (Terms.Bucket actionTypeBucket : modelTerms.getBuckets()) {
            //actionTypeBucket.getKey().toString()聚合字段的相应名称,actionTypeBucket.getDocCount()相应聚合结果
            map.put(actionTypeBucket.getKey().toString(),
                    actionTypeBucket.getDocCount());
        }

        return map;
    }

ElasticSearch 使用java代码查询

（六）聚合分析–聚合求平均
（不是绝对平均）

@Override
    public  JSONObject getEsAggOrderSearch(ItemEsQueryVO esQuery, PagerPlugin pagerPlugin) {
        NativeSearchQuery nativeSearchQuery = buildNativeSearchQueryWithPage(esQuery)
                .withPageable(PageRequest.of(pagerPlugin.getCurrPage() - 1, pagerPlugin.getPageSize()))
                .addAggregation(
                        AggregationBuilders
                                .terms("groupNum")
                                .order(
                                        BucketOrder.aggregation(ItemEsQueryCommon.PRICE, false)
                                )
                                .field(ItemEsQueryCommon.ITEM)
                                .subAggregation(
                                        AggregationBuilders
                                                .avg(ItemEsQueryCommon.PRICE)
                                                .field(ItemEsQueryCommon.PRICE)
                                )
                ).build();

        Aggregations aggregations = elasticsearchTemplate.query(nativeSearchQuery,
                new ResultsExtractor<Aggregations>() {
                    @Override
                    public Aggregations extract(SearchResponse response) {
                        return response.getAggregations();
                    }
                });

        StringTerms modelTerms = (StringTerms) aggregations.asMap().get("groupNum");
        return JSONObject.parseObject(modelTerms.toString());


    }

ElasticSearch 使用java代码查询
（七）聚合分析–先分组、再聚合求平均

@Override
    public JSONObject getEsAggRangeOrderSearch(ItemEsQueryVO esQuery, PagerPlugin pagerPlugin) {
        NativeSearchQuery nativeSearchQuery = buildNativeSearchQueryWithPage(esQuery)
                .withPageable(PageRequest.of(pagerPlugin.getCurrPage() - 1, pagerPlugin.getPageSize()))
                .addAggregation(
                        AggregationBuilders
                                .range("range_by_price")
                                .field(ItemEsQueryCommon.PRICE)
                                .addRange(0, 0.5)
                                .addRange(0.5, 1)
                                .addRange(1, 2)
                                .subAggregation(
                                        AggregationBuilders
                                                .terms("groupNum")
                                                .field(ItemEsQueryCommon.ITEM)
                                                .order(
                                                        BucketOrder.aggregation(ItemEsQueryCommon.PRICE, false)
                                                )
                                                .subAggregation(
                                                        AggregationBuilders
                                                                .avg(ItemEsQueryCommon.PRICE)
                                                                .field(ItemEsQueryCommon.PRICE)
                                                )

                                )
                ).build();

        Aggregations aggregations = elasticsearchTemplate.query(nativeSearchQuery,
                new ResultsExtractor<Aggregations>() {
                    @Override
                    public Aggregations extract(SearchResponse response) {
                        return response.getAggregations();
                    }
                });

        Aggregation aggregation = aggregations.asMap().get("range_by_price");
        return JSONObject.parseObject(aggregation.toString());

ElasticSearch 使用java代码查询

相关标签：大数据 elasticsearch 查询 scroll 高亮分组

上一篇： Openresty通过Lua+Redis 实现动态封禁IP

下一篇： nginx ip黑名单动态封禁

ElasticSearch 使用java代码查询

准备

使用Gradle做Java代码质量检查的方法示例

java使用淘宝API读写json实现手机归属地查询功能代码

java使用dom4j操作xml示例代码

java使用链表来模拟栈的入栈出栈操作实例代码

使用java操作elasticsearch的具体方法

java使用es查询的示例代码

使用JAVA判断凸多边形的示例代码

java 使用线程做的一个简单的ATM存取款实例代码

Java编程使用UDP建立群聊系统代码实例

在java中使用dom4j解析xml(示例代码)

ElasticSearch 使用java代码 查询

准备

使用Gradle做Java代码质量检查的方法示例

java使用淘宝API读写json实现手机归属地查询功能代码

java使用dom4j操作xml示例代码

java使用链表来模拟栈的入栈出栈操作实例代码

使用java操作elasticsearch的具体方法

java使用es查询的示例代码

使用JAVA判断凸多边形的示例代码

java 使用线程做的一个简单的ATM存取款实例代码

Java编程使用UDP建立群聊系统代码实例

在java中使用dom4j解析xml(示例代码)

ElasticSearch 使用java代码查询