欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

ELK日志分析系统(4)-elasticsearch数据存储

程序员文章站 2022-03-20 16:40:51
1. 概述 logstash把格式化的数据发送到elasticsearch以后,elasticsearch负责存储搜索日志数据 elasticsearch的搜索接口还是很强大的,这边不详细展开,因为kibana会去调用el的接口; 本文将讲解elasticsearch的相关配置和遇到的问题,至于el ......

 

1. 概述

  logstash把格式化的数据发送到elasticsearch以后,elasticsearch负责存储搜索日志数据

      elasticsearch的搜索接口还是很强大的,这边不详细展开,因为kibana会去调用el的接口;

  本文将讲解elasticsearch的相关配置和遇到的问题,至于elasticsearch的相关搜索使用,后面会找个时间整理一下。

2. 配置

  

  配置路径:docker-elk/elasticsearch/config/elasticsearch.yml

  • 关闭安全验证,否则kibana连接不上:xpack.security.enabled:false
  • 配置支持跨域调用,否则kibana会提示连接不上: http.cors.enabled: true

  另外由于elasticsearch很容易被攻击,所以建议不要把elasticsearch的端口对外开放

  

cluster.name: "docker-cluster"
network.host: 0.0.0.0

## use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

## x-pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: trial
xpack.security.enabled: false
xpack.monitoring.collection.enabled: true

http.cors.enabled: true
http.cors.allow-origin: "*"

 

  elasticsearch的缓存路径是/usr/share/elasticsearch/data

 

  验证是否成功:

  访问 ,如果得到以下数据表示成功: 

 

   ELK日志分析系统(4)-elasticsearch数据存储

 3. 异常处理

 

  3.1. index has exceeded [1000000] - maximum allowed to be analyzed for highlighting

  详细的出错内容是这样:

    {"type":"illegal_argument_exception","reason":"the length of [message] field of [l60zgw0bv9xmtlnx27a_] doc of [syslog] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. this maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. for large texts, indexing with offsets or term vectors is recommended!”}}

   错误原因:索引偏移量默认是100000,超过了

  

  最大迁移索引不能配置在配置文件中,只能接口修改

  

# 修改最大索引迁移

curl -xput "http://192.168.1.165:9200/_settings" -h 'content-type: application/json' -d' {
    "index" : {
        "highlight.max_analyzed_offset" : 100000000
    }
}’

 

  3.1. circuit_breaking_exception', '[parent] data too large, data for [<http_request>] would be [246901928/235.4mb], which is larger than the limit of [246546432/235.1mb]

  详细的出错内容是这样:

    elasticsearch.exceptions.transporterror: transporterror(429, 'circuit_breaking_exception', '[parent] data too large, data for [<http_request>] would be [246901928/235.4mb], which is larger than the limit of [246546432/235.1mb], real usage: [246901768/235.4mb], new bytes reserved: [160/160b], usages [request=0/0b, fielddata=11733/11.4kb, in_flight_requests=160/160b, accounting=6120593/5.8mb]')

   

  错误原因:

    堆内存不够当前查询加载数据所以会报 https://github.com/docker-library/elasticsearch/issues/98

   解决方案:

  • 提高堆栈内存

    在宿主机执行:sudo sysctl -w vm.max_map_count=262144

    docker增加命令参数设置java的虚拟机初始化堆栈大小1g,和最大堆栈大小3g

    docker-compose路径:配置路径:docker-elk/docker-compose.yml   

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        elk_version: $elk_version
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      es_java_opts: "-xms1g -xmx3g"
      elastic_password: changeme
      logspout: ignore
    networks:
      - elk

 

  • 增加堆内存的使用率,默认70%
curl -x put "http://192.168.1.165:9200/_cluster/settings" -h 'content-type: application/json' -d'

 {

     "transient" : {

         "indices.breaker.total.limit" : "90%"

     }

 }’

 

3. 安装可视化插件

  使用docker启动

  docker run -d --name elasticsearch-head -p 9100:9100 mobz/elasticsearch-head:5

  elasticsearch需要配置支持跨域调用,否则会提示连接不上

  elasticsearch head入口:

  插件效果如下:

ELK日志分析系统(4)-elasticsearch数据存储

 

  这个插件估计对新版本的elasticsearch支持不好,后面可以换一个支持新版本elsticsearch的插件。