167. Filebeat-modules 日志分析
程序员文章站
2024-01-20 08:44:16
...
filebeat module
将非结构化数据,转为结构化数据. module logstashfilebeat module --> ES.
1. 使用filebeat module采集Nginx日志 (Nginx日志是非结构数据)
1)修改filebeat配置文件
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#reload.period: 10s
setup.kibana:
host: "10.0.0.161:5601"
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
2) 启用nginx的module
[root@web01 filebeat]# filebeat modules enable nginx
3) 初始化环境
[root@web01 filebeat]# filebeat setup -e
4) 配置采集nginx日志的具体路径
[root@web01 ~]# cat /etc/filebeat/modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-nginx.html
- module: nginx
# Access logs
access:
enabled: true
var.paths: ["/var/log/nginx/access.log"]
# Error logs
error:
enabled: true
var.paths: ["/var/log/nginx/error.log"]
5) 启动filebeat
> /var/log/nginx/error.log
> /var/log/nginx/access.log
systemctl restart filebeat
2. 使用filebeat采集mysql的慢查询语句
1) 开启mysql慢日志以及错误日志
[root@web01 ~]# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
server-id=7
slow_query_log=ON
slow_query_log_file=/var/log/mariadb/slow.log
long_query_time=0.5
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
2) 配置filebeat.yml
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#reload.period: 10s
setup.kibana:
host: "10.0.0.161:5601"
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
3) 启动filebeat的module
[root@web01 ~]# filebeat modules enable mysql
4) 配置mysql module指向的日志路径
[root@web01 filebeat]# cat /etc/filebeat/modules.d/mysql.yml
# Module: mysql
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-mysql.html
- module: mysql
# Error logs
error:
enabled: true
var.paths: ["/var/log/mariadb/mariadb.log"]
# Slow logs
slowlog:
enabled: true
var.paths: ["/var/log/mariadb/slow.log"]
5) 初始化环境(一次即可)
[root@web01 ~]# filebeat setup -e
6) 重载filebeat
[root@web01 ~]# systemctl restart filebeat