欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

logstash收集nginx日志、filebeat

程序员文章站 2022-06-11 11:30:21
...

logstash收集nginx日志

使用Beats采集日志(filebeat)

 

 

 

logstash收集nginx日志

 

 

 

1.

133上(在logstash这台机器上加入这个配置文件)

编辑配置文件 vi /etc/logstash/conf.d/nginx.conf//加入如下内容

input {
  file {
    path => "/tmp/elk_access.log"
    start_position => "beginning"
    type => "nginx"
  }
}
filter {
    grok {
        match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["192.168.193.133:9200"]
	index => "nginx-test-%{+YYYY.MM.dd}"
  }
   
  }

input { #输入

file { #此处指定一个文件,把这个文件的内容,作为logstash的输入

path => "/tmp/elk_access.log" #指定这个文件的路径

start_position => "beginning" #指定这个文件从什么时候开始收集

type => "nginx" #自定义

}

}

filter { #这个是对这个日志对一个过滤(比如输出格式)。所以访问日志的格式也要定义,下面会编辑nginx日志的格式(第3步骤)

grok {

match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}

}

geoip {

source => "clientip"

}

}

output { #输出

stdout { codec => rubydebug }

elasticsearch {

hosts => ["192.168.193.133:9200"]

index => "nginx-test-%{+YYYY.MM.dd}"

}                                                                                                                                                                                                                                          

}

#(注意花括号)配置不对会找不到nginx-test

2.

下面是配置代理。我们要产生日志,得去配置。配置之前要先检测一下,上面写的日志是否正确:

以下在132上操作

检测配置文件是否有错

cd /usr/share/logstash/bin

./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit

yum install -y nginx #没有nginx要安装

vi /etc/nginx/conf.d/elk.conf//写入如下内容 #写一个虚拟主机配置文件

server {


listen 80;

server_name elk.aming.com;




location / {

proxy_pass http://192.168.193.128:5601; #代理的目标

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

access_log /tmp/elk_access.log main2; #自动生成

}

3.

以下在132上操作

vim /etc/nginx/nginx.conf//增加如下内容 #定义nginx的日志格式

log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$upstream_addr" $request_time';

 

nginx -t #解释

/usr/local/nginx/sbin/nginx -t

 /usr/local/nginx/sbin/nginx -s reload

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

systemctl start nginx

绑定hosts 192.168.193.132 elk.aming.com

浏览器访问,检查是否有日志产生

systemctl restart logstash

4.

128上curl  '192.168.193.128:9200/_cat/indices?v'

health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana               HUhL8JS6Sgqxr8mSq9UgoQ   1   1          2            0      7.2kb          7.2kb
yellow open   system-syslog-2019.06 fqLpMdxRTG2EAV5DC8eIMw   5   1        840            0    526.9kb        526.9kb

查看有没有日志

ls /tmp/elk_access.log

/tmp/elk_access.log

wc -l !$

wc -l /tmp/elk_access.log
0 /tmp/elk_access.log

cat !$

检查是否有nginx-test开头的索引生成

如果有,才能到kibana里去配置该索引

左侧点击“Managerment”-> “Index Patterns”-> “Create Index Pattern”

Index pattern这里写nginx-test-* #支持统配,直接这样写就可以

之后点击左侧的Discover

 

 

 

 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

 

 

 

使用Beats采集日志(filebeat)

 

 

 

beats属于一个轻量的日志采集器。而logstash在启动的时候很慢,会消耗资源。那我们可以尝试着使用beats去采集日志。

beats可以自定义我们想要的beats

 

 

https://www.elastic.co/cn/products/beats

filebeat metricbeat packetbeat winlogbeat auditbeat heartbeat #filebeat的成员组件(我们用到的是filebeat,他是针对日志的)

可扩展,支持自定义构建

在133上执行

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm

rpm -ivh filebeat-6.0.0-x86_64.rpm #下载rpm包安装filebeat

以下是在屏幕上的输出:

首先编辑配置文件

vim /etc/filebeat/filebeat.yml //增加或者更改 #注意文件的格式,空格

filebeat.prospectors:

- input_type: log #定义input类型是log(此处type前面的input字样,在新版本不会出现,需注意)

paths: #定义日志的路径

- /var/log/messages

output.console: #此处文件里可能没有output.console:,需要添加这两行进去。但是添加完,要将out.elasticsearch:注释掉。如下图:

enable: true

 

/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml //可以在屏幕上看到对应的日志信息 #-c指定配置文件,启动filebeat

 

以下是以服务的角色出现:

再编辑配置文件

vim /etc/filebeat/filebeat.yml //增加或者更改

filebeat.prospectors:

- input_type: log

paths:

- /var/log/messages #针对这个日志

output.elasticsearch: #注意此处跟上面有区分,就不是output.console:了

hosts: ["192.168.133.130:9200"] #如下图:

 

systemctl start filebeat

ps aux |grep filebeat

root      41834  0.2  0.5 299580  5632 ?        Ssl  16:54   0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root      41916  0.1  0.0 112720   680 pts/0    D+   16:57   0:00 grep --color=auto filebeat

ls /var/log/filebeat/filebeat 

/var/log/filebeat/filebeat

less !$
 


2019-06-21T16:54:31+08:00 INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-06-21T16:54:31+08:00 INFO Beat UUID: 77c66cbe-be20-482b-8e31-4dc91e3390d2
2019-06-21T16:54:31+08:00 INFO Setup Beat: filebeat; Version: 6.0.0
2019-06-21T16:54:31+08:00 INFO Elasticsearch url: http://192.168.193.128:9200
2019-06-21T16:54:31+08:00 INFO Metrics logging every 30s
2019-06-21T16:54:31+08:00 INFO Beat name: axinlinux-03
2019-06-21T16:54:31+08:00 INFO filebeat start running.
2019-06-21T16:54:31+08:00 INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2019-06-21T16:54:31+08:00 INFO Loading registrar data from /var/lib/filebeat/registry
2019-06-21T16:54:31+08:00 INFO States Loaded from registrar: 0
2019-06-21T16:54:31+08:00 INFO Loading Prospectors: 1
2019-06-21T16:54:31+08:00 INFO Starting prospector of type: log; id: 13761481236662083215 
2019-06-21T16:54:31+08:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2019-06-21T16:54:31+08:00 INFO Starting Registrar
2019-06-21T16:54:31+08:00 INFO Harvester started for file: /var/log/elasticsearch/aminglinux.log
2019-06-21T16:54:31+08:00 INFO Config reloader started
2019-06-21T16:54:31+08:00 INFO Loading of config files completed.
2019-06-21T16:54:40+08:00 INFO Connected to Elasticsearch version 6.8.0
2019-06-21T16:54:43+08:00 INFO Loading template for Elasticsearch version: 6.8.0
/var/log/filebeat/filebeat

curl '192.168.193.128:9200/_cat/indices?v'

health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   system-syslog-2019.06     fqLpMdxRTG2EAV5DC8eIMw   5   1                                                  
yellow open   .kibana                   HUhL8JS6Sgqxr8mSq9UgoQ   1   1                                                  
yellow open   nginx-test-2019.06.21     MHZFKf9CTH-wY6YzgD6A7g   5   1                                                  
yellow open   filebeat-6.0.0-2019.06.21 y0O_XA9gQZ-kCvx_IguilQ   3   1 

然后,可以在kibana上创建新的索引

 

 

 

 

 

 

 

 

 

 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

 

 

 

 

扩展部分

 

 

 

x-pack 收费,免费  http://www.jianshu.com/p/a49d93212eca

https://www.elastic.co/subscriptions

Elastic stack演进 http://70data.net/1505.html

基于kafka和elasticsearch,linkedin构建实时日志分析系统 http://t.cn/RYffDoE  

使用redis http://blog.lishiming.net/?p=463

ELK+Filebeat+Kafka+ZooKeeper 构建海量日志分析平台 https://www.cnblogs.com/delgyd/p/elk.html

http://www.jianshu.com/p/d65aed756587

转载于:https://my.oschina.net/u/3866192/blog/3064838