Filebeat 采集 nginx日志

来自Linux78|wiki
Bob（讨论 | 贡献）2019年11月15日 (五) 17:50的版本 （创建页面，内容为“nginx 日志格式规范： 将日志配置组合添加至nginx主配置文件； log_format eslog '$remote_addr - $remote_user [$time_local] "$request" ' '$s…”）
(差异) ←上一版本 | 最后版本 (差异) | 下一版本→ (差异)

nginx 日志格式规范： 将日志配置组合添加至nginx主配置文件；

log_format eslog '$remote_addr - $remote_user [$time_local] "$request" ' 

   '$status $body_bytes_sent "$http_referer" '
   '"$http_user_agent" "$http_x_forwarded_for"';

ES集群安装插件 集群收集nginx日志需要安装 ingest-user-agent ，ingest-geoip ，安装完成之后需要重启es 服务，否则数据无法正常录入es集群。

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip /usr/share/elasticsearch/bin/elasticsearch-plugin list 升级java至1.8 yum install java-1.8 -y 安装filebeat 导入key rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch 创建repo 文件 more lostash.repo

[elastic-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md

安装filebeat yum install filebeat -y 修改filebeat主配置文件 more /etc/filebeat/filebeat.yml filebeat.config: prospectors: path: ${path.config}/prospectors.d/.yml reload.enabled: false modules: path: /etc/filebeat/modules.d/.yml reload.enabled: false output.elasticsearch: hosts: ['10.2.3.30:9200'] setup.kibana: host: "10.2.3.13:5601" 配置filebeat nginx 模块 Access 日志：/data/nginxlog/eslog/es-access.* Error 日志：/data/nginxlog/eslog/es-error.* more /etc/filebeat/modules.d/nginx.yml - module: nginx 

 # Access logs
 access:
   enabled: true
   # Set custom paths for the log files. If left empty,
   # Filebeat will choose the paths depending on your OS.
   var.paths: ["/data/nginxlog/eslog/es-access.*"]
 # Error logs
 error:
   enabled: true
   # Set custom paths for the log files. If left empty,
   # Filebeat will choose the paths depending on your OS.
   var.paths: ["/data/nginxlog/eslog/es-error.*"]

添加启动

systemctl enable filebeat 开启nginx 模块

cd /etc/filebeat filebeat modules enable nginx filebeat modules list 初始化环境 此操作会自动导入filebeat模板和nginx dashboard 到es 集群：

Set up the initial environment: Loaded index template Loading dashboards (Kibana must be running and reachable) Loaded dashboards Loaded machine learning job configurations ./filebeat setup -e 启动服务

systemctl start filebeat 启动服务报错 内容如下：

Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern: 

 error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: returned 400 to import file: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"Request Timeout after 30000ms"}

2018-10-31T16:35:45.659+0900 INFO kibana/client.go:113 Kibana url: http://10.2.3.13:5601 2018-10-31T16:37:15.664+0900 ERROR instance/beat.go:743 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern: 

 error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: fail to execute the HTTP POST request: Post http://10.2.3.13:5601/api/kibana/dashboards/import?force=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response:

处理办法：

登陆kibana 进入Dev tools 删除 删除旧模板 

   #DELETE _template/filebeat-6.X.X

删除旧数据 

   #DELETE filebeat-6.4.2-*

重新导入模板： cd /etc/filebeat/ filebeat setup Loaded index template Loading dashboards (Kibana must be running and reachable) Loaded dashboards Loaded machine learning job configurations 重启服务： systemctl restart filebeat

取自“http://www.linux78.com/index.php?title=Filebeat_采集_nginx日志&oldid=491”