Filebeat收集Tomcat日志

我们只需要安装好tomcat，然后将 tomcat 修改为 json 格式日志，在使用 filebeat 进行收集即可；

mkdir -p /soft/ && cd /soft
wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.26/bin/apache-tomcat9.0.26.tar.gz
tar xf apache-tomcat-9.0.26.tar.gz
ln -s /soft/apache-tomcat-9.0.26 /soft/tomcat

修改tomcat server.xml文件，修改日志格式
<Host name="elk.qingchen.net"  appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
  prefix="json_elk_log" suffix=".txt"
    pattern="
     {"clientip":"%h","
     ClientUser":"%l","
     authenticated":"%u","
     AccessTime":"%t","
     method":"%r","
     status":"%s","
     SendBytes":"%b","
     Query?string":"%q","
     partner":"%{Referer}i","
     AgentVersion":"%{User-Agent}i"}" /> 
</Host>

启动tomcat
/soft/tomcat/bin/startup.sh

检查访问日志是否为json格式
cat /soft/tomcat/logs/json_elk_log.2021-10-30.txt 
{"clientip":"10.0.0.1","      ClientUser":"-","    authenticated":"-","    AccessTime":"[30/Oct/2021:11:08:45 +0800]","    
method":"GET / HTTP/1.1","    status":"200","    SendBytes":"200","    Query?string":"","    partner":"-","    
AgentVersion":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"}

cat filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths: /soft/tomcat/logs/json_elk_log*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

output.elasticsearch:
  hosts: ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
  index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.ilm.enabled: false
setup.template.name: "tomcat"       #定义模板名称
setup.template.pattern: "tomcat-*"  #定义模板的匹配索引名称

systemctl restart filebeat

例1: Tomcat正常日志是以 “日期” 开头的。而报错日志中间的错误都不是以 “日期” 开头的。
所以我们可以匹配以 “日期” 开头的一直到下一个日期出现则为一个事件日志。
例2: Elasticsearch正常日志是以 [] 开头的。而报错日志中间的错误信息不是以 [] 开头，所以可以匹配以 [开头的行，一直到下一个 [开头的出现则为一个事件日志。

官方例子
https://www.elastic.co/guide/en/beats/filebeat/7.4/multiline-examples.html

cat filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths: /soft/tomcat/logs/json_elk_log*.txt
  json.keys_under_root: true    # 默认为False; 就是将所有的日志记录到Message字段；true不存储至Message字段
  json.overwrite_keys: true     # 会覆盖掉Message字段的内容，然后使用自行定义的Json格式的Key作为字段，来存储对应的值
  tags: ["access"]

- type: log
  enabled: true
  paths: /soft/tomcat/logs/catalina.out
  tags: ["error"]
  multiline.pattern: '^\d{2}'  #匹配以两个数字开头的
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000       # 最大的合并行数 默认合并的数量是500
output.elasticsearch:
  hosts: ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
  indices:
    - index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称
      when.contains:
        tags: "access"
    - index: "tomcat-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "error"
setup.ilm.enabled: false
setup.template.name: "tomcat"       #定义模板名称
setup.template.pattern: "tomcat-*"  #定义模板的匹配索引名称

cat filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths: /var/logs/elasticsearch/my-es.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
output.elasticsearch:
  hosts: ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
  index: "es-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称
setup.ilm.enabled: false
setup.template.name: "es"       #定义模板名称
setup.template.pattern: "es-*"  #定义模板的匹配索引名称