ELK使用问题汇总

关于ElasticSearch的聚类时出现fielddata=true问题

实例：
PUT nginx-211error-6.3.0-2018.07.09/_mapping/doc
{
   "properties":{ 
    "nginx.error.level":{
      "type":"text",
      "fielddata":true 
    } 
  } 
}

举例：
PUT $my_index/_mapping/$my_type
{
   "properties":{ 
    "my_field":{
      "type":"text",
      "fielddata":true 
    } 
  } 
}

关于filebeat output ElasticSearch时区分索引的配置：

Examples elasticsearch output with indices:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "logs-%{[beat.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "critical-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "CRITICAL"
    - index: "error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "ERR"

Example elasticsearch output with pipelines:

filebeat.inputs:
- type: log
  paths: ["/var/log/app/normal/*.log"]
  fields:
    type: "normal"
- type: log
  paths: ["/var/log/app/critical/*.log"]
  fields:
    type: "critical"

output.elasticsearch:

hosts: ["http://localhost:9200"]
  index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
  pipelines:
    - pipeline: critical_pipeline
      when.equals:
        fields.type: "critical"
    - pipeline: normal_pipeline
      when.equals:
        fields.type: "normal"

关于kibana展示日志相差8小时的解决方案：

https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html

https://elasticsearch.cn/question/4754

{
    "date": {
      "field": "nginx.error.time",
      "target_field": "@timetamp",
      "formats": ["YYYY/MM/dd H:m:s"]
      "timezone": "Asia/Shanghai"
    }
  }

关于logstash输出到es展示日志缺少时间戳的解决方案(可实现动态索引)：

filter {
....
  date {
    match => [ "tomcat_datestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
    target => "@timestamp"
  }
  ruby { 
    code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)" 
  } 
  mutate { 
    add_field => { "remote_ip" => "192.168.2.15" }
    convert => ["timestamp", "string"] 
    gsub => [ "message", "\r", "" ]  
    gsub => ["timestamp", "T([\S\s]*?)Z", ""] 
    gsub => ["timestamp", "-", "."] 
  }
}

output {
  if [type] == "java-log-utf-8" {
    elasticsearch {
    hosts => ["192.168.2.245:9200"]
    index => "java-log-utf-8-%{timestamp}"
    }
  }
}
....

关于logstash收集java日志乱码/错误堆栈/排除文件的解决方案：

input {
  file {
    path => "/data/java_log/*"
    type => "java-log-utf-8"
    #排除目录下.gz的文件
    exclude => "*.gz"
      start_position => "beginning"
      stat_interval => "1"
      codec => multiline {
      #文件编码
      charset => "UTF-8" 
      #匹配开头是时间戳的行
      pattern => "^%{TIMESTAMP_ISO8601} "
      #正则生效，满足则放弃，反之亦然
      negate => true
      #不匹配则与上一行合并
      what => "previous"
      }
  }

单节点ElasticSearch出现unassigned的原因及解决办法

导致的原因：副本分片的主要目的就是为了故障转移，正如在 集群内的原理 中讨论的：如果持有主分片的节点挂掉了，一个副本分片就会晋升为主分片的角色。那么可以看出来副本分片和主分片是不能放到一个节点上面的，可是在只有一个节点的集群里，副本分片没有办法分配到其他的节点上，所以出现所有副本分片都unassigned得情况。因为只有一个节点，如果存在主分片节点挂掉了，那么整个集群理应就挂掉了，不存在副本分片升为主分片的情况。解决办法就是，在单节点的elasticsearch集群，删除存在副本分片的索引，新建索引的副本都设为0

#使用PUT方法设置索引副本数为0即可
PUT nginx-128-errorlogs-2018.10.31/_settings
{
  "number_of_replicas": 0
}

ELK中文分词器的安装与使用

项目地址：https://github.com/medcl/elasticsearch-analysis-ik

kiabna开发工具中-GROK的使用方法

• (?<TOMCAT_DATESTAMP>20(?>\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])(?:2[0123]|[01]?[0-9]):?(?:[0-5][0-9])(?::?(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))
• (? \S+)
• (?<LOG_LEVEL> ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)\s?)
• (?<COM_RONGLIAN> \S+)
• (? \S+)
• (? .*)

logstash output elasticsearch 索引分片数的设置

#官网文档如是解释：
The number of primary shards that an index should have. Defaults to 5. This setting can only be set at index creation time.

curl -XPUT 'http://192.168.1.241:9200/_template/logstash-*' -H 'Content-Type: application/json' -d'{
    "index_patterns" : ["*"],
    "order" : 0,
    "settings" : {
        "number_of_shards" : 2
    }
}'