How to create Logstash configuration ?

Create Logstash configuration

Preferably create a configuration file under Logstash installation (may be in a conf dir)

> cd /home/osadmin/ELK/logstash-2.3.2

> vim myapp-logstash.conf

-rw-r--r--  1  root root 809 Mar 2 06:01  myapp-logstash.conf

Specify the input log file / stream, parsing of the log and output to ES in 3 parts :

• input : Provide Source file path and start position
  

     OR

              Filebeat as source
              OR
              rsyslog as input stream
              etc.

• filter : Filter the logs and drop other logs
             Example, Filter the logs containing |FUNC| and drop other logs
• Use GROK to tokenize the message to different fields (using DATA or NUMBER and remaining string using GREEDYDATA)
• Use "mutate" for :
• Field value format conversions
• Add a new field, if required
• Modify (Replace) any field, if required
• Remove extra or unused field

• output : Provide Target ES instance (ip and port), Index naming convention, Template file

input {

  file {

    path => "/usr/logs/myapp.log.*"

    exclude => "*.tar"

    start_position => "beginning"

  }

filter {

  if [message] =~ "\|FUNC\|" {

    grok {    

       match => { "message" => "%{TIMESTAMP_ISO8601:timestamp},%{NUMBER:lineNumber:int}\|%{DATA:logLevel}\|%{DATA:eventSource}\|%{DATA:mco}\|%{DATA:userId}\|%{DATA:role}\|%{DATA:logType}\|%{DATA:service}\|%{DATA:operation}\#%{DATA:operationLevel}\|%{NUMBER:resultCode:int}\(%{DATA:result}\)\|%{NUMBER:timeConsumed:int}\|%{ GREEDYDATA:data}"

       }

    }

    mutate {

          convert => { "timeConsumed" => "float" }

    }

  } else {

     drop { }

  }

  

  if [resultCode] == 0 {

        mutate {

                add_field => {"status" => "OK"}

        }

  } else {

        mutate {

                add_field => {"status" => "KO"}

        }

  }

  if [operationLevel] == "0" {

        mutate {

                replace => {"operationLevel" => "Read"}

        }

  } else if [operationLevel] == "1" {

        mutate {

                replace => {"operationLevel" => "Search"}

        }

  } else if [operationLevel] == "2" {

        mutate {

                replace => {"operationLevel" => "Change"}

        }

  } else if [operationLevel] == "3" {

        mutate {

                replace => {"operationLevel" => "Admin"}

        }

  }

   mutate {

      remove_field => [ "message", "data" ]

   }

}

output {

 elasticsearch {

   hosts => ["10.170.200.53:9200"]

   index => "myapp-logs-%{+YYYY.MM.dd}"

 }

 stdout { codec => rubydebug }

}

* Template can be used to define data parsing rules for ES : 
Example : Define inside ES definition

   hosts => ["10.170.208.53:9200"]

   index => "myapp-logs-%{+YYYY.MM.dd}"

   template_name => "myapp-template"

   template => "/home/osadmin/ELK/logstash-2.3.2/myapp-template.json"

   template_overwrite => true