Monday, 30 May 2016

How to create GROK patterns ?


Use online Grok Debugger : http://grokdebug.herokuapp.com/

Provide input and pattern, it will parse and provide fields in the JSON string.
Always perform parsing in baby steps so that you will end up with correct pattern.


Grok pattern examples

Data
2016-03-11 09:25:21,165|INFO |com.shaan.Logging [log]|India|143011|Admin|TECHNICAL|cashout|Cash Out|0(success)|34|MSISDN: 8883039090

Pattern #1

%{GREEDYDATA:data}

Output
{
  "data": [
    [
      "2016-03-11 09:25:21,165|INFO |com.shaan.Logging [log]|India|143011|Admin|TECHNICAL|cashout|Cash Out|0(success)|34|MSISDN: 8883039090"
    ]
  ]
}


Pattern #2
%{TIMESTAMP_ISO8601:timestamp}\|%{GREEDYDATA:data}

Output
{
  "timestamp": [
    [
      "2016-03-11 09:25:21,165"
    ]
  ],
  "YEAR": [
    [
      "2016"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],

  ...

  "data": [
    [
"INFO |com.shaan.Logging [log]|India|143011|Admin|TECHNICAL|cashout|Cash Out|0(success)|34|MSISDN: 8883039090"

    ]
  ]
}


Pattern #3
%{TIMESTAMP_ISO8601:timestamp}\|%{DATA:logLevel}\|%{GREEDYDATA:data}

Output
{
  "timestamp": [
    [
      "2016-03-11 09:25:21,165"
    ]
  ],
  "YEAR": [
    [
      "2016"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],

  ...


  "logLevel": [
    [
      "INFO "
    ]
  ],
  "data": [
    [
"com.shaan.Logging [log]|India|143011|Admin|TECHNICAL|cashout|Cash Out|0(success)|34|MSISDN: 8883039090"
    ]
  ]
}



Pattern #4
%{TIMESTAMP_ISO8601:timestamp}\|%{DATA:logLevel}\|%{DATA:eventSource}\|%{DATA:country}\|%{DATA:userId}\|%{DATA:role}\|%{DATA:logType}\|%{DATA:operation}\|%{INT:resultCode}\(%{DATA:result}\)\|%{NUMBER:timeConsumed}\|%{GREEDYDATA:data}

Output
{
  "timestamp": [
    [
      "2016-03-11 09:25:21,165"
    ]
  ],
  "YEAR": [
    [
      "2016"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],

  ...

  "logLevel": [
    [
      "INFO "
    ]
  ],
  "eventSource": [
    [
      "com.shaan.Logging [log]"
    ]
  ],
  "country": [
    [
      "India"
    ]
  ],
  "userId": [
    [
      "143011"
    ]
  ],

  "role": [
    [
      "Admin"
    ]
  ],
  "logType": [
    [
      "TECHNICAL"
    ]
  ],
  "operation": [
    [
      "Cash Out"
    ]
  ],
  "resultCode": [
    [
      "0"
    ]
  ],
  "result": [
    [
      "success"
    ]
  ],
  "timeConsumed": [
    [
      "34"
    ]
  ],
   "data": [
    [
      "MSISDN: 8883039090"
    ]
  ]
}

No comments:

Post a Comment

Note: only a member of this blog may post a comment.