Parsing Quoted Strings and DateTime Offset - GROK and Logstash - docker

Question

With Grok Debuger I am trying to parse some custom data:
1 1 "Device 1" 1 "Input 1" 0 "On" "Off" "2020-01-01T00:00:00.1124303+00:00"
So far I have:
%{INT:id} %{INT:device} %{QUOTEDSTRING:device_name} %{INT:input}
%{QUOTEDSTRING:input_name} %{INT:state} %{QUOTEDSTRING:on_phrase}
%{QUOTEDSTRING:off_phrase} \"%{TIMESTAMP_ISO8601:when}\"
However, I am getting things like double quotes around strings %{QUOTEDSTRING), and two lots of hours and minutes with the time and date %{TIMESTAMP_ISO8601:when}
{
"id": [
[
"1"
]
],
"device": [
[
"1"
]
],
"device_name": [
[
""Device 1""
]
],
"input": [
[
"1"
]
],
"input_name": [
[
""Input 1""
]
],
"state": [
[
"0"
]
],
"on_phrase": [
[
""On""
]
],
"off_phrase": [
[
""Off""
]
],
"when": [
[
"2020-01-01T00:00:00.1124303+00:00"
]
],
"YEAR": [
[
"2020"
]
],
"MONTHNUM": [
[
"01"
]
],
"MONTHDAY": [
[
"01"
]
],
"HOUR": [
[
"00",
"00"
]
],
"MINUTE": [
[
"00",
"00"
]
],
"SECOND": [
[
"00.1124303"
]
],
"ISO8601_TIMEZONE": [
[
"+00:00"
]
]
}
Also, I am a little stuck when it comes to the logstash.conf as I am not sure what I would put as the index in the output. The following code is from a previous example from github:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
manage_template => false
index => "sample-%{+YYYY.MM.dd}"
}
}
I'm guessing mine would look something like this:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{INT:id} %{INT:device} %{QUOTEDSTRING:device_name} %{INT:input} %{QUOTEDSTRING:input_name} %{INT:state} %{QUOTEDSTRING:on_phrase} %{QUOTEDSTRING:off_phrase} \"%{TIMESTAMP_ISO8601:when}\"" }
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
manage_template => false
index => "sample-%{????????}"
}
}
Again I'm unclear as to what I am supposed to do with "sample-%{????????}"

Answer 1

In regard to the double-double-quotes: just use DATA instead of QUOTEDSTRING:
"%{DATA:device_name}"
Duplicated entries in the hours and minutes come from the timezone: first entry is the actual hour, the second one is the hour of the timezone. Same for the minutes.
To get rid of it you would need a custom pattern:
"(?<when>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?(?<ISO8601_TIMEZONE>Z|[+-](?:2[0123]|[01]?[0-9])(?::?(?:[0-5][0-9])))?)"
(if you are not interested in parsing the timestamp at all, just use DATA again).
So, your pattern might look like this:
%{INT:id} %{INT:device} "%{DATA:device_name}" %{INT:input} "%{DATA:input_name}" %{INT:state} "%{DATA:on_phrase}" "%{DATA:off_phrase}" "(?<when>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?(?<ISO8601_TIMEZONE>Z|[+-](?:2[0123]|[01]?[0-9])(?::?(?:[0-5][0-9])))?)"
Regarding index:
you can omit it completely then the default one is used: logstash-%{+YYYY.MM.dd}
you can use sample-%{+YYYY.MM.dd} if you want to have separate indexes for each day
you can use sample- to have just one index
you can use any other combination of the fields in your index pattern