I'm really new to fluentd configurations and need help to convert this logstash config to fluentd to get started
filter {
if [syslog5424_host] =~ /apilog/ {
if [syslog5424_msg] =~ /\"ApplicationType\"\:\"API\"/ {
json {
source => "syslog5424_msg"
# Remove syslog5424_msg field only if json filter is successful
remove_field => ["syslog5424_msg", "syslog5424_sd", "syslog5424_proc", "syslog5424_pri", "syslog5424_ver", "syslog_facility", "syslog_facility_code"]
}
mutate {
add_tag => ["API"]
replace => { "type" => "api-dev" }
}
}
else {
mutate {
add_tag => ["API"]
}
}
}
}
Related
I use ELK and filebeat. I send a lot of logs with distincte fields.
logstash config:
input {
beats {
port => 5044
include_codec_tag => false
}
}
filter {
if [type] == "json" {
json {
source => "message"
target => "msg"
}
mutate {
remove_field => ["msg.ecs.version", "ecs.version", "#version"]
}
}
if [type] != "json" {
grok {
match => {
message => ["time=\"%{TIMESTAMP_ISO8601:time}\""]
}
}
date {
match => [ "time", "YYYY-MM-dd'T'HH:mm:ssZZ"]
target => "time"
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
sniffing => true
manage_template => false
index => "%{[source][project]}-%{[source][application]}-%{+YYYY.MM.dd}"
}
}
Some of my message contain location
{
"location": {
"lat": 11.11,
"lon": 22.22
}
}
In elastic I can see my location, (msg.location.lat and msg.location.lon), but I don't know how convert my location to geo_point.
As I understand current index mapping is created by logstash plugin or by elastic search by default template. What and where shoud I write to use my location as geo_point?
I have ror application that in admin dashboard, admin could observe the location of his employee, in my case, I use elk to gather information of employees that contains latitude and longitude and which send to my map based on his movement, My problem is, I have a template that logstash based on template create daily index but recently I found every field in my index that have type changed to text when indexed created.
this is my json that logstash reads:
{"driver_id": 31,"driver_email": "ankith.ravindran#mailinator.com","location": {"latitude": "-35.2824767","longitude": "149.1326453"},"created_at": "2021-06-29 14:28:47", "required_matches": 1, "type": "location"}
this is my logstash.conf file:
input {
file {
path => ["/usr/share/logstash/MPD_LOCATION/*",
"/usr/share/logstash/MPD_LOCATION/*/*",
"/usr/share/logstash/MPD_LOCATION/*/*/*",
"/usr/share/logstash/MPD_LOCATION/*/*/*/*",
"/usr/share/logstash/MPD_LOCATION/*/*/*/*/*"]
start_position => "beginning"
type => "json"
sincedb_path => "/dev/null"
}
}
filter {
mutate {
gsub => ["message","/}+({)/", "}::{"]
}
mutate {
gsub => ["message","/}+( )/", "}::"]
}
split {
field => "message"
terminator => "::"
}
json { source => "message" }
mutate {
add_field => { "uuid" => "D%{driver_id}T%{created_at}" }
rename => {
"[location][latitude]" => "[location][lat]"
"[location][longitude]" => "[location][lon]"
}
convert => {
"[location][lat]" => "float"
"[location][lon]" => "float"
}
}
}
output {
if ([type] == "location") {
elasticsearch {
hosts => "http://elasticsearch:9200"
index => "live_locations_%{+YYYY_MM_dd}"
# manage_template => true
template => "/usr/share/logstash/Template/live_locations.json"
template_name => "live_locations"
# template_overwrite => true
document_id => "%{uuid}"
}
} else if ([type] == "app_info") {
elasticsearch {
hosts => "http://elasticsearch:9200"
index => "app_info_%{+YYYY_MM_dd}"
document_id => "%{uuid}"
}
}
stdout { codec => rubydebug }
}
this is my template file:
{
"settings": {
"index": {
"number_of_shards": 5,
"number_of_replicas": 1
}
},
"mappings": {
"properties": {
"driver_id": { "type": "integer" },
"email": { "type": "text" },
"location": { "type": "geo_point" },
"app-platform": { "type": "text" },
"app-version": { "type": "text" },
"created_at": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},
"required_matches": { "type": "integer" }
}
}
}
for example, I defined type of created_at , date but when index created this field return as text and I can't understand what happened or field of location it's return float so I could not use my index as geo_point, I have to add I use elk in the version of 7.13 and used on docker.
Updated : I have two types of JSON that one of them just returns the location of the employee the second of them just returns app_version and app_platform of the employee that used.
Updated 2 : I change my input from logstash to filebeat but I still have the same problem.
I have a csv file holding longitude and latitude for some of the records (otherwise it's " "). Now I want to use logstash 5.1.2 to ge the data into elasticsearch 5.1.2. I've written the following conf-file but the location field is still mapped to text.
input {
file {
path => "/usr/local/Cellar/logstash/5.1.2/bin/data.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ['logtime', 'text', 'user', 'country', 'location']
separator => ","
}
date {
match => ["logtime", "yyyy-MM-dd HH:mm:ss"]
timezone => "Europe/London"
target => "Date"
}
if [latitude] and [longitude] {
mutate { convert => {"latitude" => "float"} }
mutate { convert => {"longitude" => "float"} }
mutate { rename => {"latitude" => "[location][lat]"} }
mutate { rename => {"longitude" => "[location][lon]"} }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "twitter"}
}
What am I supposed to do to make the location field mapped as geo-point and be able to visualize the points on the map in Kibana 5.1.2? Thanks
You need to create a mapping that maps location to a geo_point. The easiest way to do that is with an index template so that when you start using time based indices, it will auto-create the mapping when a new index is created.
PUT /_template/twitter
{
"order": 0,
"template": "twitter*",
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
Then delete your /twitter index and re-index your data.
The above template says that any index that gets created with the name twitter* will that have any _type's location field turned into a geo_point.
**NOTE: After ES 7.0 Above : types were removed and when creating a mapping it no longer accepts types which is a breaking change
**
My entire system crashes after I change the configuration to add geo location field.
My system runs correctly when my config looks like this:
input {
syslog
{
host => "localhost4"
port => 5140
type => "system"
}
}
filter {
grok { match => { message => [ ".*ipaddr: %{IP:ipaddr}.*" ] }}
grok { match => { message => [ ".*dnsname: %{HOSTNAME:query_name}.*" ] }}
grok { match => { message => [ ".*mal_rank: %{NUMBER:malrank:int}.*" ] }}
grok { match => { message => [ ".*packet_size: %{NUMBER:packetsize:int}.*" ] }}
grok { match => { message => [ ".*source_ip: %{IP:sourceip}.*" ] }}
grok { match => { message => [ ".*dest_ip: %{IP:dest_ip}.*" ] }}
grok { match => { message => [ ".*source_ip: %{IP:src_ip}.*" ] }}
grok { match => { message => [ ".*sport: %{NUMBER:sport:int}.*" ] }}
}
output {
elasticsearch { hosts => ["localhost4:9200"] }
stdout { codec => rubydebug }
}
But when I add the code to my filter
input {
syslog
{
host => "localhost4"
port => 5140
type => "system"
}
}
filter {
grok { match => { message => [ ".*ipaddr: %{IP:ipaddr}.*" ] }}
grok { match => { message => [ ".*dnsname: %{HOSTNAME:query_name}.*" ] }}
grok { match => { message => [ ".*mal_rank: %{NUMBER:malrank:int}.*" ] }}
grok { match => { message => [ ".*packet_size: %{NUMBER:packetsize:int}.*" ] }}
grok { match => { message => [ ".*source_ip: %{IP:sourceip}.*" ] }}
grok { match => { message => [ ".*dest_ip: %{IP:dest_ip}.*" ] }}
grok { match => { message => [ ".*source_ip: %{IP:src_ip}.*" ] }}
grok { match => { message => [ ".*sport: %{NUMBER:sport:int}.*" ] }}
geoip {
source => "ipaddr"
target => "geoip"
add_tag => ["geoip"]
database => "/etc/logstash/GeoLiteCity.dat"
}
}
output {
elasticsearch { hosts => ["localhost4:9200"] }
stdout { codec => rubydebug }
}
I can run the curl command and get the correct output
curl http://localhost:9200/logstash-2016.04.19/_mapping/system/field/geoip.location?pretty
and returned:
{
"logstash-2016.04.19" : {
"mappings" : {
"system" : {
"geoip.location" : {
"full_name" : "geoip.location",
"mapping" : {
"location" : {
"type" : "geo_point"
}
}
}
}
}
}
}
But instead of getting anything, my logstash stops reading from the syslog.
Any suggestions?
I'm not sure what the delay is, but if I let the system wait for about an hour it starts processing logs again. I just wanted to to confirm that this code does function properly.
I need to convert the following hash:
{
"item[0][size]" => "12",
"item[0][count]" => "1"
}
to this:
{
"item": {
"0": {
"size": "12",
"count": "1"
}
}
}
Could you please advice on how to achieve that most gracefully? Maybe I can reuse some ActionPack's utility method that is used for parsing parameter strings?
You can reuse a rack lib method Rack::Utils.parse_nested_query
require "rack"
def p p
Rack::Utils.parse_nested_query(p)
end
p 'item[0][size]=12' # => {"item"=>{"0"=>{"size"=>"12"}}}
Found here.
After some research I found a way to parse nested query keys using http://apidock.com/rails/Rack/Utils/parse_nested_query:
Rack::Utils.parse_nested_query('item[0][size]')
=> {
"item" => {
"0" => {
"size" => nil
}
}
}
So it's now possible to do:
items_string = item_hash.to_a.map { |row| row.join('=') }.join('&')
result = Rack::Utils.parse_nested_query(items_string)
=> {
"item" => {
"0" => {
"size" => "12",
"count" => "1"
}
}
}