ELK Docker -- Logstash not working properly - docker

Question

I am trying to run the ELK stash using the popular Docker image on DockerHub, seep/elk.
In my project dir, I have the following two files:
docker-compose.up (just want to see if logstash works, so I'm reading from stdin and writing to stdout rather than elasticsearch):
input { stdin {} }
output { stdout {} }
logstash.conf:
elk:
image: sebp/elk
ports:
- "5605:5601"
- "9200:9200"
- "9300:9300"
- "5044:5044"
volumes:
- /path/to/project/dir/logstash.conf:/usr/share/logstash/config/logstash.conf
When I run docker-compose up elk, the following stack trace is displayed:
elk_1 | * Starting periodic command scheduler cron
elk_1 | ...done.
elk_1 | * Starting Elasticsearch Server
elk_1 | ...done.
elk_1 | waiting for Elasticsearch to be up (1/30)
elk_1 | waiting for Elasticsearch to be up (2/30)
elk_1 | waiting for Elasticsearch to be up (3/30)
elk_1 | waiting for Elasticsearch to be up (4/30)
elk_1 | waiting for Elasticsearch to be up (5/30)
elk_1 | waiting for Elasticsearch to be up (6/30)
elk_1 | waiting for Elasticsearch to be up (7/30)
elk_1 | waiting for Elasticsearch to be up (8/30)
elk_1 | waiting for Elasticsearch to be up (9/30)
elk_1 | waiting for Elasticsearch to be up (10/30)
elk_1 | waiting for Elasticsearch to be up (11/30)
elk_1 | Waiting for Elasticsearch cluster to respond (1/30)
elk_1 | logstash started.
elk_1 | * Starting Kibana5
elk_1 | ...done.
elk_1 | ==> /var/log/elasticsearch/elasticsearch.log <==
elk_1 | [2018-08-11T17:34:41,530][INFO ][o.e.g.GatewayService ] [pIJHFdO] recovered [0] indices into cluster_state
elk_1 | [2018-08-11T17:34:41,926][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.watches] for index patterns [.watches*]
elk_1 | [2018-08-11T17:34:42,033][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.watch-history-7] for index patterns [.watcher-history-7*]
elk_1 | [2018-08-11T17:34:42,099][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.triggered_watches] for index patterns [.triggered_watches*]
elk_1 | [2018-08-11T17:34:42,205][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
elk_1 | [2018-08-11T17:34:42,288][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
elk_1 | [2018-08-11T17:34:42,338][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
elk_1 | [2018-08-11T17:34:42,374][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
elk_1 | [2018-08-11T17:34:42,431][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pIJHFdO] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
elk_1 | [2018-08-11T17:34:42,523][INFO ][o.e.l.LicenseService ] [pIJHFdO] license [f28743a3-8cc3-46ad-8c75-7c096c7afaa7] mode [basic] - valid
elk_1 |
elk_1 | ==> /var/log/logstash/logstash-plain.log <==
elk_1 |
elk_1 | ==> /var/log/kibana/kibana5.log <==
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:kibana#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:elasticsearch#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:xpack_main#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:searchprofiler#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:ml#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:tilemap#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:watcher#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:license_management#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:index_management#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:timelion#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:graph#6.3.2","info"],"pid":247,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:55Z","tags":["status","plugin:monitoring#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:searchprofiler#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:ml#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:tilemap#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:watcher#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:index_management#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:graph#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:security#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:grokdebugger#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:logstash#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["status","plugin:reporting#6.3.2","info"],"pid":247,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":247,"message":"Starting all Kibana monitoring collectors"}
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:34:57Z","tags":["license","info","xpack"],"pid":247,"message":"Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active"}
elk_1 |
elk_1 | ==> /var/log/logstash/logstash-plain.log <==
elk_1 | [2018-08-11T17:35:08,371][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/opt/logstash/data/queue"}
elk_1 | [2018-08-11T17:35:08,380][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/opt/logstash/data/dead_letter_queue"}
elk_1 | [2018-08-11T17:35:08,990][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
elk_1 | [2018-08-11T17:35:09,025][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"aa287931-643e-47ae-bd8e-f982c75b2105", :path=>"/opt/logstash/data/uuid"}
elk_1 | [2018-08-11T17:35:09,779][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
elk_1 | [2018-08-11T17:35:13,753][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[//localhost], manage_template=>false, index=>"%{[#metadata][beat]}-%{+YYYY.MM.dd}", document_type=>"%{[#metadata][type]}", id=>"c4ee5abcf701afed0db36d4aa16c4fc10da6a92bbd615d837cccdf2f368b7802", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_21596240-07d7-4d2e-b4e5-bb68516e5a61", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
elk_1 | [2018-08-11T17:35:13,823][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
elk_1 | [2018-08-11T17:35:15,074][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
elk_1 | [2018-08-11T17:35:15,090][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
elk_1 | [2018-08-11T17:35:15,360][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
elk_1 | [2018-08-11T17:35:15,518][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
elk_1 | [2018-08-11T17:35:15,525][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
elk_1 | [2018-08-11T17:35:15,569][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
elk_1 | [2018-08-11T17:35:16,370][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
elk_1 | [2018-08-11T17:35:16,445][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2c697fd4 run>"}
elk_1 | [2018-08-11T17:35:16,602][INFO ][org.logstash.beats.Server] Starting server on port: 5044
elk_1 | [2018-08-11T17:35:16,643][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
elk_1 | [2018-08-11T17:35:17,096][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
elk_1 |
elk_1 | ==> /var/log/kibana/kibana5.log <==
elk_1 | {"type":"log","#timestamp":"2018-08-11T17:35:20Z","tags":["listening","info"],"pid":247,"message":"Server running at http://0.0.0.0:5601"}
Now, Kibana and Elasticsearch seem to be perfectly fine, while logstash isn't doing anything because when I type something in the terminal, I get no response.
Running ps aux in the container bash terminal, I get the following:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 21332 3592 ? Ss 17:50 0:00 /bin/bash /usr/local/bin/start.sh
root 20 0.0 0.0 29272 2576 ? Ss 17:50 0:00 /usr/sbin/cron
elastic+ 86 18.0 4.4 5910168 1479108 ? Sl 17:50 0:46 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -X
elastic+ 112 0.0 0.0 135668 7328 ? Sl 17:50 0:00 /opt/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
logstash 226 43.6 2.2 5714032 726940 ? SNl 17:50 1:47 /usr/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djav
kibana 243 20.0 0.4 1315812 155744 ? Sl 17:50 0:49 /opt/kibana/bin/../node/bin/node --max-old-space-size=250 --no-warnings /opt/kibana/bin/../src/cli -l /var/log/kibana/kibana5.log
root 245 0.0 0.0 7612 672 ? S 17:50 0:00 tail -f /var/log/elasticsearch/elasticsearch.log /var/log/logstash/logstash-plain.log /var/log/kibana/kibana5.log
root 323 1.3 0.0 21488 3544 pts/0 Ss 17:54 0:00 bash
root 340 0.0 0.0 37656 3300 pts/0 R+ 17:54 0:00 ps aux
Running ll /var/log/logstash/ in the container bash terminal, I get the following:
total 16
drwxr-xr-x 1 logstash logstash 4096 Aug 11 17:51 ./
drwxr-xr-x 1 root root 4096 Jul 26 14:27 ../
-rw-r--r-- 1 root root 0 Aug 11 17:50 logstash.err
-rw-r--r-- 1 logstash logstash 3873 Aug 11 17:51 logstash-plain.log
-rw-r--r-- 1 logstash logstash 0 Aug 11 17:51 logstash-slowlog-plain.log
-rw-r--r-- 1 root root 3964 Aug 11 17:51 logstash.stdout
Now, I did change logstash.conf to have the following:
input { stdin {} }
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
Still when I type something in the terminal, there is nothing in the discover section of Kibana, neither has any index pattern created...
Running ps aux in the container bash terminal, I get the following:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 21332 3600 ? Ss 17:40 0:00 /bin/bash /usr/local/bin/start.sh
root 21 0.0 0.0 29272 2568 ? Ss 17:40 0:00 /usr/sbin/cron
elastic+ 87 12.0 4.5 5912216 1484068 ? Sl 17:40 0:52 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -X
elastic+ 113 0.0 0.0 135668 7332 ? Sl 17:40 0:00 /opt/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
logstash 224 27.8 2.3 5714032 771528 ? SNl 17:40 1:58 /usr/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djav
kibana 241 12.0 0.5 1322444 181228 ? Sl 17:40 0:50 /opt/kibana/bin/../node/bin/node --max-old-space-size=250 --no-warnings /opt/kibana/bin/../src/cli -l /var/log/kibana/kibana5.log
root 246 0.0 0.0 7612 692 ? S 17:40 0:00 tail -f /var/log/elasticsearch/elasticsearch.log /var/log/logstash/logstash-plain.log /var/log/kibana/kibana5.log
root 317 1.0 0.0 21488 3744 pts/0 Ss 17:47 0:00 bash
root 334 0.0 0.0 37656 3356 pts/0 R+ 17:48 0:00 ps aux
Running ll /var/log/logstash/ in the container bash terminal, I get the following:
total 16
drwxr-xr-x 1 logstash logstash 4096 Aug 11 17:41 ./
drwxr-xr-x 1 root root 4096 Jul 26 14:27 ../
-rw-r--r-- 1 root root 0 Aug 11 17:40 logstash.err
-rw-r--r-- 1 logstash logstash 3873 Aug 11 17:41 logstash-plain.log
-rw-r--r-- 1 logstash logstash 0 Aug 11 17:41 logstash-slowlog-plain.log
-rw-r--r-- 1 root root 3964 Aug 11 17:41 logstash.stdout
I have been spending a good amount of time with no luck here, so any help would be highly appreciated!

Answer 1

So, I did find a solution thanks to the owner of the elk image repo.
I followed instructions from this page. That is, I entered the container bash by running docker exec -it <container-name> bash, and then (inside the container terminal) I ran the command /opt/logstash/bin/logstash --path.data /tmp/logstash/data -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'.
The problem was that although the Logstash service had been started, it did not have an an interactive terminal. The command above, addresses this problem.
The following logs were displayed inside the container terminal:
Sending Logstash's logs to /opt/logstash/logs which is now configured via log4j2.properties
[2018-08-12T06:28:28,941][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/tmp/logstash/data/queue"}
[2018-08-12T06:28:28,948][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/tmp/logstash/data/dead_letter_queue"}
[2018-08-12T06:28:29,592][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-12T06:28:29,656][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"29cb946b-2bed-4390-b0cb-9aad6ef5a2a2", :path=>"/tmp/logstash/data/uuid"}
[2018-08-12T06:28:30,634][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-12T06:28:32,911][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-12T06:28:33,646][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-08-12T06:28:33,663][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-08-12T06:28:34,107][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-08-12T06:28:34,205][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-08-12T06:28:34,212][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-08-12T06:28:34,268][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2018-08-12T06:28:34,364][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-08-12T06:28:34,442][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-08-12T06:28:34,496][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5dcf75c7 run>"}
[2018-08-12T06:28:34,602][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
The stdin plugin is now waiting for input:
[2018-08-12T06:28:34,727][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-08-12T06:28:35,607][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}
And the following inside my server terminal:
elk_1 | ==> /var/log/elasticsearch/elasticsearch.log <==
elk_1 | [2018-08-12T06:28:34,777][INFO ][o.e.c.m.MetaDataIndexTemplateService] [jqTz2zS] adding template [logstash] for index patterns [logstash-*]
elk_1 | [2018-08-12T06:28:35,214][INFO ][o.e.c.m.MetaDataCreateIndexService] [jqTz2zS] [logstash-2018.08.12] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]
elk_1 | [2018-08-12T06:28:36,207][INFO ][o.e.c.m.MetaDataMappingService] [jqTz2zS] [logstash-2018.08.12/hiLssj14TMKd5lzBq6tvrw] create_mapping [doc]
Doing so, an index pattern was indeed created inside Kibana and I started to receive messages inside the discover tab.