I have been using logstash with gelf already and wanted to check out fluent input (mainly due to TCP based docker log-driver for fluent as opposed to UDP only gelf). My configuration for testing is this:
input {
gelf {
port => 12345
}
tcp {
codec => fluent
port => 23456
}
}
filter {
}
output {
stdout { codec => rubydebug { metadata => true } }
}
I can send gelf logs using:
docker run -it \
--log-driver gelf \
--log-opt gelf-address=udp://localhost:12345 \
--log-opt tag=gelf-test \
ubuntu:16.04 /bin/bash -c 'echo $(date -u +"%Y-%m-%dT%H:%M:%SZ") Hello gelf'
However the fluent version does not work:
docker run -it \
--log-driver fluentd \
--log-opt fluentd-address=localhost:23456 \
--log-opt tag=fluent-test \
ubuntu:16.04 /bin/bash -c 'echo $(date -u +"%Y-%m-%dT%H:%M:%SZ") Hello fluent'
I can verify that logstash is receiving input:
echo 'Hello TCP' | nc localhost 23456
An error occurred. Closing connection {:client=>"172.17.0.1:42012", :exception=>#, :backtrace=>["org/jruby/RubyTime.java:1073:in at'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.4.0-java/lib/logstash/timestamp.rb:32:inat'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-fluent-2.0.4-java/lib/logstash/codecs/fluent.rb:41:in decode'", "org/msgpack/jruby/MessagePackLibrary.java:195:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-fluent-2.0.4-java/lib/logstash/codecs/fluent.rb:40:in decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:153:inhandle_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:143:in `server_connection_thread'"], :level=>:error}
I also disabled the fluent codec and sent fluent logs and logstash properly errors there as well and parses the fluent msgpack as message field of a regular TCP event as expected.
Received an event that has a different character encoding than you
configured.
{:text=>"\x94\xABfluent-test\xD2X¢鄣log\xD9\\"2017-03-10T12:58:17Z
Hello
fluent\r\xACcontainer_id\xD9#9cbd13eb83a02a1a4d4f83ff063d4e40b4419b7dcbcef960e4689495caa5c132\xAEcontainer_name\xAF/ecstatic_kilby\xA6source\xA6stdout\xC0",
:expected_charset=>"UTF-8", :level=>:warn}
{
"message" => "\\x94\\xABfluent-test\\xD2X¢鄣log\\xD9\\\"2017-03-10T12:58:17Z Hello fluent\\r\\xACcontainer_id\\xD9#9cbd13eb83a02a1a4d4f83ff063d4e40b4419b7dcbcef960e4689495caa5c132\\xAEcontainer_name\\xAF/ecstatic_kilby\\xA6source\\xA6stdout\\xC0",
"#version" => "1",
"#timestamp" => "2017-03-10T12:58:18.069Z",
"host" => "172.17.0.1",
"port" => 42016
}
I have no other ideas, has anybody run into this issue or have any ideas on how to debug further?
would you please try a Fluentd instance ?, on that way would be easier to determinate where the issue is. Doing a quick view looks like Logstash Fluent codec is not working properly.
Unfortunately you can't send messages from fluentd directly to logstash using the existing plugins (it's a shame really).
If you wish to do so, use this open-source plugin which sends the data directly to logstash tcp input (no need for fluentd codec) and also support sending data via secured SSL/TLS protocol.
Seen on this thread.
Related
I've googled the f out of this.
here is the log driver config for my docker container in the compose file
driver: gelf
options:
gelf-address: "http://graylog:12201"
I created a GELF HTTP input in the admin console.
I know that graylog is accessible at 12201, because if I ssh into a container and run
curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "test" }' 'http://graylog:12201/gelf'
Then I can see the log message.
The problem is, it seems like I have to add the /gelf to the address, but docker complains if I try to do that. But no other curl commands work without it, and I can't seem to get it to work with TCP or UDP at all. So.... what am I doing wrong?
I have a MySQL, Logstash, and ES setup but I need to set some fields to keyword type instead of text. I've read that it is not possible to do this in Logstash (logstash.conf) and so it needs to be done in ES. I've followed a similar question here and slightly modified it to PUT a mapping but I have got this error: "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [es.path.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
I am using docker-compose to start all the services at once under the same network, and so the mapping must be specified before logstash ports the data to ES. (Mapping can't be changed on a non-empty index).
I have seen other questions and they do seem a bit old so I wanted to ask if there is a better approach to doing this now.
My mapping.json
{
"mappings": {
"properties": {
"authors": {"type": "keyword"},
"tags": {"type": "keyword"}
}
}
}
Dockerfile
FROM elasticsearch:7.5.1
COPY ./docker-entrypoint.sh .
COPY ./mapping.json .
RUN mkdir /data && chown -R elasticsearch:elasticsearch /data && echo 'es.path.data: /data' >> config/elasticsearch.yml && echo 'path.data: /data' >> config/elasticsearch.yml
ADD https://raw.githubusercontent.com/vishnubob/wait-for-it/e1f115e4ca285c3c24e847c4dd4be955e0ed51c2/wait-for-it.sh /utils/wait-for-it.sh
# Copy the files you may need and your insert script
RUN ./docker-entrypoint.sh elasticsearch -p /tmp/epid & /bin/bash /utils/wait-for-it.sh -t 0 localhost:9200 -- curl -X PUT 'http://localhost:9200/cnas_publications' -d #./mapping.json; kill $(cat /tmp/epid) && wait $(cat /tmp/epid); exit 0;
Edit: I've used the docker-entrypoint.sh from the official repo here
It seems that I was mistaken, and it is actually possible to define the mapping in Logstash. Assuming you're using the official elasticsearch image, create a ES template and make a volume with it to the logstash container.
Here's a sample of my output of my logstash.conf
output {
stdout { codec => "rubydebug" }
elasticsearch {
hosts => "http://elasticsearch:9200"
index => "test"
template => "/logstash/mapping.json"
template_name => "mapping"
document_id => "%{[#metadata][_id]}"
}
}
and don't forget to set index_patterns in your ES template.
I'm trying to set up an interaction between running Docker container's logs and Logstash.
I run my Docker container with the following command:
docker run --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:12201 nfrankel/simplelog:1
and the Logstash config.json is:
input {
gelf {}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout {}
}
Logstash logs are fine, I see
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
Starting gelf listener (udp) ... {:address=>"0.0.0.0:12201"}
Successfully started Logstash API endpoint {:port=>9600}
Nevertheless, it doesn't work. I don't see either logs in Logstash console or that Elasticsearch index is created.
Could you help me to resolve my issue?
Worth to mention that the running Docker container produces logs and I see them in a Cygwin from where I launched it.
Perhaps try configuring the gelf input to accept udp on port 12201 e.g.
input {
gelf {
use_udp => true
port_udp => 12201
}
}
I'm trying to run the logstash container in red hat 7 with the command:
docker run -v /home/logstash/config:/conf -v /home/docker/logs:/logs logstash logstash -f /conf/logstash.conf --verbose
and the response received is:
{:timestamp=>"2016-05-05T10:21:20.765000+0000", :message=>"translation missing: en.logstash.runner.configuration.file-not-found", :level=>:error}
{:timestamp=>"2016-05-05T10:21:20.770000+0000", :message=>"starting agent", :level=>:info}
and the logstash container is not running.
If I execute the folowing command:
docker run -dit -v /home/logstash/config:/conf -v /home/docker/logs:/logs --name=logstash2 logstash logstash -e 'input { stdin { } } output { stdout { } }'
Enter in the container with the command :
docker exec -ti bash
and execute:
logstash -f /conf/logstash.conf
A new logstash proccess is now running in the container and I can manage log files setted up in the config file.
Any idea why am i having this strange behaviour?
Thanks, for all.
Problem solved. It was a directory permissions in the host machine.Thanks for helping #NOTtardy
Try making double quotes around logstash -f /conf/logstash.conf --verbose
Edit: just tried you command myself and it works fine. Could be that your logstash.conf is the reason for the error.
I've used a very simple logstash.conf
input {
udp {
port => 5000
type => syslog
}
}
output {
stdout {
codec => rubydebug
}
}
And got the following output
{:timestamp=>"2016-05-05T18:24:26.294000+0000", :message=>"starting agent", :level=>:info}
{:timestamp=>"2016-05-05T18:24:26.332000+0000", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2016-05-05T18:24:26.370000+0000", :message=>"Starting UDP listener", :address=>"0.0.0.0:5000", :level=>:info}
{:timestamp=>"2016-05-05T18:24:26.435000+0000", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_inflight=>125, :level=>:info}
{:timestamp=>"2016-05-05T18:24:26.441000+0000", :message=>"Pipeline main started"}
I have a set of dockerized applications scattered across multiple servers and trying to setup production-level centralized logging with ELK. I'm ok with the ELK part itself, but I'm a little confused about how to forward the logs to my logstashes.
I'm trying to use Filebeat, because of its loadbalance feature.
I'd also like to avoid packing Filebeat (or anything else) into all my dockers, and keep it separated, dockerized or not.
How can I proceed?
I've been trying the following. My Dockers log on stdout so with a non-dockerized Filebeat configured to read from stdin I do:
docker logs -f mycontainer | ./filebeat -e -c filebeat.yml
That appears to work at the beginning. The first logs are forwarded to my logstash. The cached one I guess. But at some point it gets stuck and keep sending the same event
Is that just a bug or am I headed in the wrong direction? What solution have you setup?
Here's one way to forward docker logs to the ELK stack (requires docker >= 1.8 for the gelf log driver):
Start a Logstash container with the gelf input plugin to reads from gelf and outputs to an Elasticsearch host (ES_HOST:port):
docker run --rm -p 12201:12201/udp logstash \
logstash -e 'input { gelf { } } output { elasticsearch { hosts => ["ES_HOST:PORT"] } }'
Now start a Docker container and use the gelf Docker logging driver. Here's a dumb example:
docker run --log-driver=gelf --log-opt gelf-address=udp://localhost:12201 busybox \
/bin/sh -c 'while true; do echo "Hello $(date)"; sleep 1; done'
Load up Kibana and things that would've landed in docker logs are now visible. The gelf source code shows that some handy fields are generated for you (hat-tip: Christophe Labouisse): _container_id, _container_name, _image_id, _image_name, _command, _tag, _created.
If you use docker-compose (make sure to use docker-compose >= 1.5) and add the appropriate settings in docker-compose.yml after starting the logstash container:
log_driver: "gelf"
log_opt:
gelf-address: "udp://localhost:12201"
Docker allows you to specify the logDriver in use. This answer does not care about Filebeat or load balancing.
In a presentation I used syslog to forward the logs to a Logstash (ELK) instance listening on port 5000.
The following command constantly sends messages through syslog to Logstash:
docker run -t -d --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:5000 ubuntu /bin/bash -c 'while true; do echo "Hello $(date)"; sleep 1; done'
Using filebeat you can just pipe docker logs output as you've described. Behavior you are seeing definitely sounds like a bug, but can also be the partial line read configuration hitting you (resend partial lines until newline symbol is found).
A problem I see with piping is possible back pressure in case no logstash is available. If filebeat can not send any events, it will buffer up events internally and at some point stop reading from stdin. No idea how/if docker protects from stdout becoming unresponsive. Another problem with piping might be restart behavior of filebeat + docker if you are using docker-compose. docker-compose by default reuses images + image state. So when you restart, you will ship all old logs again (given the underlying log file has not been rotated yet).
Instead of piping you can try to read the log files written by docker to the host system. The default docker log driver is the json log driver . You can and should configure the json log driver to do log-rotation + keep some old files (for buffering up on disk). See max-size and max-file options. The json driver puts one line of 'json' data for every line to be logged. On the docker host system the log files are written to /var/lib/docker/containers/container_id/container_id-json.log . These files will be forwarded by filebeat to logstash. If logstash or network becomes unavailable or filebeat is restarted, it continues forwarding log lines where it left of (given files have been not deleted due to log rotation). No events will be lost. In logstash you can use the json_lines codec or filter to parse the json lines and a grok filter to gain some more information from your logs.
There has been some discussion about using libbeat (used by filebeat for shipping log files) to add a new log driver to docker. Maybe it is possible to collect logs via dockerbeat in the future by using the docker logs api (I'm not aware of any plans about utilising the logs api, though).
Using syslog is also an option. Maybe you can get some syslog relay on your docker host load balancing log events. Or have syslog write log files and use filebeat to forward them. I think rsyslog has at least some failover mode. You can use logstash syslog input plugin and rsyslog to forward logs to logstash with failover support in case the active logstash instance becomes unavailable.
I created my own docker image using the Docker API to collect the logs of the containers running on the machine and ship them to Logstash thanks to Filebeat. No need to install or configure anything on the host.
Check it out and tell me if it suits your needs: https://hub.docker.com/r/bargenson/filebeat/.
The code is available here: https://github.com/bargenson/docker-filebeat
Just for helping others that need to do this, you can simply use Filebeat to ship the logs. I would use the container by #brice-argenson, but I needed SSL support so I went with a locally installed Filebeat instance.
The prospector from filebeat is (repeat for more containers):
- input_type: log
paths:
- /var/lib/docker/containers/<guid>/*.log
document_type: docker_log
fields:
dockercontainer: container_name
It sucks a bit that you need to know the GUIDs as they could change on updates.
On the logstash server, setup the usual filebeat input source for logstash, and use a filter like this:
filter {
if [type] == "docker_log" {
json {
source => "message"
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
mutate {
rename => { "log" => "message" }
}
date {
match => [ "time", "ISO8601" ]
}
}
}
This will parse the JSON from the Docker logs, and set the timestamp to the one reported by Docker.
If you are reading logs from the nginx Docker image, you can add this filter as well:
filter {
if [fields][dockercontainer] == "nginx" {
grok {
match => { "message" => "(?m)%{IPORHOST:targethost} %{COMBINEDAPACHELOG}" }
}
mutate {
convert => { "[bytes]" => "integer" }
convert => { "[response]" => "integer" }
}
mutate {
rename => { "bytes" => "http_streamlen" }
rename => { "response" => "http_statuscode" }
}
}
}
The convert/renames are optional, but fixes an oversight in the COMBINEDAPACHELOG expression where it does not cast these values to integers, making them unavailable for aggregation in Kibana.
I verified what erewok wrote above in a comment:
According to the docs, you should be able to use a pattern like this
in your prospectors.paths: /var/lib/docker/containers/*/*.log – erewok
Apr 18 at 21:03
The docker container guids, represented as the first '*', are correctly resolved when filebeat starts up. I do not know what happens as containers are added.