Docker-compose elastic stack no container tags - docker

Question

I have a setup with docker-compose and the elastic stack. My 'main' container is running a Django application (there are some more containers for metrics, certificates, and so on).
The logging itself works with this setup but I have no container labels or tags in Kibana. So I can't differentiate between logs from different containers (except when I know what I'm looking for).
How do I configure logstash or logspout to label or tag all logs with the container where they're from? In the best case tagging container image and container id.
I tried to add a label to the container but that didn't change anything. I also tried specified logging, with driver syslog and a tag, but that didn't work either.
I guess I have to make a specific logstash config and do some stuff there?
Below is my current docker-compose.yml
version: '2'
services:
# django container
web:
build: .
command: gunicorn backend.wsgi:application --bind 0.0.0.0:8001 --log-level debug
restart: unless-stopped
container_name: web
depends_on:
- logspout
expose:
- 8001
env_file:
- ./environments/web.test.env
image: mycontainer
labels:
container: "web"
com.example.service: "web"
logspout:
image: gliderlabs/logspout:v3.2.11
command: 'udp://logstash:5000'
restart: unless-stopped
links:
- logstash
volumes:
- '/var/run/docker.sock:/tmp/docker.sock'
depends_on:
- elasticsearch
- logstash
- kibana
logstash:
image: logstash:7.9.1
restart: unless-stopped
environment:
- STDOUT=true
links:
- elasticsearch
expose:
- 5000
depends_on:
- elasticsearch
- kibana
command: 'logstash -e "input { udp { port => 5000 } } output { elasticsearch { hosts => elasticsearch } }"'
kibana:
image: kibana:7.9.1
restart: unless-stopped
links:
- elasticsearch
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
ports:
- 5601:5601
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:7.9.1
restart: unless-stopped
ports:
- 9200:9200
- 9300:9300
environment:
- node.name=elasticsearch
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- cluster.initial_master_nodes=elasticsearch
Any help would be appreciated, thanks!

Answer 1

Sorry, I'm really inexperienced with the elastic stack, but I got it right.
Indeed you have to provide a logstash config with filter, at least that's how I got it working. Additionally, I had to switch from UDP to just syslog in logspout, I guess the udp connection didn't forward all it got (for example the docker image).
Here are my configurations that work (there are definitely some improvements to do).
logstash.conf
input {
syslog {
port => 5000
type => "docker"
}
}
filter {
grok {
match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:ver} +(?:%{TIMESTAMP_ISO8601:ts}|-) +(?:%{HOSTNAME:service}|-) +(?:%{NOTSPACE:containerName}|-) +(?:%{NOTSPACE:proc}|-) +(?:%{WORD:msgid}|-) +(?:%{SYSLOG5424SD:sd}|-|) +%{GREEDYDATA:msg}" }
}
syslog_pri { }
}
output {
elasticsearch { hosts => "elasticsearch" }
stdout {codec => rubydebug}
}
docker-compose.yml
version: '2'
services:
web:
build: .
command: gunicorn backend.wsgi:application --bind 0.0.0.0:8001 --log-level debug
restart: unless-stopped
container_name: web
depends_on:
- logspout
image: myimage
expose:
- 8001
env_file:
- ./environments/web.test.env
labels:
container: "web"
com.example.service: "web"
logspout:
image: gliderlabs/logspout:v3.2.11
command: 'syslog://logstash:5000'
restart: unless-stopped
links:
- logstash
volumes:
- '/var/run/docker.sock:/tmp/docker.sock'
depends_on:
- elasticsearch
- logstash
- kibana
logstash:
image: logstash:7.9.1
restart: unless-stopped
environment:
- LOGSPOUT=ignore
links:
- elasticsearch
depends_on:
- elasticsearch
- kibana
volumes:
- ./containers/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
kibana:
image: kibana:7.9.1
restart: unless-stopped
links:
- elasticsearch
environment:
- LOGSPOUT=ignore
- ELASTICSEARCH_URL=http://elasticsearch:9200
ports:
- 5601:5601
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:7.9.1
restart: unless-stopped
ports:
- 9200:9200
- 9300:9300
environment:
- LOGSPOUT=ignore
- node.name=elasticsearch
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- cluster.initial_master_nodes=elasticsearch