Distinguishing among multiple syslog docker sources - docker

I am setting up a docker-compose with several services, all of them writing to a common syslog container / service... which actually is a logstash service (a complete elk image as a matter of fact) with the logstash-input-syslog plugin enabled..
kind of as follows (using custom 5151 port since default 514 was giving me a hard time due to permission issues):
services:
elk-custom:
image: some_elk_image
ports:
- 5601:5601
- 9200:9200
- 5044:5044
- 5151:5151
service1:
image: myservice1_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5151"
service2:
image: myservice2_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5151"
My question is how can I add a field (an option rather under logging) so that each log entry in logstash ends up with a field, whose value will help determine whether the log came from service1 or service2.
I kind of managed to do this using the tag field, but the information ends up being part of the message, and not a separate field which I can use for queries in elasticsearch.
For the time being, kibana displays log entries as follows:
#timestamp:September 26th 2017, 12:00:47.684 syslog_severity_code:5
port:53,422 syslog_facility:user-level #version:1 host:172.18.0.1
syslog_facility_code:1 message:<27>Sep 26 12:00:47 7705a2f9b22a[2128]:
[pid: 94|app: 0|req: 4/7] 172.18.0.1 () {40 vars in 461 bytes} [Tue
Sep 26 09:00:47 2017] GET /api/v1/apikeys => generated 74 bytes in 5
msecs (HTTP/1.1 401) 2 headers in 81 bytes (1 switches on core 0)
type:syslog syslog_severity:notice tags:_grokparsefailure
_id:AV69atD4zBS_tKzDPfyh _type:syslog _index:logstash-2017.09.26 _score: -
From what I also know, we cannot define custom syslog-facilities since they are predefined by the syslog RFC.
Thx.

Ended up using port multiplexing and adding custom field based on this condition:
docker-compose.yml
elk-custom:
image: some_elk_image
ports:
- 5601:5601
- 9200:9200
- 5044:5044
- 5151:5151
- 5152:5152
service1:
image: myservice2_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5151"
service2:
image: myservice2_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5152"
logstash-conf
input {
tcp {
port => 5151
type => syslog
add_field => {'received_from' => 'service1'}
}
tcp {
port => 5152
type => syslog
add_field => {'received_from' => 'service2'}
}
}

Related

How to reduce the amount of chunks to prevent running out of disk space for Loki/Promtail?

I'm currently evaluating Loki and facing issues with running out of disk space due to the amount of chunks.
My instance is running in Docker containers using a docker-compose setup (Loki, Promtail, Grafana) from the official documentation (see docker-compose.yml below).
I'm more or less using the default configuration of Loki and Promtail. Except for some tweaks for the retention period (I need 3 months) plus a higher ingestion rate and ingestion burst size (see configs below).
I bind-mounted a volume containing 1TB of log files (MS Exchange logs) and set up a job in promtail using only one label.
The resulting chunks are constantly eating up disk space and I had to expand the VM disk incrementally up to 1TB.
Currently, I have 0.9 TB of chunks. Shouldn't this be far less? (Like 25% of initial log size?). Over the last weekend, I stopped the Promtail container to prevent running out of disk space. Today I started Promtail again and get the following warning.
level=warn ts=2022-01-24T08:54:57.763739304Z caller=client.go:349 component=client host=loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded (limit: 12582912 bytes/sec) while attempting to ingest '2774' lines totaling '1048373' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"
I had this warning beforehand and increasing ingestion_rate_mb to 12and ingestion_burst_size_mb to 24 fixed this...
Kind of at a dead-end here.
Docker Compose
version: "3"
networks:
loki:
services:
loki:
image: grafana/loki:2.4.1
container_name: loki
restart: always
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ${DATADIR}/loki/etc:/etc/loki:rw
networks:
- loki
promtail:
image: grafana/promtail:2.4.1
container_name: promtail
restart: always
volumes:
- /var/log/exchange:/var/log
- ${DATADIR}/promtail/etc:/etc/promtail
ports:
- "1514:1514" # for syslog-ng
- "9080:9080" # for http web interface
command: -config.file=/etc/promtail/config.yml
networks:
- loki
grafana:
image: grafana/grafana:latest
container_name: grafana
restart: always
volumes:
- grafana_var:/var/lib/grafana
ports:
- "3000:3000"
networks:
- loki
volumes:
grafana_var:
Loki Config:
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
# https://grafana.com/docs/loki/latest/configuration/#limits_config
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
ingestion_rate_mb: 12
ingestion_burst_size_mb: 24
per_stream_rate_limit: 12MB
chunk_store_config:
max_look_back_period: 336h
table_manager:
retention_deletes_enabled: true
retention_period: 2190h
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_encoding: snappy
Promtail Config
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: exchange
static_configs:
- targets:
- localhost
labels:
job: exchangelog
__path__: /var/log/*/*/*log
Issue was solved. Logs were stored on ZFS with compression enabled and were thus listed much smaller on the file system. Chunk size was actually accurate. My bad.

docker-compose.yml logging driver needs to configurable

I have a docker-compose.yml in which I have mention logging driver as fluentd.
But I want logging driver to be enabled or disabled with a flag.(That means whenever I need, I can enable the logging driver or can disable completly)
Also I want to make IP address and port configurable. How can I change them whenever I need?
srvc-name:
image: <DOCKER-REISTRY-PATH:IMAGE-NAME>
environment:
- "JAVA_OPTS=-XX:MaxRAM=4g"
logging:
driver: "fluentd"
options:
fluentd-address: localhost:24224
fluentd-async-connect: "true"
tag: tag.name
volumes:
- app-name-config:/usr/local/tomcat/conf/app1
- app-name-logs:/usr/local/tomcat/logs

rsyslog not connecting to elasticsearch in docker

I am trying to capture syslog messages sent over the network using rsyslog, and then have rsyslog capture, transform and send these messages to elasticsearch.
I found a nice article on the configuration on https://www.reddit.com/r/devops/comments/9g1nts/rsyslog_elasticsearch_logging/
Problem is that rsyslog keeps popping up an error at startup that it cannot connect to Elasticsearch on the same machine on port 9200. Error I get is
Failed to connect to localhost port 9200: Connection refused
2020-03-20T12:57:51.610444+00:00 53fd9e2560d9 rsyslogd: [origin software="rsyslogd" swVersion="8.36.0" x-pid="1" x-info="http://www.rsyslog.com"] start
rsyslogd: omelasticsearch: we are suspending ourselfs due to server failure 7: Failed to connect to localhost port 9200: Connection refused [v8.36.0 try http://www.rsyslog.com/e/2007 ]
Anyone can help on this?
Everything is running in docker on a single machine. I use below docker compose file to start the stack.
version: "3"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
environment:
- discovery.type=single-node
- xpack.security.enabled=false
ports:
- 9200:9200
networks:
- logging-network
kibana:
image: docker.elastic.co/kibana/kibana:7.6.1
depends_on:
- logstash
ports:
- 5601:5601
networks:
- logging-network
rsyslog:
image: rsyslog/syslog_appliance_alpine:8.36.0-3.7
environment:
- TZ=UTC
- xpack.security.enabled=false
ports:
- 514:514/tcp
- 514:514/udp
volumes:
- ./rsyslog.conf:/etc/rsyslog.conf:ro
- rsyslog-work:/work
- rsyslog-logs:/logs
volumes:
rsyslog-work:
rsyslog-logs:
networks:
logging-network:
driver: bridge
rsyslog.conf file below:
global(processInternalMessages="on")
#module(load="imtcp" StreamDriver.AuthMode="anon" StreamDriver.Mode="1")
module(load="impstats") # config.enabled=`echo $ENABLE_STATISTICS`)
module(load="imrelp")
module(load="imptcp")
module(load="imudp" TimeRequery="500")
module(load="omstdout")
module(load="omelasticsearch")
module(load="mmjsonparse")
module(load="mmutf8fix")
input(type="imptcp" port="514")
input(type="imudp" port="514")
input(type="imrelp" port="1601")
# includes done explicitely
include(file="/etc/rsyslog.conf.d/log_to_logsene.conf" config.enabled=`echo $ENABLE_LOGSENE`)
include(file="/etc/rsyslog.conf.d/log_to_files.conf" config.enabled=`echo $ENABLE_LOGFILES`)
#try to parse a structured log
action(type="mmjsonparse")
# this is for index names to be like: rsyslog-YYYY.MM.DD
template(name="rsyslog-index" type="string" string="rsyslog-%$YEAR%.%$MONTH%.%$DAY%")
# this is for formatting our syslog in JSON with #timestamp
template(name="json-syslog" type="list") {
constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",") property(name="$!all-json" position.from="2")
# closing brace is in all-json
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch" template="json-syslog" searchIndex="rsyslog-index" dynSearchIndex="on")
#################### default ruleset begins ####################
# we emit our own messages to docker console:
syslog.* :omstdout:
include(file="/config/droprules.conf" mode="optional") # this permits the user to easily drop unwanted messages
action(name="main_utf8fix" type="mmutf8fix" replacementChar="?")
include(text=`echo $CNF_CALL_LOG_TO_LOGFILES`)
include(text=`echo $CNF_CALL_LOG_TO_LOGSENE`)
First of all you need to run all the containers on the same docker network which in this case are not. Second , after running the containers on the same network , login to rsyslog container and check if 9200 is available.

How can I send and aggregate multiple docker container's logs to journald?

I'm running multiple containers that contain Apache. I'd like all these specific set of containers to log their log output to a single location - either a file or - or possibly journald?
Just some way in which I can aggregate their logs together - to be viewed together
I'm not looking for a heavy solution like fluentd / ELK stack.
How can I achieve the above? Currently all the containers are logging out to /dev/stdout and hence get collected in the 'docker logs'. But these are don't seem possible to aggregate together.
According to Save docker-compose logs to a file it seems I might be able to set a 'log path' - but how? Logging driver? And can this log file be shared between multiple containers?
Is the systemd logging driver a suitable option?
So I've had some luck with the journald logging driver. I've set some labels on a container like so:
version: "3"
services:
nginx-lb:
labels:
- "node_service=nginx"
logging:
driver: "journald"
options:
labels: "node_service=nginx"
restart: always
network_mode: host
build: .
ports:
- "80:80"
- "443:443"
But now, how do I filter by these lables when viewing them with journalctl?
Here is an example journald entry generated:
{ "__CURSOR" : "s=b300aa41db4946f1bcc528e2522627ce;i=1087c;b=e6decf90a91f40c2ad7507e342fda85a;m=8744b1cdfa;t=5934bdb103a24;x=9ba66ecb768eb67", "__REALTIME_TIMESTAMP" : "1569328890657316", "__MONOTONIC_TIMESTAMP" : "580973088250", "_BOOT_ID" : "e6decf90a91f40c2ad7507e342fda85a", "_MACHINE_ID" : "c1339882251041f48f4612e758675ff3", "_HOSTNAME" : "staging", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "3fffffffff", "_SELINUX_CONTEXT" : "unconfined\n", "_SYSTEMD_SLICE" : "system.slice", "_TRANSPORT" : "journal", "_PID" : "3969", "_COMM" : "dockerd", "_EXE" : "/usr/bin/dockerd", "_CMDLINE" : "/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock", "_SYSTEMD_CGROUP" : "/system.slice/docker.service", "_SYSTEMD_UNIT" : "docker.service", "_SYSTEMD_INVOCATION_ID" : "9f1488b462ae478a84bec6e64d72886b", "CONTAINER_NAME" : "3b9b51b4cda1a1e3b21a01f6fe80c7748fb3d231_apache_1", "CONTAINER_TAG" : "497b2f965b76", "SYSLOG_IDENTIFIER" : "497b2f965b76", "CONTAINER_ID" : "497b2f965b76", "CONTAINER_ID_FULL" : "497b2f965b767f897786f3bb8c4789dd91db1a91fe34e5ede368172f44fb3aac", "MESSAGE" : "192.168.240.1 - - [24/Sep/2019:12:41:30 +0000] \"GET / HTTP/1.0\" 200 2697 \"-\" \"curl/7.58.0\"", "_SOURCE_REALTIME_TIMESTAMP" : "1569328890657297" }
I instead used the tag logging option.
version: "3"
services:
nginx-lb:
labels:
- "node_service=nginx"
logging:
driver: "journald"
options:
labels: "node_service=nginx"
tag: "nginx"
restart: always
network_mode: host
build: .
ports:
- "80:80"
- "443:443"
And then to view / filter:
journalctl CONTAINER_TAG=nginx --since "1 hour ago"
I've met similar problem and this could probably work,
run journalctl command with option CONTAINER_NAME repeatedly, for example:
journalctl -f CONTAINER_NAME=containerA CONTAINER_NAME=containerB
will output recent logs from containerA and containerB.

ELK for spring boot application using docker - performance issues

We are using ELK for logging in our spring application using docker setup. I have configured log stash to read the log file from a given path(where the application generates the logs) and pass it to elastic search. The initial setup works fine and all the logs are passed to kibana instantly. However, as the size of the logs increase (or some form of application logging happens), the response time for application increases exponentially and ultimately brings down the application and everything within the docker network.
Logstash conf file:
input {
file {
type => "java"
path => ["/logs/application.log"]
}
filter {
multiline {
pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
negate => "true"
what => "previous"
periodic_flush => false
}
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
grok {
match => [ "message",
"(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(?<logmessage>.*)",
"message",
"(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?<logmessage>.*)"
]
}
#Parsing out timestamps which are in timestamp field thanks to previous grok section
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
}
}
output {
# Sending properly parsed log events to elasticsearch
elasticsearch {
hosts => ["elasticsearch:9200"] // elastic search is the name if the service in docker-compose file for elk
}
}}
Logstash Docker file:
FROM logstash
ADD config/logstash.conf /tmp/config/logstash.conf
Volume $HOME/Documents/logs /logs
RUN touch /tmp/config/logstash.conf
EXPOSE 5000
ENTRYPOINT ["logstash", "agent","-v","-f","/tmp/config/logstash.conf"]
docker compose for ELK:
version: '2'
services:
elasticsearch:
image: elasticsearch:2.3.3
command: elasticsearch -Des.network.host=0.0.0.0
ports:
- "9200:9200"
- "9300:9300"
networks:
- elk
logstash:
build: image/logstash
volumes:
- $HOME/Documents/logs:/logs
ports:
- "5000:5000"
networks:
- elk
kibana:
image: kibana:4.5.1
ports:
- "5601:5601"
networks:
- elk
networks:
elk:
Note: My spring-boot application and elk are on different networks. The performance issue remains same even if they are on same container.
Is this a performance issue because of the continuous writing/polling of a log file which is causing read/write lock issues?

Resources