Application logging with multiple docker replicas (containers) - docker

Question

We have a .NET core app, which logs its output to files, for eg. portal-20200430-000.log. In DEV environment, all is well :)
App is deployed via docker service, which initializes 3 replicas - 3 docker containers. We want to have all the logs from all the containers (replicas) in one place, so we mapped the file systems between the host machine and the containers via volumes.
docker-compose.yml:
version: "3.7"
services:
web:
image: portal:0.0.1
--- snipped content ---
volumes:
- "/home/portal/Dev/Logs:/app/Logs"
deploy:
replicas: 3
Each container (replica) is outputting its own logs to /app/Logs/portal-20200430-000.log inside container, but this folder is mapped to /home/portal/Dev/Logs on the host. So all 3 containers are writing into the same file on the host, which is not ok - some of the logs get lost, these 3 containers are overwritting each others logs, etc.
I suppose possible solutions are:
change the file name of each container's log (but logging is done via external Karambolo logger, that has filenames hardcoded inside appsettings.json. These settings are common to all container replicas)
instruct each docker replica to map a different volume - is that even possible?
Is there another solution?

Answer 1

Note - This is a partial solution.
When you start docker-compose with replicas, the only difference inside the container is environment variable hostname (unless set by docker-compose.yml to a static value).
Create docker-compose.yml as below:-
version: '3'
services:
test_logging:
image: bash
entrypoint: bash
command: -c "sleep 3600"
deploy:
replicas: 2
Run the containers:-
docker-compose up -d
Now, if you execute bash in interactive mode in the running container, you will see following environment variables.
$ docker exec -it test_test_logging_1 bash
bash-5.1# env
HOSTNAME=f000d941eab2
PWD=/
_BASH_BASELINE_PATCH=16
HOME=/root
_BASH_VERSION=5.1.16
_BASH_BASELINE=5.1.16
_BASH_LATEST_PATCH=16
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
$ docker exec -it test_test_logging_2 bash
bash-5.1# env
HOSTNAME=0b848ef70202
PWD=/
_BASH_BASELINE_PATCH=16
HOME=/root
_BASH_VERSION=5.1.16
_BASH_BASELINE=5.1.16
_BASH_LATEST_PATCH=16
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
As you can see, only hostname differs between different docker containers. You can use hostname to generate different filenames for different log files.
Disk Space Issue
However, this wont be enough. By default FileHandler (logging) will use as much space as needed. Its best to move to RotatingFileHandler or TimedRotatingFileHandler. Note that, its better to use RotatingFileHandler due to sudden surge of too many error logs which can happen due to any number of reasons. By the time, TimedRotatingFileHandler will rotate the files, you might end up using all the disk space.
Note that RotatingFileHandler doesnt provide compression of rotated log files, this needs to be additionally implemented.
P.S. Be aware that with R replicas, and each replica using B backupCount files for RotatingFileHandler and S maxBytes, you will use R x B x S
Bytes disk space with no compression.
Finally, this is a partial solution, since if you do a restart of the service, the hostname will change, and new log files will be created. However, even with this issue, the above partial solution will work if you integrate log services like ELK stack to collect older log files for further analysis of log files such as error monitoring.