Background:
For Development purposes I do a lot of docker-compose up -d and docker-compose stop.
To view logs of a container I do either
- docker logs --details --since=1m -t -f container_name
or
- docker inspect --format='{{.LogPath}}' container_name
cat path-from-previous
The problem is when I want to view 10 days older logs, there are none, the logs just have todays logs.
when I do a docker inspect container_name I get the following
"Created": "todays-timestamp"
my logging is the default config.
"LogConfig": {
"Type": "json-file",
"Config": {}
},
the reason behind this is because there is no rotation in your docker-logs.
in case you are using a linux system go to:
/etc/logrotate.d/
and create the file docker-container like this => /etc/logrotate.d/docker-container
write this into the file:
/var/lib/docker/containers/*/*.log {
rotate 7
daily
compress
missingok
delaycompress
copytruncate
}
it takes all builded images and their daily log and rotates + compress them.
you can test this with:
logrotate -fv /etc/logrotate.d/docker-container
enter your docker folder /var/lib/docker/containers/[CONTAINER ID]/ and you can see the rotation.
reference: https://sandro-keil.de/blog/logrotate-for-docker-container/
Related
I started a docker images anapsix/webdis:
sudo docker run -d -p 7379:7379 -e LOCAL_REDIS=true anapsix/webdis
and changed the etc/webdis.json to allow websockets and committed it with
sudo docker commit <container-id>
however, when I used the new image to start a container, it does not keep the changes. Is there something I'm doing wrong?
Thanks!
In this case your problem is that the anapsix/webdis image has an entrypoint script (/entrypoint.sh) that generates /etc/webdis.json when the container starts.
Looking at the script, you can set the value of websockets by setting the WEBSOCKETS variable when you start the container:
docker run -d -p 7379:7379 \
-e LOCAL_REDIS=true \
-e WEBSOCKETS=true \
anapsix/webdis
When we run it like this, the generated /etc/webdis.json looks like:
{
"redis_host": "127.0.0.1",
"redis_port": 6379,
"redis_auth": null,
"http_host": "0.0.0.0",
"http_port": 7379,
"threads": 5,
"pool_size": 10,
"daemonize": false,
"websockets": true,
"database": 0,
"acl": [
{
"disabled": ["DEBUG", "FLUSHDB", "FLUSHALL"]
},
{
"http_basic_auth": "user:password",
"enabled": ["DEBUG"]
}
],
"verbosity": 8,
"logfile": "/dev/stdout"
}
More broadly, using docker commit is almost always the wrong thing to do; you should generate custom images using a Dockerfile (this gives you a much more manageable, reproducible process for creating container images).
When accessing a remote machine I'd like to know if a container was started over docker run or docker-compose or some other means.
Is that even possible?
EDIT: the main reason for this was to find out, where these containers are getting orchestrated, i.e. if the container goes down, will be started again? Where would that configuration be?
For investigation purposes I created the most simplest docker-compose.yml:
version: "2.4"
services:
hello:
image: "hello-world"
Then run it with docker-compose up
And lastly the normal way: docker run -it --name cli hello-world
So I had two stopped containers:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6a8d53ff45a4 hello-world "/hello" 9 minutes ago Exited (0) 9 minutes ago cli
d54f7a2ae8b2 hello-world "/hello" 9 minutes ago Exited (0) 9 minutes ago compose_hello_1
Then I compared inspect output of both:
diff <(docker inspect cli) <(docker inspect compose_hello_1)
I found out that there are labels which compose creates:
"Labels": {}
---
"Labels": {
"com.docker.compose.config-hash": "251ebf43e00417fde81d3c53b9f3d8cd877e1beec00ebbffbc4a06c4db9c7b00",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "compose",
"com.docker.compose.service": "hello",
"com.docker.compose.version": "1.24.1"
}
Also compose uses another network:
"NetworkMode": "default",
---
"NetworkMode": "compose_default",
You should do it on your environment and try to find out differences where you can surely differentiate between both launch ways.
Unfortunately that's impossible. You can only guess it according to some secondary properties doing docker inspect container-name.
With the default json-file logging driver, is there a way to log rotate docker container logs with container names, instead of the container IDs?
The container IDs in the log file name look not so readable, which is when i thought of saving the logs with container names instead?
It's possible to configure the engine with log options to include labels in the logs:
# cat /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3",
"labels": "com.docker.stack.namespace,com.docker.swarm.service.name,environment"
}
}
# docker run --label environment=dev busybox echo hello logs
hello logs
root#vm-11:/etc/docker# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9615c898c2d2 busybox "echo hello logs" 8 seconds ago Exited (0) 7 seconds ago eloquent_germain
# docker logs --details 961
environment=dev hello logs
# more /var/lib/docker/containers/9615c898c2d2aa7439581e08c2e685f154e4bf2bb9fd5ded0c384da3242c6c9e/9615c898c2d2aa7439581e08c2e685f154e4bf2bb9fd5ded0c384da3242c6c9e-json.log
{"log":"hello logs\n","stream":"stdout","attrs":{"environment":"dev"},"time":"2020-09-22T11:12:41.279155826Z"}
You need to reload the docker engine after making changes to the daemon.json, and changes only apply to newly created containers. For systemd, reloading is done with systemctl reload docker.
To specifically pass the container name, which isn't a label, you can pass a "tag" setting:
# docker run --name test-log-opts --log-opt tag="{{.Name}}/{{.ID}}" busybox echo hello log opts
hello log opts
# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c201d0a2504a busybox "echo hello log opts" 6 seconds ago Exited (0) 5 seconds ago test-log-opts
# docker logs --details c20
tag=test-log-opts%2Fc201d0a2504a hello log opts
# more /var/lib/docker/containers/c201d0a2504addedb2b6785850a83e8931052d0d9778438e9dcc27391f45fec2/c201d0a2504addedb2b6785850a83e8931052d0d9778438e9dcc27391f45fec2-json.log
{"log":"hello log opts\n","stream":"stdout","attrs":{"tag":"test-log-opts/c201d0a2504a"},"time":"2020-09-22T11:15:26.998956544Z"}
For more details:
JSON log driver options: https://docs.docker.com/config/containers/logging/json-file/#options
Container logging tags: https://docs.docker.com/config/containers/logging/log_tags/
I would like to run Filebeat as Docker container in Azure IoT Edge. I would like Filebeat to get logs from others running containers.
I'm already able to run filebeat as Docker container, from the documentation (https://www.elastic.co/guide/en/beats/filebeat/6.8/running-on-docker.html#_volume_mounted_configuration)
docker run -d \
--name=filebeat \
--user=root \
--volume="$(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \
--volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
--volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
docker.elastic.co/beats/filebeat:6.8.3 filebeat -e -strict.perms=false
With this command and with the correct filebeat.yml file I'm able to collect logs for every running containers on my device.
Now I would like to deploy this configuration as Azure IoT Edge Modules.
I created a docker image having the filebeat.yml file included with the following Dockerfile:
FROM docker.elastic.co/beats/filebeat:6.8.3
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chmod go-w /usr/share/filebeat/filebeat.yml
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
USER filebeat
From documentation: https://www.elastic.co/guide/en/beats/filebeat/6.8/running-on-docker.html#_custom_image_configuration
I tested this Dockerfile by running locally
docker build -t filebeat .
and
docker run -d \
--name=filebeat \
--user=root \
--volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
--volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
filebeat:latest filebeat -e -strict.perms=false
This works fine, logs from other containers are collected as they should.
Now my question is :
In Azure IoT Edge, how can I mount volumes to access others Docker containers running on the devices, like it's done with
--volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
--volume="/var/run/docker.sock:/var/run/docker.sock:ro"
in order to collect logs?
From this other SO post (Mount path to Azure IoT Edge module) in the Azure IoT Edge portal I tried the following:
"HostConfig": {
"Mounts": [
{
"Target": "/var/lib/docker/containers",
"Source": "/var/lib/docker/containers",
"Type": "volume",
"ReadOnly: true
},
{
"Target": "/var/run/docker.sock",
"Source": "/var/run/docker.sock",
"Type": "volume",
"ReadOnly: true
}
]
}
}
But when I deploy this module I have the following error:
2019-11-25T10:09:41Z [WARN] - Could not create module FilebeatAgent
2019-11-25T10:09:41Z [WARN] - caused by: create /var/lib/docker/containers: "/var/lib/docker/containers" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path
I don't understand this error. How can I specify a path using only [a-zA-Z0-9][a-zA-Z0-9_.-] ?
Thanks for your help.
EDIT
In the Azure IoT Edge portal, createOptions json:
{
"HostConfig": {
"Binds": [
"/var/lib/docker/containers:/var/lib/docker/containers",
"/var/run/docker.sock:/var/run/docker.sock"
]
}
}
There is an article that describes how to mount storage from the host here: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-access-host-storage-from-module
I have the following problem : Trying to pull built docker image from private hub and run it as a service, but the following error appears
Failed to launch container: Failed to run 'docker -H unix:///var/run/docker.sock pull r.cfcr.io/path/to/repo/': exited with status 1; stderr='Error: Cannot perform an interactive login from a non TTY device '
here is the fetch[] config.json info that I am using to authenticate :
{
"auths": {
"r.cfcr.io": {
"auth": "=auth_token="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.06.1-ce (linux)"
}
}
Do you have any idea how to resolve the problem?
Probably not linked to the problem here, but some people might encounter the exact same message when trying a docker login from a Linux like terminal on Windows such as Git bash or Docker quickstart terminal or even Cygwin.
The trick here is to use winpty docker login
or try to use this command
docker login "${URL}" -u "${Username}" -p "${PASSWORD}"
You must keep the config.json file at .docker directory at $MESOS_SANDBOX.
So create the archieve of .docker directory with below listing of files :
$ tar tvf docker-login.tar
drwx------ parvez/parvez 0 2019-06-12 21:45 .docker/
-rw------- parvez/parvez 177 2019-06-12 21:45 .docker/config.json
Fetch and extract this archive from mesos configuration.
"fetch": [{
"uri": "https://foo.com/docker-login.tar",
"executable": false,
"extract": true,
"cache": true
}],
It will download and extract archieve at $MESOS_SANDBOX path and docker pull should be successful.