I created a docker compose *.yml file where I have many services with specified image tags. Then I let docker deploy a stack for me (on my local machine) with docker stack deploy -c .\my-compose-file.yml --with-registry-auth dev and it is able to run all services. When I have docker events running simultaneously, I can see image pull messages in the log, so docker pulls missing images. But when I run docker image ls -a, the pulled images are not displayed here.
So I wondering and want to know, what live cycle do the downloaded images have (will then be removed from my drive when I do docker stack rm or not), and when not, how do I clean up such images?
I assume you have multi-node swarm configured. In such cases, the docker image ls is running in your local machine only, while the containers from the stack are distributed across nodes. The images are pulled on the nodes that will run the container.
To get the list of the containers, you will need to go to each member of the swarm and issue the command. Easy way to do it is to use, assuming you have ssh access to the nodes with identity key:
docker node ls | cut -c 31-49 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER /usr/local/bin/docker image ls -a"
I need to run more than 70 docker containers at once. Later, these containers need to be stopped.
At the moment I can docker stop all of them with the shell command docker stop $(docker ps -f since=<last docker before>). It works OK, but if there are any containers started after mine, I have a problem as the above code will stop them too.
Is there any way I can close all of running containers with some kind of specific search?
I know there is an docker ps -f label=<some label>, but I just haven't figured out on how to use it yet.
If you're launching many containers at the same time, launch them all with
docker run --label=anyname other-docker-args-of-yours image:tag
And when you want to delete all your containers just do
docker stop $(docker ps -f label=anyname | awk 'NR>1 {print$1}')
where anyname is the label name you provide during the docker run command and
awk 'NR>1 {print$1}' ignores the column header CONTAINER_ID and just prints the values alone.
Edit-1:
I later realized that you can achieve the list of Container_ID without awk as well. I'd consider using the below line.
docker stop `docker ps -qaf label=anyname`
If you want to remove all stoppped containers also, then include a within the options, like instead of -qf use -qaf.
-q to print container IDs alone.
-a for all containers including stopped.
I have a following setup in docker compose
nginx for proxying to frontend, backend and serving static conent
backend app on port 8080 (spring boot)
frontend app on port 4000 (node for SSR)
mysql used by backend
Frontend can be updated relatively fast using
docker-compose up -d --no-deps frontend
Unfortunately backend takes about 1 minute to start.
Is there an easy way to achieve lower downtime without having to change the current setup too much? I like how simple it is right now.
I would imagine something like:
Start a new instance of backend
Wait till it starts (it could be per timer or a healthtest
Close the perviously running instance
Swarm is the right solution to go, but this is still painfully doable with docker-compose.
First, ensure your proxy can do service discovery. You can't use container_name (as you can't use it in swarm) because your will increase the number of container of the same service. Proxy like traefik or nginx-proxy uses labels to do this.
Then, docker-compose up -d --scale backend=2 --no-recreate this creates a new container with the new image without touching the running one.
After it's up and running, docker kill old_container, then docker-compose up -d --scale backend=1 --no-recreate just to reset the number.
EDIT 1
docker kill old_container should be docker rm -f old_container
EDIT 2
how to handle even and not even runs
We want to always kill the oldest containers
docker rm -f $(docker ps --format "table {{.ID}} {{.Names}} {{.CreatedAt}}" | grep backend | (read -r; printf "%s\n" "$REPLY"; sort -k 3 ) | awk -F " " '{print $1}' | head -1)
Here is the script I've ended up using:
PREVIOUS_CONTAINER=$(docker ps --format "table {{.ID}} {{.Names}} {{.CreatedAt}}" | grep backend | awk -F " " '{print $1}')
docker-compose up -d --no-deps --scale backend=2 --no-recreate backend
sleep 100
docker kill -s SIGTERM $PREVIOUS_CONTAINER
sleep 1
docker rm -f $PREVIOUS_CONTAINER
docker-compose up -d --no-deps --scale backend=1 --no-recreate backend
docker-compose stop http-nginx
docker-compose up -d --no-deps --build http-nginx
I use docker logs [container-name] to see the logs of a specific container.
Is there an elegant way to clear these logs?
First the bad answer. From this question there's a one-liner that you can run:
echo "" > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)
instead of echo, there's the simpler:
: > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)
or there's the truncate command:
truncate -s 0 $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)
I'm not a big fan of either of those since they modify Docker's files directly. The external log deletion could happen while docker is writing json formatted data to the file, resulting in a partial line, and breaking the ability to read any logs from the docker logs cli. For an example of that happening, see this comment on duketwo's answer:
after emptying the logfile, I get this error: error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value
Instead, you can have Docker automatically rotate the logs for you. This is done with additional flags to dockerd if you are using the default JSON logging driver:
dockerd ... --log-opt max-size=10m --log-opt max-file=3
You can also set this as part of your daemon.json file instead of modifying your startup scripts:
{
"log-driver": "json-file",
"log-opts": {"max-size": "10m", "max-file": "3"}
}
These options need to be configured with root access. Make sure to run a systemctl reload docker after changing this file to have the settings applied. This setting will then be the default for any newly created containers. Note, existing containers need to be deleted and recreated to receive the new log limits.
Similar log options can be passed to individual containers to override these defaults, allowing you to save more or fewer logs on individual containers. From docker run this looks like:
docker run --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 ...
or in a compose file:
version: '3.7'
services:
app:
image: ...
logging:
options:
max-size: "10m"
max-file: "3"
For additional space savings, you can switch from the json log driver to the "local" log driver. It takes the same max-size and max-file options, but instead of storing in json it uses a binary syntax that is faster and smaller. This allows you to store more logs in the same sized file. The daemon.json entry for that looks like:
{
"log-driver": "local",
"log-opts": {"max-size": "10m", "max-file": "3"}
}
The downside of the local driver is external log parsers/forwarders that depended on direct access to the json logs will no longer work. So if you use a tool like filebeat to send to Elastic, or Splunk's universal forwarder, I'd avoid the "local" driver.
I've got a bit more on this in my Tips and Tricks presentation.
Use:
truncate -s 0 /var/lib/docker/containers/**/*-json.log
You may need sudo
sudo sh -c "truncate -s 0 /var/lib/docker/containers/**/*-json.log"
ref. Jeff S. Docker: How to clear the logs properly for a Docker container?
Reference: Truncating a file while it's being used (Linux)
On Docker for Windows and Mac, and probably others too, it is possible to use the tail option. For example:
docker logs -f --tail 100
This way, only the last 100 lines are shown, and you don't have first to scroll through 1M lines...
(And thus, deleting the log is probably unnecessary)
sudo sh -c "truncate -s 0 /var/lib/docker/containers/*/*-json.log"
You can set up logrotate to clear the logs periodically.
Example file in /etc/logrotate.d/docker-logs
/var/lib/docker/containers/*/*.log {
rotate 7
daily
compress
size=50M
missingok
delaycompress
copytruncate
}
You can also supply the log-opts parameters on the docker run command line, like this:
docker run --log-opt max-size=10m --log-opt max-file=5 my-app:latest
or in a docker-compose.yml like this
my-app:
image: my-app:latest
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "5"
Credits: https://medium.com/#Quigley_Ja/rotating-docker-logs-keeping-your-overlay-folder-small-40cfa2155412 (James Quigley)
Docker4Mac, a 2018 solution:
LOGPATH=$(docker inspect --format='{{.LogPath}}' <container_name_or_id>)
docker run -it --rm --privileged --pid=host alpine:latest nsenter -t 1 -m -u -n -i -- truncate -s0 $LOGPATH
The first line gets the log file path, similar to the accepted answer.
The second line uses nsenter that allows you to run commands in the xhyve VM that servers as the host for all the docker containers under Docker4Mac. The command we run is the familiar truncate -s0 $LOGPATH from non-Mac answers.
If you're using docker-compose, the first line becomes:
local LOGPATH=$(docker inspect --format='{{.LogPath}}' $(docker-compose ps -q <service>))
and <service> is the service name from your docker-compose.yml file.
Thanks to https://github.com/justincormack/nsenter1 for the nsenter trick.
You can't do this directly through a Docker command.
You can either limit the log's size, or use a script to delete logs related to a container. You can find scripts examples here (read from the bottom): Feature: Ability to clear log history #1083
Check out the logging section of the docker-compose file reference, where you can specify options (such as log rotation and log size limit) for some logging drivers.
Here is a cross platform solution to clearing docker container logs:
docker run --rm -v /var/lib/docker:/var/lib/docker alpine sh -c "echo '' > $(docker inspect --format='{{.LogPath}}' CONTAINER_NAME)"
Paste this into your terminal and change CONTAINER_NAME to desired container name or id.
As a root user, try to run the following:
> /var/lib/docker/containers/*/*-json.log
or
cat /dev/null > /var/lib/docker/containers/*/*-json.log
or
echo "" > /var/lib/docker/containers/*/*-json.log
On my Ubuntu servers even as sudo I would get Cannot open ‘/var/lib/docker/containers/*/*-json.log’ for writing: No such file or directory
But combing the docker inspect and truncate answers worked :
sudo truncate -s 0 `docker inspect --format='{{.LogPath}}' <container>`
I do prefer this one (from solutions above):
truncate -s 0 /var/lib/docker/containers/*/*-json.log
However I'm running several systems (Ubuntu 18.x Bionic for example), where this path does not work as expected. Docker is installed through Snap, so the path to containers is more like:
truncate -s 0 /var/snap/docker/common/var-lib-docker/containers/*/*-json.log
This will delete all logfiles for all containers:
sudo find /var/lib/docker/containers/ -type f -name "*.log" -delete
Thanks to answer by #BMitch, I've just wrote a shell script to clean logs of all the containers:
#!/bin/bash
ids=$(docker ps -a --format='{{.ID}}')
for id in $ids
do
echo $(docker ps -a --format='{{.ID}} ### {{.Names}} ### {{.Image}}' | fgrep $id)
truncate -s 0 $(docker inspect --format='{{.LogPath}}' $id)
ls -llh $(docker inspect --format='{{.LogPath}}' $id)
done
Not sure if this is helpful for you, but removing the container always helps.
So, if you use docker-compose for your setup, you can simply use docker-compose down && docker-compose up -d instead of docker-compose restart. With a proper setup (make sure to use volume mounts for persistent data), you don't lose any data this way.
Sure, this is more than the OP requested. But there are various situations where the other answers cannot help (if using a remote docker server or working on a Windows machine, accessing the underlying filesystem is proprietary and difficult)
Linux/Ubuntu:
If you have several containers and you want to remove just one log but not others.
(If you have issues like "Permission denied" do first sudo su.)
List all containers: docker ps -a
Look for the container you desire and copy the CONTAINER ID. Example: E1X2A3M4P5L6.
Containers folders and real names are longer than E1X2A3M4P5L6 but first 12 characters are those resulted in docker ps -a.
Remove just that log:
> /var/lib/docker/containers/E1X2A3M4P5L6*/E1X2A3M4P5L6*-json.log (Replace E1X2A3M4P5L6 for your result !! )
As you can see, inside /containers are the containers, and logs has the same name but with -json.log at the end. You just need to know that first 12 characters, because * means "anything".
Docker for Mac users, here is the solution:
Find log file path by:
$ docker inspect | grep log
SSH into the docker machine( suppose the name is default, if not, run docker-machine ls to find out):
$ docker-machine ssh default
Change to root user(reference):
$ sudo -i
Delete the log file content:
$ echo "" > log_file_path_from_step1
I needed something I could run as one command, instead of having to write docker ps and copying over each Container ID and running the command multiple times. I've adapted BMitch's answer and thought I'd share in case someone else may find this useful.
Mixing xargs seems to pull off what I need here:
docker ps --format='{{.ID}}' | \
xargs -I {} sh -c 'echo > $(docker inspect --format="{{.LogPath}}" {})'
This grabs each Container ID listed by docker ps (will erase your logs for any container on that list!), pipes it into xargs and then echoes a blank string to replace the log path of the container.
To remove/clear docker container logs we can use below command
$(docker inspect container_id|grep "LogPath"|cut -d """ -f4)
or
$(docker inspect container_name|grep "LogPath"|cut -d """ -f4)
If you need to store a backup of the log files before deleting them, I have created a script that performs the following actions (you have to run it with sudo) for a specified container:
Creates a folder to store compressed log files as backup.
Looks for the running container's id (specified by the container's name).
Copy the container's log file to a new location (folder in step 1) using a random name.
Compress the previous log file (to save space).
Truncates the container's log file by certain size that you can define.
Notes:
It uses the shuf command. Make sure your linux distribution has it or change it to another bash-supported random generator.
Before use, change the variable CONTAINER_NAME to match your running container; it can be a partial name (doesn't have to be the exact matching name).
By default it truncates the log file to 10M (10 megabytes), but you can change this size by modifying the variable SIZE_TO_TRUNCATE.
It creates a folder in the path: /opt/your-container-name/logs, if you want to store the compressed logs somewhere else, just change the variable LOG_FOLDER.
Run some tests before running it in production.
#!/bin/bash
set -ex
############################# Main Variables Definition:
CONTAINER_NAME="your-container-name"
SIZE_TO_TRUNCATE="10M"
############################# Other Variables Definition:
CURRENT_DATE=$(date "+%d-%b-%Y-%H-%M-%S")
RANDOM_VALUE=$(shuf -i 1-1000000 -n 1)
LOG_FOLDER="/opt/${CONTAINER_NAME}/logs"
CN=$(docker ps --no-trunc -f name=${CONTAINER_NAME} | awk '{print $1}' | tail -n +2)
LOG_DOCKER_FILE="$(docker inspect --format='{{.LogPath}}' ${CN})"
LOG_FILE_NAME="${CURRENT_DATE}-${RANDOM_VALUE}"
############################# Procedure:
mkdir -p "${LOG_FOLDER}"
cp ${LOG_DOCKER_FILE} "${LOG_FOLDER}/${LOG_FILE_NAME}.log"
cd ${LOG_FOLDER}
tar -cvzf "${LOG_FILE_NAME}.tar.gz" "${LOG_FILE_NAME}.log"
rm -rf "${LOG_FILE_NAME}.log"
truncate -s ${SIZE_TO_TRUNCATE} ${LOG_DOCKER_FILE}
You can create a cronjob to run the previous script every month. First run:
sudo crontab -e
Type a in your keyboard to enter edit mode. Then add the following line:
0 0 1 * * /your-script-path/script.sh
Hit the escape key to exit Edit mode. Save the file by typing :wq and hitting enter. Make sure the script.sh file has execution permissions.
On computers with docker desktop we use:
truncate -s 0 //wsl.localhost/docker-desktop-data/data/docker/containers/*/*-json.log
For linux distributions you can use this it works for me with this path:
truncate -s 0 /var/lib/docker/containers/*/*-json.log
docker system prune
run this command in command prompt
What CLI commands do I need to use in order to check if the image in my private docker registry is a newer version than the one currently running on my server?
E.g. I have a container that I ran using docker run -d my.domain.com:5000/project1
and I would like to know if it is out-of-date.
Brownie points to #mbarthelemy and #amuino who put me on track. From that I was able to come up with the following bash script that others may find useful. It just checks if the tag on the registry is different from the currently executing container.
#!/bin/bash
# ensure running bash
if ! [ -n "$BASH_VERSION" ];then
echo "this is not bash, calling self with bash....";
SCRIPT=$(readlink -f "$0")
/bin/bash $SCRIPT
exit;
fi
REGISTRY="my.registry.com:5000"
REPOSITORY="awesome-project-of-awesomeness"
LATEST="`wget -qO- http://$REGISTRY/v1/repositories/$REPOSITORY/tags`"
LATEST=`echo $LATEST | sed "s/{//g" | sed "s/}//g" | sed "s/\"//g" | cut -d ' ' -f2`
RUNNING=`docker inspect "$REGISTRY/$REPOSITORY" | grep Id | sed "s/\"//g" | sed "s/,//g" | tr -s ' ' | cut -d ' ' -f3`
if [ "$RUNNING" == "$LATEST" ];then
echo "same, do nothing"
else
echo "update!"
echo "$RUNNING != $LATEST"
fi
Even when there is no command, you can use the API to check for tags on the registry and compare against what you are running.
$ curl --silent my.domain.com:5000/v1/repositories//project1/tags | grep latest
{"latest": "116f283e4f19716a07bbf48a562588d58ec107fe6e9af979a5b1ceac299c4370"}
$ docker images --no-trunc my.domain.com:5000/project1
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
my.domain.com:5000 latest 64d935ffade6ed1cca3de1b484549d4d278a5ca62b31165e36e72c3e6ab8a30f 4 days ago 583.2 MB
By comparing the ids, you can know that you are not running the latest version.
Not sure about the version but if you mean the tag of image, it can be easily checked through the registry v2 api . Note that in context of docker images tag has nothing to do with the version of software.
Use curl command in CLI
curl <docker_host_ip>:<docker_host_port>/v2/<repository_name>/tags/list
To get a list of repositories pushed on the private registry, use
curl <docker_host_ip>:<docker_host_port>/v2/_catalog
AFAIK, this is not possible right now.
The only thing I see would be to pull the registry to check if there is a new version of your image (would then have a different ID than your locally stored image):
docker pull your/image:tag
But yes, that would mean fetching the new images (if any).
If you have a look at the registry API documentation, you'll see that if you don't mind scripting a bit, you could get this information without actually downloading the image, by fetching the image tags and check if the returned ID for the tag matches the ID of the local image you have with the same tag.
That being said, having something to "check for updates" integrated into the docker CLI would be a nice addition.
I don't know if this works as advertised. Just a quick hack I just put together.
But this will at least give you a little push on how this might be done.
#!/bin/bash
container=$1
imageid=$(docker inspect --format '{{.Config.Image}}' ${container})
echo "Running version from: $(docker inspect --format '{{.Created}}' ${container})"
echo "Image version from: $(docker inspect --format '{{.Created}}' ${imageid})"
Example output:
[root#server ~]# sh version_check.sh 9e500019b9d4
Running version from: 2014-05-30T08:24:08.761178656Z
Image version from: 2014-05-01T16:48:24.163628504Z
You can use a bash script running in a cron scheduled task:
#!/bin/bash
docker_instance='YOUR_RUNNING_INSTANCE'
instance_id=$(docker ps -qa --filter name=$docker_instance)
image_name_tag=$(docker inspect $instance_id | jq -r [] |.Config.Image')
if [ "-${image_name_tag}-" != "--" ]; then
status=$(docker pull $image_name_tag | grep "Downloaded newer image")
if [ "-${status}-" != "--" ]; then
echo ">>> There is one update for this image ... "
# stop the docker instance
docker stop $docker_instance
# remove the docker instance
docker rm $docker_instance
# restart the docker using the last command, using the new image from the remote repository
run-my-docker-instance.sh
fi
fi
An older question, but this sounds like a problem that Watchtower can solve for you. It is another dockerized application that runs adjacent to your other containers and periodically check to see if their base images are updated. When they are, it downloads the new image and restarts them.
If given the correct credentials, it can work with a local registry.
FWIW I solved it with the bash script below for a while until I decided that Watchtower was the easier way to go (by the way: note the maintainer switched from v2tec to containrrr a while ago, the v2tec one isn't getting updates anymore). Watchtower gave me an easy way to schedule things without having to rely on cron (which gets blown away in a reinstall - granted, you could have something like Ansible recreate that for you, but this was easier for me). It also adds easy notifications (I like using Telegram) for updates, which I appreciate knowing about so that if something goes sideways at least I know there was an update that could be to blame.
I'm not saying this is will never cause issues, but I've been running Watchtower on various Docker hosts (3 of them, 2 in my homelab, one on Linode) for about a year now and I have yet to have an issue with it. I prefer this to having to manually update my containers on a regular basis. For me the risk of something getting screwed up is lower than the risks of running outdated containers, so this is what I chose for myself. YMMV.
I honestly don't get the apparent hate for automated update solutions like Watchtower - I see so many comments saying that you shouldn't use automated updates because they'll cause problems... I don't know what folks have been burned by - would love to hear more about where this caused problems for you! I mean that, I genuinely don't understand and would love to learn more. I keep having some vague unease about doing automated updates, but given my experience so far I can honestly only recommend it. I used to use Diun for getting notified about updates and then would go and manually update my containers. That got real old after it became a daily chore! (With ~45 different containers running, you can pretty much guarantee that at least one of them will have an update every single day.)
If I really need to keep a container from updating, I can always stick a com.centurylinklabs.watchtower.enable=false label on the container. Or you can whitelist only the containers you want automatically updated, or... There are loads of possibilities with Watchtower.
However, for reference if you still want to use it, see my script below. I used docker-compose pull to get the latest version - it does a check first to see if there is a new image, so doesn't waste a whole lot of bandwidth if there is nothing to update. It's effectively like doing the curl you guys used. Also I prefer the docker inspect -f commands to check the versions to the solutions that pipe through grep, sed, and co. since that is less likely to get broken by changes to docker inspect output format.
#!/usr/bin/env bash
cd /directory/with/docker-compose.yml/
image_name=your-awesome-image
docker-compose pull
container_version=$(docker inspect -f '{{ index .Config.Labels "org.opencontainers.image.version" }}' "$image_name")
latest_image_version=$(docker inspect -f '{{ index .Config.Labels "org.opencontainers.image.version" }}' "$image_name")
if [[ "$container_version" != "$latest_image_version" ]]; then
echo "Upgrading ${image_name} from ${container_version} to ${latest_image_version}"
docker-compose down
docker-compose up -d
fi