Remove 'old' docker config files with wildcards and a single command - docker

I'm currently working with docker config files and I'm looking for a way to automate the removal of 'old' config file. I use a script that will update all my config files version with +1;
app1-config-file-1.2.1
app1-config-file-1.2.2
app1-config-file-1.2.3
etc etc.
Now I would like to remove app1-config-file-1.2.2 when app1-config-file-1.2.3 is created.
This command gets me kinda close;
docker config rm $(docker config ls | grep -e 'app1'.*'-1.2.2')
Where 'app1' is the prefix and the '-1.2.2', well, is the version I want to remove.
The thing is that the command above does remove the correct config, but it will also try to remove the following 'configs'
Error response from daemon: access denied
Error: No such config: hours
Error: No such config: ago
Error response from daemon: access denied
Error: No such config: hours
Error: No such config: ago
Which is what comes back from my query:
$(docker config ls | grep -e 'app1'.*'-1.2.2')
I tried adding the --quiet or -q to only get back the config IDs. But for some reason I can't use it with config files the way it works with container.
docker config ls -q | grep -e 'app1'.*'-1.2.2'
results in nothing.
docker config ls | grep -e 'app1'.*'-1.2.2'
results in a list with ID, configfile names, age etc. Basically all the config files I would like to remove, but I just need the ID's
So my question is,
What is the correct way to get the IDs of specific docker config file while using wildcards?
Thank you

When you write
docker config rm $(docker config ls | grep -e 'app1'.*'-1.2.2')
This does three things:
Run docker config ls, producing its formatted output.
Filter (grep) that output to only lines that contain app1...-1.2.2.
Pass the resulting output, split into words, as arguments to docker config rm.
The ... | grep invocation is "dumb" in the sense that it doesn't know anything in particular about the input it has, it just knows that it's lines of text and it should try to filter it.
The better option here is to more directly tell docker config ls what you're looking for. That command takes a --filter option; that's somewhat underdocumented, but if you know the exact name of the config object, you can combine it with --quiet to tell Docker (not the shell) to first filter to a specific config object then only print out its ID.
docker config rm $(docker config ls -f name=app1-config-file-1.2.2 -q)
# But really this is the same as
docker config rm app1-config-file-1.2.2
The "shellier" way is to use another tool to clean up the formatted output. The examples in the Docker documentation show the config ID as the first whitespace-separated column, so you could use awk(1) like
docker config ls | \
grep 'app1.*-1.2.2' | \
awk '{ print $1 }' \
xargs docker config rm
(Other tools work fine here too; personally I'd use sed(1) but it's harder to explain in an SO answer.)

Related

How to follow a log until a string is matched in PowerShell

I would like to be able to do something similar to this is PowerShell
docker logs -f my-container | grep -q "string-to-match"
i.e. to follow a log file and when a string is matched you stop follow the file. My idea was to try something like this
docker logs -f my-container | Select-String "string-to-match".
I know it's not complete but I can't figure out how to make it work. I also did try to use WSL2 like this
docker logs -f my-container | wsl grep -q "string-to-match".
but it keeps following the log even after a match has been found!
A WSL-solution would solve my problem but a native PowerShell-solution would be preferable!
I tried with ugrep.exe in PowerShell:
docker logs -f my-container | ugrep -q "string-to-match"
and that appears to work fine as it stops at the first match. This can also be done with ugrep -m1 "string-to-match" to report the line that matched, since -m1 stops searching further after the first hit.

Apply completion options dynamically

I have a fish shell completion like this for docker:
# 'docker-subcommands.txt' is an exhaustive list derived from the "Usage" message.
set -l subcoms (cat /etc/fish/completions/docker-subcommands.txt)
# Found '__fish_seen_subcommand_from' in the completion for `systemctl`
# (in /usr/share/fish/completions/...).
complete -f -c docker -n "not __fish_seen_subcommand_from $subcoms" -a "$subcoms"
function _docker_container_action
# The placeholder is needed here because fish will choke on an empty completion list.
echo (docker ps -aq) '{z-placeholder}'
end
function _docker_image_action
echo (docker images -q) '{z-placeholder}'
end
for sc in cp exec inspect restart rm start stop
complete -c docker -n "contains $sc (commandline -poc)" -x -a (_docker_container_action)
end
for sc in rmi run
complete -c docker -n "contains $sc (commandline -poc)" -x -a (_docker_image_action)
end
The problem is that the completion for image and container IDs needs to be dynamic (like file completion), whereas the above seems to run the -a command when the completion file is (re-)sourced.
How can I make this refresh the options when the completion is actually being applied?
I should mention that I've tried quoting the -a command, as seems to be common practice. But then instead of an option for each ID, I get only one completion option, a long string with space escapes in it (\) containing the IDs and \{z-placeholder} at the end. :(
[...] Quoting the -a command does work as long as its output is not produced by echo (which stringifies it). Turns out -a's "choking" problem is not having an argument, which is what happens if you put docker ps -aq there directly and there are no container IDs to list. So this pattern works:
function _docker_container_action
docker ps -aq
end
complete -c docker -n "contains $sc (commandline -poc)" -x -a "(_docker_container_action)"
And no need for my placeholder either.
whereas the above seems to run the -a command when the completion file is (re-sourced).
That's because you've included it here as a command substitution, so when the file is sourced, it expands, and the completion system has no idea what it was before.
But the -a option knows to expand its argument at completion-time, so all you need to do is quote it:
complete -c docker -n "contains $sc (commandline -poc)" -x -a "(_docker_container_action)"

how to get docker container to read from stdin?

I have a script that I want to use to use Sigil (based on Go's template engine) to populate template files
I'm using a dockerized sigil for this via:
docker run -v ${TEMPLATE_DIR}:/tmp/sigil mikegrass/gliderlabs_sigil-docker/sigil -f prometheus-configmap.yaml -p API_SERVER=$api_server_url > $TEMP_FILE
This seems a bit clunky with having to map a volume, so I'd rather use STDIN to pass in the file....
So what I'd like is
cat ./prometheus-configmap.yaml | docker run mikegrass/gliderlabs_sigil-docker -p API_SERVER=$api_server_url > $TEMP_FILE
Unfortunately this doesn't work, I get no output.
Googling around I see possible solutions but haven't gotten any to work...
You need to run the container in interactive mode with --interactive or -i:
cat ./prometheus-configmap.yaml | docker run -i mikegrass/gliderlabs_sigil-docker -p API_SERVER=$api_server_url > $TEMP_FILE
Without cat:
docker run -i mikegrass/gliderlabs_sigil-docker -p API_SERVER=$api_server_url < ./prometheus-configmap.yaml

How to clean Docker container logs?

I read my Docker container log output using
docker logs -f <container_name>
I log lots of data to the log in my node.js app via calls to console.log(). I need to clean the log, because it's gotten too long and the docker logs command first runs through the existing lines of the log before getting to the end. How do I clean it to make it short again? I'd like to see a command like:
docker logs clean <container_name>
But it doesn't seem to exist.
First, if you just need to see less output, you can have docker only show you the more recent lines:
docker logs --since 30s -f <container_name_or_id>
Or you can put a number of lines to limit:
docker logs --tail 20 -f <container_name_or_id>
To delete the logs on a Docker for Linux install, you can run the following for a single container:
echo "" > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)
Note that this requires root, and I do not recommend this. You could potentially corrupt the logfile if you null the file in the middle of docker writing a log to the same file. Instead you should configure docker to rotate the logs.
Lastly, you can configure docker to automatically rotate logs with the following in an /etc/docker/daemon.json file:
{
"log-driver": "json-file",
"log-opts": {"max-size": "10m", "max-file": "3"}
}
That allows docker to keep up to 3 log files per container, with each file limited to 10 megs (so a limit between 20 and 30 megs of logs per container). You will need to run a systemctl reload docker to apply those changes. And these changes are the defaults for any newly created container, they do not apply to already created containers. You will need to remove and recreate any existing containers to have these settings apply.
The best script I found is
sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log'
It cleans all logs and you don't need to stop the containers.
Credit goes to https://bytefreaks.net/applications/docker/horrible-solution-how-to-delete-all-docker-logs
If you want to remove all log files, not only for a specific container's log, you can use:
docker system prune
But, note that this does not clear logs for running containers.
This is not the ideal solution, but until Docker builds in a command to do it, this is a good workaround.
Create a script file docker-clean-logs.sh with this content:
#!/bin/bash
rm $(docker inspect $1 | grep -G '"LogPath": "*"' | sed -e 's/.*"LogPath": "//g' | sed -e 's/",//g');
Grant the execute permission to it:
chmod +x ./docker-clean-logs.sh
Stop the Docker container that you want to clean:
docker stop <container_name>
Then run the above script:
./docker-clean-logs.sh <container_name>
And finally run your container again:
docker start ...
Credit goes to the user sgarbesi on this page: https://github.com/docker/compose/issues/1083
You can use logrotate as explained in this article
https://sandro-keil.de/blog/2015/03/11/logrotate-for-docker-container/
This needs to be done before launching the container.
Run:
docker inspect {containerId}
Copy LogPath value
truncate -s 0 {LogaPath}
Solution for a docker swarm service:
logging:
options:
max-size: "10m"
max-file: "10"
In order to do this on OSX, you need to get to the virtual machine the Docker containers are running in.
You can use the walkerlee/nsenter image to run commands inside the VM like so:
docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n sh
Combining that with a simplified version of the accepted answer you get:
#!/bin/sh
docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n \
cp /dev/null $(docker inspect -f '{{.LogPath}}' $1)
Save it, chmod +x it, run it.
As far as I can tell this doesn't require the container to be stopped. Also, it clears out the log file (instead of deleting it) avoiding errors when doing docker logs right after cleanup.
On Windows 10 none of the solutions worked for me, I kept getting 'No such file or directory'
This worked
Get container ID (inspect the container)
In file explorer open docker-desktop-data (in WSL)
Navigate to version-pack-data\community\docker\containers\CONTAINER_ID
Stop the container
Open the file CONTAINER_ID-json.log file and trim it or just create a blank file with same name
source

Piping a file into docker run

I need to pipe (inject) a file or some data into docker as part of the run command and have it written to a file within the container as part of the startup. Is there best practise way to do this ?
I've tried this.
cat data.txt | docker run -a stdin -a stdout -i -t ubuntu /bin/bash -c 'cat >/data.txt'
But can't seem to get it to work.
cat setup.json | docker run -i ubuntu /bin/bash -c 'cat'
This worked for me. Remove the -t. Don't need the -a's either.
The better solution is to make (mount) you host folder be accessible to docker container. E.g. like this
docker run -v /Users/<path>:/<container path> ...
Here is /Users/<path> is a folder on your host computer and <container path> mounted path inside container.
Also see Manage data in containers manual page.
UPDATE another Accessing External Files from Docker Containers example.

Resources