I need to mount host directory /data to 6 containers h1,h2,h3,h4,h5,h6
/data is an external hard disk mounted on the host. The 6 containers can be opened and closed easily.
The 6 containers will go into their own sub-directories of /data to analyze data independently and produce new data locally. All sub-directories have nothing to do with each other.
A relevant question is here, but no preferred answer is given.
How to do that? Below are the containers and images I have now.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9bd9334a1e7 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h6
23679fe7252b ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h5
e2864e38e746 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h4
c8996a304638 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h3
9acd2a223d86 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h2
5690b8c7b6da ubuntu "/usr/bin/bash" 2 days ago Up 12 hours h1
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/hello-world latest f2a91732366c 2 months ago 1.85 kB
docker.io/ubuntu 27 422dc563ca32 2 months ago 252 MB
docker.io/ubuntu latest 422dc563ca32 2 months ago 252 MB
IF those containers are already running, you cannot easily add /data to them.
Except maybe with docker cp.
But the best practice remains either:
make images with /data already in it (Dockerfile ADD)
or use the existing image and launch your container, but with the -v (volume) option. See Use volumes.
Related
I want to run container and receive error:
docker run -ti --rm grafana/promtail:2.5.0 -config.file=/etc/promtail/config.yml
docker: Error response from daemon: mkdir /var/lib/docker/overlay2/0cad6a6645e2445a9985d5c9e9c6909fa74ee1a30425b407ddfac13684bd9d31-init: no space left on device.
At first, I thought I have a lot of volumes and images cached. So I clean docker with:
docker prune
docker builder prune
But in a while, the same error occur. When I check my Docker Desktop configuration, I can see I am using all available disk size for images:
Disk image size:
59.6 GB (59.5 GB used)
I have 13 images on my system and together its less than 5GB:
REPOSITORY TAG IMAGE ID CREATED SIZE
logstashloki latest 157966144f3b 3 days ago 761MB
minio/minio <none> 717586e37f7f 4 days ago 232MB
grafana/grafana <none> 31a8875955e5 9 days ago 277MB
docker.elastic.co/beats/filebeat 8.3.2 e7b210caf528 3 weeks ago 295MB
k8s.gcr.io/kube-apiserver v1.24.0 b62a103951f4 2 months ago 126MB
k8s.gcr.io/kube-scheduler v1.24.0 b81513b3bfb4 2 months ago 50MB
k8s.gcr.io/kube-controller-manager v1.24.0 59fad34d4fe0 2 months ago 116MB
k8s.gcr.io/kube-proxy v1.24.0 66e1443684b0 2 months ago 106MB
k8s.gcr.io/etcd 3.5.3-0 a9a710bb96df 3 months ago 178MB
grafana/promtail 2.5.0 aa21fd577ae2 3 months ago 177MB
grafana/loki 2.5.0 369cbd28ef9b 3 months ago 60MB
k8s.gcr.io/pause 3.7 e5a475a03805 4 months ago 514kB
k8s.gcr.io/coredns/coredns v1.8.6 edaa71f2aee8 9 months ago 46.8MB
From output of docker system df there is no suspicious size of container, images or volumes:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 13 13 2.35GB 69.57MB (2%)
Containers 21 21 35.15kB 0B (0%)
Local Volumes 2 0 2.186MB 2.186MB (100%)
Build Cache 20 0 0B 0B
I am new to MacOS and cannot determine what take all my space and how to clean all that space and where are all that data stored on system?
I have docker in version 18.03.1-ce that support command docker system df. His output:
Images space usage:
REPOSITORY TAG IMAGE ID CREATED ago SIZE SHARED SIZE UNIQUE SiZE CONTAINERS
registry.gitlab.com/precisesale/app latest b7833546c2cf About an hour ago ago 252.1MB 123.8MB 128.4MB 1
healthdiary/app latest 565c6d3906e6 2 days ago ago 312.2MB 123.8MB 188.4MB 1
mongo latest f93ff881751f 5 days ago ago 367.6MB 0B 367.6MB 2
nginx latest b175e7467d66 6 weeks ago ago 108.9MB 0B 108.9MB 1
jwilder/docker-gen latest 8959ee34c769 2 months ago ago 19.91MB 4.148MB 15.77MB 1
jrcs/letsencrypt-nginx-proxy-companion latest 17939ceb7a52 2 months ago ago 86.86MB 4.148MB 82.71MB 1
Containers space usage:
CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED ago STATUS NAMES
c20dc3438552 healthdiary/app "./entrypoint.sh nod…" 0 0B 8 minutes ago ago Up 8 minutes healthdiary_app_1
bf8c4307dcbb mongo:latest "docker-entrypoint.s…" 1 0B 8 minutes ago ago Up 8 minutes healthdiary_mongo_1
47fced8d18fe registry.gitlab.com/precisesale/app "./entrypoint.sh nod…" 0 0B 9 minutes ago ago Up 9 minutes precisesale_app_1
597d97d5c1fa mongo:latest "docker-entrypoint.s…" 1 0B 9 minutes ago ago Up 9 minutes precisesale_db_1
b5bb14faa910 jwilder/docker-gen "/usr/local/bin/dock…" 0 0B 7 hours ago ago Up 19 minutes nginx-gen
8eee2bee084a nginx "nginx -g 'daemon of…" 0 2B 7 hours ago ago Up 19 minutes nginx-web
6b8b0cd5d938 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/entr…" 0 1.66kB 7 hours ago ago Up 19 minutes nginx-letsencrypt
Local Volumes space usage:
VOLUME NAME LINKS SIZE
0a40fac6ca98e776dad972c8193362a51a485b3305979e58996545d97310a3c7 1 0B
929b0b88849ad4d390efd4666e6a0e5f82e0e6dd34f7a09f609de90b190e6148 1 0B
Build cache usage: 0B
Even if I do not take into account savings from shared space from two first containers summary size is 1147.5 MB
But if I measure size of docker overlay2 on disc by du I get
du -hs /var/lib/docker/overlay2/
2.7G /var/lib/docker/overlay2/
Where is reason of difference in size of containers measured by docker system df and du?
I was wondering the same thing some time ago.
It’s not a bug, it’s a feature :-)
du -sh /var/lib/docker/overlay2
is not showing objective value because merge folders have been mounted using overlay driver and du output is not actual disk allocation size.
You can see the actual disk allocation size by examining only diff folders like:
du -shc /var/lib/docker/overlay2/*/diff
You can test this in your environment like this:
run
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
Now start 20 centos containers and observe what has change:
for i in {1..20}; do docker run -itd centos bash; done
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
You can see that the actual disk allocation (df command) is just cca 200MB more than before, but “du” on whole folder outputs 4.2G allocation.
“du” on “diff” folders shows 212M what is correct.
This is how Docker works and what makes it great!
I've been toying with a docker image for Tensorflow.
To summarize, I first installed the standard image, then realized I needed nodejs, so added it and did a docker commit. Then realized I needed expressJS, added it an did a commit
I am running docker v1.12.5 (so the new gc/prune commands are not there)
At this stage, docker images -a shows:
REPOSITORY TAG IMAGE ID CREATED SIZE
tensor-node-express latest f2f59eb61aae 15 hours ago 2.104 GB
gcr.io/tensorflow/tensorflow latest-devel 308238445d5c 2 days ago 1.995 GB
gcr.io/tensorflow/tensorflow <none> 74435614a991 9 days ago 1.52 GB
I only want to keep tensor-node-express and delete the older images.
$ docker rmi 308238445d5c
Error response from daemon: conflict: unable to delete 308238445d5c (cannot be forced) - image has dependent child images
$docker rmi gcr.io/tensorflow/tensorflow:latest-devel
Error response from daemon: conflict: unable to remove repository reference "gcr.io/tensorflow/tensorflow:latest-devel" (must force) - container 03de9d864e31 is using its referenced image 308238445d5c
I assumed that this means docker commits store differential images, but when I go to ~/.docker/machine/machines/default, I see:
40894464 Mar 13 13:57 boot2docker.iso
5043847168 Mar 16 08:34 disk.vmdk
I suppose the 5G file is a composite of my images, which seems to show each docker commit is the full image!
Any thoughts on how I can only use the latest docker image (tensor-node-express) and free my HD of the invasion of docker?
Supplementary info - here is the output of docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6dcd2915991 tensor-node-express "/bin/bash" 15 hours ago Exited (130) 15 hours ago flamboyant_bose
fb44b19a21c2 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 18 hours ago Exited (130) 15 hours ago compassionate_bose
075001a687e3 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 18 hours ago Exited (0) 18 hours ago nervous_sinoussi
a80ce2d2e688 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 19 hours ago Exited (130) 18 hours ago happy_euclid
f493bd3c8712 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 19 hours ago Exited (1) 19 hours ago friendly_cori
03de9d864e31 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 2 days ago Exited (255) 23 minutes ago 6006/tcp, 8888/tcp tender_hopper
2dd1e83d62d3 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 2 days ago Exited (0) 15 hours ago modest_einstein
3067ed171b1c gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 2 days ago Exited (0) 2 days ago dazzling_bhabha
62c699afd3fd 74435614a991 "/bin/bash" 2 days ago Exited (127) 2 days ago inspiring_austin
9523ffe2945c 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago kickass_leakey
e06958ea517c 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago objective_euler
ccf922954667 74435614a991 "/bin/bash" 2 days ago Exited (255) 2 days ago dreamy_bartik
fad0d92a07a3 74435614a991 "/bin/bash" 2 days ago Exited (130) 2 days ago elastic_dubinsky
f2a98d4e11ea 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago heuristic_kilby
f07e46367b17 74435614a991 "/bin/bash" 2 days ago Exited (130) 2 days ago trusting_darwin
5bbf9cf992b8 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago flamboyant_knuth
I tried
docker ps --filter "status=exited" | grep "days ago" | awk '{print $1}' | xargs docker rm (credit)
I ran the above manually as well for some of the containers it missed
That pruned the ps list to:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6dcd2915991 tensor-node-express "/bin/bash" 15 hours ago Exited (130) 15 hours ago flamboyant_bose
But even then I can't delete old images - same error.
Further update, I tried to list dependencies in images (credit)
using this script:
for i in $(docker images -q)
do
docker history $i | grep -q 74435614a991 && echo $i
done | sort -u
And it told me:
308238445d5c
74435614a991
f2f59eb61aae
This means my new images are child images of the old image. But the size is not a differential looking at the disk size.
Thoughts?
docker-machine uses a Linux VM
When you looked at the docker-machine .vdmk and .iso files, what you are looking at is files for a Linux VM running on your Mac. This is needed because Docker requires Linux kernel features to run, it cannot run directly on the Mac's microkernel.
So your Mac is running a Linux virtual machine, and inside that virutal machine is running the Docker daemon and all of your containers.
Therefore the file size of the .vmdk and .iso tell you nothing about any one image.
docker images have parent/child relationships
As you may already know, docker images have parents and/or children. For instance when you build an image with a Dockerfile like this:
FROM ubuntu:latest
RUN apt-get update && apt-get install nginx
You will end up with a new image that you have perhaps tagged my-nginx. But it requires the ubuntu:latest image as its parent; you cannot delete ubuntu:latest with this image still around, as it requires its parent.
docker commit creates those relationships
When you use docker commit, you are basically doing a dynamic snapshot build. It is similar to the above, except there's no Dockerfile involved.
The above example has a FROM line which indicates the image to use as a base. When using commit, there is a base implied - whatever image was used to launch the running container that you are committing.
The above example has a RUN command which will create new contents in the built image, above and beyond the base image. In a real Dockerfile there are usually multiple commands that do various things which build on the base image. When you use commit, you don't have that. Instead, anything that has been written to the container on top of the base image is your new content. It exists in a read-write filesystem layer in the container. That is the thing you are committing; it is written as a new read-only layer and you get that back as a new (immutable, read-only) docker image. With a parent.
Based on your comments, and the question itself, you appear to have believed that using docker commit would create a new full image that had no dependencies on other images. That is not true. You can craft images like that if you build them yourself from scratch, just not this way.
You can untag the image
If what you want is for the image to not show up in your list, that's easy. Just untag it.
docker rmi gcr.io/tensorflow/tensorflow:latest-devel
However, this is more or less cosmetic. The image will still be there, as another image requires it. All this does is remove the tag, so it doesn't appear in the docker images list anymore without the -a flag.
The reason trying this did not work for you is you tried to rmi the image using its ID, not using its tag.
Using the command docker images, you can list all images on your host:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
scdockerdemo_php latest 155d80ea7642 4 minutes ago 265.3 MB
scdockerdemo_node latest 6189bc65c3fe 8 minutes ago 861.4 MB
php 5.6-apache fc50bce69ea0 3 days ago 481.3 MB
node 4.1 fc81e574af43 3 days ago 641.1 MB
With docker images -f "tag=latest", you can filter for images with a certain tag.
How can I filter for a repository name? E.g. docker images -f "repository=scdockerdemo_*"
This command always return Invalid filter 'repository'
https://docs.docker.com/reference/commandline/images/
If you want to filter by repository name(eg: testImage), just use the following:
docker images testImage
When you have multiple images with same repository name but diffetent tags, you can specify tags on the repository name by using a ':' ( For example, testImage:<whatever tag>)
Source: Docker images
According to the documentation at https://docs.docker.com/engine/reference/commandline/images/ you can match by wildcard by enclosing in double quotes:
docker images "scdockerdemo_*"
** special chars **
when you have special characters like '/' in the repository name, you should escape it to filter by repository name.
ex)
> docker images "zirho6\/*"
REPOSITORY TAG IMAGE ID CREATED SIZE
zirho6/aaaa latest 946bf5cc28fc 2 days ago 997MB
zirho6/bbbb latest 741a6e368d07 10 days ago 866MB
zirho6/ccc latest 173b36570522 12 days ago 853MB
zirho6/dddd latest e08e5c202e9b 13 days ago 853MB
You can filter images by reference (combination of name and tag):
docker image ls --filter 'reference=scdockerdemo_*'
According to this answer to a similar question, the filter option currently only supports "dangling=true".
If you're using Bash, the easiest thing to do is probably:
$ docker images | grep scdockerdemo
Or, you can try using awk to match on a string in the first column:
$ docker images | awk '$1 ~ /scdockerdemo/ { print }'
There is my solution for filter Docker images by name
List all my Docker images
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED
pay-with-crypto-apis_store-management latest fd78ce3a6869 27 minutes ago
pay-with-crypto-apis_webhooks latest a305d3a8bd5c 28 minutes ago
pay-with-crypto-apis_invoices latest 1ded36a0dfb1 29 minutes ago
pay-with-crypto-apis_authentication latest 73cf83569a45 30 minutes ago
pay-with-crypto-apis_gateway latest f6a8ccadd645 34 minutes ago
node 14-alpine 47afee183159 2 months ago
node 18 e3cb0fb99b7c 2 months ago
btcpayserver/btcpayserver 1.5.4 f4a433460739 3 months ago
nicolasdorier/nbxplorer 2.3.26 121100cdb31b 3 months ago
saubyk/c-lightning-rest 0.7.2 7924309b4d7d 3 months ago
shahanafarooqui/rtl 0.12.3 da155f98b998 4 months ago
mongo latest 27dd1be4bed1 4 months ago
I wanna to filter by pay-with-crypto-apis
$ sudo docker images "pay-with-crypto-apis_*"
REPOSITORY TAG IMAGE ID CREATED
pay-with-crypto-apis_store-management latest fd78ce3a6869 26 minutes ago
pay-with-crypto-apis_webhooks latest a305d3a8bd5c 27 minutes ago
pay-with-crypto-apis_invoices latest 1ded36a0dfb1 28 minutes ago
pay-with-crypto-apis_authentication latest 73cf83569a45 29 minutes ago
pay-with-crypto-apis_gateway latest f6a8ccadd645 33 minutes ago
Provide another option for reference
docker images | ruby -ne ' puts $_ if $_ =~ /harbor/ '
Let the commands speak for themselves:
on a host called: coreworker
core#coreworker-1 ~ $ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059 1 bbd5d4d98156 31 minutes ago 139.3 MB
ec2-54-169-239-164.ap-southeast-1.compute.amazonaws.com:5000/hub-action-repository latest 66ecb895d185 14 hours ago 856.4 MB
ec2-54-169-239-164.ap-southeast-1.compute.amazonaws.com:5000/hub-ext-node-base 0.12.7 f2f1afc202e4 8 days ago 136.6 MB
...
core#coreworker-1 ~ $
on a host called devhost
core#devhost ~ $ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
<none> <none> 67e45ce93dee 48 minutes ago 725 MB
node 0.12.7 a4b45afffe4a 5 days ago 642.2 MB
jwnintex/mesos-worker latest 42f1b41b0089 5 weeks ago 504 MB
jwnintex/nginx-port-router latest 11edcdf1a5fc 9 weeks ago 126.4 MB
jwnintex/consul latest e66fb6787628 10 weeks ago 69.4 MB
jwnintex/mesos-master latest 187d84106a3e 3 months ago 561.8 MB
jwnintex/marathon latest b1d8dd91146a 3 months ago 699.3 MB
jwnintex/zookeeper latest 9b72d56707c9 4 months ago 304.3 MB
jwnintex/registrator latest b1c29d1a74a9 6 months ago 11.79 MB
but when I do:
core#devhost ~ $ docker images -a hub-docker-repo:5000
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
now run the image on devhost residing in hub-docker-repo
veryify it's up:
core#devhost ~ $ ping hub-docker-repo
PING hub-docker-repo.service.consul (172.17.8.150) 56(84) bytes of data.
64 bytes from coreworker-1.node.dc1.consul (172.17.8.150): icmp_seq=1 ttl=63 time=8.27 ms
64 bytes from coreworker-1.node.dc1.consul (172.17.8.150): icmp_seq=2 ttl=63 time=5.19 ms
now try it
core#devhost ~ $ docker run -d hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059:1
Unable to find image 'hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059:1' locally
Pulling repository hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059
FATA[0000] Error: image 485d5874-c786-4b90-93ac-8db5342a6059:1 not found
Now try it on coreworker which is actually hosting the registry (as a docker image)
core#coreworker-1 ~ $ docker run -d hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059:1
98d642c1bafd30569d853e92167c7b4fe720bd67f65ec0d0719ec5a36bb6616f
Why am I not able to run that remote image? Or better, even discover it?
It turns out the image had never been pushed and so couldn't be seen by the other host.