I am running WSL and I want to find where the docker images are stored in my filesystem.
I've tried docker info and it shows Docker Root Dir: /var/lib/docker
but when I do ls /var/lib/docker it shows ls: cannot access '/var/lib/docker/': No such file or directory
Tried finding some info and came across this fcc blog regarding Docker on Windows:
but mine contains only txt files and a tmp folder.
I tried inspecting an image I've built and it shows: `
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/tm64pahi18wlse3ibs2uex6dm/diff:/var/lib/docker/overlay2/s4jgik32xo1e9qoqie33bt6d7/diff:/var/lib/docker/overlay2/m9ea6j2mrehihllvnm1s51uyi/diff:/var/lib/docker/overlay2/1o1586gytwxyzhuap98m5a9v4/diff:/var/lib/docker/overlay2/437dfxzj5gmfrbs6f5a9pj0xt/diff:/var/lib/docker/overlay2/4d53adbcad7cc93f261de7f36303a7e1c54ae1cf1accb2768c881be550dd4e95/diff:/var/lib/docker/overlay2/62106246a0f1977a89f193792c0f066a10bc8179e1406d1931cb8c8f15dc47f4/diff:/var/lib/docker/overlay2/28cf7a8fe3d1eab10f09a1056a7a36f88425c331cd0681b6cf1159156238cf4c/diff:/var/lib/docker/overlay2/6fd3c82e02de85abceb47d0a58a06405c1cab0301c73a70939f507a3810fa540/diff:/var/lib/docker/overlay2/4d5eb280e3c8445439d9ee40ba3c066e6c11caf0a76f104e2795195c50ac8389/diff",
"MergedDir": "/var/lib/docker/overlay2/9az4itvoy9m4jv669niircfar/merged",
"UpperDir": "/var/lib/docker/overlay2/9az4itvoy9m4jv669niircfar/diff",
"WorkDir": "/var/lib/docker/overlay2/9az4itvoy9m4jv669niircfar/work"
},
"Name": "overlay2"
}
but doing ls gives the same ls: cannot access '/var/lib/docker/': No such file or directory
The only place I can see anything that I can understand is using docker images and in Docker Desktop where I can see the size an image takes up.
My OS info is below
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
Related
All my team members use the same server as docker remote context. I have set up a project using VSCode-Devcontainer with a devcontainer.json like this:
{
"name": "MyProject - DevContainer",
"dockerFile": "../Dockerfile",
"context": "..",
"workspaceMount": "source=vsc-myprojekt-${localEnv:USERNAME},target=/workspace,type=volume",
"workspaceFolder": "/workspace",
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance"
],
"postCreateCommand": "/opt/entrypoint.sh",
"mounts": [
"source=/media/Pool/,target=/Pool,type=bind",
"source=cache,target=/cache,type=volume"
]
}
This worked fine for me, but now as my colleges start their devcontainers, we have the problem, that a newly started devcontainer kill other already running devcontainers.
We found that the local folder of the projekt seems to by the way to identify already running devcontainers:
[3216 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=d:\develop\myproject
[3839 ms] Start: Run: docker inspect --type container 8ca7d3a44662
[4469 ms] Start: Removing Existing Container
As we all use the same path this identification based on the local folder is problematic. Is there a way to use other labels?
Seems to be a bug, because the issue I opened, was accepted as a bug report.
I'm using the vscode command Remote-contains: Open Folder in container...
I'm trying to mount bind a file into the docker container.
~/.config/dart/pub-tokens.json
The host file is under my HOME directory and I need it mounted in the same location within the container's HOME directory.
Here is my mount command from the vscode devcontainer.json
"mounts": [
"source=${localEnv:HOME}/.config/dart/pub-tokens.json,target=${containerEnv:HOME}/.config/dart/pub-tokens.json,type=bind,consistency=cached",
]
Note the 'containerEnv' in the target clause.
Launching the container via the vscode Remote-contains: Open Folder in container...
produces the following error: (for readability I've added some newlines)
Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR
--mount type=bind,source=/home/bsutton/git/onepub/onepub,target=/workspaces/onepub
--mount source=/home/bsutton/.config/dart/pub-tokens.json,target=${containerEnv:HOME}/.config/dart/pub-tokens.json,type=bind,consistency=cached
--mount source=/home/bsutton/.onepub/onepub.yaml,target=${containerEnv:HOME}/.onepub/onepub.yaml,type=bind,consistency=cached
--mount type=volume,src=vscode,dst=/vscode -l devcontainer.local_folder=/home/bsutton/git/onepub/onepub
--entrypoint /bin/sh vsc-onepub-7ff341664d5755895634c2f74983ff45-uid -c echo Container started
docker: Error response from daemon:
invalid mount config for type "bind": invalid mount path: '${containerEnv:HOME}/.config/dart/pub-tokens.json' mount path must be absolute.
It would appear that vscode isn't expanding the the containerEnv.
If I replace containerEnv it with localEnv it does get expanded (but the wrong path).
i.e. the following works:
"mounts": [
"source=${localEnv:HOME}/.config/dart/pub-tokens.json,target=${localEnv:HOME}/.config/dart/pub-tokens.json,type=bind,consistency=cached",
]
Here is the complete devcontainer.json
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.245.0/containers/ubuntu
{
"name": "Ubuntu",
"build": {
"dockerfile": "Dockerfile",
// Update 'VARIANT' to pick an Ubuntu version: jammy / ubuntu-22.04, focal / ubuntu-20.04, bionic /ubuntu-18.04
// Use ubuntu-22.04 or ubuntu-18.04 on local arm64/Apple Silicon.
"args": { "VARIANT": "ubuntu-22.04" }
},
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "uname -a",
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "vscode",
"features": {
"git": "latest",
"github-cli": "latest"
},
"mounts": [
"source=${localEnv:HOME}/.config/dart/pub-tokens.json,target=${containerEnv:HOME}/.config/dart/pub-tokens.json,type=bind,consistency=cached",
"source=${localEnv:HOME}/.onepub/onepub.yaml,target=${containerEnv:HOME}/.onepub/onepub.yaml,type=bind,consistency=cached"
]
}
I have been frustrated by this issue for a while because this has been asked multiple times here, such as in How to deal with persistent storage (e.g. databases) in Docker and What is the (best) way to manage permissions for Docker shared volumes?, but the answers do not address the issue at all.
The first "answer" says to just use named volumes instead of traditional bind mounts. That solves nothing because when the named volume is mounted on the host, for instance at the default location /var/lib/docker/volumes/<volume name>/_data, then that mount point will have the uid/gid of the mount point inside the container.
The other "answer" given, before docker had named volumes, was to use a data-only container. This exhibits the same exact problem.
The reason this is a huge problem for me is that I have many embedded machines on which I want to run the docker host, and the user may have a different uid/gid on each of these. Therefore I cannot hardcode a uid/gid in a Dockerfile for the mount points for my persistent volumes, to achieve matching ids.
Here's an example of the problem: Say my user is foo on the host, with uid 1001 and gid 1001, and the user writing files to the volume inside the container has uid 1002. When I run the container, docker will chown 1002:1002 the mount point dir on the host, and write files with this uid, which I can't even read/write with my user foo.
Visually (all these operations on the host):
$ docker volume create --driver local --opt type=volume --opt device=/home/<my_host_user>/logs --opt o=bind logs
logs
$ docker volume inspect logs
[
{
"CreatedAt": "2020-08-26T16:26:08+01:00",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/logs/_data",
"Name": "logs",
"Options": {
"device": "/home/<myhostuser>/logs",
"o": "bind",
"type": "volume"
},
"Scope": "local"
}
]
$ pwd
/home/foo
$ mkdir logs && ls -ld logs
drwxr-xr-x 2 foo foo 4096 Aug 26 17:24 logs
Then running the container:
$ docker run --rm --name <cont_name> -it --net="host" --mount src=logs,target=/home/<container_user>/logs <my docker image>
And now the mount point:
$ ls -ld logs
drwxr-xr-x 2 1002 1002 4096 Aug 26 17:30 logs
$ ls -l logs/
total 4
-rw-r----- 1 1002 1002 0 Aug 26 17:30 log
-rw-r----- 1 1002 1002 2967 Aug 26 17:27 log.1
As you can see, the logs written to the volume have a uid/gid which doesn't correspond to something that exists on the host and which I can't access without root/sudo.
Now then, is there ANY way that docker can be told to map uid/gids in the container to uid/gids on the host, or even simpler to just use the specified uid/gid for the host mount point?
my env:
Ubuntu 22.04
Docker version 20.10.17, build 100c701
create mount piont path with suitable permission.
# docker file
RUN mkdir --parents '$volumeDir' ; chown --recursive '$userName':'$userGroup' '$volumeDir'
next, create container and mount volume .
# terminal
docker run --name=containerName --interactive
--user=$userName:$userGroup --mount='source=volumeName,target==$volumeDir,readonly=false'
imageName /bin/bash
you will got suitable permission
Below is the manifest file entry snippet(docker inspect image redis) of redis image
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/bd512eb256c8aa56cbe9243d440a311820712d1a245fe6f523d39d19cd6c862d/diff:/var/lib/docker/overlay2/7fa1e90f35c78fc83c3a
4b86e36e45d742383b394adf9ce4cf9b339d919c9cbe/diff:/var/lib/docker/overlay2/2c1869386b5b8542959da4f0173a5272b9703326d619f27258b4edff7a1dbbf9/diff:/var/lib/docker/overlay2
/23ba3955c5b72ec17b9c409bd5233a3d92cbd75543c7d144b364f8188765788e/diff:/var/lib/docker/overlay2/87d8a92919103e8ff723221200acb36e17c611fa499571ab183d0f51458e6f24/diff",
"MergedDir": "/var/lib/docker/overlay2/e503ed41978e99fe9b71a4225763a40b7988e9a4f31d4c06ef1ec1af46b0b6ab/merged",
"UpperDir": "/var/lib/docker/overlay2/e503ed41978e99fe9b71a4225763a40b7988e9a4f31d4c06ef1ec1af46b0b6ab/diff",
"WorkDir": "/var/lib/docker/overlay2/e503ed41978e99fe9b71a4225763a40b7988e9a4f31d4c06ef1ec1af46b0b6ab/work"
},
"Name": "overlay2"
},
where overlay2 filesystem is used by docker image and container.
Within GraphDriver entry of manifest,
what does LowerDir / MergedDir / UpperDir / WorkDir indicate?
LowerDir: these are the read-only layers of an overlay filesystem. For docker, these are the image layers assembled in order.
UpperDir: this is the read-write layer of an overlay filesystem. For docker, that is equivalent to the container specific layer which contains changes made by that container.
WorkDir: this is a required directory for overlay, it needs an empty directory for internal use.
MergedDir: this is the result of the overlay filesystem. Docker effectively chroot's into this directory when running the container.
For more on overlay filesystems (overlay2 is a newer release, but I don't believe there are any user visible changes), see the kernel docs: https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt
Background:
For Development purposes I do a lot of docker-compose up -d and docker-compose stop.
To view logs of a container I do either
- docker logs --details --since=1m -t -f container_name
or
- docker inspect --format='{{.LogPath}}' container_name
cat path-from-previous
The problem is when I want to view 10 days older logs, there are none, the logs just have todays logs.
when I do a docker inspect container_name I get the following
"Created": "todays-timestamp"
my logging is the default config.
"LogConfig": {
"Type": "json-file",
"Config": {}
},
the reason behind this is because there is no rotation in your docker-logs.
in case you are using a linux system go to:
/etc/logrotate.d/
and create the file docker-container like this => /etc/logrotate.d/docker-container
write this into the file:
/var/lib/docker/containers/*/*.log {
rotate 7
daily
compress
missingok
delaycompress
copytruncate
}
it takes all builded images and their daily log and rotates + compress them.
you can test this with:
logrotate -fv /etc/logrotate.d/docker-container
enter your docker folder /var/lib/docker/containers/[CONTAINER ID]/ and you can see the rotation.
reference: https://sandro-keil.de/blog/logrotate-for-docker-container/