I'm playing around with Docker on OS X (with boot2docker) and can't figure out where these images are being stored. I'm just curious.
This question answers it for Linux which is apparently in /var/lib/docker/graph/<id>/layer
I installed via Homebrew and poked around in /usr/local/Cellar/docker but it doesn't look like it's there. There's a boot2docker-vm.vmdk in ~/.boot2docker. Are the images actually stored in this VM?
CORRECT ANSWER BY CREACK BELOW.
THIS IS JUST SOME ELABORATION:
To get into the boot2docker VM:
boot2docker ssh
user: docker
pass: tcuser
I had to sudo su to get access to the directory
And voilà, there are the images.
Boot2docker is a Virtal Machine (via VirtualBox). So the images are stored within the virtualbox image (the vmdk file). Once mounter by VirtualBox, it is linux and the images are stored at the place you stated.
You can store the images outside of the VirtualBox images, by using VirtualBox shared folder option.
I was able to use a folder in C:\ drive, for all the data that docker needs.
To do so you have to mount /var/lib/docker to a local folder in your host machine.
Set "Auto-Mount", but do not set "Read-Only".
Related
I am running Windows Subsystem Linux (WSL) with Ubuntu as client OS under Windows 10. Now I installed Docker Desktop on the Windows host and enabled the WSL integration in the Docker settings. That works fine so far, I can access the Docker daemon running on the Windows host from my WSL Ubuntu client.
Now I am wondering where all the Docker volumes and other data is stored in this setup. Usually these are under /var/lib/docker, but it seems when using WSL this is not the case. When running df -h I can see the following Docker-related lines:
/dev/sdd 251G 3.1G 236G 2% /mnt/wsl/docker-desktop-data/isocache
/dev/sdc 251G 120M 239G 1% /mnt/wsl/docker-desktop/shared-sockets
/dev/loop0 244M 244M 0 100% /mnt/wsl/docker-desktop/cli-tools
So they are somewhere on the Windows host it seems.
... but where?
When I create a volume named shared_data in docker, I can find it under
\\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\shared_data\\_data
You can find WSL2 volumes under a hidden network share. Open Windows Explorer, and type \\wsl$ into the location bar. Hit enter, and it should display your WSL volumes, including the ones for Docker for Windows.
If you are wondering where on the Windows host the docker volumes are located, for me they seem to be at:
C:\Users\username\AppData\Local\Docker\wsl\data\ext4.vhdx
and
C:\Users\username\AppData\Local\Docker\wsl\distro\ext4.vhdx
presumably, these are docker-desktop-data and docker-desktop respectively.
In theory, these WSL2 instances can be re-located to an alternate drive to free disk space as per this post; that is the standard method for exporting, unregistering, and re-importing an instance from a new location. This process is also described here (with regard to standard WSL instances).
(Caveat - I haven't yet done this with the docker WSL2 instances yet myself, only for Ubuntu using the method in the second link.)
Windows 10 + WSL2
I run docker-desktop on Windows 10 + WSL2. Just make sure you run the docker desktop, so the path would be accessible from a network.
I found my volume data under
\\wsl$\docker-desktop-data\data\docker\volumes
Note that you need to have docker desktop running before you will be able to discover those network direcotories:
Docker Desktop's WSL2 feature creates two new wsl2 containers docker-desktop and docker-desktop-data, which can be seen by the command wsl -l -v
NAME STATE VERSION
* Ubuntu-18.04 Running 2
docker-desktop Running 2
docker-desktop-data Running 2
This is where the docker daemon actually runs and where you can find the data you are looking for.
The volumes in the wsl2 kernel are mapped as follows:
docker run -ti -v host_dir:/app amazing-container will get mapped to /mnt/wsl/docker-desktop-data/data/docker/volumes/host_dir/_data/
The above is the right path, even though docker volume inspect amazing-container will tell you differently (/var/lib/docker/volumes/).
To conclude, the volumes are mapped to: /mnt/wsl/docker-desktop-data/data/docker/volumes/
Most answers on this topic are about the location from the Windows side, I needed to access the container log files (the issue is the same as for volumes) from my WSL distribution, the Windows path \\wsl$ was not an option.
The files could be found on Windows in \\wsl$\docker-desktop-data\version-pack-data\community\docker\containers.
From the WSL distribution, I could go to /mnt/wsl/docker-desktop-data/version-pack-data but it was empty.
Finally found a solution here:
From Windows, create a disk for docker-desktop-data:
net use w: \\wsl$\docker-desktop-data
From your WSL distribution, mount it to docker:
sudo mkdir /mnt/docker
sudo mount -t drvfs w: /mnt/docker
Now you can get everything you want, in my case log files:
ls -l /mnt/docker/version-pack-data/community/docker/containers/
total 0
drwxrwxrwx 4 root root 512 May 19 15:06 3f41ade0891c06725e828853524d73f185b415d035262f9c51d6b6e03654d505
In my case, i install docker-desktop on wsl2, windows 10 home. i find my image files in
\\wsl$\docker-desktop-data\version-pack-data\community\docker\overlay2
All image files are stored there, and have been seperated into several folders with long string names. When i look into every folder, i can find all the real image files in "diff" folders.
Although the terminal show the path "var/lib/docker", but the folder doesn't exsit and the actual files are not stored there. i think there is no error, the "var/lib/docker" is just linked or mapped to the real folder, kind like that.
In windows, we also use mklink to link two folders, it is similar, right?
You can find volumes and others data when using docker with WSL under docker-desktop-data
If you are running Docker on Windows host, using Docker Desktop, you can access the volumes at \\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\ (search this path from windows explorer and make sure docker engine is running).
When running Docker desktop app, the app creates its own Linux VM or using WSL to run the docker container and the path /var/lib/docker/volumes/ is from within that VM I think. The volumes are created as mountable .vhdx file at
C:\Users\username\AppData\Local\Docker\wsl\distro\
but accessing this directly is tricky.
Ref: Google how to access WSl files from Windows
Windows 10 + WSL2, Docker Desktop v4.13.1, free service tier, 2022-11-03:
I found my volumes at \\wsl$\docker-desktop-data\data\docker\volumes
Need to know where docker volumes are located when using the docker machine on macOS.
The installation is using boot2docker, so the VM works behind.
Example:
docker volume create test-data
docker inspect shows a path, but where can I find the specific (physical) location?
It’s inside the virtual machine and isn’t directly accessible from the host.
Debug-level commands like docker volume inspect will give you a path, but they really are only for emergency debugging and not for routine use. If you have a way to get a shell in the VM you can see that path, but you really shouldn’t be directly accessing files there, and you shouldn’t be routinely docker inspecting anything.
macOS use a virtual machine it's different to linux where you can access to volumes from /var/lib/docker/volumes.
For macOS you should connect to a VM to find your volumes.
If you use persistent data volumes in Docker, and you want to access them with command-line.
If your docker host is Linux, that’s not a problem; you can find Docker volumes by /var/lib/docker/volumes path.
However, that’s not the case when you use Docker for Mac.
Try to cd /var/lib/docker/volumes from your MacOS terminal, you ‘ll get nothing.
You see, your Mac machine isn’t a real Docker host. Docker for Mac runs a virtual machine and hides it from you to make things simple.
So, to access persistent volumes created by Docker for Mac, you need to connect on that VM.
In order to accomplish this, we need to use a serial terminal on Mac. There’s a terminal application called “screen” that’s going to help us.
We need to “screen into” the Docker driver by executing a command:
screen
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
You should see a blank screen, just press Enter , and after a while, you should see a command line prompt
Now you’re inside Docker’s VM and you can cd into volumes dir by typing: cd /var/lib/docker/volumes
Profit, you got there!
If you need to transfer files from your MacOS host into Docker host you can refer to File Sharing
Hope this helps you!
If you have installed docker using snap then volumes are located at:
/var/snap/docker/common/var-lib-docker/volumes/
location of volumes when using docker official install
/var/lib/docker/volumes/
Normally, if you want to "know" where a volume lives, you would want to map a volume to the local filesystem. When you create a named volume you are just allocating "shared" storage. However, if your really need to know, run this command:
docker volume inspect test-data
I'm trying create a container to run a program. I'm using a pre configurate image and now I need run the program. However, it's a machine learning program and I need a dataset from my computer to run.
The file is too large to be copied to the container. It would be best if the program running in the container searched the dataset in a local directory of my computer, but I don't know how I can do this.
Is there any way to do this reference with some docker command? Or using Dockerfile?
Yes, you can do this. What you are describing is a bind mount. See https://docs.docker.com/storage/bind-mounts/ for documentation on the subject.
For example, if I want to mount a folder from my home directory into /mnt/mydata in a container, I can do:
docker run -v /Users/andy/mydata:/mnt/mydata myimage
Now, /mnt/mydata inside the container will have access to /Users/andy/mydata on my host.
Keep in mind, if you are using Docker for Mac or Docker for Windows there are specific directories on the host that are allowed by default:
If you are using Docker Machine on Mac or Windows, your Docker Engine daemon has only limited access to your macOS or Windows filesystem. Docker Machine tries to auto-share your /Users (macOS) or C:\Users (Windows) directory. So, you can mount files or directories on macOS using.
Update July 2019:
I've updated the documentation link and naming to be correct. These type of mounts are called "bind mounts". The snippet about Docker for Mac or Windows no longer appears in the documentation but it should still apply. I'm not sure why they removed it (my Docker for Mac still has an explicit list of allowed mounting paths on the host).
When I start docker on windows, it says:
Downloading C:\Users\dhblah.docker\machine\cache\boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v1.11.0/boot2docker.iso...
How do I change location of .docker directory to, say, C:\dev\.docker?
DOCKER_OPTS=-g /your/folder/here
-> restart the docker daemon with these settings.
this changes the directory where images are saved.
Now, you're running boot2docker o Windows I presume; boot2docker is a Linux VM. The docker images will be saved in the VM.
If what you're trying to do is change where the boot2docker image is deployed by your VM engine, refer to the engine you use (is it VirtualBox?)
I have to install Docker on windows 7 in a private netwrok with no internet access.
I can download anything and bring it in by usb from another computer.
How do I intall and use docker?
Meaning: From installation, (what to install and how to setup) to creating the first image.
Most of the instruction I found use proxy and I cant use a proxy.
The installation itself involve copying docker-machine-Windows-x86_64.exe, renaming it to docker-machine.exe, and creating a virtualbox machine with it.
The issue is that it will attempt to download boot2docker.iso (the tinyCore-based linux image which includes docker pre-installed)
That means you need to copy that file on your usb key first, from boot2docker/boot2docker/releases.
From issue 539:
docker-machine create mydocker --virtualbox-boot2docker-url=file:///Users/auser/Downloads/boot2docker.iso --driver=virtualbox
You will need a similar docker setup on a machine with internet access in order to:
docker pull the images you want
docker save them
copy them on the USB key
copy them on your offline server, in C:\Users... (which is the only folders mounted in boot2docker VM)
Then you need to open an ssh session
docker-machine ssh default
And within that session, you can access the folder where the saved images are copied, and docker load them.