I'm currently experimenting with Docker containers on Windows Server. I've created a number of containers, and I want to see where they are actually saved on the host's file system (like a .vhd file for Hyper-V). Is there a default location I can look, or a way to find that out using Docker CLI?
Other answers suggest the data might be stored in:
C:\Users\Public\Documents\Hyper-V\Virtual hard disks\MobyLinuxVM.vhdx
or since the Windows 10 Anniversary Update:
C:\ProgramData\docker\containers
You can find out by entering:
docker info
Credit to / More info:
https://stackoverflow.com/a/38419398/331637
https://stackoverflow.com/a/39971954/331637
Related
I am a beginner with Docker and I have been searching for 2 days now and I do not understand which would be a better solution.
I have a docker container on a Ubuntu server. I need to copy many large video files to the Ubuntu host via FTP. Docker via cron will process the videos using ffmpeg and save the result to the Ubuntu host somehow so the files are accessible via FTP.
What is the best solution:
create a bind drive - I understand the host may change files in the bind drive
create a volume but I do not understand how may I add files to the volume
create a folder on the Ubuntu and have a cron that will copy using "docker cp" command and after a video has been processed to copy it to the host?
Thank you in advance.
Bind-mounting a host directory for this is probably the best approach, for exactly the reasons you lay out: both the host and container can directly read and write to it, but the host can't easily write to a named volume. docker cp is tricky, you note the problem of knowing when the process is completed, and anyone who can run any docker command at all can pretty trivially root the host; you don't want to give this permission to something network-facing.
If you're designing a larger-scale system, you also might consider an approach where no files are actually shared at all. The upload server sends the files (maybe via HTTP POST) to an internal storage service, then posts a message to a message queue (maybe RabbitMQ). That then retrieves the files from the storage service, does its work, uploads the result, and posts a response message. The big advantages of this approach are being able to run it on multiple systems, easily being able to scale the individual components of it, and not needing to worry about filesystem permissions. But, it's a much more involved design.
When my Docker containers start, I receive the following notification that reads:
Docker Desktop has detected that you shared a Windows file into a WSL 2 container, which may perform poorly. Click here for more details.
My questions are:
What does this mean?
What is the better practice / how should this be avoided?
If the message has been closed, or I've clicked "Don't show again", how can I get to the details of this warning?
I am happy to share the Dockerfile or Docker-Compose setup if needed, but I simply cannot find anything either here on SO or by a Google search that points me in any direction, so I'm not sure where to start. I'm assuming the issue lies in the Dockerfile since that is where we running COPY to move some files around.
Docker Version: Docker Desktop 2.4.0.0 (48506) Community
Operating System: Windows 10 Pro (version 10.0.19041)
This error means that accessing files on the Windows host file system from a Linux container will perform a little slower than accessing files that are already in a Linux filesystem. Accessing Windows files from the Linux container will perform like accessing files on a remote file share.
Docker and Microsoft recommend avoiding this by storing your source files in a WSL2 distro's file system (which you can bind mount to the container) or building your container image to include all the files needed rather than storing your files in the Windows file system.
If you've clicked "Don't show again", you can get to the details of this message by going to Develop with Docker and WSL 2.
For more information, Docker for Windows Best Practices says:
Linux containers only receive file change events (“inotify events”) if the original files are stored in the Linux filesystem. For example, some web development workflows rely on inotify events for automatic reloading when files have changed.
Performance is much higher when files are bind-mounted from the Linux filesystem, rather than remoted from the Windows host. Therefore avoid docker run -v /mnt/c/users:/users (where /mnt/c is mounted from Windows).
Instead, from a Linux shell use a command like docker run -v ~/my-project:/sources <my-image> where ~ is expanded by the Linux shell to $HOME.
Microsoft's Comparing WSL 1 and WSL 2 article has a whole section on Performance across OS file systems, and its opening paragraph says:
We recommend against working across operating systems with your files, unless you have a specific reason for doing so. For the fastest performance speed, store your files in the WSL file system if you are working in a Linux command line (Ubuntu, OpenSUSE, etc). If you're working in a Windows command line (PowerShell, Command Prompt), store your files in the Windows file system.
Also, the Docker blog article Docker Desktop: WSL 2 Best practices has an "Awesome mounts performance" section that says:
Both your own WSL 2 distro and docker-desktop run on the same utility VM. They share the same Kernel, VFS cache etc. They just run in separate namespaces so that they have the illusion of running totally independently. Docker Desktop leverages that to handle bind mounts from a WSL 2 distro without involving any remote file sharing system. This means that when you mount your project files in a container (with docker run -v ~/my-project:/sources <...>), docker will propagate inotify events and share the same cache as your own distro to avoid reading file content from disk repeatedly.
A little warning though: if you mount files that live in the Windows file system (such as with docker run -v /mnt/c/Users/Simon/windows-project:/sources <...>), you won’t get those performance benefits, as /mnt/c is actually a mountpoint exposing Windows files through a Plan9 file share.
All of that advice is great if you want your primary development workflow to be in Linux. Docker wants you to go "all in" on Linux containers. But if you work primarily in Windows and just want to use a Linux container for a specialized task, then it's fine to click "Don't show again". As Microsoft said, "If you're working in a Windows command line, store your files in the Windows file system."
I run with my main development folder in Windows, and I bind mount it to a Linux container that's just used to execute unit tests. So my full build runs in Windows, then I run all my unit tests in Windows, and I finish by running all my unit tests in a Linux container too. Having Linux bind mount to my Windows folder works fast and great for this scenario where the "dotnet test" call in Linux is just loading and executing the required DLLs from my Windows volume.
This setup may sound like heresy to those that believe containers must be used everywhere, but I love containers for application deployment. I'm not convinced that you need to go all in and do all your development inside a container too. I'm happy with Windows (and VS 2019) as my development environment, and then I use Linux containers for application testing and deployment. So the Windows/WSL2 file system performance hit is a minimal impact to me.
After having read about the performance improvements when running Docker on wsl2, I have been waiting for the official release of Windows 10 that supports wsl2.
I updated Windows and Docker and switched on the Docker flag to use wsl2 and was hoping for some performance boost for my Oracle Database running in a Docker container but unfortunately the change slowed down the container and my laptop dramatically.
The performance of the container is about 10x slower and my laptop is pretty much stuck when starting the container.
It seems as if the memory consumption would completely use up my 8GB and heavy memory swapping starts to take place.
Is there anything I can do to improve the performance of Docker on wsl2 or at least to better understand what's wrong in my setup?
My environment:
Processor Intel(R) Core(TM) i7-2620M CPU # 2.70GHz, 2 Core(s)
Installed Physical Memory (RAM) 8.00 GB
Microsoft Windows 10 Pro Version 10.0.19041 Build 19041
Docker version 19.03.8, build afacb8b
This comes from the "vmmem" which consumes as much resource as it can.
To solve the problem just go to your user file
for me in
C:\Users\userName
In this directory create a file named ".wslconfig" in which you will configure how many resources can consume WSL2:
[wsl2]
memory=900MB #Limits VM memory in WSL 2 to 900MB
processors=1 #Makes the WSL 2 VM use one virtual processors
Now close your docker and wait for "vmmem" to close in the task manager.
then You can restart docker and normally "vmmem" will not exceed the limit you have set (here 900MB)
If don't work restart your computer.
I hope it helped you.
You probably have your code stored on the Windows machine in a folder similar to this...
C:\\Users\YourName\projects\blahfu
But you are using Docker on WSL 2 which is a different (Linux) filesystem. So, when you do a Docker build all of the code/context gets copied from the Windows filesystem to Linux filesystem and then from there to the Docker container. This is what takes the most time and is incredibly slow.
Try to put your project into a folder like this...
/home/YouName/projects/blahfu
You should get quite a performance boost.
wsl container have they proper filesystem isolated from the windows filesystem.
The base idea is to copy your source code from windows file systeme to wsl file systeme.
from window you can acces the wsl container and copy your project to a wslcontainer :
navigate with explorer to \\wsl$
rebuild the container from this location this will do the trick !
If the data for the actual docker container is stored on a windows file system (i.e. NTFS) instead of stored on a native linux filesystem (regardless of what the docker container contents are, which are likely already linux based), then I think you are going to see slow performance because you're running WSL and using the docker container from a mounted WINDOWS file system (i.e. /c/mnt/...).
If you copy your docker container to something like /usr/local, or /home//docker on WSL then you may see a 10x performance INCREASE. Try that and see if it works?
you need edit "vmmem" resource
just add file .wslconfig in path
C:\Users<yourUserName>.wslconfig
Configure global options with .wslconfig
Available in Windows Build 19041 and later
You can configure global WSL options by placing a .wslconfig file into the root directory of your users folder: C:\Users<yourUserName>.wslconfig. Many of these files are related to WSL 2, please keep in mind you may need to run
wsl --shutdown
to shut down the WSL 2 VM and then restart your WSL instance for these changes to take affect.
Here is a sample .wslconfig file:
Console
Copy
[wsl2]
kernel=C:\\temp\\myCustomKernel
memory=4GB # Limits VM memory in WSL 2 to 4 GB
processors=2 # Makes the WSL 2 VM use two virtual processors
see this https://learn.microsoft.com/en-us/windows/wsl/wsl-config
Open your wsl2 distribution (Ubuntu for example) and set the ~/.docker/config.json file.
Only you need to change:
{
"credsStore": "docker.exe"
}
"credsStore": "desktop.exe" : ultra-slow (over 2 minutes)
"credsStore": "wincred.exe" : fast
"credsStore": "" : fast
It works very well.
If you are using VS Code, there is a command named "Remote-Containers: Clone Repository in Container Volume..." which assures you have full speed file access.
Form the documentation:
Repository Containers use isolated, local Docker volumes instead binding to the local filesystem. In addition to not polluting your file tree, local volumes have the added benefit of improved performance on Windows and macOS.
As mentioned by Claudio above, setting below lines in ~/.docker/config.json of wsl ubuntu server solved the problem for me.
{
"credsStore": "wincred.exe"
}
Earlier it was taking 5-10 min to build any simple image, now it is done in 1-2 seconds.
Downside: You have to make this change every time you open the server. I have tried every solution mentioned in https://github.com/docker/for-win/issues/9843 to solve this but nothing works for me.
I am learning CDH and Docker and didn't have prior experiene in setting up both tools. After reading documentation i managed to run CDH docker in mac environment and also completed example given in quick start guid. But when next day when i started mac book again to learn something new but i didn't find my previous work which i found very strange and even couldn't see container running which seems fine to me.
What i really want to do is i don't want to loose my work even after stoping docker container. could you please guid me how do i configure docker so that i will not loose my work even after restarting docker again?
Every instance of a docker run will allocate a new filesystem, essentially starting from scratch.
If you actually want to "save" your work, then you need to volume mount (using -v docker flag) your local filesystem into the container for at least the following directories.
HDFS Data Directory
NameNode Data Directory
/home/cloudera
I think the hadoop data folders are somewhere under /var/lib/hadoop-*, by default
The better alternative for saving your workloads would be the CDH VM, where it actually has a persistent HDD associated with it.
this might seems a stupid question, but here I am :
I'm running Ubuntu 16.04 and managed to install windows 10 in dual boot.
Running docker exclusively in linux so far, I decided to give it a try on Windows 10.
As I already downloaded several docker images on my Linux system, I'm willing to have a "shared" like development environment. I must admit this would be a waste of time and disk space to download Docker images I already downloaded before (on linux) on my fresh windows install.
So my question is simple : Can I use my linux images / containers on windows. I'm thinking of something like a global path variable pointing to my linux images to configure on docker windows.
Any idea if this is possible, and if yes, the pros and cons and the caveats ?
Thanks for helping me on this one.
Well i would suggest to create your local registry and then push these images there and pull it in your windows docker.
Sonatype nexus(artifact storage repository) can be used to store your docker images. Check if this helps.
I guess it's not possible to share the same folder (to reduce disk usage) since the stored files are totally different:
Under Windows the file is:
C:\Users\Public\Documents\Hyper-V\Virtual hard disks \MobyLinuxVM.vhdx
the vhdx extension is specific to MS systems.
and under linux it consist of 2 files:
/var/lib/docker/devicemapper/devicemapper/data
/var/lib/docker/devicemapper/devicemapper/metadata
see here for details
Where are Docker images stored on the host machine?
The technology under this is to have a specific fileSystem optimal for docker. Even if they used the same fileSystem storage, it wouldn't be a good idea imho.
If the purpose is only to gain time for resintalling, just dump all the images from on system, and re-pull them on the other one.
docker images --format "{{.Repository}}" > image-list.txt
then loop on the other OS
while read p; do
docker pull $p
done < image-listtxt