I am trying to pull and save a few docker images in parallel (they are quite big and parallelism can save a lot of time - it is done within a Python script activating docker pull and then save within each thread). However, it fails all the time with the message like this:
Client Error: Not Found ("open /var/lib/docker/overlay2/41693d132695cd5ada8cf37f210d5b70bc1bac1b2cedfa5a4f352efa5ff00fc6/merged/some_file_name: no such file or directory")
the specific file on which it complains ('no such file or directory') varies.
In /var/log/messages (even after adding the debug flag to docker daemon options) I can't see anything valuable.
e.g.
level=error msg="Handler for GET /v1.35/images/xxx/xxx:xxx/get returned error: open /var/lib/docker/overlay2/41693d132695cd5ada8cf37f210d5b70bc1bac1b2cedfa5a4f352efa5ff00fc6/merged/opt/external/postgresql-42.2.5/postgresql-42.2.5.ja
r: no such file or directory"
Important (probably) note: - the images share many layers in common as they are built based on the same parent images (is this the reason for the collision in overlay FS?).
Running the same sequentually (number of parallel threads set to 1) works perfectly
OS: centos 7.9
Docker:
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Related
I have a Google Cloud VM that installed with my application. The installation step is completed and I:
Turned off the VM instance.
Exported the disk to disk image called MY_CUSTOM_IMAGE_1
My wish now is to use MY_CUSTOM_IMAGE_1 as the starting image of my docker image build. For building the images I'm using Google Cloud Build.
My docker file should look like this:
FROM MY_CUSTOM_IMAGE_1 AS BUILD_ENV
...
When I tried to use this image I got the build error:
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1
ERROR
pull access denied for MY_CUSTOM_IMAGE_1, repository does not exist or may require 'docker login'
Step 1/43 : FROM MY_CUSTOM_IMAGE_1 AS BUILD_ENV
The reason is that VM images are not the same as Docker images.
Is this possible to make this transform (GCP VM Image -> Docker image), without external tools (outside GCP, like "docker private repositories")?
Thanks!
If you know all the installed things on your VM (and all the commands), do the same thing in a Dokerfile. Use as base image, the same OS version as your current VM. Perform some tests and it should be quickly equivalent.
If you have statefull files in your VM application, it's a little bit more complex, you have to mount a disk in your container and to update your application's configuration to write in the correct mounted folder. It's more "complex" but there is tons of example on internet!
No, this is not possible without a tool to extract your application out of the virtual machine image and recreate in a container. To the best of my knowledge, there is no general-purpose tool that exists.
There is a big difference between a container image and a virtual machine image. Container images do not have an operating system, virtual machine images are a complete operating system and device data. The two conceptually are similar, but extremely different in how they are implemented at the software and hardware level.
I have a Google Cloud VM that installed with my application. The installation step is completed and I:
Turned off the VM instance.
Exported the disk to disk image called MY_CUSTOM_IMAGE_1
My wish now is to use MY_CUSTOM_IMAGE_1 as the starting image of my docker image build. For building the images I'm using Google Cloud Build.
My docker file should look like this:
FROM MY_CUSTOM_IMAGE_1 AS BUILD_ENV
...
When I tried to use this image I got the build error:
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1
ERROR
pull access denied for MY_CUSTOM_IMAGE_1, repository does not exist or may require 'docker login'
Step 1/43 : FROM MY_CUSTOM_IMAGE_1 AS BUILD_ENV
The reason is that VM images are not the same as Docker images.
Is this possible to make this transform (GCP VM Image -> Docker image), without external tools (outside GCP, like "docker private repositories")?
Thanks!
If you know all the installed things on your VM (and all the commands), do the same thing in a Dokerfile. Use as base image, the same OS version as your current VM. Perform some tests and it should be quickly equivalent.
If you have statefull files in your VM application, it's a little bit more complex, you have to mount a disk in your container and to update your application's configuration to write in the correct mounted folder. It's more "complex" but there is tons of example on internet!
No, this is not possible without a tool to extract your application out of the virtual machine image and recreate in a container. To the best of my knowledge, there is no general-purpose tool that exists.
There is a big difference between a container image and a virtual machine image. Container images do not have an operating system, virtual machine images are a complete operating system and device data. The two conceptually are similar, but extremely different in how they are implemented at the software and hardware level.
I have two build servers building docker containers, a Windows10-1709 OS and Server2016-LTS, they both build containers based on microsoft/dotnet-framework:latest
These containers are then deployed by our CD system to a test Server2016-LTS host. The host is a VM and is restored to a checkpoint prior to each deployment. The checkpoint has the latest microsoft/dotnet-framework:latest image pulled and stored in it, checkpoint was updated today.
When the container from Server2016-LTS build server deploys, it just pulls our part of the image and is up and running in under 60 seconds.
When the image from the Windows10-1709 build server deploys, it takes ~10 minutes to deploy an image that contains ~10Mb of code. It's pull a different base image (I'm assuming one based on 1709). Once the pull completes, the image fails to run with the following error:
2018-02-15T23:15:57.3170769Z failed to register layer: re-exec error: exit status 1: output: time="2018-02-15T23:15:48Z" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\?\C:\ProgramData\docker\windowsfilter\d7defcca1ec427b77fca7528840e442a596598002140b30afb4b5bb52311c8c6 flavour=1 folder=C:\Windows\TEMP\hcs025707919"
2018-02-15T23:15:57.3171830Z hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\?\C:\ProgramData\docker\windowsfilter\d7defcca1ec427b77fca7528840e442a596598002140b30afb4b5bb52311c8c6 flavour=1 folder=C:\Windows\TEMP\hcs025707919
My assumption was that all the Microsoft/dotnet-framework:latest images are LTS, and you have to specify 1709 to get that base.
So why do my two docker images, which both have the same FROM in their docker file, behave so differently?
Thanks.
My DockerFile contains the following instruction:
rm -f plugins.7z
This command worked as expected in earlier versions of docker but fails with version 1.13. I see the error:
cannot access plugins.7z: No such file or directory
If I bring up a container with the base image and execute the command manually, I see the same error.
Trying to list the folder contents displays:
# ls -lrt
ls: cannot access plugins.7z: No such file or directory
total 12
??????????? ? ? ? ? ? plugins.7z
This is not listed as a known issue in Docker Issues. How do I debug the issue further?
Edit:
For reasons of IP, I cannot post the full Dockerfile here. Also, it may not be necessary. As I mentioned, I am able to simulate the issue even by manually running the container and trying to execute the command
The file exists before I attempt to delete it
I was wrong about there not being a similar bug in the issues list. Here is one
The issue may not be to do with that file. Deleting other files/folders in the folder also makes them appear with ??? permissions
The user performing the operation is root
The reason removing directories fails is that the backing (xfs) filesystem was not formatted with d_type support ("ftype=1"); you can find a discussion on github; https://github.com/docker/docker/issues/27358.
To verify if d_type support is available on your system, check the output of docker info;
Server Version: 1.13.1
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
This requirement is also described in the release notes for RHEL/CentOS
Note that XFS file systems must be created with the -n ftype=1 option enabled for use as an overlay. With the rootfs and any file systems created during system installation, set the --mkfsoptions=-n ftype=1 parameters in the Anaconda kickstart. When creating a new file system after the installation, run the # mkfs -t xfs -n ftype=1 /PATH/TO/DEVICE command. To determine whether an existing file system is eligible for use as an overlay, run the # xfs_info /PATH/TO/DEVICE | grep ftype command to see if the ftype=1 option is enabled.
To resolve the issue, either;
re-format the device with ftype=1
use a different storage driver. Note that the default device mapper configuration (which uses loopback devices) is not recommended for production use, so requires manual configuration.
For backward-compatibility (older versions of docker allowed running overlay on systems without d_type), docker 1.13 will only log a warning in the daemon logs (https://github.com/docker/docker/pull/27433), but will no longer be supported in a future version.
Was able to get past the issue.
The change log for 1.13 says
"IMPORTANT: On Linux distributions where devicemapper was the default
storage driver, the overlay2, or overlay is now used by default
(if the kernel supports it)."
So I tried putting back devicemapper and it is now working as expected.
is there a way to make docker download the layers of an image sequentially instead of in parallel. I require this due to our repository being very strict (or dodgey) on networking issues. I get a lot of the EOF errors like:
time="2016-06-14T13:15:52.936846635Z" level=debug msg="Error contacting registry http://repo.server/v1/: Get http://repo.server/v1/images/b6...be/layer: EOF"
time="2016-06-14T13:15:52.936924310Z" level=error msg="Download failed: Server error: Status 0 while fetching image layer (b6...be)"
This is when running Docker 1.11.2 on windows.
But on a Centos7 VM it all works fine with the default 1.9.1.
I noticed one difference was that 1.9.1 does the downloads sequentially. So I tried to install 1.9.1 on windows, but the quick start terminal automatically downloaded and installed the 1.11.2 version of the boot2docker ISO.
So is there some arg, config, or environment variable I can set to make docker download the layers one at a time?
Or am I jumping to the wrong conclusion assuming the concurrent downloads are causing my network errors?
Thanks
It seems that there was recently added a max-concurrent-downloads option to the configuration of the docker daemon. Here is the link to the docs although I did not have a chance to test it yet myself.