I am getting error as error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged
Please find debug info below.
OS
root#cks-master:/vagrant/files# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
Dockerfile
# cat ch5Dockerfile
FROM bash
CMD ["ping", "killer.sh"]
Build log
root#cks-master:/vagrant/files# podman build -t simple -f ch5Dockerfile .
STEP 1/2: FROM bash
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged: invalid argument
Error: error mounting new container: error mounting build container "6c0f88a6da54d713e18283e16521385fff736bc1a1072938fddfc6be4b3d43cc": error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/BNREFG6CRAAHJ7VSYG3EUXV5UO:/var/lib/containers/storage/overlay/l/MDSWZVRVZNCOW75JF32K6D4QQC:/var/lib/containers/storage/overlay/l/4G3NS52LYHWPTKA4FURHLYMAPF,upperdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/diff,workdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/work": invalid argument
root#cks-master:/vagrant/files#
podman version
root#cks-master:/vagrant/files# podman version
Version: 3.4.2
API Version: 3.4.2
Go Version: go1.15.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
root#cks-master:/vagrant/files#
Issue resolved.
I also posted the issue in podman room at https://app.element.io/#/room/#podman:fedoraproject.org
I am suggested with podman reset(command trace is below), then its complained about storage.conf, I removed that file and did reset again. Then its worked.
I still wonder whats inside of storage.conf causing this issue but I deleted before looking into it. Finally its worked and I am able to continue. Hope it helps.
Note: Post the deleting storage.conf file and podman reset, I tried with docker build as well just to check if docker has any dependency over storage.conf file, but none, docker build also executed successfully( command trace below)
root#cks-master:~# podman system reset -f
A storage.conf file exists at /etc/containers/storage.conf
You should remove this file if you did not modified the configuration.
root#cks-master:~# rm /etc/containers/storage.conf
root#cks-master:~# podman system reset -f
root#cks-master:~# podman build -t simple -f /vagrant/files/ch5Dockerfile .
STEP 1/2: FROM bash
Resolving "bash" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/bash:latest...
Getting image source signatures
Copying blob 9621f1afde84 done
Copying blob 1dd831616e40 done
Copying blob fd6cd28e0879 done
Copying config 8b332999f6 done
Writing manifest to image destination
Storing signatures
STEP 2/2: CMD ["ping", "killer.sh"]
COMMIT simple
--> cd1407a69ea
Successfully tagged localhost/simple:latest
cd1407a69ea490496d6635700958f2b5fcf2b1d01f8dd218dea0f83187e55872
root#cks-master:~# podman run --name simple simple
PING killer.sh (35.227.196.29): 56 data bytes
64 bytes from 35.227.196.29: seq=0 ttl=42 time=15.689 ms
64 bytes from 35.227.196.29: seq=1 ttl=42 time=14.662 ms
64 bytes from 35.227.196.29: seq=2 ttl=42 time=15.161 ms
^C
--- killer.sh ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 14.662/15.170/15.689 ms
root#cks-master:~# docker build -t simple -f /vagrant/files/ch5Dockerfile .
Sending build context to Docker daemon 3.141MB
Step 1/2 : FROM bash
latest: Pulling from library/bash
9621f1afde84: Pull complete
1dd831616e40: Pull complete
fd6cd28e0879: Pull complete
Digest: sha256:e4624241e953934fc4c396217253d8322ebda53be3b1863cd7795541d168034f
Status: Downloaded newer image for bash:latest
---> 8b332999f684
Step 2/2 : CMD ["ping", "killer.sh"]
---> Running in 306963a83d1c
Removing intermediate container 306963a83d1c
---> 51dee555fd57
Successfully built 51dee555fd57
Successfully tagged simple:latest
root#cks-master:~# ^C
root#cks-master:~#
Related
UPDATE 3 (Solution):
I installed the latest Windows updates on my host and specified the exact servercore image to match my updated Windows Server version:
mcr.microsoft.com/windows/servercore:10.0.17763.973
When running the container everything worked as expected.
Original question:
I cannot figure out why nginx doesn't start in my container running on Windows Server 2019.
Nothing is written to the nginx error.log and inspecting the System Event using this answer doesn't provide any information regarding nginx.
When I run nginx directly on the server (i.e. without the container) it starts up fine.
Here are the contents of the dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
WORKDIR C:/nginx
COPY /. ./
CMD ["nginx"]
I run the container using the following command:
docker run -d --rm -p 80:80 --name nginx `
-v C:/Data/_config/nginx/conf:C:/nginx/conf `
-v C:/Data/_config/nginx/temp:C:/nginx/temp `
-v C:/Data/_config/nginx/logs:C:/nginx/logs `
nginx-2019
If I exec into the running container I can see that the directory structure is as expected:
Microsoft Windows [Version 10.0.17763.1040]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\nginx>dir
Volume in drive C has no label.
Volume Serial Number is 72BD-907D
Directory of C:\nginx
02/27/2020 06:05 AM <DIR> .
02/27/2020 06:05 AM <DIR> ..
02/27/2020 06:05 AM <DIR> conf
02/27/2020 05:11 AM <DIR> contrib
02/27/2020 05:11 AM <DIR> docs
02/27/2020 05:11 AM <DIR> html
02/27/2020 05:55 AM <DIR> logs
02/27/2020 05:14 AM <DIR> conf
01/21/2020 03:30 PM 3,712,512 nginx.exe
01/21/2020 04:41 PM <DIR> temp
1 File(s) 3,716,608 bytes
9 Dir(s) 21,206,409,216 bytes free
UPDATE 1:
As part of my troubleshooting process I started up a clean VM on Azure and after installing Docker and recreating the Docker image using the exact same files, it starts up as expected.
This means that the issue is specific to my server and not a general issue.
UPDATE 2:
By removing the --rm from the run command I find the following info by running docker ps -a after the container exits:
Exited (3221225785) 4 seconds ago
I can't find any info on what the exit code means.
I was having the same issue, but for me it wasn't docker or nginx, it was the image.
image mcr.microsoft.com/windows/servercore:ltsc2019 was updated on 2/11/2020 and both container and host must have the same update (KB4532691 I think) or some processes may fail silently.
I updated my host, and all is well.
See microsoft-windows-servercore
and you-might-encounter-issues-when-using-windows-server-containers-with-t for more info
I just bought a QNAP TS-832X NAS (Firmware: 4.3.4.0695 Build 20180830).
This machine comes with an ARM CPU (Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU # 1.70GHz).
I bought it only to install GitLab on it, but the official image doesn't seem to work.
When I try to run the image it fails.
[~] # docker run -d --name gitlab-server --hostname build1 -p 10080:10080 -p 10022:22 -p 10443:443 -v /share/GitLab/config:/etc/gitlab -v /share/GitLab/logs:/var/log/gitlab -v /share/GitLab/data:/var/opt/gitlab --restart always gitlab/gitlab-ce:latest
[~] # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a176158729ad gitlab/gitlab-ce:latest "/assets/wrapper" 5 seconds ago Restarting (1) 1 second ago gitlab-server
[~] # docker logs a1
standard_init_linux.go:185: exec user process caused "exec format error"
standard_init_linux.go:185: exec user process caused "exec format error"
standard_init_linux.go:185: exec user process caused "exec format error"
standard_init_linux.go:185: exec user process caused "exec format error"
standard_init_linux.go:185: exec user process caused "exec format error"
standard_init_linux.go:185: exec user process caused "exec format error"
standard_init_linux.go:185: exec user process caused "exec format error"
After googling I figured it might be caused by the host architecture, so I tried running ulm0/gitlab, but with the same result.
I also tried other images with "ARM" in their tags like arm64v8/ubuntu. This one didn't even give any logs.
[~] # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2b2b68bc912c arm64v8/ubuntu:latest "/bin/bash" 7 seconds ago Restarting (0) 1 second ago ubuntu-arm
a176158729ad gitlab/gitlab-ce:latest "/assets/wrapper" 2 hours ago Restarting (1) 51 seconds ago gitlab-server
[~] # docker logs 2b
[~] #
uname -a
Linux build1 4.2.8 #2 SMP Thu Aug 30 07:33:01 CST 2018 aarch64 GNU/Linux
docker version
Client:
Version: 17.09.1-ce
API version: 1.32
Go version: go1.8.3
Git commit: a9fd393
Built: Fri Aug 3 04:31:20 2018
OS/Arch: linux/arm64
Server:
Version: 17.09.1-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: a9fd393
Built: Fri Aug 3 04:31:20 2018
OS/Arch: linux/arm64
Experimental: false
Sorry to hear about your problem, unfortunately I don't believe there is any official GitLab Docker image for ARM devices.
From personal experience I've found that most developers will make a Docker image for Intel devices but not work on ARM Devices.
This topic has been discussed on the QNAP Forums already:
My QNAP is Intel based, so I can't corroborate your results, but quoting a few sentences from a page about docker on Raspberry Pi:
"Docker-based apps you use have to be packaged specifically for ARM architecture! Docker-based apps packaged for x86/x64 will not work and will result in an error such as:
FATA[0003] Error response from daemon: Cannot start container 0f0fa3f8e510e53908e6a459e817d600b9649e621e7dede974d6a65761ad39e5: exec format error
Keep this in mind when searching for apps on the Docker Hub - the source for Docker apps/images. If you see the keyword RPI or ARM in the heading or description, this app can usually be used for the Raspberry Pi."
The TS-831X has a "AnnapurnaLabs, an Amazon company Alpine AL-314 Quad-core 1.7 GHz Cortex-A15 processor" CPU, which is an ARM architecture much like the Raspberry Pi..
So, I suspect you may be limited in what Docker images you have access to and unless an official/canonical maintainer of an app also makes an ARM build, you may be stuck with either rolling your own or trusting a 3rd party hobbyist to do so...
I hate to say this but I'd say you should have picked up an Intel one instead.
I have a QNAP TS-251+ (Intel based) with 8GB RAM and 2x8TB in a RAID Configuration and this works perfectly for my Gitlab instance, in addition to running PLEX and using it as a Webserver as well.
I would also suggest when you do finally get it up and running to map the volumes to directories that are easy to access so you can make configuration changes easily.
I'm trying to create a new docker image with the following dockerfile, but it's taking an awful long time to finish one of the steps:
FROM microsoft/dotnet-framework:4.7
SHELL ["powershell"]
# Note: Get MSBuild 12.
RUN Invoke-WebRequest "https://download.microsoft.com/download/9/B/B/9BB1309E-1A8F-4A47-A6C5-ECF76672A3B3/BuildTools_Full.exe" -OutFile "$env:TEMP\BuildTools_Full.exe" -UseBasicParsing
RUN & "$env:TEMP\BuildTools_Full.exe" /Silent /Full
# Todo: delete the BuildTools_Full.exe file in this layer
# Note: Add .NET
## RUN Install-WindowsFeature NET-Framework-45-Features ; \
# Note: Add NuGet
RUN Invoke-WebRequest "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe" -OutFile "C:\windows\nuget.exe" -UseBasicParsing
WORKDIR "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0"
# Note: Add Msbuild to path
RUN setx PATH '%PATH%;C:\\Program Files (x86)\\MSBuild\\12.0\\Bin\\msbuild.exe'
CMD ["C:\\Program Files (x86)\\MSBuild\\12.0\\Bin\\msbuild.exe"]
Here is the output so far:
PS C:\MyWorkspace\images\msbuild> docker build -t msbuild .
Sending build context to Docker daemon 2.56kB
Step 1/9 : FROM microsoft/dotnet-framework:4.7
---> 91abbfdc50cb
Step 2/9 : MAINTAINER mohamed.elkammar#gmail.com
---> Using cache
---> fbf720101007
Step 3/9 : SHELL powershell
---> Using cache
---> 642cf0e08730
Step 4/9 : RUN Invoke-WebRequest "https://download.microsoft.com/download/9/B/B/9BB1309E-1A8F-4A47-A6C5-ECF76672A3B3/BuildTools_Full.exe" -OutFile "$env:TEMP\BuildTools_Full.exe" -UseBasicParsing
---> Using cache
---> a722c88fee0f
Step 5/9 : RUN & "$env:TEMP\BuildTools_Full.exe" /Silent /Full
---> Using cache
---> 4fda7448f2e4
Step 6/9 : RUN Invoke-WebRequest "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe" -OutFile "C:\windows\nuget.exe" -UseBasicParsing
---> Running in eec036874574
Additionally, here is the output of docker info:
C:\Windows\system32>docker info
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 5
Server Version: 17.06.1-ee-2
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1715.amd64fre.rs1_release_inmarket.170906-1810)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 1
Total Memory: 4.75GiB
Name: instance-1
ID: B2BG:6AW5:Y32S:YLIO:FE25:WWDO:ZAGQ:CZ3M:S5XM:LSHB:U5GM:VYEM
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
What would cause a simple download step to take forever?
I had face a similar issue with Docker and Windows. Tried the following options:
Check if DNS is too slow. I replaced few nameserver entries in /etc/resolve.conf with 8.8.8.8 and did a force-reload. This helped reduce the load time.
Check if docker is connecting to dockerd through IPv6; which is in my case wasn't the listener. Disabling IPv6 on network interfaces and rebooting the machine helped reduce the load time.
A docker build process will take quite a bit longer than running a container. It also depends on docker has cached.
So usually the first time a container is build when there is no cache it takes the longest, then subsequent docker builds the Docker Engine will use its cache if available. When you run a container it is faster once built.
When you build especially the first time docker will pull microsoft/dotnet-framework:4.7 from the public registry unless a copy is already pulled and available locally in docker images. If it is the Docker Engine will use it, if not it will put it from remote repository. If you have a local copy it is faster. You can pull it before the build if you wish or put it maybe in a local private registry and change your tag to your registry url.
Which step exactly is slow? The very last one where it is getting that nuget.exe? If so the first thing that comes to mind maybe is name resolution, DNS, firewalls, or network time outs reaching the internet.
Try to run this container to check name resolution, and see if it can get that file quickly.
FROM centos
MAINTAINER Blake Russo
RUN yum install -y dig wget bind-utils nc ; dig -x dist.nuget.org; nslookup dist.nuget.org; wget dist.nuget.org/win-x86-commandline/latest/nuget.exe; ls -la nug*;
I'm trying to dockerize a Stardog 3.1.3 community edition server. The container fails to start because it sees a directory instead of a license file. For the record, I'm on Windows. These are the steps I'm following:
Create a data container
docker create -v /data/stardog:/data/stardog --name stardog_data busybox /bin/true
Copy the local license key to the data container (not done in the Dockerfile that is mentioned below as the license is environment specific)
docker cp .\stardog\stardog-license-key.bin stardog_data:/stardog-license-key.bin
Create an image based on the following Dockerfile
docker build -t me/stardog .
FROM java:openjdk-8-jdk
ENV STARDOG_VER stardog-3.1.3
ENV STARDOG_HOME /data/stardog
COPY ${LOCAL_PATH}/${STARDOG_VER}.zip /
RUN unzip ${STARDOG_VER}.zip
WORKDIR /${STARDOG_VER}
CMD rm $STARDOG_HOME/system.lock || true && bin/stardog-admin server start && (tail -f $STARDOG_HOME/stardog.log &) && while (pidof java > /dev/null); do sleep 1; done
Run a Stardog container
docker run -d -p 5820:5820 --volumes-from stardog_data --name stardog me/stardog
When I execute docker ps -a, I see that the container is stopped after a couple of seconds:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9559b22473e1 me/stardog "/bin/sh -c 'rm $STAR" 26 minutes ago Exited (1) 26 minutes ago stardog
2b929329e35e busybox "/bin/true" 32 minutes ago Created stardog_data
When I check the logs with docker logs stardog, I'm getting this:
com.clarkparsia.license.InvalidLicenseException: java.io.FileNotFoundException: /data/stardog/stardog-license-key.bin (Is a directory)
at com.clarkparsia.license.LicenseValidator.validate(LicenseValidator.java:157)
at com.complexible.stardog.StardogLicense.findLicense(StardogLicense.java:127)
at com.complexible.stardog.StardogLicense.<init>(StardogLicense.java:70)
at com.complexible.stardog.Stardog.<init>(Stardog.java:158)
at com.complexible.stardog.Stardog.initialize(Stardog.java:263)
at com.complexible.stardog.Stardog.initialize(Stardog.java:254)
at com.complexible.stardog.Stardog.buildServer(Stardog.java:247)
at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:144)
at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:47)
at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:54)
at com.complexible.stardog.cli.admin.CLI.main(CLI.java:194)
Caused by: java.io.FileNotFoundException: /data/stardog/stardog-license-key.bin (Is a directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at com.clarkparsia.license.LicenseValidator.validate(LicenseValidator.java:113)
... 10 more
Your Stardog license is invalid. Please contact support#clarkparsia.com for information on obtaining a new license.
It seems that the license file is considered to be a directory. What am I doing wrong?
I'm using the following Docker version:
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 21:49:11 2016
OS/Arch: windows/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 21:49:11 2016
OS/Arch: linux/amd64
This seems to be a Windows related problem. I've tried these exact same steps on a native Ubuntu (14.04) machine and it works as expected. I've filed this as a bug and hopefully this gets fixed soon.
I've created a Docker machine to run TensorFlow:
$ docker-machine create --driver virtualbox tensorflow
$ eval $(docker-machine env tensorflow)
$ docker-machine ls
NAME ACTIVE URL STATE URL SWARM DOCKER ERRORS
tensorflow * virtualbox Running tcp://193.168.99.101:2376 v1.9.1
Then I try to launch a container with the TesnorFlow dev image:
$ docker run -it b.gcr.io/tensorflow/tensorflow:latest-devel
But I get the following printout and error:
Unable to find image 'b.gcr.io/tensorflow/tensorflow:latest-devel' locally
latest-devel: Pulling from tensorflow/tensorflow
d5abe3500fe7: Verifying Checksum
b4906bf7cb96: Download complete
bfd1864b7d78: Verifying Checksum
b5834ff0edff: Download complete
37c70795c272: Download complete
fd1295db72c3: Verifying Checksum
0851dd369c75: Download complete
f090bc83193e: Download complete
3505c3e79922: Download complete
5f7f94f4a89a: Download complete
82d880401459: Download complete
3f8ab9378c1a: Verifying Checksum
cd2e7af6dd36: Verifying Checksum
0fbd1aaa7f93: Download complete
a073e9eadc31: Verifying Checksum
eac8972ce5cd: Download complete
d20000bda44f: Download complete
7abc7a47a494: Download complete
6a90f0a0e005: Download complete
Pulling repository b.gcr.io/tensorflow/tensorflow
74c849ee103e: Error pulling image (latest-devel) from b.gcr.io/tensorflow/tensorflow, Untar re-exec error: exit status 1: output: write /usr/lib/gcc/x86_64-linux-gnu/4.8/libitm.a: no space left on device bitm.a: no space left on device
a82f81f25750: Download complete
b207c06aba70: Download complete
d55e68e6cc9c: Download complete
0aa714ad37e0: Download complete
28f77615a692: Error downloading dependent layers
Error pulling image (latest-devel) from b.gcr.io/tensorflow/tensorflow, Untar re-exec error: exit status 1: output: write /usr/lib/gcc/x86_64-linux-gnu/4.8/libitm.a: no space left on device
I'm not sure what to do here... Will someone explain what this means and how I should proceed?
Extra info: I'm on Mac OSX 10.9,
$ docker info
Containers: 0
Images: 12
Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: tmpfs
Dirs: 12
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: tensorflow
Debug mode (server): true
File Descriptors: 11
Goroutines: 19
System Time: 2016-01-24T02:54:29.677797424Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox
Instead of the default docker create, try this:
docker-machine create --driver virtualbox --virtualbox-disk-size "20000" tensorflow
Then:
eval $(docker-machine env tensorflow)
docker run -it b.gcr.io/tensorflow/tensorflow:latest-devel
Note:
See Docker increase disk space
Make sure you have 20 GB on your physical disk