Why is nginx not starting in Windows Server 2019 docker container? - docker

UPDATE 3 (Solution):
I installed the latest Windows updates on my host and specified the exact servercore image to match my updated Windows Server version:
mcr.microsoft.com/windows/servercore:10.0.17763.973
When running the container everything worked as expected.
Original question:
I cannot figure out why nginx doesn't start in my container running on Windows Server 2019.
Nothing is written to the nginx error.log and inspecting the System Event using this answer doesn't provide any information regarding nginx.
When I run nginx directly on the server (i.e. without the container) it starts up fine.
Here are the contents of the dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
WORKDIR C:/nginx
COPY /. ./
CMD ["nginx"]
I run the container using the following command:
docker run -d --rm -p 80:80 --name nginx `
-v C:/Data/_config/nginx/conf:C:/nginx/conf `
-v C:/Data/_config/nginx/temp:C:/nginx/temp `
-v C:/Data/_config/nginx/logs:C:/nginx/logs `
nginx-2019
If I exec into the running container I can see that the directory structure is as expected:
Microsoft Windows [Version 10.0.17763.1040]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\nginx>dir
Volume in drive C has no label.
Volume Serial Number is 72BD-907D
Directory of C:\nginx
02/27/2020 06:05 AM <DIR> .
02/27/2020 06:05 AM <DIR> ..
02/27/2020 06:05 AM <DIR> conf
02/27/2020 05:11 AM <DIR> contrib
02/27/2020 05:11 AM <DIR> docs
02/27/2020 05:11 AM <DIR> html
02/27/2020 05:55 AM <DIR> logs
02/27/2020 05:14 AM <DIR> conf
01/21/2020 03:30 PM 3,712,512 nginx.exe
01/21/2020 04:41 PM <DIR> temp
1 File(s) 3,716,608 bytes
9 Dir(s) 21,206,409,216 bytes free
UPDATE 1:
As part of my troubleshooting process I started up a clean VM on Azure and after installing Docker and recreating the Docker image using the exact same files, it starts up as expected.
This means that the issue is specific to my server and not a general issue.
UPDATE 2:
By removing the --rm from the run command I find the following info by running docker ps -a after the container exits:
Exited (3221225785) 4 seconds ago
I can't find any info on what the exit code means.

I was having the same issue, but for me it wasn't docker or nginx, it was the image.
image mcr.microsoft.com/windows/servercore:ltsc2019 was updated on 2/11/2020 and both container and host must have the same update (KB4532691 I think) or some processes may fail silently.
I updated my host, and all is well.
See microsoft-windows-servercore
and you-might-encounter-issues-when-using-windows-server-containers-with-t for more info

Related

podman unable to build image from Dockerfile error creating overlay mount

I am getting error as error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged
Please find debug info below.
OS
root#cks-master:/vagrant/files# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
Dockerfile
# cat ch5Dockerfile
FROM bash
CMD ["ping", "killer.sh"]
Build log
root#cks-master:/vagrant/files# podman build -t simple -f ch5Dockerfile .
STEP 1/2: FROM bash
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged: invalid argument
Error: error mounting new container: error mounting build container "6c0f88a6da54d713e18283e16521385fff736bc1a1072938fddfc6be4b3d43cc": error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/BNREFG6CRAAHJ7VSYG3EUXV5UO:/var/lib/containers/storage/overlay/l/MDSWZVRVZNCOW75JF32K6D4QQC:/var/lib/containers/storage/overlay/l/4G3NS52LYHWPTKA4FURHLYMAPF,upperdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/diff,workdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/work": invalid argument
root#cks-master:/vagrant/files#
podman version
root#cks-master:/vagrant/files# podman version
Version: 3.4.2
API Version: 3.4.2
Go Version: go1.15.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
root#cks-master:/vagrant/files#
Issue resolved.
I also posted the issue in podman room at https://app.element.io/#/room/#podman:fedoraproject.org
I am suggested with podman reset(command trace is below), then its complained about storage.conf, I removed that file and did reset again. Then its worked.
I still wonder whats inside of storage.conf causing this issue but I deleted before looking into it. Finally its worked and I am able to continue. Hope it helps.
Note: Post the deleting storage.conf file and podman reset, I tried with docker build as well just to check if docker has any dependency over storage.conf file, but none, docker build also executed successfully( command trace below)
root#cks-master:~# podman system reset -f
A storage.conf file exists at /etc/containers/storage.conf
You should remove this file if you did not modified the configuration.
root#cks-master:~# rm /etc/containers/storage.conf
root#cks-master:~# podman system reset -f
root#cks-master:~# podman build -t simple -f /vagrant/files/ch5Dockerfile .
STEP 1/2: FROM bash
Resolving "bash" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/bash:latest...
Getting image source signatures
Copying blob 9621f1afde84 done
Copying blob 1dd831616e40 done
Copying blob fd6cd28e0879 done
Copying config 8b332999f6 done
Writing manifest to image destination
Storing signatures
STEP 2/2: CMD ["ping", "killer.sh"]
COMMIT simple
--> cd1407a69ea
Successfully tagged localhost/simple:latest
cd1407a69ea490496d6635700958f2b5fcf2b1d01f8dd218dea0f83187e55872
root#cks-master:~# podman run --name simple simple
PING killer.sh (35.227.196.29): 56 data bytes
64 bytes from 35.227.196.29: seq=0 ttl=42 time=15.689 ms
64 bytes from 35.227.196.29: seq=1 ttl=42 time=14.662 ms
64 bytes from 35.227.196.29: seq=2 ttl=42 time=15.161 ms
^C
--- killer.sh ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 14.662/15.170/15.689 ms
root#cks-master:~# docker build -t simple -f /vagrant/files/ch5Dockerfile .
Sending build context to Docker daemon 3.141MB
Step 1/2 : FROM bash
latest: Pulling from library/bash
9621f1afde84: Pull complete
1dd831616e40: Pull complete
fd6cd28e0879: Pull complete
Digest: sha256:e4624241e953934fc4c396217253d8322ebda53be3b1863cd7795541d168034f
Status: Downloaded newer image for bash:latest
---> 8b332999f684
Step 2/2 : CMD ["ping", "killer.sh"]
---> Running in 306963a83d1c
Removing intermediate container 306963a83d1c
---> 51dee555fd57
Successfully built 51dee555fd57
Successfully tagged simple:latest
root#cks-master:~# ^C
root#cks-master:~#

docker command not available in custom pipe for BitBucket Pipeline

I'm working on a build step that handles common deployment tasks in a Docker Swarm Mode cluster. As this is a common problem for us and for others, we've shared this build step as a BitBucket pipe: https://bitbucket.org/matchory/swarm-secret-pipe/
The pipe needs to use the docker command to work with a remote Docker installation. This doesn't work, however, because the docker executable cannot be found when the pipe runs.
The following holds true for our test repository pipeline:
The docker option is set to true:
options:
docker: true
The docker service is enabled for the build step:
main:
- step:
services:
- docker: true
Docker works fine in the repository pipeline itself, but not within the pipe.
Pipeline log shows the docker path being mounted into the pipe container:
docker container run \
--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \
--volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \
--volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/matchory/swarm-secret-pipe:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/matchory/swarm-secret-pipe \
--workdir=$(pwd) \
--label=org.bitbucket.pipelines.system=true \
radiergummi/swarm-secret-pipe:1.3.7#sha256:baf05b25b38f2a59b044e07f4ad07065de90257a000137a0e1eb71cbe1a438e5
The pipe is pretty standard and uses a recent Alpine image; nothing special in that regard. The PATH is never overwritten. Now for the fun part: If I do ls /usr/local/bin/docker inside the pipe, it shows an empty directory:
ls /usr/local/bin
total 16K
drwxr-xr-x 1 root root 4.0K May 13 13:06 .
drwxr-xr-x 1 root root 4.0K Apr 4 16:06 ..
drwxr-xr-x 2 root root 4.0K Apr 29 09:30 docker
ls /usr/local/bin/docker
total 8K
drwxr-xr-x 2 root root 4.0K Apr 29 09:30 .
drwxr-xr-x 1 root root 4.0K May 13 13:06 ..
ls: /usr/local/bin/docker/docker: No such file or directory
As far as I understand pipelines and Docker, /usr/local/bin/docker should be the docker binary file. Instead, it appears to be an empty directory for some reason.
What is going on here?
I've also looked at other, official, pipes. They don't do anything differently, but seem to be using the docker command just fine (eg. the Azure pipe).
After talking to BitBucket support, I solved the issue. As it turns out, if the docker context is changed, any docker command is sent straight to the remote docker binary, which (on our services) lives at a different path than in BitBucket Pipelines!
As we changed the docker context before using the pipe, and the docker instance mounted into the pipe still has the remote context set, but the pipe searches for the docker binary at another place, the No such file or directory error is thrown.
TL;DR: Always restore the default docker host/context before passing control to a pipe, e.g.:
script:
- export DEFAULT_DOCKER_HOST=$DOCKER_HOST
- unset DOCKER_HOST
- docker context create remote --docker "host=ssh://${DEPLOY_SSH_USER}#${DEPLOY_SSH_HOST}"
- docker context use remote
# do your thing
- export DOCKER_HOST=$DEFAULT_DOCKER_HOST # <------ restore the default host
- pipe: matchory/swarm-secret-pipe:1.3.16

Time in Docker container out of sync with host machine

I'm trying to connect to CosmosDB through my SpringBoot app. I have all of this working if I run the app with Spring or via Intellij. But, when I run the app in Docker I get the following error message:
com.azure.data.cosmos.CosmosClientException: The authorization token is not valid at the current time.
Please create another token and retry
(token start time: Thu, 26 Mar 2020 04:32:10 GMT,
token expiry time: Thu, 26 Mar 2020 04:47:10 GMT, current server time: Tue, 31 Mar 2020 20:12:42 GMT).
Note that in the above error message the current server time is correct but the other times are 5 days behind.
What I find interesting is that I only ever receive this in the docker container.
FROM {copy of zulu-jdk11}
ARG JAR_FILE
#.crt file in the same folder as your Dockerfile
ARG CERT="cosmos.cer"
ARG ALIAS="cosmos2"
#import cert into java
COPY $CERT /
RUN chmod +x /$CERT
WORKDIR $JAVA_HOME/lib/security
RUN keytool -importcert -file /$CERT -alias $ALIAS -cacerts -storepass changeit -noprompt
WORKDIR /
COPY /target/${JAR_FILE} app.jar
COPY run-java.sh /
RUN chmod +x /run-java.sh
ENV JAVA_OPTIONS "-Duser.timezone=UTC"
ENV JAVA_APP_JAR "/app.jar"
# run as non-root to mitigate some security risks
RUN addgroup -S pcc && adduser -S nonroot -G nonroot
USER nonroot:nonroot
ENTRYPOINT ["/run-java.sh"]
One thing to note is ENV JAVA_OPTIONS "-Duser.timezone=UTC" but removing this didn't help me at all
I basically run the same step from IntelliJ and I have no issues with it but in docker the expiry date seems to be 5 days behind.
version: "3.7"
services:
orchestration-agent:
image: {image-name}
ports:
- "8080:8080"
network_mode: host
environment:
- COSMOSDB_URI=https://host.docker.internal:8081/
- COSMOSDB_KEY={key}
- COSMOSDB_DATABASE={database}
- COSMOSDB_POPULATEQUERYMETRICS=true
- COSMOSDB_ITEMLEVELTTL=60
I think it should also be mentioned that I changed the network_mode to host. And I also changed the CosmosDB URI from https://localhost:8081 to https://host.docker.internal:8081/
I would also like to mention that I built my dockerfile with the help of:
Importing self-signed cert into Docker's JRE cacert is not recognized by the service
How to add a SSL self-signed cert to Jenkins for LDAPS within Dockerfile?
Docker containers don't maintain a separate clock, it's identical to the Linux host since time is not a namespaced value. This is also why Docker removes the permission to change the time inside the container, since that would impact the host and other containers, breaking the isolation model.
However, on Docker Desktop, docker runs inside of a VM (allowing you to run Linux containers on non-Linux desktops), and that VM's time can get out of sync when the laptop is suspended. This is currently being tracked in an issue over on github which you can follow to see the progress: https://github.com/docker/for-win/issues/4526
Potential solutions include restarting your computer, restarting docker's VM, running NTP as a privileged container, or resetting the time sync in the windows VM with the following PowerShell:
Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Disable-VMIntegrationService
Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Enable-VMIntegrationService
With WSL 2, restarting the VM involves:
wsl --shutdown
wsl
There is recent known problem with WSL 2 time shift after sleep which has been fixed in 5.10.16.3 WSL 2 Linux kernel which is still not included in Windows 10 version 21H1 update but can be installed manually.
How to check WSL kernel version:
> wsl uname -r
Temporal workaround for the old kernel that helps until next sleep:
> wsl hwclock -s
Here's an alternative that worked for me on WSL2 with Docker Desktop on Windows:
Since it's not possible to set the date inside a Docker container, I just opened Ubuntu in WSL2 and ran the following command to synchronize the clock:
sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
It worked well, so I added the following line in my root user's crontab:
# Edit root user's crontab
sudo crontab -e
# Add the following line to run it every minute of every day:
* * * * * sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
After that, I just restarted my Docker containers, and the dates were correct since they seemed to use the WSL2 Ubuntu dates.
Date before (incorrect):
date
Thu Feb 4 21:50:35 UTC 2021
Date after (correct):
date
Fri Feb 5 19:01:05 UTC 2021

How to see file from docker for Windows host on a running container and vice versa

I'm trying to get a Linux container to share files with the Windows host, primarily because I want to build some Linux libs, and put the resulting output where I can see them in the file system. I open a cmd.exe window and do the following.
Microsoft Windows [Version 10.0.16299.1565]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\Users\alanmur>mkdir \dev\test
C:\Users\alanmur>cd \dev\test
C:\dev\test>echo for the container >> testfile.out
C:\dev\test>dir
Volume in drive C has no label.
Volume Serial Number is 0C3F-DCE2
Directory of C:\dev\test
2020-01-07 21:45 <DIR> .
2020-01-07 21:45 <DIR> ..
2020-01-07 21:45 20 testfile.out
1 File(s) 20 bytes
2 Dir(s) 52,950,069,248 bytes free
C:\dev\test>docker run -it --rm -v C:/dev/test:/app/data:rw alpine /bin/sh
/ # cd /app/data
/app/data # ls
/app/data # echo for the host >> testfile2.out
/app/data # ls
testfile2.out
/app/data # exit
C:\dev\test>dir
Volume in drive C has no label.
Volume Serial Number is 0C3F-DCE2
Directory of C:\dev\test
2020-01-07 21:45 <DIR> .
2020-01-07 21:45 <DIR> ..
2020-01-07 21:45 20 testfile.out
1 File(s) 20 bytes
2 Dir(s) 52,942,929,920 bytes free
C:\dev\test>
How would I set this up so that the first ls on the container shows testfile.out and then when I exit the container I can see testfile2.out in the host C:\dev\test directory? I swear I had this working before, but I can't figure out what I'm doing wrong now, as I am doing everything the same.
My guess is that you haven't done is with this particular location/drive. You need to allow docker to access it.
Here is how (for windows)
For you it is drive C:
In the System Tray, you should have the
cute Docker whale swimming. Right click and select Settings.
Docker Settings Menu
In the Settings dialog that comes up, click on Shared Drives. This should be able to list down the drives that you have available on your
Windows machine. In my case, I have C and D drives and I have chosen
to share D:\ drive since I want to expose the D:\data folder to my
containers.
Docker for Windows : Shared Drives
3. Click on Apply. This will bring up the Credentials dialog and you will need to provide your current Windows credentials. Ensure that you
give it correctly. I also suspect that you might need to be an
Administrator.
Source: https://rominirani.com/docker-on-windows-mounting-host-directories-d96f3f056a2c

Mounting Folders in Docker: Windows Host, Windows Image

How to use the docker mount option to share a local folder in Docker Container? Currently, I am using this command but I am not being successful.
docker run --mount source='c:\temp',target='c:\temp' -i newname:latest
I get this error -
C:\Program Files\Docker\docker.exe: Error response from daemon: invalid mount config for type "volume": invalid volume name.
My environment:
Host: Windows Server, version 1709
Docker Container: Windows Server Core, v1709
You need to use bind mount. Example below maps your host directory c:\users\public\ to the one which is inside container c:\users\public and then outputs content of that directory.
PS C:\Users\gsuvalia> docker run --rm --mount type=bind,source=c:\users\public\,destination=c:\users\public\ microsoft/nanoserver powershell get-childitem c:\users\public
Directory: C:\users\public
Mode LastWriteTime Length Name
---- ------------- ------ ----
d-r--- 12/1/2017 10:16 PM Documents
d-r--- 7/16/2016 6:47 AM Downloads
d-r--- 7/16/2016 6:47 AM Music
d-r--- 12/1/2017 10:16 PM Pictures
d----- 8/22/2017 10:26 PM Roaming
d-r--- 7/16/2016 6:47 AM Videos

Resources