Cannot access the docker registry - docker

I stopped the docker service on my Linux Lubuntu 16.04 and restarted it in debug mode:
sudo service docker stop
Edited the /etc/init.d/docker file to have DOCKER_OPTS="--debug"
sudo service docker start
I launched my registry with the commands:
stephane#ubuntu-512mb-fra1-01:~/dev/certificates$ ll
total 20K
-rw-rw-r-- 1 stephane 962 Oct 22 20:34 certificates.txt
drwxr-xr-x 3 root 4.0K Oct 22 20:46 home/
-rw-rw-r-- 1 stephane 316 Oct 22 20:57 registry-start.sh
-rw-r--r-- 1 root 1.8K Oct 22 20:44 thalasoft.com.crt
-rw-r--r-- 1 root 1.7K Oct 22 20:44 thalasoft.com.key
stephane#ubuntu-512mb-fra1-01:~/dev/certificates$ sudo docker run -d \
> --restart=always \
> --name registry \
> -v `pwd`:/certs \
> -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
> -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/thalasoft.com.crt \
> -e REGISTRY_HTTP_TLS_KEY=/certs/thalasoft.com.key \
> -p 443:443 \
> registry:2
e9ac1a734212779dec14674957efd8daaa034fcd7972c9c0ae9ad6fd8ef89efb
Then I try to push an image into it but it fails as the connection is refused:
stephane#ubuntu-512mb-fra1-01:~/dev/certificates$ sudo docker push localhost:5000/alpine
The push refers to a repository [localhost:5000/alpine]
Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: getsockopt: connection refused
The docker deamon log has this to say:
Oct 23 20:23:39 ubuntu-512mb-fra1-01 systemd[1]: Started Docker Application Container Engine.
Oct 23 20:23:39 ubuntu-512mb-fra1-01 dockerd[23449]: time="2017-10-23T20:23:39.253867742Z" level=info msg="API listen on /var/run/docker.soc
Oct 23 20:23:44 ubuntu-512mb-fra1-01 dockerd[23449]: time="2017-10-23T20:23:44.918635986Z" level=info msg="Attempting next endpoint for push
Oct 23 20:23:44 ubuntu-512mb-fra1-01 dockerd[23449]: time="2017-10-23T20:23:44.919271470Z" level=info msg="Attempting next endpoint for push
The registry container log says:
time="2017-10-23T20:23:39Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.7.6 instance.id=af32ad34-7fb5-419a-ad0c-66ef04471caa version=v2.6.2
time="2017-10-23T20:23:39Z" level=info msg="redis not configured" go.version=go1.7.6 instance.id=af32ad34-7fb5-419a-ad0c-66ef04471caa version=v2.6.2
time="2017-10-23T20:23:39Z" level=info msg="Starting upload purge in 35m0s" go.version=go1.7.6 instance.id=af32ad34-7fb5-419a-ad0c-66ef04471caa version=v2.6.2
time="2017-10-23T20:23:39Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.7.6 instance.id=af32ad34-7fb5-419a-ad0c-66ef04471caa version=v2.6.2
time="2017-10-23T20:23:39Z" level=info msg="listening on [::]:443, tls" go.version=go1.7.6 instance.id=af32ad34-7fb5-419a-ad0c-66ef04471caa version=v2.6.2
My docker version is:
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false
The docker info says:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 17.09.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 8
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-93-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.3MiB
Name: ubuntu-512mb-fra1-01
ID: FTOG:OZBQ:SDIQ:VDF6:Z4UW:7LCA:BOY2:E532:V44N:KECN:TQDR:TUIJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
The registry version is:
registry github.com/docker/distribution v2.6.2
I had created the certificate and key with letsencrypt.org

You run registry container with -p 443:443 and then try to access it on localhost:5000. You should use port 443 and full domain name for which you have certificate (assuming that DNS points to this server). Without using full domain name there is no sense in setting up SSL in first place.

Related

Docker command hang on container

I have created one container by using msql-server:5.7. After creating, I am able to access it without a problem. But after a couple of minutes, all Docker commands in the container becomes unresponsive. I cannot inspect, stop or kill. Is there a way I can debug? I have some other containers running as well, but those are OK.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ca471079614b mysql/mysql-server:5.7 "/entrypoint.sh mysq…" 2 hours ago Up 2 hours (healthy) 33060/tcp db
$ docker info
Containers: 8
Running: 8
Paused: 0
Stopped: 0
Images: 94
Server Version: 17.12.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1047-aws
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.42GiB
Name: ip-172-31-12-60
ID: I7FQ:XUUN:UU5C:KVJI:JPDT:L2BV:B3EQ:5LHI:5XD5:PSWP:NI7Y:BDX7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
$ docker version
Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:11:19 2017
OS/Arch: linux/amd64
Server:
Engine:
Version: 17.12.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:09:53 2017
OS/Arch: linux/amd64
Experimental: false
$ uname -a
Linux ip-172-31-12-60 4.4.0-1047-aws #56-Ubuntu SMP Sat Jan 6 19:39:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Are you double sure this is not docker specific issue? Maybe unsupported kernel version? All logs are dumped to a log file inside the volume in the /var/lib/docker/volumes directory. Try running the container without the -d flag so all output goes to stdout directly to identify the problem.
after couple of day trying, it turns out I change my instance-type from m4.2xlarge to m4.4xlarge. and after trying 1 days. it seems the problem solved. I have no idea why but it is good so far

Docker push continuously retries with large files

Docker push is failing when I have large HDF5 files in the image. One file is between 2.4 and 3.2 GB. It doesn't matter which file it is, it will always fail. Small HDF5 files (19MB) will succeed.
I actually do not get any error message if I attempt to push to the docker registry. The UI looks like it is pushing the file, then when it has pushed all the bytes (100%), it says "Retrying in 15s", and then starts all over again.
I have some error logs from the gitlab-registry below, but a "docker push --verbose" method would be quite helpful.
==> /var/log/gitlab/registry/current <== 2017-08-09_09:23:22.28799 time="2017-08-09T09:23:22.287891292Z" level=error msg="client
disconnected during blob PATCH" auth.user.name=root
contentLength=2000110499 copied=1432460362 environment=production
error="unexpected EOF" go.version=go1.8.1
http.request.host=registry.gitlab.bignut.ai
http.request.id=cb09cab1-8a4c-4001-9aa4-7c11dc9c04c9
http.request.method=PATCH http.request.remoteaddr=35.189.251.28
http.request.uri="/v2/root/gitlab-docker-test/wa_tr/wa_tr_test_data/blobs/uploads/d463eda2-4d84-4b0a-a76a-37dfc043d750?_state=ZTIcKZ18tNeK8HAEIyt4iy0zpqDTC3L6h7phBly07S97Ik5hbWUiOiJyb290L2dpdGxhYi1kb2NrZXItdGVzdC93YV90cmFkaW5nL3dhX3RyYWRpbmdfdGVzdF9kYXRhIiwiVVVJRCI6ImQ0NjNlZGEyLTRkODQtNGIwYS1hNzZhLTM3ZGZjMDQzZDc1MCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNy0wOC0wOVQwOToxOTowNC43ODE1NzIwMTlaIn0%3D"
http.request.useragent="docker/17.06.0-ce go/go1.8.3
git-commit/02c1d87 kernel/4.4.0-89-generic os/linux arch/amd64
UpstreamClient(Docker-Client/17.06.0-ce (linux))"
instance.id=070c4d24-6572-472d-81a4-0ecccd173789 service=registry
vars.name="root/gitlab-docker-test/wa_tr/wa_tr_test_data"
vars.uuid=d463eda2-4d84-4b0a-a76a-37dfc043d750
version=v2.6.1-1-gdd544a8 2017-08-09_09:23:23.05762 127.0.0.1 - -
[09/Aug/2017:09:22:42 +0000] "PATCH
/v2/root/gitlab-docker-test/wa_tr/wa_tr_test_data/blobs/uploads/d463eda2-4d84-4b0a-a76a-37dfc043d750?_state=ZTIcKZ18tNeK8HAEIyt4iy0zpqDTC3L6h7phBly07S97Ik5hbWUiOiJyb290L2dpdGxhYi1kb2NrZXItdGVzdC93YV90cmTM3ZGZjMDRhcnRlZEF0IjoiMjAxOToxOTowNC43ODE1NzIwMTlaIn0%3D
HTTP/1.0" 499 0 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87
kernel/4.4.0-89-generic os/linux arch/amd64
UpstreamClient(Docker-Client/17.06.0-ce (linux))"
Output of docker version:
Client: Version: 17.06.0-ce API version:
1.30 Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:23:31 2017 OS/Arch: linux/amd64
Server: Version: 17.06.0-ce API version: 1.30 (minimum version 1.12)
Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:19:04
2017 OS/Arch: linux/amd64 Experimental: false
Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 28 Server
Version: 17.06.0-ce Storage Driver: aufs Root Dir:
/var/lib/docker/aufs Backing Filesystem: extfs Dirs: 40 Dirperm1
Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs
Plugins: Volume: local Network: bridge host macvlan null overlay Log:
awslogs fluentd gcplogs gelf journald json-file logentries splunk
syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init
Binary: docker-init containerd version:
cfb82a876ecc11b5ca0977d1733adbe58599088a runc version:
2d41c047c83e09a6d61d464906feb2a2f3c52aa4 init version: 949e6fa
Security Options: apparmor seccomp Profile: default Kernel Version:
4.4.0-89-generic Operating System: Ubuntu 16.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 91.47GiB Name: amitt ID:
5PGY:FXXK:WB4M:YBZP:SQZU:DIZD:6WYC:CZ6T:NZZ5:3C7D:WQHX:O3UB Docker
Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode
(server): false Registry: https://index.docker.io/v1/ Experimental:
false Insecure Registries:
127.0.0.0/8 Live Restore Enabled: false
WARNING: No swap limit support

Docker command with build / run / etc. "too many open files"

Im installed Docker on server machine (with Ubuntu 16.04), but trying run any commands, I get this error:
"Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: too many open files"
Please, who know how solve this issue?
Docker Info:
Containers: 18
Running: 0
Paused: 0
Stopped: 18
Images: 4
Server Version: 17.03.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 76
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-ini
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953 GiB
Name: web.renderdeal.com
ID: LXNZ:I6HH:ZVBB:KS3V:3WCT:ADWY:C2MZ:QJ37:VZUU:EZ6T:PYWO:66WQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: animarender
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
/Docker Info
docker version
Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 11:02:43 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 11:02:43 2017
OS/Arch: linux/amd64
Experimental: false
/ docker version
Considering Docker status seems OK, check if the issue is on your side (as in this ticket)
Let's check on the current open files
lsof | grep "rclone" | wc -l
returns 6458
Now, Check on current ulimit
ulimit -n
returns 1024
Set that to 9000 and Check again
ulimit -n 9000 && ulimit -n
returns 9000
To make the ulimit more persistent you can either edit /etc/security/limits.conf
and add :
* soft nofile 9000 + reboot
or write it into your user's .bashrc

Docker container can't resolve DNS to reach another AWS Ec2 Machine

I am not able to ping another machine/host App2 by resolving the DNS from the container running on host App1. Though the /etc/resolv.conf is same as that of host. I am making use of AWS Route 53 private hosted DNS to allow intercommunication by resolving DNS and not IPs.
Some basic info for this :
ubuntu#app1:~$ docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 10
Server Version: 1.13.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 31
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.13.0-106-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797 GiB
Name: app1
ID: 6GYC:GI6M:JNTM:MMSL:7LRD:BEUZ:RTRD:Q4AG:NEQU:XC5C:ALOK:N3LM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
############################################
ubuntu#app1:~$ docker version
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:42:29 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:42:29 2017
OS/Arch: linux/amd64
Experimental: false
###########################################
ubuntu#app1:~$ docker exec -it conatiner1 sh
/data # ping app2
ping: bad address 'app2'
/data # ping app2.mydomain
PING app2.mydomain (10.xx.xx.xx): 56 data bytes
##############################################
resolv.conf on conatiner
/data # cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.xx.xx.xx
search mydomain
resolv.conf on host
ubuntu#app1:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.XX.XX.xx [ same as of container ]
search mydomain
From the docker host i am able to ping App2 wihtout giving full domain like app2.mydomain but same is not working from the container
When you call docker run, please add --net=host option to use host's network stack. It will do the trick.

docker-proxy - Error starting userland proxy while trying to bind on 443

I'm trying to install discourse with docker in an Ubuntu 16.04 LTS with Apache listening to port 80 and 443.
When I try to lunch the app I get the following error:
starting up existing container
+ /usr/bin/docker start app Error response from daemon: driver failed programming external connectivity on endpoint app
(dade361e77fbf29f4d9667febe57a06f168f916148e10cc1365093d8f97026bb):
Error starting userland proxy: listen tcp 0.0.0.0:443: listen: address
already in use Error: failed to start containers: app
For what I'v found docker-proxy is the one that is trying to bind on 443.
How can I solve this?
Some details...
docker version
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
docker info
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 4
Server Version: 1.11.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 25
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.0-28-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.39 GiB
Name: sd-12345
ID: 6OLH:SAG5:VWTW:BL7U:6QYH:4BBS:QHBN:37MY:DLXA:W64E:4EVZ:WBAK
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
perhaps, stop apache? – vitr Jul 22 '16 at 2:56
^^^ This comment from vitr should be the Accepted Answer:
Docker cannot proxy a service from within a container to the port on the host without first stopping any services that are already using that port.
In this case, Apache must be stopped with a command such as sudo service apache2 stop.
Then docker start app can then be run and docker should do its thing unhindered.
See the related question: docker run -> name is already in use by container
Edit /etc/docker/daemon.json and add:
{
"userland-proxy": false
}

Resources