Docker fails pulling repository with error: Too Many Requests (HAP429) - docker

I was trying to install gitlab using docker containers and was able to bring up gitlab successfully using docker compose file from sameersbn.
However after few uninstalls and (docker rm ) reinstalls (docker-compose up) as part of CI testing, I started getting this weird error while running docker-compose up or docker run
[root#server.com ~]# docker run java
Unable to find image 'java:latest' locally
Pulling repository docker.io/library/java
docker: Error while pulling image: Get https://index.docker.io/v1/repositories/library/java/images: malformed MIME header line: Too Many Requests (HAP429)..
See 'docker run --help'.
I can't seem to be able to pull any of the docker containers using docker run or docker-compose.
Couldn't find much help online reg this issue.
As per the docker hub forum the issue https://forums.docker.com/t/429-too-many-requests-how-to-fix-this-isssue/3971/7 should disappear after an hour but I waited half a day without much luck!
Here are the details of my installation:
[root#server build]# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:
OS/Arch: linux/amd64
[root#server build]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 15
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 3.077 GB
Data Space Total: 61.2 GB
Data Space Available: 58.12 GB
Metadata Space Used: 1.204 MB
Metadata Space Total: 641.7 MB
Metadata Space Available: 640.5 MB
Thin Pool Minimum Free Space: 6.119 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64 GiB
Name: server.com
ID: SDFS:SDEF:GKY5:UKGK:QHWR:H4EC:wEFw:YVAS:JE2V:A5YB:FDSW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 23
System Time: 2016-10-09T18:34:43.969512367-05:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Any help would be much appreciated. I'm stuck with this error and can't proceed any further with my gitlab.
Thanks.

This may or may not be relevant to your situation, but I can report that I had the same error (didn't go away within an hour) and it was related to the fact that I was on a VPN to my office. I don't know if the VPN was the issue, or the NAT of my workplace, but when I turned off the VPN, the issue went away.
Note, I was running Docker for Windows (W7), so my circumstances are quite different from yours. But perhaps this answer will be useful to you or to anyone else looking for an answer.
Bottom line: If you are using a VPN, switch it off and try again. If you are inside a corporate filewall, try from outside.

Related

Docker pull: operation not permitted

I'm getting this error when pulling some docker images (but not all):
failed to register layer: Error processing tar file(exit status 1): operation not permitted
For example: docker pull nginx works, but not docker pull redis.
I get the same result wether i run the command with a user that is part of the docker group, using sudo or as root.
If i run dockerd in debug mode i see this in the logs:
DEBU[0025] Downloaded 5233d9aed181 to tempfile /var/lib/docker/tmp/GetImageBlob023191751
DEBU[0025] Applying tar in /var/lib/docker/overlay2/e5290b8c50d601918458c912d937a4f6d4801ecaa90afb3b729a5dc0fc405afc/diff
DEBU[0027] Applied tar sha256:16ada34affd41b053ca08a51a3ca92a1a63379c1b04e5bbe59ef27c9af98e5c6 to e5290b8c50d601918458c912d937a4f6d4801ecaa90afb3b729a5dc0fc405afc, size: 79185732
(...)
DEBU[0029] Applying tar in /var/lib/docker/overlay2/c5c0cfb9907a591dc57b1b7ba0e99ae48d0d7309d96d80861d499504af94b21d/diff
DEBU[0029] Cleaning up layer c5c0cfb9907a591dc57b1b7ba0e99ae48d0d7309d96d80861d499504af94b21d: Error processing tar file(exit status 1): operation not permitted
INFO[0029] Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): operation not permitted
INFO[0029] Layer sha256:938f1cd4eae26ed4fc51c37fa2f7b358418b6bd59c906119e0816ff74a934052 cleaned up
(...)
If i run watch -n 0 "sudo ls -lt /var/lib/docker/overlay2/" while the image is pulling, i can see new folders appearing (and disappearing after it fails) and the permissions on /var/lib/docker/overlay2/ are root:root:700 so i don't think it's exactly a permission issue.
Here are some detail about the environment:
I have a proxmox running the LXC container where i'm having the issue.
The container itself is running Debian 8.
And here are the various versions:
$> uname -a
Linux [redacted-hostname] 4.10.15-1-pve #1 SMP PVE 4.10.15-15 (Fri, 23 Jun 2017 08:57:55 +0200) x86_64 GNU/Linux
$> docker version
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:20:04 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:18:59 2017
OS/Arch: linux/amd64
Experimental: false
$>docker info
Containers: 20
Running: 0
Paused: 0
Stopped: 20
Images: 28
Server Version: 17.06.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Kernel Version: 4.10.15-1-pve
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.906GiB
Name: resumed-dev
ID: EBJ6:AFVS:L3RC:ZEE7:A6ZJ:WDQE:GTIZ:RXHA:P4AQ:QJD7:H6GG:YIQB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 16
Goroutines: 24
System Time: 2017-08-17T14:17:07.800849127+02:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
EDIT: This will be fixed by any release after December 18, 2017 of Moby via this merge. Will update again when fully incorporated into Docker.
If your container is unprivileged, this appears to be an issue with the overlay2 storage driver for Docker. This does not appear to be an issue with overlay (GitHub issue). So either utilize the overlay storage driver instead of overlay2, or make your container privileged.
I have almost the same environment as you, and met the same problem.
Some image works perfectly (alpine), while some images fails at cleaning up (ubuntu).
strace -f dockerd -D then docker pull or docker load gives the reason:
mknodat(AT_FDCWD, "/dev/agpgart", S_IFCHR|0660, makedev(10, 175)) = -1 EPERM (Operation not permitted)
Unprivileged container prohibit mknod by design. If you insist nesting Docker inside lxc, you will have to choose privileged container. (And notice that existing unprivileged container cannot be converted to privileged container directly due to uid/gid mapping)

Docker build leads to "no space left on device" on Windows 10

I´m facing a weird problem when I want to build my image on Windows. I haven´t used Docker for anything else, so the installation can be considered as fresh. There are no volumes at all and no images yet.
When I´m trying to build my application from my Dockerfile, it finishes with this error
docker build ./
Sending build context to Docker daemon 1.4 GB
Error response from daemon: Error processing tar file(exit status 1): write /.bowerrc: no space left on device
I´ve read that you can increase the basesize of docker, but I haven´t found a solution for that for Windows (Why is this even limited by default?)
docker info prints some stuff, but it doesn´t show anything about the basesize under "Storage Driver" at all
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: tmpfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.8-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.934 GiB
Name: moby
ID: GUXQ:KPKS:PHBV:BMEF:QHHM:B2YG:MWPB:2W5H:Z3GX:27YS:QBT6:O4RV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 13
Goroutines: 21
System Time: 2017-02-19T20:15:57.8764828Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
According to some posts in the internet, this was once or is the way to go on Linux, but it doesn´t work on Windows
docker daemon --storage-opt dm.basesize=20G
What is wrong with my docker installation and how can I increase the basesize?
I had the same problem on Windows. One of this two settings should solve this problem:
Increase Memory that Docker is using. If is it 2GB, add more
Increase "Disk image max size" - initially is 60, move it to 80 GB
This should be enough, but depends on complexity of what you are building. In some cases increase of Swap would be needed. The initial Swap of Docker is 1024 MB
Thanks
I reset to factory default , then select : delete all image.
At advanced tab : increase 'Disk image max size' (I modify to 80 GB)
Then build docker again
I´ve reinstalled Docker and now it seems to work
This wouldn't have been the underlying issue for the OP back in 2017, but if you're experiencing this problem with Windows 10 1903, then you may be running into Docker for Windows issue #4100 ("Windows 1903 fails when storage-opt used").
On January 6th, 2020, a commit was pushed to the docker-ce GitHub repository that, from the comments (below), appears to implement a workaround to the underlying bug in Windows.
microsoft/hcsshim#718 wclayer: Work around Windows bug when expanding sandbox size
fixes microsoft/hcsshim#708 Windows Host Compute Service bug breaks docker (and other) sandboxes bigger than 20G on Windows 1903
fixes microsoft/hcsshim#624The hcsshim on Windows 10 1903 always fails to build Docker image
fixes/addresses docker/for-win#3884 An error occurred while attempting to build Docker image (especially this comment and the next comments after: docker/for-win#3884 (comment))
fixes/addresses docker/for-win#4100 Windows 1903 fails when storage-opt used
fixes moby/moby#36831 hcsshim::PrepareLayer failed in Win32: The parameter is incorrect (moby/moby#36831 (comment))
fixes Stannieman/audacity-with-asio-builder#5 Docker won't build container
fixes MicrosoftDocs/visualstudio-docs#3523 Error when running build with storage-opts set
fixes moby/moby#39524 Docker build windows 19.03 --storage-opt size>20G
Unfortunately, no new release of Docker has yet been made that includes this commit, so those of us on 1903 and experiencing this bug are kind of stuck for the time being.
Assuming you're doing this with a dev environment, just run:
docker system prune -a -f
Caution: Do not run this on a production environment

Docker remove container error

When I want to rerun container with another volumes or update image.
I stop and try to remove container, but often geterror on rm command
# docker rm containername
Error response from daemon: Driver devicemapper failed to remove root filesystem dbe6....f91f: Device is Busy
I need to restart docker daemon to remove container.
~ # docker version root#CentOS-72-64-minimal
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
------------------------------------------------------------
~ # docker info root#CentOS-72-64-minimal
Containers: 40
Running: 11
Paused: 0
Stopped: 29
Images: 32
Server Version: 1.12.5
Storage Driver: devicemapper
Pool Name: docker-8:3-28705145-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 14.83 GB
Data Space Total: 107.4 GB
Data Space Available: 92.54 GB
Metadata Space Used: 21.15 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.126 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-09-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge overlay host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.96 GiB
Name: CentOS-72-64-minimal
ID: SMTY:72HJ:5QIS:AT63:6GPI:U2UQ:KUYY:C7M6:UIOY:37AR:JS53:JAGA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
I've been experiencing this issue quite a bit on a Red Hat host. The fix according to the reported issue on this is to upgrade to a newer kernel. As a workaround for places where that's not an option, I've been using docker rm -f ... which still throws the error but the container does get cleaned up. Much quicker and less intrusive than a restart of the daemon.
I experienced the same problem, service docker restart almost always fixes the issue.(restarting docker service)
More information about this issue availbale here.

Flag provided but not defined -d while running docker container

I am trying to setup this https://github.com/jwasham/computer-science-flash-cards on my local pc using docker but after I have built my image,when I try
docker run -d -p 8000:8000 --name cs-flash-cards cs-flash-cards
it says
flag provided but not defined: -d
Any ideas how to fix this and run this container?
EDIT(on docker info;docker version I get the following info):
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 10
Server Version: 1.12.5
Storage Driver: devicemapper
Pool Name: docker-202:1-312980-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 877.5 MB
Data Space Total: 107.4 GB
Data Space Available: 2.019 GB
Metadata Space Used: 1.913 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.019 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 990.7 MiB
Name: ip-172-31-33-253
ID: QPUK:E7BB:Y2PW:MPJR:L2X4:4AMT:VHAT:SOXK:3A2N:UKI2:ZXRK:QF4S
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:42:17 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:42:17 2016
OS/Arch: linux/amd64
Just for the next person who has the same problem.
this was for SystemD on Ubuntu LTS and the docker.service file
was the fix.
If you are on Centos7, you may have an extra file called override.conf
this is used to override the docker.service in systemd.
This file also has the -d flag
and is located in
/etc/systemd/system/docker.service.d/override.conf
replace -d with deamon here
do systemctl daemon-reload and docker will start up.

Building docker image that downloads large file fails with error

I'm new to docker. I've been trying it for less than two weeks. We have a service that we'd like to migrate into a container. The service makes use of about 50G worth of data, so we expect the image to be very large. We've written a Dockerfile for it. When we run the build it fails with the ff:
ApplyLayer exit status 1 stdout: stderr: write /mnt/spine_features/spine_features_subset.lmdb/data.mdb: input/output error
When we check docker ps -a for containers, we can see the build container listed with status:
Exited (1) About a minute ago
When we try to commit the container, we get the same error:
Error response from daemon: ApplyLayer exit status 1 stdout: stderr: write /mnt/spine_features/spine_features_subset.lmdb/data.mdb: input/output error
We can also docker inspect the container. When we exclude downloading the largest files, we are able to complete building the service image. Is there some sort of configuration we can change succeed while still including the larger files?
docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.7
Git commit: 23cf638
Built: Fri Aug 19 02:03:02 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.7
Git commit: 23cf638
Built: Fri Aug 19 02:03:02 2016
OS/Arch: linux/amd64
docker info
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 22
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:2-7603782-pool
Pool Blocksize: 65.54 kB
Base Device Size: 214.7 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 67.78 GB
Data Space Total: 107.4 GB
Data Space Available: 39.59 GB
Metadata Space Used: 37.04 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.11 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.134 (2016-09-07)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.7.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 15.58 GiB
Name: mega-haro
ID: MDQ5:JIT3:BVQX:XYO6:YTXI:HTRE:N2UQ:ML4V:ENIE:DDCO:ZGYF:3P5F
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
misty:5000
127.0.0.0/8
As Haoming Zhang recommended, mounting the host directory into the container is an acceptable solution. We are also exploring the possibility of using FUSE to load the data into the container at runtime instead of baking it into the image during build or having the host pass it into the container when the container is run.

Resources