I am brand new to Docker and I am trying to follow the Getting Started tutorial from Docker. I am using Docker 17.05-ce under Ubuntu 17.04. The problem appears to be network related. When I try to push I get the following results:
jonathan#poseidon:~/DockerTest$ sudo docker push jgossage/get-started:part1
The push refers to a repository [docker.io/jgossage/get-started]
1770f1c9a8cf: Pushed
61fd1d8cd138: Pushed
e0f735a5e86f: Layer already exists
1de570a07fb5: Pushed
b3640b6d4ac2: Layer already exists
08d4c9ccebfd: Pushed
007ab444b234: Retrying in 1 second
dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: dial udp 127.0.0.53:53: i/o timeout
jonathan#poseidon:~/DockerTest$ sudo docker logs 58e8df0a7426
* Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
172.17.0.1 - - [20/Jun/2017 15:12:24] "GET / HTTP/1.1" 200 -
172.17.0.1 - - [20/Jun/2017 15:13:17] "GET / HTTP/1.1" 200 -
The push runs for some time with several retries before timing out.
This is on a home network with one computer connected to the router via WiFi and then normal TCP to my ISP and the Internet. What steps can I take to make Docker run reliably?
It looks like a DNS issue similar to this one: https://forums.docker.com/t/fata-0025-io-timeout-on-docker-image-push/1742/9
The suggestion is to replace your current DNS (127.0.0.53) by the Google DNS (8.8.8.8).
I'm not sure if there is an open issue concerning this problem. I couldn't find one.
I resolved this issue by replacing the standard DNS caching and resolving DNS server with a third party implementation unbound. The following web page contains complete instructions for doing this at the end of the document. As also suggested by others, it is a good idea to change to use the public Google DNS servers
Related
Overview of my setup/problem
I'm running a K3s cluster inside a Docker container on a Rhel 7.9 box. This is all on an air gapped network so bare with me if you don't see copy and pasted examples below.
I'm trying to install Istio on the cluster but the install hangs on setting up the ingress gateway deployment. The Istio install hangs on gateway deployment because its unable to resolve the Istiod Kubernetes service from inside the Ingress Gateway pod.
What I've tried
I tested the image on a Ubuntu vagrant box and the Istio install works fine there. I've also tested the install on a Windows 10 machine using Rancher Desktop and it works fine there as well. At one point it worked on the Rhel box but my team did some security hardening over a two period but naturally they have no idea what change broke my cluster. So I'm trying to narrow down the search.
I've determined that the issue is with CoreDNS in my K3s cluster. I used the dnsutils docker image and ran a nslookup kubernetes.default. I checked the logs of the CoreDNS pod and it shows the lookup but the response it sends back to nslookup has the ip of the CoreDNS pod rather than the kube-dns Kubernetes service. nslookup correctly sees that and says
nslookup kubernetes.default
;; reply from unexpected source: 10.42.0.4#53, expected 10.43.0.2#53
;; reply from unexpected source: 10.42.0.4#53, expected 10.43.0.2#53
;; reply from unexpected source: 10.42.0.4#53, expected 10.43.0.2#53
;; connection timed out; no servers could be reached
10.42.0.4 being the CoreDNS pod and the 10.43.0.2 being the kube-dns Kubernetes Service for that pod.
The logs from the failing Istio Ingress Gateway pod are saying that its failing to retrieve a certificate from the Istiod pod because the Istiod kubernetes service connection is timing out. Which makes sense considering I can't resolve kubernetes.default correctly either.
2021-05-27T10:28:07.342344Z warn ca ca request failed, starting attempt 1 in 91.589072ms
2021-05-27T10:28:07.434806Z warn ca ca request failed, starting attempt 2 in 203.792343ms
2021-05-27T10:28:07.639557Z warn ca ca request failed, starting attempt 3 in 364.729652ms
2021-05-27T10:28:08.005300Z warn ca ca request failed, starting attempt 4 in 830.723933ms
And then states that the request to the Istiod service timed out
transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.153.113:41187->10.96.0.10:53: i/o timeout
Again my setup is on an air gapped network so ignore the IP addresses in the example above. These were copied from other posts that are related to my issue.
Where to go from here?
I'm trying to figure out what could be causing this problem. DNS resolution should be out of the box functionality for K3s and its not working correctly. As I stated before its not the Docker image I'm running k3s out of since I've gotten k3s and Istio to work on other machines.
Any suggestions on what to do next or advice on how to troubleshoot this would be greatly appreciated. Let me know if there is any other info I can provide to help. Thanks!
TLDR - bridge-nf-call-iptables and bridge-nf-callip6tables were disabled. They need to be enabled.
I found this using docker info. This listed a warning about bridge-nf-call-iptables and bridge-nf-callip6tables being disabled. I found lots of talk on the CoreDNS and k3s github about issues caused by iptables and our suspicions were correct.
This link was the solution for us.
Running the Chainlink Node with local docker/postgres in OSX Catalina is quite cumbersome due to failed ORM connection or any others.
Doc used: https://docs.chain.link/docs/running-a-chainlink-node
To check if my local db is indeed working ok. I've ran these commands with successful results:
psql postgresql://suchain:docker#127.0.0.1:5432/chainlink
psql -h localhost -U suchain -d chainlink
What have been tried so far
Adding --network host haven't resolve the connection issue
Error Message: Incorrect Usage. flag provided but not defined: -network
Note: Tried with --network=host - same result
Changing the db_url from 127.0.0.1 to localhost
Error Message: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)
Changing the localhost/127.0.0.1 to docker instance name (like pg-docker)
Error Message: hostname resolving error (lookup pg-docker on 192.168.65.1:53: no such host)
Which other options can be used?
Much thanks in advance
What pages have been checked before filing this one:
Running a Chainlink Node - Can't connect to database
CHAINLINK NODE: How might I approach fixing "unable to lock ORM" errors?
https://youtu.be/jJOjyDpg1aA?t=521
Thanks to Patrick. The root cause is the same as in this link
Replacing the db link from localhost/127.0.0.1 to the private/local IP(192.168.0.x) fixed the issue.
FYI: in mac os to find your IP is ifconfig. You'll need to find the en0
How do you set up environment variable for config. Could some one please explain in details. I am using windows home and trying to docker-compose.yml to k8s but when I do kompose up it says:
I have installed kubectl and minikube and dont know how to set the config file so this api can be started
Error while deploying application: Get http://localhost:8080/api: dial tcp [::1]:8080: connectex: No connection could be made because the target machine actively refused it.
Thanks by advance
Kompose always refer to http://localhost:8080/ by default. The problem is that as you are using minikube, your api server is in a different address.
To check the address of your API, run any kubectl command and get your API server address:
$ kubectl get nodes -v6
Output:
I0518 07:27:05.109476 3656 loader.go:375] Config loaded from file: /home/christofoletti/.kube/config
I0518 07:27:05.138651 3656 round_trippers.go:443] GET https://192.168.39.6:8443/api/v1/nodes?limit=500 200 OK in 19 milliseconds
NAME STATUS ROLES AGE VERSION
cluster2 Ready master 3d19h v1.18.2
As you can see, we have GET https://192.168.39.36:8443/api/v1/nodes?limit=500 200 OK.
So, my API server address is https://192.168.39.26:8443/.
Now you can run $ kompose up --server https://192.168.39.26:8443/ and Kompose will know where to send the request.
I'm running a kubernetes cluster on docker-desktop (mac).
It has a local docker registry inside it.
I'm able to query the registry no problem via the API calls to get the list of tags.
I was able to push an image before, but it took multiple attempts to push.
I can't push new changes now. It looks like it pushes successfully for layers, but then doesn't acknowledge the layer has been pushed and then retries.
Repo is called localhost:5000 and I am correctly port forwarding as per instructions on https://blog.hasura.io/sharing-a-local-registry-for-minikube-37c7240d0615/
I'm ot using ssl certs as this is for development on local machine.
(The port forwarding is proven to work otherwise API call would fail)
e086a4af6e6b: Retrying in 1 second
35c20f26d188: Layer already exists
c3fe59dd9556: Pushing [========================> ] 169.3MB/351.5MB
6ed1a81ba5b6: Layer already exists
a3483ce177ce: Retrying in 16 seconds
ce6c8756685b: Layer already exists
30339f20ced0: Retrying in 1 second
0eb22bfb707d: Pushing [==================================================>] 45.18MB
a2ae92ffcd29: Waiting
received unexpected HTTP status: 502 Bad Gateway
workaround (this will suffice but not ideal, as have to build each container
apiVersion: v1
kind: Pod
metadata:
name: producer
namespace: aetasa
spec:
containers:
- name: kafkaproducer
image: localhost:5000/aetasa/cta-user-create-app
imagePullPolicy: Never // this line uses the built container in docker
ports:
- containerPort: 5005
Kubectl logs for registry
10.1.0.1 - - [20/Feb/2019:19:18:03 +0000] "POST /v2/aetasa/cta-user-create-app/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/18.09.2 go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \x5C(darwin\x5C))" "-"
2019/02/20 19:18:03 [warn] 12#12: *293 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000011, client: 10.1.0.1, server: localhost, request: "PATCH /v2/aetasa/cta-user-create-app/blobs/uploads/16ad0e41-9af3-48c8-bdbe-e19e2b478278?_state=qjngrtaLCTal-7-hLwL9mvkmhOTHu4xvOv12gxYfgPx7Ik5hbWUiOiJhZXRhc2EvY3RhLXVzZXItY3JlYXRlLWFwcCIsIlVVSUQiOiIxNmFkMGU0MS05YWYzLTQ4YzgtYmRiZS1lMTllMmI0NzgyNzgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDItMjBUMTk6MTg6MDMuMTU2ODYxNloifQ%3D%3D HTTP/1.1", host: "localhost:5000"
2019/02/20 19:18:03 [error] 12#12: *293 connect() failed (111: Connection refused) while connecting to upstream, client: 10.1.0.1, server: localhost, request: "PATCH /v2/aetasa/cta-user-create-app/blobs/uploads/16ad0e41-9af3-48c8-bdbe-e19e2b478278?_state=qjngrtaLCTal-7-hLwL9mvkmhOTHu4xvOv12gxYfgPx7Ik5hbWUiOiJhZXRhc2EvY3RhLXVzZXItY3JlYXRlLWFwcCIsIlVVSUQiOiIxNmFkMGU0MS05YWYzLTQ4YzgtYmRiZS1lMTllMmI0NzgyNzgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDItMjBUMTk6MTg6MDMuMTU2ODYxNloifQ%3D%3D HTTP/1.1", upstream: "http://10.104.68.90:5000/v2/aetasa/cta-user-create-app/blobs/uploads/16ad0e41-9af3-48c8-bdbe-e19e2b478278?_state=qjngrtaLCTal-7-hLwL9mvkmhOTHu4xvOv12gxYfgPx7Ik5hbWUiOiJhZXRhc2EvY3RhLXVzZXItY3JlYXRlLWFwcCIsIlVVSUQiOiIxNmFkMGU0MS05YWYzLTQ4YzgtYmRiZS1lMTllMmI0NzgyNzgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDItMjBUMTk6MTg6MDMuMTU2ODYxNloifQ%3D%3D", host: "localhost:5000"
Try configure --max-concurrent-uploads=1 for your docker client. You are pushing quite large layers (350MB), so probably you are hitting some limits (request sizes, timeouts) somewhere. Single concurrent upload may help you, but it is only a work around. Real solution will be configuration (buffer sizes, timeouts, ...) of registry + reverse proxy in front of registry eventually.
It may be a disk space issue. If you store docker images inside the Docker VM you can fill up the disk space quite fast.
By default, docker-desktop VM disk space is limited to 64 gigabytes. You can increase it up to 112Gb on the "Disk" tab in Docker Preferences.
I have encountered this issues quite few times and unfortunately couldn't get to the permanent fix.
Most likely the image should have been corrupted in the registry. As a work around, i suggest you delete the image from registry and do a fresh push. it would work and subsequent pushes would work too.
This issue must be related to the missing layers of the image. sometimes we delete the image using --force option, in that case it is possible that some of the common layers might get deleted and would affect other images that share the deleted layers.
When I go docker pull hello-world I get the below error message:
Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
Installed Docker version:
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64
Installed using: curl -fsSL https://get.docker.com/ | sh
I have ensured that network exists and I can reach other sites. Please help
Update 1: The issue cannot be of setting MTU because I could pull the images from Docker hub a few days back on the same machine.
The issue cannot be of HTTP_PROXY either because I am in my home network
I have run across this issue a couple times with Raspberry Pi boards running various flavors of Debian/Raspbian (RPi model info was obtained by cat /proc/cpuinfo | grep Model):
Raspberry Pi Model B Rev 1 with Raspbian based on Debian 11 (bullseye)
Raspberry Pi 4 MOdel B Rev 1.4 with Debian 10 (buster)
In both cases, running docker run --rm hello-world resulted in the 408 HTTP status code reported in the original question in this thread:
$ docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: error parsing
HTTP 408 response body: invalid character '<'
looking for beginning of value: "<html><body>
<h1>408 Request Time-out</h1>\nYour browser
didn't send a complete request in time.\n</body>
</html>\n".
See 'docker run --help'.
The solution (noted as an aside by #Romaan) was to adjust the MTU. I did this as follows:
sudo ip link set dev eth0 mtu 1400
docker run --rm hello-world
and the hello-world container was successfully pulled and executed.
Examples of how to permanently adjust the MTU for a network interface on Debian may be found here.
That error message looks like it's coming from a proxy server. From the docker pull documentation
Proxy configuration
If you are behind an HTTP proxy server, for example in corporate
settings, before open a connect to registry, you may need to configure
the Docker daemon’s proxy settings, using the HTTP_PROXY, HTTPS_PROXY,
and NO_PROXY environment variables. To set these environment variables
on a host using systemd, refer to the control and configure Docker
with systemd for variables configuration.
The link to the instructions for configuring systemd with a proxy is straightforward.
The error message is little misleading. The problem was not that there was invalid character, but the network was misconfigured. I had one LAN interface and one WLAN interface.
LAN interface connects to a router A which forward requests to router B which was connected to internet. While the WLAN interface was directly connected to router B. I forgot to remove the WLAN configuration.
Once I ensured the WLAN interface is removed, things worked smoothly.
In short: Ensured DNS resolution works and that MTU is set right.
Another possible reason for error
If you are using Mac, please ensure to allow Unrestriction Access to Web Content like below:
Another possible step in troubleshooting
Ensure there is no proxy or web filter in your network, that is, if possible connect to your 3G network and try again to see if the results are different
I ran into this problem on Ubuntu. I managed to solve it by disconnecting from NordVPN:
$ nordvpn disconnect
You are disconnected from NordVPN.
It seems the VPN somehow slowed down the dockerhub traffic and broke my docker pulls.
There is a high chance that this is caused due to internet connectivity issue, try to rerun when the internet connection is stable.