I have a docker container running Caddy, and another web server.
In my Dockerfile I have
EXPOSE 80 10240 # 10240 is the port of the other webserver.
And I run docker like this (don't ask my why you need EXPOSE and -p).
docker run -p 80:80 -p 10240:1024 -it <hash>
This starts the two servers. On my host machine (it's a Mac btw) I can connect to localhost:10240 fine. However if I connect to localhost:80, I get an empty response (dropped connection).
Netstat in the docker container shows:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:10240 0.0.0.0:* LISTEN 19/node
tcp6 0 0 :::80 :::* LISTEN 9/caddy
Here's where it starts to get weird. If I curl -L localhost within the docker container, it works fine - I get the web page from the 10240 server.
If I curl -L 127.0.0.1 from within the docker container it returns 404 Site 127.0.0.1 is not served on this interface. Ok fine.
If I curl -L 127.0.0.1 from outside the container it also returns 404 Site 127.0.0.1 is not served on this interface. So somehow my requests are getting through, but Caddy drops localhost requests from outside the container, and it doesn't from inside it. I have logging enabled but it prints nothing.
Can anyone tell me what the hell is going on? All this docker port forwarding stuff is ridiculous.
Here's my Caddyfile (and I've tried about a billion other combinations of localhost, 127.0.0.1, etc.):
localhost:80
bind 0.0.0.0
proxy / 127.0.0.1:10240
I am not sure but I suspect this was because Docker for Mac's networking is kind of broken. I gave up on Caddy and tried to do the same think with Traefik, which also didn't work (though it gave a "Gateway error" instead of totally dropping the connection).
As soon as moved everything to Linux, it worked perfectly.
Related
Trying investigate my issue with docker container. I lost a day when I thought that issue is in nodejs code (it has server and I am trying to connect to this server).
After investigations I found interesting thing for me.
For example - Lets run some test docker image:
docker run -p 888:888 -it ubuntu:16 /bin/bash
After that, prepare and install "simple server to listen our port":
apt-get update
apt-get install -y netcat
nc -l 888
After that I going to try to telnet localhost 888 from my system and got telnet: connect to address 127.0.0.1: Connection refused. The same with nodejs image.
But if you try to use, for example, nginx container -
docker run -p 888:888 -it nginx /bin/bash
I will be successfull:
$telnet 127.0.0.1 888
Trying 127.0.0.1...
Connected to localhost.
How it is possible, what I am missing? Why I can bind and use any port in nginx but not for other images?
When you run nc -l 888, you are creating a port that is listening explicitly for IPv4 connections. If we run ss -tln, we will see:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1 0.0.0.0:888 0.0.0.0:*
When you run telnet localhost 888 on your host, there's a good chance it's trying to connect to the IPv6 localhost address, ::1. This connection fails if you're trying to connect an IPv4-only socket.
If you explicitly use the IPv4 loopback address by typing telnet 127.0.0.1 888, it should work as expected.
If you enable IPv6 support in nc by adding the -6 parameter:
nc -6 -l 8888
Then you get a socket that listen for both IPv4 and IPv6 connections:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1 *:888 *:*
And if you attempt to connect to this socket using telnet localhost 888, it will work as expected (as will telnet 127.0.0.1 888).
Most programs (like nginx) open multi-protocol sockets by default, so this isn't normally an issue.
I would like to run the VSCode OSS Web Server within a Docker Container, as described here: https://github.com/microsoft/vscode/wiki/How-to-Contribute#vs-code-for-the-web
The Container is running, but the port mapping doesn't work. I run my image with
docker run -it -p 9888:9888 -p 5877:5877 vscode-server
but I got nothing with curl -I http://localhost:9888 on my machine. The VScode server is running, but the mapping to the host will not work. I think the problem is the binding. It looks like the VScode Server will bind to 127.0.0.1 but should bind to 0.0.0.0
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:9888 0.0.0.0:* LISTEN 870/node
tcp 0 0 127.0.0.1:5877 0.0.0.0:* LISTEN 881/node
Can anybody help here?
I'm running a setup of 3 Ubuntu virtual machines. Two running the Python production code base and the other has a Memcached Docker container. On the Memcached machine I ran docker run -dit --name production-memcached --publish 11211:11211 memcached:latest.
The code base gets the following error message when trying to interact with it:
"exception": "TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None)"
I have ran docker exec -it production-memcached memcached stats and get the error message below.
failed to listen on TCP port 11211: Address already in use
However I've ran netstat -plnt and get tcp6 0 0 :::11211 :::* LISTEN 35030/docker-proxy, which looks fine to me.
I was able to get this to work by opening port 80 and using iptables to forward incoming port 80 to port 11211 but would prefer to use the Memcached port number.
The Memcached client is created by the following line:
client = base.Client(("domain.co.uk", 11211))
Any help would be appreciated, thanks.
DigitalOcean doesn't allow in-traffic on port 11211 to its virtual machines. If you want a Memcached machine you'll also need a virtual machine to act a proxy between you and it. I hope this saves someone the headache it caused me!
Suddenly when I deployed some new containers with docker-compose the internal hostname resolution didn't work.
When I tried to ping one container from the other using the service name from the docker-compose.yaml file I got ping: bad address 'myhostname'
I checked that the /etc/resolv.conf was correct and it was using 127.0.0.11
When I tried to manually resolve my hostname with either nslookup myhostname. or nslookup myhostname.docker.internal I got error
nslookup: write to '127.0.0.11': Connection refused
;; connection timed out; no servers could be reached
Okay so the issue is that the docker DNS server has stopped working. All already started containers still function, but any new ones started has this issue.
I am running Docker version 19.03.6-ce, build 369ce74
I could of course just restart docker to see if it solves it, but I am also keen on understanding why this issue happened and how to avoid it in the future.
I have a lot of containers started on the server and a total of 25 docker networks currently.
Any ideas on what can be done to troubleshoot? Any known issues that could explain this?
The docker-compose.yaml file I use has worked before and no changes has been done to it.
Edit: No DNS names at all can be resolved. 127.0.0.11 refuses all connections. I can ping any external IP addresses, as well as the IP of other containers on the same docker network. It is only the 127.0.0.11 DNS server that is not working. 127.0.0.11 still replies to ping from within the container.
Make sure you're using a custom bridge network, NOT the default one. As per the Docker docs (https://docs.docker.com/network/bridge/), the default bridge network does not allow automatic DNS resolution:
Containers on the default bridge network can only access each other by IP addresses, unless you use the --link option, which is considered legacy. On a user-defined bridge network, containers can resolve each other by name or alias.
I have the same problem. I am using the pihole/pihole docker container as the sole dns server on my network. Docker containers on the same host as the pihole server could not resolve domain names.
I resolved the issue based on "hmario"'s response to this forum post.
In brief, modify the pihole docker-compose.yml from:
---
version: '3.7'
services:
unbound:
image: mvance/unbound-rpi:1.13.0
hostname: unbound
restart: unless-stopped
ports:
- 53:53/udp
- 53:53/tcp
volumes: [...]
to
---
version: '3.7'
services:
unbound:
image: mvance/unbound-rpi:1.13.0
hostname: unbound
restart: unless-stopped
ports:
- 192.168.1.30:53:53/udp
- 192.168.1.30:53:53/tcp
volumes: [...]
Where 192.168.1.30 is a ip address of the docker host.
I'm having exactly the same problem. According to the comment here I could reproduce the setting without docker-compose, only using docker:
docker network create alpine_net
docker run -it --network alpine_net alpine /bin/sh -c "cat /etc/resolv.conf; ping -c 4 www.google.com"
stopping docker (systemctl stop docker) and enabling debug output it gives
> dockerd --debug
[...]
[resolver] read from DNS server failed, read udp 172.19.0.2:40868->192.168.177.1:53: i/o timeout
[...]
where 192.168.177.1 is my local network ip for the host that docker runs on and where also pi-hole as dns server is running and working for all of my systems.
I played around with fixing iptables configuration. but even switching them off completely and opening everything did not help.
The solution I found, without fully understanding the root case, was to move the dns to another server. I installed dnsmasq on a second system with ip 192.168.177.2 that nothing else than forwarding all dns queries back to my pi-hole server on 192.168.177.1
starting docker on 192.168.177.1 again with dns configured to use 192.168.177.2 everything was working again
with this in one terminal
dockerd --debug --dns 192.168.177.2
and the command from above in another it worked again.
> docker run -it --network alpine_net alpine /bin/sh -c "cat /etc/resolv.conf; ping -c 4 www.google.com"
search mydomain.local
nameserver 127.0.0.11
options ndots:0
PING www.google.com (172.217.23.4): 56 data bytes
64 bytes from 172.217.23.4: seq=0 ttl=118 time=8.201 ms
--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 8.201/8.201/8.201 ms
So moving the the dns server to another host and adding "dns" : ["192.168.177.2"] to my /etc/docker/daemon.json fixed it for me
Maybe someone else can help me to explain the root cause behind the problem with running the dns server on the same host as docker.
First, make sure your container is connected to a custom bridged network. I suppose by default in a custom network DNS request inside the container will be sent to 127.0.0.11#53 and forwarded to the DNS server of the host machine.
Second, check iptables -L to see if there are docker-related rules. If there is not, probably that's because iptables are restarted/reset. You'll need to restart docker demon to re-add the rules to make DNS request forwarding working.
I had same problem, the problem was host machine's hostname. I have checked hostnamectl result and it was ok but problem solved with stupid reboot. before reboot result of cat /etc/hosts was like this:
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 localhost HostnameSetupByISP
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
# The following lines are desirable for IPv6 capable hosts
::1 localhost HostnameSetupByISP
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
and after reboot, I've got this result:
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 hostnameIHaveSetuped HostnameSetupByISP
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
# The following lines are desirable for IPv6 capable hosts
::1 hostnameIHaveSetuped HostnameSetupByISP
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
I have a MariaDB docker container running on Synology DS918+ and redirects traffic from container port 3306 to external port 3333
When I see how it binds to the port, it seems different than a working example I have for another service that doesn't run on docker
Working :
ash-4.3# netstat -nao | grep 5000
tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN off (0.00/0/0)
tcp6 0 0 :::5000 :::* LISTEN
Not working:
ash-4.3# netstat -nao | grep 3333
tcp6 0 0 :::3333 :::* LISTEN off (0.00/0/0)
When I try to access port 3333 from my laptop to the remote machine running docker I'm able to do so, the issue is when trying to access the machine's private IP from within the machine itself, this one fails
Any help is appreciated here
To clarify, although your docker is only binding to the ipv6 interface(“:::”) not the ipv4(“0.0.0.0”), Docker forbids a loopback connection to its docker-proxy from the host. I believe this also fails in all networking modes.
If you’re connecting from container to another container, use the container name via the docker-dns and private LAN. For example, if your MariaDB container is named “maria”, I believe docker’s DNS on 127.0.0.11 offers a lookup for the name “maria” to a 172...* ipv4 to which other containers may connect if in the same 172.{subnet}../16 as your MariaDB host. Connect to “maria” in another container and the tcp magically gets to the right place.
If you’re trying to connect from the docker host to a container, this is a problem that I have resigned to proxying off my router in a hairpin NAT to the same upnp ports that I’ve exported via External Access on Synology, which feels like a poor solution but works today.