I am running a Debian server (stable), with the docker.io Debian package.This is the one distributed by Debian, not the one from the Docker developers. Since docker.io is only available in sid, I have installed from there (apt install -t unstable docker.io).
My firewall does allow connections to/from docker containers:
$ sudo ufw status
(...)
Anywhere ALLOW 172.17.0.0/16
172.17.0.0/16 ALLOW Anywhere
I also have this in /etc/ufw/before.rules :
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 172.17.0.0/16 -o eth0 -j MASQUERADE
So -- I have created an image with
$ sudo debootstrap stable ./stable-chroot http://deb.debian.org/debian > /dev/null
$ sudo tar -C stable-chroot -c . | docker import - debian-stable
Then started a container and installed apache2 and netcat. Port 1111 on the host machine will be redirected to port 80 on the container:
$ docker run -ti -p 1111:80 debian-stable bash
root#dc4996de9fe6:/# apt update
(... usual output from apt update ...)
root#dc4996de9fe6:/# apt install apache2 netcat
(... expected output, installation successful ...)
root#dc4996de9fe6:/# service apache2 start
root#dc4996de9fe6:/# service apache2 status
[ ok ] apache2 is running.
And from the host machine I can connect to the apache server:
$ curl 127.0.0.1:1111
(... HTML from the Debian apache placeholder page ...)
$ telnet 127.0.0.1 1111
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
And it waits for me to type (if I type GET / I get the Debian apache placeholder page). Ok. And if I stop apache inside the container,
root#06da401a5724:/# service apache2 stop
[ ok ] Stopping Apache httpd web server: apache2.
root#06da401a5724:/# service apache2 status
[FAIL] apache2 is not running ... failed!
Then connections to port 1111 on the host will be rejected (as expected):
$ telnet 127.0.0.1 1111
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.
Now, if I start netcat on the container, listening on port 80:
root#06da401a5724:/# nc -l 172.17.0.2 80
Then I cannot connect from the host!
$ telnet localhost 1111
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
The same happens if I try nc -l 127.0.0.1 80 in the container.
What could be happening? Both apache and netcat were listening on port 80. What have I missed?
I'd appreciate any hints...
update: if I try this:
root#12b8fd142e00:/# nc -vv -l -p 80
listening on [any] 80 ...
172.17.0.1: inverse host lookup failed: Unknown host
invalid connection to [172.17.0.2] from (UNKNOWN) [172.17.0.1] 54876
Then it works!
Now it's weird... ifconfig inside the container tells me it has IP 172.17.0.2, but I can only use netcat binding to 172.17.0.1:
root#12b8fd142e00:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.2 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::42:acff:fe11:2 prefixlen 64 scopeid 0x20<link>
And Apache seems to want to 172.17.0.2 instead:
2AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
but it actually uses 172.17.0.1:
root#12b8fd142e00:/# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 12b8fd142e00:http 172.17.0.1:54942 TIME_WAIT
tcp 0 0 12b8fd142e00:39528 151.101.48.204:http TIME_WAIT
Apache is not listening on 172.17.0.1, that's the address of the host (in the docker bridge).
In the netstat output, the local address has been resolved to 12b8fd142e00. Use the -n option with netstat to see unresolved (numeric) addresses (for example netstat -plnet to see listening sockets). 172.17.0.1 is the foreign address that connected to Apache (an it's indeed the host).
The last line in the netstat output shows that some process made a connection to 151.101.48.204:80, probably to make an HTTP request. You can see the PID/name of the process with netstat -p.
Related
Trying investigate my issue with docker container. I lost a day when I thought that issue is in nodejs code (it has server and I am trying to connect to this server).
After investigations I found interesting thing for me.
For example - Lets run some test docker image:
docker run -p 888:888 -it ubuntu:16 /bin/bash
After that, prepare and install "simple server to listen our port":
apt-get update
apt-get install -y netcat
nc -l 888
After that I going to try to telnet localhost 888 from my system and got telnet: connect to address 127.0.0.1: Connection refused. The same with nodejs image.
But if you try to use, for example, nginx container -
docker run -p 888:888 -it nginx /bin/bash
I will be successfull:
$telnet 127.0.0.1 888
Trying 127.0.0.1...
Connected to localhost.
How it is possible, what I am missing? Why I can bind and use any port in nginx but not for other images?
When you run nc -l 888, you are creating a port that is listening explicitly for IPv4 connections. If we run ss -tln, we will see:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1 0.0.0.0:888 0.0.0.0:*
When you run telnet localhost 888 on your host, there's a good chance it's trying to connect to the IPv6 localhost address, ::1. This connection fails if you're trying to connect an IPv4-only socket.
If you explicitly use the IPv4 loopback address by typing telnet 127.0.0.1 888, it should work as expected.
If you enable IPv6 support in nc by adding the -6 parameter:
nc -6 -l 8888
Then you get a socket that listen for both IPv4 and IPv6 connections:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1 *:888 *:*
And if you attempt to connect to this socket using telnet localhost 888, it will work as expected (as will telnet 127.0.0.1 888).
Most programs (like nginx) open multi-protocol sockets by default, so this isn't normally an issue.
My setup is:
Debian, Docker
Host machine running Protonmail Bridge as a service
Docker container running Discourse with their default recommended setup
Issue: From the Docker container, I cannot connect to the SMTP server exposed by the Protonmail Bridge on the host machine.
I checked open ports on the host machine, all good:
ss -plnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.1:1025 0.0.0.0:* users:(("proton-bridge",pid=953,fd=12))
How I test
Host machine:
openssl s_client -connect 127.0.01:1025 -starttls smtp
Works.
Docker container:
openssl s_client -connect 172.17.0.1:1025 -starttls smtp
Connection refused.
I’m wondering if the Protonmail Bridge service that’s listening on 127.0.0.1:1025 is not accepting connections from the Docker container because they are not coming from 127.0.0.1 exactly? If this is the problem, how to validate and fix? If this is not the problem, what am I doing wrong?
Other tests
nmap 127.0.0.1 on the host machine outputs:
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000010s latency).
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
1025/tcp open NFS-or-IIS
1042/tcp open afrog
Note that it lists the open port 1025.
nmap 172.17.0.1 in the docker container does not output any 1025 port. I'm not sure if this is the problem either.
Output of route in the Docker container:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
This may be impossible currently, but should be solved by this pull request.
If you're comfortable compiling the proton-bridge package from source, you only have to change 1 line in the internal/bridge/constants.go file to say
Host = '127.0.0.1'
To
Host = '0.0.0.0'
Then recompile with make build-nogui (to build the "headless" version).
And you should be good to go!
Docker provides a way to map ports between the container and host.
As per the official documentation its also possible to mention host-ip while port mapping.
-p 192.168.1.100:8080:80 - Map TCP port 80 in the container to port 8080 on the Docker host for connections to host IP 192.168.1.100.
I tried this option to figure out what's the difference with/without the host-ip.
Using just -p 80:80
$ docker run -itd -p 80:80 nginx:alpine
$ curl localhost:80
$ curl 127.0.0.1:80
$ curl 0.0.0.0:80
$ curl 192.168.0.13:80
$ ps -ef | grep docker-proxy
16723 root 0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.1 -container-port 80
$
All the curl commands return the output.
Using host-ip like -p 192.168.0.13:80:80
$ docker run -itd -p 192.168.0.13:80:80 nginx:alpine
$ curl localhost:80
curl: (7) Failed to connect to localhost port 80: Connection refused
$ curl 127.0.0.1:80
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused
$ curl 0.0.0.0:80
curl: (7) Failed to connect to 0.0.0.0 port 80: Connection refused
$ curl 192.168.0.13:80 # return output
$ ps -ef | grep docker-proxy
4914 root 0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 192.168.0.13 -host-port 80 -container-ip 172.17.0.2 -container-port 80
$
All the curl commands failed except 192.168.0.13:80.
Is there any there any other difference apart for the one I mentioned here.
Wondering when to use host-ip based port mapping. Any use cases?
A docker host may have multiple NICs. In the data center, this may be too segregate traffic, e.g. management, storage, and application/public. On your laptop, this may be for wireless and wired interfaces. There are also virtual NICs for things like loopback (127.0.0.1) and VPN tunnels.
When you do not specify an IP in the port publish command, by default docker will bind to all interfaces on the host. In IPv4, this is commonly notated as 0.0.0.0 which means listen on any interface (and this is why I don't connect to this address because there's no such thing as connecting to any IP). With the IP address specified, you manually specify which interface to use. Why would you want to specify this? Several reasons I can think of:
Listening on only 127.0.0.1 to prevent external access
Listening on 0.0.0.0 to explicitly bind to all IPv4 interfaces (it is possible to change docker's default behavior, so this could be necessary for some).
Listening on one physical NIC, allowing other NICs to be bound by other services on the same port.
Listening on only IPv4 interfaces if the app does not work for IPv6.
While there are lots of possible reasons, other than listening on loopback for security, these use cases are very rare and most users leave docker to listen on all interfaces.
I ran docker daemon for using it with global IPv6 for containers:
docker daemon --ipv6 --fixed-cidr-v6="xxxx:xxxx:xxxx:xxxx::/64"
After it I ran docker container:
docker run -d --name my-container some-image
It successfully got Global IPv6 address( I checked by docker inspect my-container). But I can't to ping my container by this ip:
Destination unreachable: Address unreachable
But I can successfully ping docker0 bridge by it's IPv6 address.
Output of route -n -6 contains next lines:
Destination Next Hop Flag Met Ref Use If
xxxx:xxxx:xxxx:xxxx::/64 :: U 256 0 0 docker0
xxxx:xxxx:xxxx:xxxx::/64 :: U 1024 0 0 docker0
fe80::/64 :: U 256 0 0 docker0
docker0 interface has global IPv6 address:
inet6 addr: xxxx:xxxx:xxxx:xxxx::1/64 Scope:Global
xxxx:xxxx:xxxx:xxxx:: everywhere is the same, and it's global IPv6 address of my eth0 interface
Does docker required something additional configs for accessing my containers via IPv6?
Assuming IPv6 in your guest OS is properly configured probably you are pinging the container not from host OS, but outside and network discovery protocol is not configured. Other hosts does not know if your container is behind of your host. I'm doing this after start of container with IPv6 (in host OS) (in ExecStartPost clauses of Systemd .service file)
/usr/sbin/sysctl net.ipv6.conf.interface_name.proxy_ndp=1
/usr/bin/ip -6 neigh add proxy $(docker inspect --format {{.NetworkSettings.GlobalIPv6Address}} container_name) dev interface_name"
Beware of IPv6: docker developers say in replies to bug reports they do not have enough time to make IPv6 production-ready in version 1.10 and say nothing about 1.11.
Mb you use wrong ping command. For ipv6 is ping6.
$ ping6 2607:f0d0:1002:51::4
On a mac system, I start the register following the deploy document:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
And its ip is "IPAddress": "172.17.0.2", which is from the command docker inspect <cid>
But I can't telnet its port:
$ telnet 172.17.0.2 5000
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Can't assign requested address
telnet: Unable to connect to remote host
Or
$ curl http://172.17.0.2:5000
curl: (7) Couldn't connect to server
I ssh to the container to check it:
$ docker exec -it cda bash
root#cda4c64efd06:/# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 0 :::5000 :::* LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
root#cda4c64efd06:/# curl -I http://localhost:5000
HTTP/1.1 200 OK
Cache-Control: no-cache
Date: Fri, 11 Dec 2015 07:34:17 GMT
Content-Type: text/plain; charset=utf-8
root#cda4c64efd06:/#
Seems like it's binding tcp6 not normally tcp.
Where is wrong?
If you are using docker-machine, see "Port forwarding in docker-machine?"
either port forward the 5000 port on the VirtualBox level (meaning localhost:5000 will work)
VBoxManage controlvm "boot2docker-vm" natpf1 "tcp-port5000,tcp,,5000,,5000";
or use the ip returned by $(docker-machine ip <yourMachine>)