I'm on CentOS 7.9 and I need to get dockers to work with a firewalld configured.
However, the port is always filtered when firewalld is started... Unless I tried to connect to it over localhost.
On this machine (myhost), I have a running container listening on port 55123 :
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6878f1d6152d registry.example.com/my-image:22.1.1 "/entrypoint.sh" 2 weeks ago Up 10 minutes 0.0.0.0:55123->55123/tcp myhost_myservice
# netstat -lntp |grep docker-proxy
tcp 0 0 0.0.0.0:55123 0.0.0.0:* LISTEN 27091/docker-proxy
I did add the port to firewalld :
# firewall-cmd --zone=public --add-port=55123/tcp --permanent
# firewall-cmd --reload
I have a zone docker and a zone public, and I added my interfaces to the public zone to be sure :
# firewall-cmd --get-active-zones
docker
interfaces: br-e7b57dXXXXXX docker0
public
interfaces: ens192 ens224 ens256
I tried a restart of firewalld and then docker service but still getting filtered :
# nmap myhost -Pn -p 55123
Nmap scan report for myhost (XXX.XXX.XXX.23)
Host is up.
PORT STATE SERVICE
55123/tcp filtered unknown
Nmap done: 1 IP address (1 host up) scanned in 0.25 seconds
but all good on localhost :
# nmap localhost -Pn -p 55123
Nmap scan report for myhost (XXX.XXX.XXX.23)
Host is up.
PORT STATE SERVICE
55123/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 0.22 seconds
I have enabled docker metrics on port 9323 directly on the host (myhost), and this port is well opened
# nmap myhost -Pn -p 9323
Nmap scan report for myhost (XXX.XXX.XXX.23)
Host is up.
PORT STATE SERVICE
9323/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 0.22 seconds
If I stop firewalld, I immediatly get :
# nmap myhost -Pn -p 55123
Nmap scan report for myhost (XXX.XXX.XXX.23)
Host is up.
PORT STATE SERVICE
55123/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds
My Firewalld configuration :
# firewall-cmd --zone=docker --list-all
docker (active)
target: ACCEPT
icmp-block-inversion: no
interfaces: br-e7b57dXXXXXX docker0
sources:
services:
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
# firewall-cmd --zone=public --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192 ens224 ens256
sources:
services: dhcpv6-client ssh
ports: 9323/tcp 55123/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
I'm a bit lost of why it isn't working.
There is one thing bit weird, we have a network with non-standard private network. Our network is on something like 110.2.0.0/16.
Related
My setup is:
Debian, Docker
Host machine running Protonmail Bridge as a service
Docker container running Discourse with their default recommended setup
Issue: From the Docker container, I cannot connect to the SMTP server exposed by the Protonmail Bridge on the host machine.
I checked open ports on the host machine, all good:
ss -plnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.1:1025 0.0.0.0:* users:(("proton-bridge",pid=953,fd=12))
How I test
Host machine:
openssl s_client -connect 127.0.01:1025 -starttls smtp
Works.
Docker container:
openssl s_client -connect 172.17.0.1:1025 -starttls smtp
Connection refused.
I’m wondering if the Protonmail Bridge service that’s listening on 127.0.0.1:1025 is not accepting connections from the Docker container because they are not coming from 127.0.0.1 exactly? If this is the problem, how to validate and fix? If this is not the problem, what am I doing wrong?
Other tests
nmap 127.0.0.1 on the host machine outputs:
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000010s latency).
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
1025/tcp open NFS-or-IIS
1042/tcp open afrog
Note that it lists the open port 1025.
nmap 172.17.0.1 in the docker container does not output any 1025 port. I'm not sure if this is the problem either.
Output of route in the Docker container:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
This may be impossible currently, but should be solved by this pull request.
If you're comfortable compiling the proton-bridge package from source, you only have to change 1 line in the internal/bridge/constants.go file to say
Host = '127.0.0.1'
To
Host = '0.0.0.0'
Then recompile with make build-nogui (to build the "headless" version).
And you should be good to go!
I've run into an issue that seems similar too this one; https://forums.docker.com/t/cant-access-service-in-swarm/63876. My setup is a little bit different though and I haven't found a solution to my problem yet.
The minimal, reproducible example
Build a swarm cluster between atleast 3 Ubuntu 20.04 docker swarm managers.
Deploy a service docker service create --name test_web --replicas 3 --publish published=8080,target=80 nginxdemos/hello
Check that the containers and services were created properly and observe the failure of connecting to that service:
demi-ubu01:~/stacks$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d4a12a3c5448 nginxdemos/hello:latest "nginx -g 'daemon of…" About a minute ago Up About a minute 80/tcp test_web.2.yul33wdycarig3qoxnehgrjrz
demi-ubu01:~/stacks$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
0yqd7gvggwuh test_web replicated 3/3 nginxdemos/hello:latest *:8080->80/tcp
# External test:
demi-ubu01:~/stacks$ curl -I 10.100.4.5:8080
curl: (7) Failed to connect to 10.100.4.5 port 8080: Connection refused
# Inside container to published service port:
demi-ubu01:~/stacks$ docker exec -it d4a12a3c5448 wget http://test_web:8080
Connecting to test_web:8080 (10.0.4.2:8080)
wget: can't connect to remote host (10.0.4.2): Host is unreachable
# Inside container to apps exposed port:
demi-ubu01:~/stacks$ docker exec -it d4a12a3c5448 wget http://localhost:80
Connecting to localhost:80 (127.0.0.1:80)
index.html 100% |****************************| 7217 0:00:00 ETA
The expected result of the first curl command should be a Status 200 Ok.
The detailed report
My setup is 4 nodes in total. They are identical Ubuntu 20.04 KVM virtual machines all on the same network. There are no firewalls between them. I have 3 Managers and 1 Worker (which i've only added as a step during troubleshooting).
:~/stacks$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
kcm5v64psntjxngnqkfdj1jzh * demi-ubu01 Ready Active Reachable 20.10.1
uo3rljg6ax5qkjm898pyym9t1 demi-ubu02 Ready Active Leader 20.10.1
pysnl8sohdp4fv67gui156z4k demi-ubu03 Ready Active Reachable 20.10.1
rp2otsqpnxkgbmxbpkv21yjs6 demi-ubu04 Ready Active 20.10.1
I can run a container normally and reach it on the local host fine.
demi-ubu01:~/stacks$ docker run -p 8080:80 -d nginxdemos/hello
de4d0a937710acb1d6d8ae3b7eb9175860b6614dfd9ce92bc972efe619ae095f
demi-ubu01:~/stacks$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
de4d0a937710 nginxdemos/hello "nginx -g 'daemon of…" 4 seconds ago Up 2 seconds 0.0.0.0:8080->80/tcp pedantic_wiles
demi-ubu01:~/stacks$ curl -I 10.100.4.5:8080
HTTP/1.1 200 OK
Server: nginx/1.13.8
Date: Sat, 19 Dec 2020 17:59:23 GMT
Content-Type: text/html
Connection: keep-alive
Expires: Sat, 19 Dec 2020 17:59:22 GMT
Cache-Control: no-cache
However the same app deployed as a service using the following compose file:
demi-ubu01:~/stacks$ cat test.yml
version: "3.6"
services:
web:
image: nginxdemos/hello:latest
deploy:
replicas: 3
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- target: 80
published: 8080
protocol: tcp
mode: ingress
networks:
- webnet
networks:
webnet:
driver: overlay
It does not become reachable from any of the hosts at all:
demi-ubu01:~/stacks$ docker stack deploy -c test.yml test
Creating network test_webnet
Creating service test_web
demi-ubu01:~/stacks$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
05030ef897a1 nginxdemos/hello:latest "nginx -g 'daemon of…" 10 seconds ago Up 7 seconds 80/tcp test_web.1.kobrpkp68f2qbs4jhd6o8aebg
# Trying on all of the hosts in the cluster. No firewalls here.
demi-ubu01:~/stacks$ curl -I 10.100.4.5:8080
curl: (7) Failed to connect to 10.100.4.5 port 8080: Connection refused
demi-ubu01:~/stacks$ curl -I 10.100.4.9:8080
curl: (7) Failed to connect to 10.100.4.9 port 8080: Connection refused
demi-ubu01:~/stacks$ curl -I 10.100.4.10:8080
curl: (7) Failed to connect to 10.100.4.10 port 8080: Connection refused
demi-ubu01:~/stacks$ curl -I 10.100.4.11:8080
curl: (7) Failed to connect to 10.100.4.11 port 8080: Connection refused
demi-ubu01:~/stacks$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
elvfm7o4v4zo test_web replicated 3/3 nginxdemos/hello:latest *:8080->80/tcp
I also don't see any port bindings being made on those hosts at all, so it doesn't look like any ports are being published.
INeed2Poo#demi-ubu01:~/stacks$ docker service inspect test_web
[
## https://pastebin.com/WqqyDnVS ##
]
demi-ubu01:~/stacks$ netstat -na | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:49152 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
demi-ubu01:~/stacks$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6e5f7e7cebc3 bridge bridge local
7a1155f87a62 docker_gwbridge bridge local
ab32da8ac1ec host host local
46id8wzw4ayf ingress overlay swarm
a24a40ef78f4 none null local
d9l7msysdx8m test_webnet overlay swarm
INeed2Poo#demi-ubu01:~/stacks$ docker network inspect 46id8wzw4ayf
[
https://pastebin.com/JPA0ZBjE
]
I also can't reach the service while exec'ed into a container for that service. Execing into a container, I'm able to hit the LOCAL app port, however I cannot hit the service by name. The container CAN resolve the service name.
## Testing the app's service from the local container fails:
demi-ubu01:~/stacks$ docker exec -it 05030ef897a1 wget http://test_web:8080
Connecting to test_web:8080 (10.0.4.2:8080)
wget: can't connect to remote host (10.0.4.2): Host is unreachable
## Testing the app's local port from the local container is sucessful:
demi-ubu01:~/stacks$ docker exec -it 05030ef897a1 wget http://localhost:80
Connecting to localhost:80 (127.0.0.1:80)
index.html 100% |****************************| 7217 0:00:00 ETA
demi-ubu01:~/stacks$ docker --version
Docker version 20.10.1, build 831ebea
I've changed the default-addr-pool for the swarm cluster from the original 10.0.0.0/8 network:
demi-ubu01:~$ docker info --format '{{json .Swarm.Cluster.DefaultAddrPool}}'
["10.135.0.0/16"]
I've gone and made sure that I'm not using any overlapping networks that might be causing this and have gone so far as to completely redeploy the cluster. I've just about exhausted all of my troubleshooting idea's. Any Idea's?
Edit: Update: I redeployed using Ubuntu 18.04 as my base image, and the same exact setup on that (deployed using ansible) seems to work fine... So this is an issue with the current version of Docker on Ubuntu 20.04.
Let me add my response from the docker forum here as well, as it is high likely the solution:
Is it safe to assume that 10.100.4.5 is one of your nodes ip?
The default address pool is 10.0.0.0/8, see: docker info --format '{{json .Swarm.Cluster.DefaultAddrPool}}'
If this is the case, you might find this blog post helpful - you can safely ignore that it refers to Docker EE, the problem and solution is valid for Docker CE as well. You need to alter default-addr-pool either when initiating the swarm or by modifying each node’s /etc/docker/daemon.json configuration file (and restart the daemon then).
I am running a Debian server (stable), with the docker.io Debian package.This is the one distributed by Debian, not the one from the Docker developers. Since docker.io is only available in sid, I have installed from there (apt install -t unstable docker.io).
My firewall does allow connections to/from docker containers:
$ sudo ufw status
(...)
Anywhere ALLOW 172.17.0.0/16
172.17.0.0/16 ALLOW Anywhere
I also have this in /etc/ufw/before.rules :
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 172.17.0.0/16 -o eth0 -j MASQUERADE
So -- I have created an image with
$ sudo debootstrap stable ./stable-chroot http://deb.debian.org/debian > /dev/null
$ sudo tar -C stable-chroot -c . | docker import - debian-stable
Then started a container and installed apache2 and netcat. Port 1111 on the host machine will be redirected to port 80 on the container:
$ docker run -ti -p 1111:80 debian-stable bash
root#dc4996de9fe6:/# apt update
(... usual output from apt update ...)
root#dc4996de9fe6:/# apt install apache2 netcat
(... expected output, installation successful ...)
root#dc4996de9fe6:/# service apache2 start
root#dc4996de9fe6:/# service apache2 status
[ ok ] apache2 is running.
And from the host machine I can connect to the apache server:
$ curl 127.0.0.1:1111
(... HTML from the Debian apache placeholder page ...)
$ telnet 127.0.0.1 1111
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
And it waits for me to type (if I type GET / I get the Debian apache placeholder page). Ok. And if I stop apache inside the container,
root#06da401a5724:/# service apache2 stop
[ ok ] Stopping Apache httpd web server: apache2.
root#06da401a5724:/# service apache2 status
[FAIL] apache2 is not running ... failed!
Then connections to port 1111 on the host will be rejected (as expected):
$ telnet 127.0.0.1 1111
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.
Now, if I start netcat on the container, listening on port 80:
root#06da401a5724:/# nc -l 172.17.0.2 80
Then I cannot connect from the host!
$ telnet localhost 1111
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
The same happens if I try nc -l 127.0.0.1 80 in the container.
What could be happening? Both apache and netcat were listening on port 80. What have I missed?
I'd appreciate any hints...
update: if I try this:
root#12b8fd142e00:/# nc -vv -l -p 80
listening on [any] 80 ...
172.17.0.1: inverse host lookup failed: Unknown host
invalid connection to [172.17.0.2] from (UNKNOWN) [172.17.0.1] 54876
Then it works!
Now it's weird... ifconfig inside the container tells me it has IP 172.17.0.2, but I can only use netcat binding to 172.17.0.1:
root#12b8fd142e00:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.2 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::42:acff:fe11:2 prefixlen 64 scopeid 0x20<link>
And Apache seems to want to 172.17.0.2 instead:
2AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
but it actually uses 172.17.0.1:
root#12b8fd142e00:/# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 12b8fd142e00:http 172.17.0.1:54942 TIME_WAIT
tcp 0 0 12b8fd142e00:39528 151.101.48.204:http TIME_WAIT
Apache is not listening on 172.17.0.1, that's the address of the host (in the docker bridge).
In the netstat output, the local address has been resolved to 12b8fd142e00. Use the -n option with netstat to see unresolved (numeric) addresses (for example netstat -plnet to see listening sockets). 172.17.0.1 is the foreign address that connected to Apache (an it's indeed the host).
The last line in the netstat output shows that some process made a connection to 151.101.48.204:80, probably to make an HTTP request. You can see the PID/name of the process with netstat -p.
I have setup a docker swarm with 3 nodes (docker 18.03). These nodes use an overlay network to communicate.
node1:
laptop
host tun0 172.16.0.6 --> openvpn -> nat gateway
container n1
ip = 192.169.1.10
node2:
aws ec2
host eth2 10.0.30.62
container n2
ip = 192.169.1.9
node3:
aws ec2
host eth2 10.0.140.122
container n3
ip = 192.169.1.12
nat-gateway:
aws ec2
tun0 172.16.0.1 --> openvpn --> laptop
eth0 10.0.30.198
The scheme is partly working:
1. Containers can ping eachother using name (n1,n2,n3)
2. Docker swarm commands are working, services can be deployed
The overlay is partly working. Some nodes cannot communicate with each other either using tcp/ip or udp. I tried all combinations of the 3 nodes with udp and tcp/ip:
I did a tcpdump on the nat gateway to monitor overlay vxlan network activity (port 4789):
tcpdump -l -n -i eth0 "port 4789"
tcpdump -l -n -i tun0 "port 4789"
Then I tried tcp/ip communication from node2 to node3. On node3:
nc -l -s 0.0.0.0 -p 8999
On node1:
telnet 192.169.1.12 8999
Node1 will then try to connect to node3. I see packets coming in on the nat-gateway over the tun0 interface:
on the nat-gateway eth0 interface:
it seems that the nat-gateway is not sending replies back over the tun0 interface.
The iptables configuration the nat-gateway
The routing of the nat-gateway
Can you help me solve this issue?
I have been able to fix the issue using the following configuration on the NAT gateway:
and
No masquerading of 172.16.0.0/22 is needed. All the workers and managers will route their traffic for 172.16.0.0/22 via the NAT gateway, and it knows how to send the packets over tun0.
Masquerading of eth0 was just wrong...
All the containers can now ping and establish tcp/ip connections to each other.
I have installed a fresh Centos 7 box on virtualbox running in a bridged network adapter. I have installed ruby on rails and setup a simple app. I started the server on port 3000, but when I try to reach it from my host machine hitting the IP I get no response.
On the server I can do a
wget "http://127.0.0.1:3000"
and I get the right index.html file. So I figured my port was getting blocked.
So I installed firewalld and issued the following commands
sudo firewall-cmd --zone=public --add-port=3000/tcp --permanent
sudo firewall-cmd --reload
firewall-cmd --list-all
The list all shows the following ;
public (default, active)
interfaces: enp0s3
sources:
services: dhcpv6-client ssh
ports: 3000/tcp 80/tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
However when I run nmap I see it's closed
sudo nmap -p 3000 127.0.0.1
Starting Nmap 6.40 ( http://nmap.org ) at 2016-11-26 01:47 EST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000071s latency).
PORT STATE SERVICE
3000/tcp closed ppp
Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds
From my host machine I can ping the machine but when I nmap to port 3000 it says the host is unreachable.
I don't know how to go any further. Any thoughts?