how DOCKER chain in iptables works? - docker

I'm new to docker.
Recently I experienced somewhat strange scenario.
I created ubuntu instance in OCI(Oracle Cloud Infrastructure).
When I started nginx process in this ubuntu instance, access from external network fails.
When I started nginx with docker container, access from external network successes.
After some search, I found that INPUT chain in iptables does not allow tcp 80.
Additionally, docker will update my iptables to ADD rule in PREROUTING chain.
Here is result of command sudo iptables -nL -t nat.
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
But as you can see in DOCKER chain, I think, RETURN rule is always applied former than DNAT rule.
This means, DOCKER chain does not do anything!
Can anybody give any explanations on this?
One possible assumption is that RETURN rule is applied last regardless of order of rule. (But I'm not sure this is true or not).
Thanks!

RETURN rule actually will be applied at first, but it is not mean to ignore other rules. There also have another two variables IN and OUT which describe the rule's input and output.
You can add -v to your command to show it.
$ iptables -nL -v -t nat
......
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
12 720 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
1691 107K DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8601 to:172.17.0.2:8601
......

Related

How do I prevent Docker from source NAT'ing traffic on Synology NAS

On a Synology NAS, it appears the default setup for docker/iptables is to source NAT traffic going to the container to the gateway IP. This fundamentally causes problems when the container needs to see the correct/real client IP. I do not see this issue when running on an ubuntu system, for example.
Let me walk you though what I'm seeing. On the Synology NAS, I'll run a simple nginx container:
docker run --rm -ti -p 80:80 nginx
(I have disabled the Synology NAS from using port 80)
I'll do a curl from my laptop and I see the following log line:
172.17.0.1 - - [05/May/2020:17:02:44 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
Hm... the client IP (172.17.0.1) is the docker0 interface on the NAS.
user#synology:~$ ifconfig docker0
docker0 Link encap:Ethernet HWaddr 02:42:B5:BE:C5:51
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
<snip>
When I first saw this, I was quite confused because I did not recall docker networking working this way. So I fired up my ubuntu VM and tried the same test (same docker run command from earlier).
The log line from my curl test off box:
172.16.207.1 - - [05/May/2020:17:12:04 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
In this case the docker0 IP was not the source of the traffic as seen by the container.
user#ubuntu-vm:~# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
<snip>
So it appears that the docker network setup for a Synolgy NAS is different vs a more standard ubuntu deployment. Okay, neat. So how does one fix this?
This is where I'm struggling. Clearly something is happening with iptables. Running the same iptables -t nat -L -n command on both systems shows vastly different results.
user#synology:~$ iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DEFAULT_OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
DEFAULT_POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
Chain DEFAULT_OUTPUT (1 references)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain DEFAULT_POSTROUTING (1 references)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
user#ubuntu-vm:~# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
I do not really understand how these chains work or why they're different for different systems. I haven't changed any docker level settings on either system, these are default configs.
Has anyone ran into this before? Is there a quick way to toggle this?
I've experienced this issue myself while using Pi-hole (described here) and this iptables rule seems to fix it:
sudo iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
Be aware that this is not permanent, so if you reboot the NAS you will have to apply it again.
Update: Here's a more "permanent" fix: https://gist.github.com/PedroLamas/db809a2b9112166da4a2dbf8e3a72ae9

Why Docker NAT iptables has RETURN in first row?

Hello StackOverflow users.
Now I'm studying docker nat network especially in iptables.
I executed docker command like below.
Make it to accessible from remote server computer.
docker run -d -p 32170:80 --name nginx_server nginx
When I investigate iptables nat table. I found something weird.
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
RETURN all -- 192.168.123.0/24 base-address.mcast.net/24
RETURN all -- 192.168.123.0/24 255.255.255.255
MASQUERADE tcp -- 192.168.123.0/24 !192.168.123.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.123.0/24 !192.168.123.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.123.0/24 !192.168.123.0/24
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:5000
MASQUERADE tcp -- 172.17.0.3 172.17.0.3 tcp dpt:http
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
DNAT tcp -- anywhere anywhere tcp dpt:5000 to:172.17.0.2:5000
DNAT tcp -- anywhere anywhere tcp dpt:32170 to:172.17.0.3:80
As I searched, packet first checked in PREROUTING.
So its rule is like below and packet should go to DOCKER chain.
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
After it came to DOCKER chain.
My queistion is here... As like below all packets are matched to RETURN and back to previous chain.
If it is right, then when does its source address is changed to docker container and when does its port is changed to docker port?
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
To summarize I have two questions
What is ADDRTYPE match dst-type LOCAL meaning?
When and where does outside packet's source and port will be changed in iptables?
Is there any reference about iptables NAT in docker?
You can try use iptables -t nat -L -v.
The result contins in.
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
3088 185K RETURN all -- docker0 any anywhere anywhere
0 0 RETURN all -- br-5db28a02ac5c any anywhere anywhere
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:postgres to:172.17.0.2:5432
28 1792 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:8084 to:172.17.0.3:8082
14 860 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:us-srv to:172.17.0.3:8081
Only package from docker0 will return

Unable to reach container webserver from host localhost

Using Docker v 17.03.1-ce, on a linux mint machine, i'm unable to reach the container web server (container port 5000) with my browser (localhost port 9000) on the host.
Container launched with command :
sudo docker run -d -p 9000:5000 --name myContainer imageName
I started by checking that the server (flask) on my container was properly launched. It's launched.
I wanted to check that the server was working properly, so in the container, using curl, i sent a GET request on localhost, port 5000. The server returned the web page
So, the server is working, therefore the issue lies somewhere in the communication between container and host.
I checked iptables, but am not sure what to make of it:
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (2 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:5000
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
sudo iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:5000
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9000 to:172.17.0.2:5000
Expected result : using my browser, with url "localhost:9000", i can receive the homepage sent from the container, through port 5000.
edit: Adding docker logs and docker ps
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
59a20248c5b2 apptest "python3 src/jboos..." 12 hours ago Up 12 hours 0.0.0.0:9000->5000/tcp jboost
sudo docker logs jboost
* Serving Flask app "jboost_app" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 310-292-856
127.0.0.1 - - [03/Jul/2019 04:12:54] "GET / HTTP/1.1" 200 -
edit 2: adding results for curl localhost:9000 on host machine
So when connecting with my web browser, the connection doesn't work, but curl gives a more specific message:
curl localhost:9000
curl: (56) Recv failure: Connection reset by peer
I found the solution in this post : https://devops.stackexchange.com/questions/3380/dockerized-flask-connection-reset-by-peer
The Docker networking and port forwarding were working correctly. The problem was with my flask server. It seems that by default, the server is configured to only accept requests from local host.
When launching your flash server, with the "run" command, you must specify host='0.0.0.0' , so that any ip can be served.
if __name__ == "__main__":
app.run(host='0.0.0.0')

Proxying Docker container for outgoing connections

My application need access webservice requires white list IP.
Since I have many app servers, i end up a solution like this
iptables -t nat -A OUTPUT -p tcp --dport 80 -d a.webservices.dimension.abc.com -j DNAT --to 52.1.2.3:9999
that means outgoing connection to a.webservices.dimension.abc.com will be routed to 52.1.2.3:9999 ( a proxy server )
It works.
And now I move my application server ( ec2 vm) to Docker container. But doing this, the iptables rule doesn't work.
here is the final rules
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCALChain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
DNAT tcp -- anywhere 1.2.3.4 tcp dpt:http to:52.1.2.3:9999
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- ip-172-17-0-0.us-west-2.compute.internal/16 anywhere
MASQUERADE tcp -- ip-172-17-0-2.us-west-2.compute.internal ip-172-17-0-2.us-west-2.compute.internal tcp dpt:http
Chain DOCKER (2 references)
target prot opt source destination
LOG all -- anywhere anywhere limit: avg 2/min burst 5 LOG level warning prefix "DOCKER CHAIN "
RETURN all -- anywhere anywhere
DNAT tcp -- anywhere anywhere tcp dpt:9090 to:172.17.0.2:80
As a result, the following command d
docker exec -it some-nginx curl -iL 1.2.3.4
doesn't work
BUT
curl -iL 1.2.3.4
works
I don't know why.
Proxying outgoing connection from Docker container should be common but I haven't found out any existing solution.
Please help.
UPDATE
The following link
https://serverfault.com/questions/734526/whitelisting-outgoing-traffic-from-docker-containers
shed me some lights.
use this rule to make it work
iptables -t nat -A PREROUTING -i docker0 -p tcp a.webservices.dimension.abc.com -j DNAT --to 52.1.2.3:9999

Block port from the outside except for Docker

I installed a jenkins via Docker on my server and assigned to a specific domain (jenkins.mydomain.com), which works perfectly fine. But I can also reach jenkins (and every other service in docker) if I browse my domain with the service's port, for example: mydomain.com:8181
I'v already tried a few thing to block the port from the outside and make it only accessible via domain, but no luck.
First I tried to block the port for the eth0 interface:
iptables -A INPUT -i eth0 -p tcp --destination-port 8181 -j DROP
But it didn't work because when I tried to reach jenkins from the domain, I'v got an error 503.
Also tried to block the port for every incoming requests except docker's ip. It didn't work either.
So how can I make the ports unaccessible from the outside but accessible for Docker?
iptables -L -n --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
1 DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
5 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain DOCKER (1 references)
num target prot opt source destination
1 ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:3000
2 ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:22
3 ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:8081
4 ACCEPT tcp -- 0.0.0.0/0 172.17.0.4 tcp dpt:50000
5 ACCEPT tcp -- 0.0.0.0/0 172.17.0.4 tcp dpt:8080
Chain DOCKER-ISOLATION (1 references)
num target prot opt source destination
1 RETURN all -- 0.0.0.0/0 0.0.0.0/0

Resources