I have 4 virtual machines in the same subnet, which are part of a docker-swarm.
Now I want connect another node (virtual machine), which is located in a different country (not the same subnet).
I am an IP noob and it is hard for me to set up an overlay network in docker, which is able to handle this connection.
Which aspects I need to keep in mind, by setting up this kind of docker-swarm?
You need the following ports open between your swarm nodes:
2377/tcp: Swarm mode api
7946/both: Overlay networking control
4789/udp: Overlay networking data
protocol 50 for ipsec (secure option) of overlay networking
The following iptables commands can be used for this (you may want to limit the source host to only your other docker swarm nodes):
iptables -A INPUT -p tcp -m tcp --dport 2377 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 7946 -j ACCEPT
iptables -A INPUT -p tcp -m udp --dport 7946 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 4789 -j ACCEPT
iptables -A INPUT -p 50 -j ACCEPT
This needs to be configured on all of your swarm nodes if they have a restrictive host firewall, and on the network firewalls protecting your subnets.
Related
I have an issue in regards to my iptables setup. My goal is to reach the https based webserver inside a docker container from the server machine itself.
The setup is the following:
The server is connected to the internet via eth0 and serves http via port 443.
Any users from the outside (internet) reach the server via the ip address 1.2.3.4.
It is connected to the internal network via eth1 and serves dhcp, dns and some more services.
Any users from the inside (intranet) reach the server via the ip address 10.0.0.1.
The docker container is connected via docker1 on the server. The later has the ip address 10.8.0.2 inside the docker network.
The docker container serves the webserver on port 1443, but iptables forwards (NAT) requests on port 443 to its address 10.8.0.1 and the destination port 1443.
What is working:
The webserver is perfectly reachable from the internet and the intranet.
The webserver can be reached from the server itself using the address 10.8.0.1:1443.
What is not working:
Any client which is working directly on the server can not reach the docker webserver using https://example.com:443. Using https://10.8.0.1:8443 would work, but fails due to a certificate error. It is not a goal to skip the certificate check as a workaround.
Excerpt of the iptable configuration:
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -P FORWARD DROP
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i docker1 -o docker1 -j ACCEPT
iptables -A PREROUTING -t nat -p tcp -d 1.2.3.4 --dport 443 -j DNAT --to-destination 10.1.0.1:1443
iptables -A FORWARD -o docker1 -p tcp --dport 1443 -j ACCEPT
iptables -A INPUT -i docker1 -j DROP
iptables -A FORWARD -i docker1 -j DROP
Due to that "complicated" setup I am no longer able to understand which of the iptable rules and chains need to be applied to make this work so I am seeking for your help to solve that issue.
Brainstorming about the issue using a simplified model and my understanding of the iptable chains the way of the packages might/should look like this:
Origin is a local application (wget).
The packages go through the OUTPUT table.
The packages go through the POSTROUTING table.
Magic happens...
The packages arrive again in the PREROUTING table.
The packages might go trough INPUT again.
The packages might arrive at the target application (webserver).
I tried to create services in docker swarm model by following this document
I created two nodes in the swarm:
Then create the deploy the service, I use jwilder/whoami here instead of nginx in the document,
docker service create --name my-web --publish published=8888,target=8000 --replicas 2 jwilder/whoami
Seems like they started successfully:
As the document said:
When you access port 8080 on any node, Docker routes your request to
an active container.
SO in my opinion, I can access the my-web service from any of the node, however I found that only one node work:
What's going on?
This can be caused by ports being blocked between the nodes. The swarm mesh networking uses the "ingress" network to connect the published port to a VIP for the service. That ingress network is an overlay network implemented with vxlan. For that you need:
TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
UDP port 4789 for overlay network traffic
Reference: https://docs.docker.com/network/overlay/
It's possible for these ports to be blocked at many levels, including iptables, firewalls on the routers, and I've even seen VMware block this with their NSX tool that also implemented vxlan.
For iptables, I typically use the following commands:
iptables -A INPUT -p tcp -m tcp --dport 2376 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 2377 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 7946 -j ACCEPT
iptables -A INPUT -p udp -m udp --dport 7946 -j ACCEPT
iptables -A INPUT -p tcp -m udp --dport 4789 -j ACCEPT
iptables -A INPUT -p 50 -j ACCEPT
The above will differ if you use firewalld or need to change firewall rules on the network routers.
Just realized a few days ago that Docker seems to bypass my iptable rules. I am not incredible experienced with Docker nor iptables. Tried a lot of different things the last days. Also saw that there was big change in recent docker versions with a special DOCKER-chain that should allow me to do that. However not sure what I am doing wrong but it never does what I expect it to do.
So what I want is quite simple. I want that it behaves like expected. That if I have an ACCEPT-Rule to go through and if not it gets blocked.
My iptable looked originally like that (so before my many unsuccessful attempts):
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [779:162776]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -s 1.2.3.4 -p tcp -m tcp --dport 123 -j ACCEPT
-A INPUT -j DROP
COMMIT
Hoped that it does exactly what I want. Just allow access to ports 22 and 80 and also allow port 123 from the ip 1.2.3.4. However If I create a container with "-p 123:123" everybody can access it. Can anybody help me and tell me how I have to change the above file?
Thanks!
Docker-Version: 1.6.2
Edit:
Left initially my different tries out to not overcomplicate the question. However adding at least one of them could maybe be helpful.
*nat
:PREROUTING ACCEPT [319:17164]
:INPUT ACCEPT [8:436]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [16:960]
:DOCKER - [0:0]
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [779:162776]
:DOCKER - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -s 1.2.3.4 -p tcp -m tcp --dport 123 -j ACCEPT
-A DOCKER -j DROP
-A INPUT -j DROP
COMMIT
The above kind of works. However get then a lot of other problems. For example do I get problems with container linking, DNS does not work anymore, and so on. So then end up adding a lot of additional rules to fix that issues but I get never to a state where it runs properly. So I guess there most be better and easier solution out there.
Solution:
Ended up doing more or less exactly what larsks said. Just did not add it to the FORWARD chain, I added it to the DOCKER chain instead. The problem with the FORWARD chain is that Docker adds its stuff in there when it restarts in first position. Which results in having my rules getting pushed down and not having any effect. However for the DOCKER chain it seems Docker appends only additional rules so mine stay in effect. So when I save my rules and then restart the server everything still works fine.
So now it looks more or less like that:
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [779:162776]
:DOCKER - [0:0]
# That I can access from IP 1.2.3.4
-A DOCKER -s 1.2.3.4/32 -p tcp -m tcp --dport 123 -j ACCEPT
# That I can access from other Docker containers
-A DOCKER -o docker0 -p tcp -m tcp --dport 123 -j ACCEPT
# Does not allow it for anything else
-A DOCKER -p tcp --dport 123 -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -j DROP
COMMIT
I'm not an expert on iptables but I know that if you run the container with -p 127.0.0.1:123:123 then the port won't be exposed on all interfaces, just on the loopback.
Your iptables configuration looks a little broken right now, as if you cleared it out at some point without restarting Docker. For example, you have a DOCKER chain available in both the filter and nat tables, but no rules that reference it, so rules placed in that chain will have no affect.
In general, if you want to implement iptables rules that affect your Docker containers they need to go in the FORWARD chain of the filter table. Each container has it's own ip address, which means that your host is simply accepting packets and then FORWARDing them to the container address.
Rules in the INPUT chain are only for packets with a final destination of an address on an interface in the host's global network namespace.
However, I'm not sure that iptables is actually your problem.
If you are trying to expose services in containers such that they are available to other systems, you need to publish those ports using the -p flag to docker run. You can read more about that
in this section of the documentation.
If you want to update your question with a specific example of what you are trying to accomplish I can provide a more targeted answer.
Update
It's true that when you publish a container port using -p it will generally be available to any source ip address. In order to restrict access to a published port you would need to add a new rule to your FORWARD chain. For example, if I start a web server:
docker run --name web -p 80:8080 larsks/mini-httpd
The web server in the container is now available on port 8080 on my host. If I want to block access to this port, I need to insert a rule into the FORWARD chain that blocks access to port 80 on the container ip. So first I need the container ip address:
$ web_ip=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' web)
$ echo $web_ip
172.17.0.5
The rule I create in the FORWARD chain needs to come before the rules that docker creates, so I will need to specify an explicit position:
iptables -I FORWARD 1 -d $web_ip -p tcp --dport 80 \! -s 192.168.1.10 -j DROP
This would block all traffic from hosts other than 192.168.1.10.
If you want a rule to apply to all containers, rather than a specific container, you can bind it to the docker0 interface rather than a specific ip address:
-A FORWARD -o docker0 -p tcp --dport 80 \! -s 192.168.1.10 -j DROP
This would prohibit access to port 80 on any container.
Ended up doing more or less exactly what larsks said. Just did not add
it to the FORWARD chain, I added it to the DOCKER chain instead.
I've found the same in the docs: https://docs.docker.com/v1.5/articles/networking/#the-world
Docker will not delete or modify any pre-existing rules from the
DOCKER filter chain. This allows the user to create in advance any
rules required to further restrict access to the containers.
Docker's forward rules permit all external source IPs by default. To
allow only a specific IP or network to access the containers, insert a
negated rule at the top of the DOCKER filter chain. For example, to
restrict external access such that only source IP 8.8.8.8 can access
the containers, the following rule could be added:
$ iptables -I DOCKER -i ext_if ! -s 8.8.8.8 -j DROP
To use iptables on published ports from docker containers, you need a combination of things:
DOCKER-USER table: docker uses this table for iptables rules that affect containers and is reserved specifically for user provided rules that won't be overwritten by the docker engine when it restarts.
conntrack: port forwarding can publish on one port and forward to another in the container. You can have multiple containers all listening on port 80 with different published ports on the host.
To use these, the resulting iptables rule looks like:
iptables -I DOCKER-USER -i eth0 -s 10.0.0.0/24 -p tcp \
-m conntrack --ctorigdstport 8080 -j ACCEPT
iptables -I DOCKER-USER -i eth0 ! -s 10.0.0.0/24 -p tcp \
-m conntrack --ctorigdstport 8080 -j DROP
This handles requests to the published port 8080/tcp (that's on the host, the container could be listening on 80 or any other port), and only accepts the requests from the 10.0.0.0/24 subnet. Everything outside of that subnet is dropped.
Note that the DOCKER-USER table has a default rule to immediately return, so all changes should be inserted before that default rule in the table.
Given: Debian stretch, docker 18.06, and a docker process created via
docker run ... -p 5678:1234 ...
Required: Access to the docker container restricted to multiple external subnetworks.
Solution: (using some example subnetworks)
iptables -I DOCKER-USER -p tcp --dport 1234 -j REJECT
iptables -I DOCKER-USER -s 18.204.0.0/16 -p tcp --dport 1234 -j RETURN
iptables -I DOCKER-USER -s 34.192.0.0/16 -p tcp --dport 1234 -j RETURN
iptables -I DOCKER-USER -s 35.153.0.0/16 -p tcp --dport 1234 -j RETURN
iptables -I DOCKER-USER -s 13.56.63.0/24 -p tcp --dport 1234 -j RETURN
Persist the changed rules using iptables-save:
iptables-save > /etc/iptables/rules.v4
Result:
iptables -L DOCKER-USER -n -v --line-numbers
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp -- * * 13.56.63.0/24 0.0.0.0/0 tcp dpt:1234
0 0 RETURN tcp -- * * 35.153.0.0/16 0.0.0.0/0 tcp dpt:1234
0 0 RETURN tcp -- * * 34.192.0.0/16 0.0.0.0/0 tcp dpt:1234
0 0 RETURN tcp -- * * 18.204.0.0/16 0.0.0.0/0 tcp dpt:1234
0 0 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1234 reject-with icmp-port-unreachable
Background 1: Rules
All rules are added to the chain DOCKER-USER as recommended by the current docker documentation.
The first rule targets REJECT and will end up as the last rule, since the other rules are added on top (option -I without a position number corresponds to adding a rule at position 1). All packages with destination port 1234 reaching this rule will be rejected.
The other rules target RETURN, i.e. a package with destination port 1234 and source IP from one of the given subnetworks will be returned to the calling chain, which is the FORWARD chain.
iptables -L FORWARD -n -v
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
16471 4568K DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
16413 4565K DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
7173 2060K ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
45 2340 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
From the FORWARD chain, it will be processed by the DOCKER chain, where it is forwarded as desired to the docker container:
iptables -L DOCKER -n -v
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
45 2340 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:1234
Background 2: Port value
We don't use the external port value 5678 in the forward rule, because the destination port is changed via a rule automatically created by docker and applied before the forward chain is executed. See top section of /etc/iptables/rules.v4:
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
...
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5678 -j DNAT --to-destination 172.17.0.2:1234
Could someone please tell me, what can I do to give my application a simple URL. Right now I call my app with this URL -
http://localhostname:8080/MyProject
I would like to call it with this URL -
http://localhostname/MyProject
I'am using JBoss 7.1.0 Final version
As said above in Alexander Pavlov's comments the easiest way to go about this is using port 80.
Applications servers normally have a config file (usually xml) in which you specify the port for your application to use. The default for most application servers is port 8080 so your url will look like this: http://<server IP or name>:8080.
If you modify your applications server configuration to make the default port 80 then you will only have to do this: http://<server IP or Name>
I was in the process of researching this very topic (for a single instance though) and came across a recommendation from the RedHat Discussions.
This is Linux specific for a single instance. OP didn't specify environment. But this should point OP down the right path if using Linux.
Using port 80 & 443 requires root to run the JBoss instance. Chances are, the SA isn't going to grant this to the user, so an alternative is to have the SA modify the iptables. Credit goes to PixelDrift.NET Support over in the RedHat Discussions for the great lead.
iptables -I INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
iptables -I INPUT -i eth0 -p tcp --dport 8443 -j ACCEPT
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8443
My SA modified the iptables to our needs.
iptables -I INPUT -p tcp --dport 8380 -j ACCEPT
iptables -I INPUT -p tcp --dport 8443 -j ACCEPT
iptables -I INPUT -p tcp --dport 9990 -j ACCEPT
iptables -I INPUT -p tcp --dport 9443 -j ACCEPT
iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8380
iptables -A PREROUTING -t nat -p tcp --dport 443 -j REDIRECT --to-port 8443
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
After the changes were applied, I was successfully able to access the application using http://bar.foo/baz without the necessity of including the port number.
I have two rails apps on separate virtual servers, but in the same facility. Both apps can communicate via local ip addresses.
This is a two part question:
1) How do I check where the request is originating and limit requests only to those from that location?
2) Do you think this would be secure enough?
My gut is telling me this isn't secure enough because of IP spoofing, but I'm thinking OAuth or similar is a little too hardcore for my needs. Though, maybe not.
This is the first time I've approached something like this and I'm looking for anyone that can push me in the right direction here.
Thanks.
Depending on who's hosting you, the local network (to which your local addresses belong) could be a private network only accessible to your instances or, more likely, it would be shared with other virtual machines that do not belong to you. You would not be open to direct external attacks, but any compromised virtual machine sharing the same local network as you can be a springboard for attack, so your concerns are absolutely valid.
Answering, in order, your two concerns:
Configure iptables for the local interfaces to only accept requests coming on specific ports from specific local IPs (read a tutorial for a better understanding of iptables configuration.) All other virtual machines on the local network should not be able to probe you, although they might be able to intercept your traffic (addressed below.)
No; you should use SSL over all intra-node connections. This will protect you in two ways: firstly it will protect you from spoofing (an attacker will be rejected if he does not have your certificate, even if he bypasses iptables by spoofing his address, or because your iptables config gets overwritten by an admin), and secondly it will protect your data from prying eyes (e.g. an attacker will not be able to snoop traffic for your passwords.) Some applications (e.g. most database engines, net-snmpd set up in v3 mode, etc.) support SSL natively. Alternatively, establish and use ssh tunnels, or use stunnel
Sample base iptables configuration allowing basic services (HTTP, SSH etc.) on the public (internet) interface, as well as allowing www1 and www2 to connect to this node's MySQL on port 3306 on the eth0 interface (www1 and www2 are defined in /etc/hosts so they resolve to the appropriate IP addresses.):
# * raw
#
# Allows internal traffic without loading conntrack
# -A PREROUTING -i lo -d 127.0.0.0/8 -j NOTRACK
*filter
# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -i ! lo -d 127.0.0.0/8 -j DROP
# Accepts all established inbound connections (TCP, UDP, ICMP incl. "network unreachable" etc.)
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allows all outbound traffic
# You can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT
# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
# Allows SSH connections
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
# Allow ping
-A INPUT -p icmp -m icmp --icmp-type echo-request -j ACCEPT
# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level debug
# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT
# Allows MySQL within our cluster ONLY
-A INPUT -p tcp -s www1 -i eth0 --dport 3306 -j ACCEPT
-A INPUT -p udp -s www1 -i eth0 --dport 3306 -j ACCEPT
-A INPUT -p tcp -s www2 -i eth0 --dport 3306 -j ACCEPT
-A INPUT -p udp -s www2 -i eth0 --dport 3306 -j ACCEPT
COMMIT
This doesn't really sound like a Rails question, it's more of a question about web architecture. I'm assuming that both machines are accessible to the outside world via HTTP. If that's the case, you may want to consider putting a firewall in front of both machines to create a local network that the two machines are on.
Once you've done that, you should be able to configure the firewall to disallow requests based on any criteria you specify. Given that this is a Rails application I'm going to assume that the API is a set of resources. If that's the case you could configure your firewall to filter requests to the private API.
This way, the machines on the local network can communicate freely as their requests to one another aren't going through the firewall.