Multi host Container communication with kubernetes and flannel - docker

I am following the following guide
http://kubernetes.io/v1.0/docs/getting-started-guides/fedora/flannel_multi_node_cluster.html
I get all the way to the very last step when it says to ping a container on another host from a separate container on another host. I am supposed to see something like this (like it shows in the documentation)
bash-4.3# ping 18.16.90.4
PING 18.16.90.4 (18.16.90.4) 56(84) bytes of data.
64 bytes from 18.16.90.4: icmp_seq=1 ttl=62 time=0.275 ms
64 bytes from 18.16.90.4: icmp_seq=2 ttl=62 time=0.372 ms
I get no response on the ping, I don't know how to troubleshoot this and where I should start.

The first thing to try is to ping the container IP from outside the container, but on a different machine. Then try typical network debugging tools like 'traceroute', make sure traffic is getting into the flannel bridge.
Also, use 'ifconfig' to make sure that the settings for the docker0 bridge match the settings that flannel created in its config file.

Related

Docker docs says "traffic from containers connected to the default bridge network is not forwarded to the outside world". What does this mean?

If we create a container in default bridge network, we are able to access internet from within this container. (Below example copied from Networking with Standalone containers)
docker run -dit --name alpine1 alpine ash
docker attach alpine1
# ping -c 2 google.com
PING google.com (172.217.3.174): 56 data bytes
64 bytes from 172.217.3.174: seq=0 ttl=41 time=9.841 ms
64 bytes from 172.217.3.174: seq=1 ttl=41 time=9.897 ms
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 9.841/9.869/9.897 ms
pinging google.com from within the container, we see successful ping response.
This behaviour seems to contradict the statement from Docker docs. Or I am not understanding something here ?
Link to Docker docs
Based on #Turing85's comment , I was checking if IPForwarding is enabled on my windows. Used this link How to enable IP forwarding in windows. Looks like its not enabled.

Docker container is not able to resolve internal or external dns or ips

I am running build inside a docker container hosted on Jenkins master machine(jenkins.internal.dns).
When i am making any http call from within the container it is getting routed to Jenkins master machine as in it returns result from http://jenkin.internal.dns.
I tried another internal server by running "curl http://another.internal.dns" and also external "curl http://google.com" but it returned result from "http://jenkin.internal.dns"
But when i pinged google.com looks like i am able to reach it from within container .
ping google.com
PING google.com (172.253.115.101) 56(84) bytes of data.
64 bytes from bg-in-f101.1e100.net (172.253.115.101): icmp_seq=1 ttl=98 time=10.4 ms
64 bytes from bg-in-f101.1e100.net (172.253.115.101): icmp_seq=2 ttl=98 time=10.5 ms
Can anyone suggest what is wrong here ?

Cannot ping from one Docker container to another

We've been experiencing a long-standing networking issue. In short, one container cannot ping (or ssh) another. Does anybody have an extra moment to think along with me?
Our setup:
Docker CE 18.06.03 (while trying to fix the issue, we've upgraded from 17.03, but it has not helped)
Swarm Classic (Standalone) 1.2.9
Consul as a Swarm backend, running with members on five nodes
Seven nodes in total, six of which host containers
Each container is connected to an overlay network when it is started
What we've tried so far:
This issue has largely stumped us. We've spent a lot of time on it and done much of the basic troubleshooting, and some more advanced troubleshooting (happy to elaborate). (But I don't expect that I've exhausted our options, so please don't hesitate to suggest anything you may think will work.)
It's inconsistent (happening to different images, different nodes), intermittent, and long-standing (several months). We've made two changes, one of which was a workaround for MAC address assignment (explained here: https://github.com/docker/libnetwork/pull/2380; the actual workaround: https://github.com/systemd/systemd/issues/3374#issuecomment-452718898), which did improve the situation, including removing MAC address assignment errors from the logs. We also upgraded to get this fix (https://github.com/docker/libnetwork/pull/1935), which deals with IP reuse. This also decreased the problem (at the time, no containers could communicate). I've also run through some basics tests using the netshoot container (let me know if you want more info on that).
We have a workaround for a given container that is broken: we delete the Consul data for this container and then stop and restart it. From what I can tell, it does not seem to be an issue with the Consul data per se but instead comes from Docker/Swarm resetting several network configurations when the container is started (I can say more if this seems to trigger a thought for anybody reading). Then, the container can often ping other containers, but not always.
Specific question:
It seems like there's a window of time during which this can be worse. It's not necessarily tied to starting several containers at once, but there's a somewhat clear pattern: during some windows of time, containers do not get configured properly to communicate with each other. What troubleshooting steps come to mind for you?
The content below is the output from trying to ping one container (82afb0dccbcc) from two other containers. It fails at first, but then is successful.
The first time I try to ping the container, at 2019-12-10T23:57:52+00:00:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
82afb0dccbcc: user___92397089 crccheck/hello-world
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PING 82afb0dccbcc (172.24.0.165) 56(84) bytes of data.^M
^M
--- 82afb0dccbcc ping statistics ---^M
4 packets transmitted, 0 received, 100% packet loss, time 3033ms^M
^M
PING 82afb0dccbcc (172.24.0.165) 56(84) bytes of data.^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=2 ttl=64 time=0.083 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=3 ttl=64 time=0.072 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=4 ttl=64 time=0.073 ms^M
^M
--- 82afb0dccbcc ping statistics ---^M
4 packets transmitted, 3 received, 25% packet loss, time 3023ms^M
rtt min/avg/max/mdev = 0.072/0.076/0.083/0.005 ms^M
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In this first ping test, above, we note that the packet loss from the first container is 100% and from the second container, it is 25%.
A few minutes later (2019-12-10T23:57:52+00:00), however, 82afb0dccbcc can be successfully pinged from both containers:
82afb0dccbcc: user___92397089 crccheck/hello-world
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ping from ansible-provisioner:
PING 82afb0dccbcc (172.24.0.165) 56(84) bytes of data.^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=1 ttl=64 time=0.056 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=2 ttl=64 time=0.073 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=3 ttl=64 time=0.077 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=4 ttl=64 time=0.087 ms^M
^M
--- 82afb0dccbcc ping statistics ---^M
4 packets transmitted, 4 received, 0% packet loss, time 3063ms^M
rtt min/avg/max/mdev = 0.056/0.073/0.087/0.012 ms^M
ping from ansible_container:
PING 82afb0dccbcc (172.24.0.165) 56(84) bytes of data.^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=1 ttl=64 time=0.055 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=2 ttl=64 time=0.055 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=3 ttl=64 time=0.060 ms^M
64 bytes from user___92397089.wharf (172.24.0.165): icmp_seq=4 ttl=64 time=0.085 ms^M
^M
--- 82afb0dccbcc ping statistics ---^M
4 packets transmitted, 4 received, 0% packet loss, time 3062ms^M
rtt min/avg/max/mdev = 0.055/0.063/0.085/0.015 ms^M
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
You need to create a network and connect both the containers to that network.
The Docker embedded DNS server enables name resolution for containers connected to a given network. This means that any connected container can ping another container on the same network by its container name.
From within container1, you can ping container2 by name.So, its important to explicitly specify names for the containers otherwise this would not work.
Create two containers:
docker run -d --name container1 -p 8001:80 test/apache-php
docker run -d --name container2 -p 8002:80 test/apache-php
Now create a network:
docker network create myNetwork
After that connect your containers to the network:
docker network connect myNetwork container1
docker network connect myNetwork container2
Check if your containers are part of the new network:
docker network inspect myNetwork
Now test the connection, you will be able to ping container2 from container1:
docker exec -ti container1 ping container2
I actually ran into this issue randomly, but in my case both containers were already on the same network so it was puzzling me why one container couldn't ping another.
until I ran docker network inspect myNetwork and randomly noticed that for some reason both containers were assigned the SAME mac address... no idea why that happened or even how. Obviously that would preclude pinging since on a LAN mac addresses are used by switching logic to route traffic.
I had to stop and remove the container then recreate it to change the mac address.
In case, if there is any webapp is running on any one of your container and you want to ping/call any endpoint from another container and want to use response then you can follow steps as mentioned below -
First establish inter-container communications using docker network
1. docker network create dockerContainerCommunication
Now connect containers to network dockerContainerCommunication
2. docker network connect dockerContainerCommunication container1
3. docker network connect dockerContainerCommunication container2
Now start your containers (if not started)
4. docker start container1
5. docker start container2
Inspect your network. Here you can also find out IP address of the containers.
docker network inspect dockerContainerCommunication
Now attach to any one of the container from where you want to use web application, then ping other container using curl + IP address you found out in step 6.
or
docker attach container1
OR
docker attach container2
and then run curl command
curl http://IP_ADDRESS:PORT_ON_WHICH_APP_IS_RUNNING/api/endpointPath
I hope it helps.

Having problems sending data to statsd / graphite database on docker from outside container

I'm having issues getting data sent into a statsd container. I can successfully send data while on the command line inside the container itself. I need to be able to send statistic data to it from the host machine or from another Docker container.
I'm using Kitematic, I can see that the selection for 'bridge' network is checked on both containers. Using a bridge network was a suggestion I found for this issue.
I also tried passing '-P' to Docker while running the command to build the container, as that was supposed to expose the ports. I didn't notice a difference in the way it behaved when sending data from the other container.
Example of code that runs to create fake statistics using port 8125 on localhost (taken from this Docker container webpage https://hub.docker.com/r/graphiteapp/graphite-statsd)
Let's fake some stats with a random counter to prove things are working.
while true; do echo -n "example:$((RANDOM % 100))|c" | nc -w 1 -u 127.0.0.1 8125; done
The container is created using the following command:
docker run -d --name graphite --restart=always -p 80:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 graphiteapp/graphite-statsd
I've tried making sure both are on the same 'bridge' network. I'm running Docker Desktop on Windows 10 Enterprise. I've found several commands dealing with iptables and networking on Linux, but I feel like I'm missing something. I might also mention that statsd uses UDP connection on port 8125 by default.
If I try running the example command from another container on the bridge network, I don't get any result. I know the data (from another container) is not getting over correctly because I can't see it in the metrics that get received on the statsd dashboard.
I can ping localhost:8125 and get a response from within another container. From the outside (Powershell window on the host machine) it won't resolve.
PING localhost:8125 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.024 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.052 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.031 ms
^C
--- localhost:8125 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.024/0.035/0.052 ms>
If I run docker container ls then I get the following:
I found that I needed to get the specific IP address for each container, which can be found by running docker inspect (name of network). In this case, bridge.
Then I needed to specify the IP address of the container. I replaced the suggested address with that IP address and it worked.

Not able to connect to network inside docker container

I have a CentOS 7 host on which I am running Docker. When I do a ping from my host to 8.8.8.8, ping was successful whereas same inside a docker container is not working.
From Host
[root#linux1 ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=47 time=31.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=47 time=31.6 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 31.592/31.617/31.643/0.179 ms
From Docker Container (I am using basic ubuntu image):
[root#linux1 ~]# docker run ubuntu ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 172.17.0.1 icmp_seq=1 Destination Host Unreachable
From 172.17.0.1 icmp_seq=2 Destination Host Unreachable
From 172.17.0.1 icmp_seq=3 Destination Host Unreachable
From 172.17.0.1 icmp_seq=4 Destination Host Unreachable
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5000ms
pipe 4
Any suggestions would be helpful. Thanks
Restart the Docker daemon on Debian9
service docker restart
and the connections and networks works fine
Recently I faced a similar network issue. The other answers here didn't help: DNS was working fine and restarting Docker wouldn't change a thing. I've found that specifying the network as host solved it.
There are three ways of doing it:
In docker-compose:
By setting network_mode in the yaml file:
services:
worker:
build: .
network_mode: host
In the image building stage for RUN commands:
docker build --network=host
In the execution stage for the application:
docker run --network=host <image>
Try this:
docker run --dns=8.8.8.8 -it ubuntu ping 8.8.8.8
Ref: DOCKER DNS
I figured out the issue. It is not an issue with the DNS but an issue with the network connection itself inside Docker containers. Drilled down the issue is the default IP assigned to docker0 interface, which conflicted with my network address. Forced docker daemon to assign an IP so that it won't conflict and my issue is resolved.
Thanks
I had the same issue when stop and start container separately. I have just rebuild and re up containers.
docker-compose down
docker-compose build
docker-compose up -d
And then problem gone.

Resources