IP link to connect host to macvlan containers breaks after a couple of seconds - docker

I have a macvlan network where i put all my containers in. It was created like this:
docker network create -d macvlan -o parent=eno2 \
--subnet 10.0.2.0/24 \
--gateway 10.0.2.1 \
--ip-range 10.0.2.0/24 \
mynet
This is allows for container <-> container communication and from external computers <-> container communication. The problem is that host <-> container communication is not possible.
Searching about how to fix this, i found this blog.
That solution appears to work, running this commands:
ip link add zrz-ch link eno2 type macvlan mode bridge
ip addr add 10.0.2.15/24 dev zrz-ch
ifconfig zrz-ch up
I can successfully ping any container, and any container can ping the host.
The problem is that after 5-10 seconds the link breaks and the communication does not work anymore.
Ping from host -> container:
root#pluto:/home/zrz# ping 10.0.2.10
PING 10.0.2.10 (10.0.2.10) 56(84) bytes of data.
64 bytes from 10.0.2.10: icmp_seq=1 ttl=64 time=0.106 ms
64 bytes from 10.0.2.10: icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 10.0.2.10: icmp_seq=3 ttl=64 time=0.050 ms
64 bytes from 10.0.2.10: icmp_seq=4 ttl=64 time=0.068 ms
From 10.0.2.15 icmp_seq=5 Destination Host Unreachable
From 10.0.2.15 icmp_seq=6 Destination Host Unreachable
From 10.0.2.15 icmp_seq=7 Destination Host Unreachable
From 10.0.2.15 icmp_seq=8 Destination Host Unreachable
From 10.0.2.15 icmp_seq=9 Destination Host Unreachable
From 10.0.2.15 icmp_seq=10 Destination Host Unreachable
As you see after 5 seconds~ it stops working, the same happens when pinging from container -> host (after running the above commands on the host)
I can ping again if i do
ip link delete zrz-ch
And then run the commands above again. But as i said, it breaks after a couple of seconds.
How can i fix this?
Any ideas of how can i fix this?

Related

Wrong IP address in docker swarm service

I have very weird problem:
I have the swarm cluster and one of my service have wrong ip:
$ docker service inspect nginx_backend | grep Addr
"Addr": "10.0.0.107/24"
From any container in the cluster:
/ # ping nginx_backend
PING nginx_backend (10.0.0.107): 56 data bytes
64 bytes from 10.0.0.107: seq=0 ttl=64 time=0.057 ms
64 bytes from 10.0.0.107: seq=1 ttl=64 time=0.061 ms
64 bytes from 10.0.0.107: seq=2 ttl=64 time=0.064 ms
64 bytes from 10.0.0.107: seq=3 ttl=64 time=0.083 ms
^C
--- nginx_backend ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.057/0.066/0.083 ms
But in the server which hosted nginx_backend container:
root#backend:~# docker inspect nginx_backend.1.myzy10psfdl9r4jljrsz5zd5t | grep IPv4
"IPv4Address": "10.0.0.87"
And when some service try connect by name it got connect error, but if I manually put record like 10.0.0.87 nginx_backend to /etc/hosts inside a container, it have successful connect.
What I did wrong?)
Docker creates (by default) a Virtual IP (VIP) for each service. That's the 10.0.0.107. It then balances requests between the backend containers. In the second example (10.0.0.87) you're seeing the IP address of one of the containers. That's routable within Docker as well (which is why hitting the IP works). However the name (nginx_backend.1.myzy10psfdl9r4jljrsz5zd5t) is not DNS resolvable so that's why that fails.
You can find a list of the 'backing' containers for a service by doing a DNS lookup on tasks.nginx_backend.
Some more background here: https://docs.docker.com/network/overlay/

Docker: Connection from inside the container to localhost:port Refused

I'm trying to insure the connection between the different containers and the localhost address (127.0.0.1) used with port 8040.( My web application container run using this port.)
root#a70b20fbda00:~# curl -v http://127.0.0.1
* Rebuilt URL to: http://127.0.0.1/
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to 127.0.0.1 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused
This is what I get when I want to connect to localhost from inside the container
root#a70b20fbda00:~# curl -v http://127.0.0.1:8040
* Rebuilt URL to: http://127.0.0.1:8040/
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* connect to 127.0.0.1 port 8040 failed: Connection refused
* Failed to connect to 127.0.0.1 port 8040: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 8040: Connection refused
About iptables in each container:
root#a70b20fbda00:~# iptables
bash: iptables: command not found
Connection between the container is good
root#635114ca18b7:~# ping 172.17.0.1
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=0.253 ms
--- 172.17.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
root#635114ca18b7:~# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.100 ms
--- 127.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
root#635114ca18b7:~# ping 172.17.0.3
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.149 ms
64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.180 ms
--- 172.17.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.149/0.164/0.180/0.020 ms
Ping the 127.0.0.1:8040
root#635114ca18b7:~# ping 127.0.01:8040
ping: unknown host 127.0.0.1:8040
What I need to do in this case?
So the Global image that there is two containers ,
The first container contains a tomcat server that deploy my web application and it turnes perfectly.
The second is a container that need to connect to the web application
URL. http://127.0.0.1:8040/my_app
you will have to use docker run --network host IMAGE:TAG for achieving the desired connection
further read here
example:-
docker run --network host --name CONTAINER1 IMAGE:tag
docker run --network host --name CONTAINER2 IMAGE:tag
inside container - CONTAINER2 you will be able to access other container as host CONTAINER1
And for accessing the service you will have to do CONTAINER:
Based on the information provided, looks like there are two containers. If these two containers are started by docker without --net=host then each of them get two different IP addresses. Say your first container got 172.17.0.2 and the second one 172.17.0.3.
In this scenario each container gets it's own networking stack. So 127.0.0.1 refers to it's own networking stack not the same.
As pointed out by #kakabali, it's possible to run the containers with host network, sharing the networking stack of the host.
One of the other options is to use the actual IP address of the first container in the second one.
second-container# curl http://172.17.0.2
Or another option is to run the second container as the sidekick/sidecar container sharing the networking stack of the first one.
docker run --net=container:${ID_OF_FIRST_CONTAINER} ${IMAGE_SECOND}:${IMAGE_TAG_SECOND}
Or if you use links correctly:
docker run --name web -itd ${IMAGE_FIRST}:${TAG_FIRST}
docker run --link web -itd ${IMAGE_SECOND}:${TAG_SECOND}
Note: docker --link feature is deprecated.
Another option is to use container management platforms which take care of service discovery for you automatically.
PS: You cannot ping an IP address on a different port. For more info, click here.

Resolution by container name

I have two containers connected to the default bridge network:
» docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3cc528ddbe7e gitlab/gitlab-runner:latest "/usr/bin/dumb-ini..." 25 minutes ago Up 25 minutes gitlab-runner
3c01073065c7 gitlab/gitlab-ee:latest "/assets/wrapper" About an hour ago Up About an hour (healthy) 0.0.0.0:45022->22/tcp, 0.0.0.0:45080->80/tcp, 0.0.0.0:45443->443/tcp gitlab
I have found the corresponsing IP addresses with docker inspect (any better method of obtaining them?), and I can ping from one container to the other, by IP address:
» docker exec -it gitlab-runner bash
root#3cc528ddbe7e:/# ping 172.17.0.3
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.079 ms
64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.063 ms
64 bytes from 172.17.0.3: icmp_seq=3 ttl=64 time=0.060 ms
^C
--- 172.17.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.060/0.067/0.079/0.010 ms
But I cannot ping by name:
root#3cc528ddbe7e:/# ping gitlab
ping: unknown host gitlab
Why is this? I thought docker provides DNS by container name.
I have two containers connected to the default bridge network...
I can ping from one container to the other, by IP address...
But I cannot ping by name...
This is the default behavior for the default bridge network.
From: Docker docs
Differences between user-defined bridges and the default bridge
User-defined bridges provide automatic DNS resolution between containers.
Containers on the default bridge network can only access each other by IP addresses, unless you use the --link option, which is considered legacy. On a user-defined bridge network, containers can resolve each other by name or alias.

docker multiply hostnames for one container

How can i add multiply hostnames for some docker in docker-compose? Container should see by self many hostnames, for example:
ping name: should resolve 127.0.0.1 (or $docker_ip)
ping another.name: should resolve 127.0.0.1 (or $docker_ip)
Use extra-hosts:
extra_hosts:
- "name:127.0.0.1"
- "another.name:127.0.0.1"
Has the same effect as docker run --add-host:
> docker run --add-host another.name:127.0.0.1 sixeyed/ubuntu-with-utils ping another.name
PING another.name (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.057 ms

docker network - ping 255.255.255.255

When I setup a network with docker create network test1 and then start a few containers, for example
docker run -d --net=test1 --name=t1 elasticsearch
docker run -d --net=test1 elasticsearch
docker run -d --net=test1 elasticsearch
I can't broadcast ping any of these containers with docker exec -ti t1 ping 255.255.255.255.
Any idea how I can change this?
This is currently followed in issue 17814
UDP broadcasts don't work in multi-host network between hosts.
UDP broadcasts only work if both containers run on the same host.
Playing with icmp broadcast by pinging on 255.255.255.255, I receive replies only from the local host:
# ping -b 255.255.255.255
WARNING: pinging broadcast address
PING 255.255.255.255 (255.255.255.255) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_req=1 ttl=64 time=0.601 ms
64 bytes from 172.18.0.1: icmp_req=2 ttl=64 time=0.424 ms
64 bytes from 172.18.0.1: icmp_req=3 ttl=64 time=0.420 ms
64 bytes from 172.18.0.1: icmp_req=4 ttl=64 time=0.427 ms
(I made sure /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts is set to 0 on both hosts.)
It also seems impossible to set a broadcast address on the interface connected to the shared network:
# ifconfig eth0 broadcast 10.0.0.255
SIOCSIFBRDADDR: Operation not permitted
SIOCSIFFLAGS: Operation not permitted
This ability to multicast in overlay driver is discussed in docker/libnetwork issue 552.
(help wanted)

Resources