I'm having a few docker containers (Using docker-compose and a single network - network-sol)
One of the containers is a Spring Boot application that sends UDP broadcast to the local network. 255.255.255.255 fails because It's the local broadcast address of network-sol
How can I broadcast UDP messages such as the "top local network" Will get those packets? Do i have to use directed broadcast address for that?
P.S
broadcast works if the application is deployed outside of docker (part of the local network
You should either run the service defined in your docker-compose.yml file with network_mode: host.
Alternatively you can publish the port of the container you intended to communicate with by publishing it using the following configuration. Note that the /udp is required for UDP communication to work.
service:
ports:
- "8080:8080/udp"
I have gotten some luck out of this. The guide specifies sysctl parameters that are needed for broadcast forwarding from a docker network, you should then be able to either use his script or specify these parameters when running docker.
Related
I'm learning Docker networking. I'm using Docker Desktop on Windows.
I'm trying to understand the following observations:
Short version in a picture:
Longer version:
First setup (data from container to host)
I have a simple app running in a container. It sends one UDP-datagram to a specific port on the host (using "host.docker.internal")
I have a corresponding app running on the host. It listens to the port and is supposed to receive the UDP-datagram.
That works without publishing any ports in docker (expected behavior!).
Second setup (data from host to container)
I have a simple app on the host. It sends one UDP-datagram to a specific port on the loopback network (using "localhost")
I have a corresponding app running in a container. It listens to the port and is supposed to receives the UDP-datagram.
That works only if the container is run with option -p port:port/udp (expected behavior!).
Third setup (combination of the other two)
I have an app "Requestor" running in a container. It sends a UDP request-message to a specific port on the host and then wants to receive a response-message.
I have a corresponding app "Responder" running on the host. It listens to the port and is supposed to receive the request-message. Then it sends a UDP response-message to the endpoint of the request-message.
This works as well, and - that's what I don't understand - without publishing the port for the response-message!
How does this work? I'm pretty sure there's some basic networking-knowledge that I simply don't have already to explain this. I would be pleased to learn some background on this.
Sidenote:
Since I can do curl www.google.com successfully from inside a container, I realize that a container definitely must not publish ports to receive any data. But there's TCP involved here to establish a connection. UDP on the other hand is "connectionless", so that can't be the (whole) explanation.
After further investigation, NAT seems to be the answer.
According to these explanations, a NAT is involved between the loopback interface and the docker0 bridge.
This is less recognizable with Docker Desktop for Windows because of the following (source):
Because of the way networking is implemented in Docker Desktop for Windows, you cannot see a docker0 interface on the host. This interface is actually within the virtual machine.
I have a a docker container with a few images running there. I run them via docker-compose up command. On my device everything works well with localhost but I want to make so that other devices in the same network will be able to access the MQTT broker as well. How do I do that?
Currently, in my code I do this:
ws:localhost:9001
But since this localhost applies only for the device that runs docker, another laptop won't be able to use it. How do I solve that?
You use the LAN IP address of your machine (the one hosting the docker containers) in place of localhost.
We have no way of knowing what that address may be, but it could start with 192.168.x.x or may be 10.x.x.x
By default, Docker has a "bridge" network that will bridge your container to the outside world. Just use the IP address of the computer where your MQTT Broker Container is running, and port 9001, and it should work fine.
If you need to run it on an internal Docker network, you will have to use something like an ADC or TCP Proxy of some sort to allow access to it.
I'm running Docker Desktop for Mac on host, it is running two containers.
Container-1: linux-based OS, running UDP-based server program listening on 14xxx port (udp://:14xxx/).
Container-2: linux-based OS, python application sending/receiving data via UDP address as udp://14xxx/ without any specific hostname.
Question: My python app on Container-2 is able to send on UDP port, but never receives anything back from Container-1.
Given UDP works differently from TCP & HTTP protocols..
How can I establish successful UDP communication between two docker containers running on same host (MacOS)?
Various things that I have tried, but no luck.
Tried running both containers using --network host option.
Tried creating a new docker network testnet and started containers using --network testnet option.
Never mind. I found the solution.
First, it was not a docker thing at all.
In my python application on Container-2, I used environment variables to determine the UDP address. Apparently, these variables were not set properly. Hence, the confusion/error.
Second, "--network host" is still a VALID argument to have for both running Docker containers to make sure they discover/talk to each other.
Hope it helps!
I am using Docker 18.06.1-ce-win73 on windows 10 and trying to perform the following udp operation:
Docker port 10001 --------------> host port 10620
It is mandatory for the application running on the host to receive packets from the port 10001.
Inside the docker container, using python I bind on the IP ('0.0.0.0', 10001) and use the socket to send my packets to the host IP on port 16020.
I have also started the container with the argument -p 10001:10001/udp.
Unfortunately, when receiving the packet on the Host application, the origin port is not 10001 but a random one.
Is it possible to force docker to use a specific source port when using UDP from inside the container ?
You can control the container source port, but when you communicate outside of docker, even to your host, the request will go through a NAT layer that will change the source to be the host with a random port. You may be able to modify the iptables rules to work around this NAT effect.
However, if you really need control of the source port like this, you may be better off switching to host networking (--net=host or network_mode: host depending on how you run your containers), or change to a networking driver like macvlan that exposes the container directly without going through the NAT rules.
Few days ago I tried to configure Kafka Docker container with Docker Compose and port mapping and discover interesting behavior which I do not fully understand:
Kafka broker seems to connect to itself. Why ?
My set up is:
Ubuntu 14.04, Docker 1.13.1, Docker-Compose 1.5.2
Kafka 0.10 listens on port 9092, this port is exposed by container.
In Docker Compose I have port mapping from container port 9092 to local port 4005.
I configured host name of my Docker Host machine and local port from Compose in advertised.listeners (docker-host:4005) since broker should be visible from my company network.
With this set up when I try to send/fetch data to/from Kafka, all attempts end up with:
Topic metadata fetch included errors: {topic_name=LEADER_NOT_AVAILABLE}
After trying various combinations of ports and host names in advertised.listeners, I discovered that sole working combination is localhost:9092. Any attempt to change hostname or port led to the error mentioned above.
This made me think that Kafka tries to connect to address configured in advertised.listeners and this is somehow related to topic metadata.
So inside Docker container I did:
redirect traffic to "docker-host" to loopback
echo "127.0.0.1 $ADVERTISED_HOST" >> /etc/hosts
configure Kafka to listen on all interfaces and port (exact as advertised)
sed -r -i "s/#(listeners)=(.*)/\1=PLAINTEXT:\/\/0.0.0.0:4005/g" $KAFKA_HOME/config/server.properties
advertise "docker-host" and external port
sed -r -i "s/#(advertised.listeners)=(.*)/\1=PLAINTEXT:\/\/$ADVERTISED_HOST:4005/g" $KAFKA_HOME/config/server.properties
And now it works like a charm.
However I still do not understand:
Why Kafka broker might need to connect to itself via address configured in advertised.listeners ?
Is there a way to disable this or at least configure it to use address from 'listeners' property (with default Kafka port) ?
UPD
Worth to mention, following setup does not work: Kafka listens on 0.0.0.0:9092, advertised listener is configured to docker-host:4005.
In this case whenever consumer or producer connects to kafka it receives LEADER_NOT_AVAILABLE.
There is also connection shown by netstat (within container) to docker-host:4005 in state SYN_SENT.
UPD 2
Looks like there is similar problem with Kafka but inside AWS described here.
Difference is that in my case I want to use different Kafka port.
UPD 3
Ok, the reason why setup mentioned in the first UPD paragraph does not work is - UFW, for some reasons it blocks traffic which goes from docker container to itself via host machine.
Why Kafka broker might need to connect to itself via address
configured in advertised.listeners ?
When a Kafka broker is first connected by a client, it replies back with the address that it expects that client to use in the future to talk to the broker. This is what is set in the advertised.listeners property. If you don't set this property, the value from listeners will be used instead (which answers your second question).
So your "issue" is, that remote clients connect to yourhost:9092, reach the Kafka broker, because you forwarded the port, the broker then responds with "you can reach me at localhost:9092" and when the client sends the next packet there it just connects back to itself.
The metadata is not really related here, its just the first request that gets made.
Your solution is correct for this setup I think, have Kafka listen on local interfaces and set the advertised.listeners to the host that someone from your company network would connect to.
I don't 100% know if the broker needs to connect to itself as well, pretty sure thats not the case though. I think your setup would also work without the entry of the external hostname in your /etc/hosts file.
Is there a way to disable this or at least configure it to use address
from 'listeners' property (with default Kafka port) ?
see above