I'm working locally on OSX.
I'm using Kafka and zookeeper in local mode. Meaning zookeeper from my Kafka installation. One node cluster.
Both work on the loopback localhost
zookeeper.connect=localhost:2181
My /etc/host looks as follows:
127.0.0.1 localhost MaatPro.local
255.255.255.255 broadcasthost
::1 localhost MaatPro.local
fe80::1%lo0 localhost MaatPro.local
I have docker for Mac set on my machine, with the Kubernetes extension.
My scenario
I have an Akka-stream micro-service dockerized, that reads data from an external database and write it in a Kafka topic. It uses as bootstrap server:
"localhost:9092"
Issue
When I run my service on my machine directly (e.g. command line or from within Intellij) everything works fine. When I run it on my Local Docker or Kubernetes I get the following error:
(o.a.k.clients.NetworkClient) [Producer clientId=producer-1] Connection to node 0 could not be established. The broker may not be available.
With Kubernetes I build the following YAML File to deploy my pod:
apiVersion: v1
kind: Pod
metadata:
name: fetcher
spec:
hostNetwork: true
containers:
- image: entellectextractorsfetchers:0.1.0-SNAPSHOT
name: fetcher
I took the precaution to set hostNetwork: true
With Docker daemon directly I originally tried to set the network as host too but discovered that this does not work with docker for mac. Hence, I abandoned that route. I understood that it has to do with virtualization.
1) Does the virtualization issue that happens with docker is actually the same as my local kubernetes? Basically, the host network is the virtual machine and not my mac?
2) I try to change my code and add as a bootstrap server the following address: host.docker.internal as per the documentation. But the problem persists. Is the fundamental problem the fact that I am working on a loopback address? Shall I work on my network address indeed? To which address does host.docker.internal point to? How can I make it work with the loopback address? If I'm completely off, any idea of what I need to implement to get this working?
Thank you so much for any help with this.
Based on #cricket_007 guidance Kafka Listeners - Explained and the many read I have had here and there over this issue, including the official docker documentation I Want to connect from a container to a servicer on the host
I came up with the following solution.
I added the following to my default local kafka configuration (i.e. server.properties)
listeners=EXTERNAL://0.0.0.0:19092,INTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://127.0.0.1:9092,EXTERNAL://docker.for.mac.localhost:19092
inter.broker.listener.name=INTERNAL
In fact External here, is expected to be the docker network. This config is only for my OSX machine for my local development purpose. I do not expect people connecting to my laptop to use my local kafka hence i can use EXTERNAL://docker.for.mac.localhost:19092. This is what is advertised to my container in docker/kube. From within that network, docker.for.mac.localhost is reachable.
Note this would probably not work with Minikube. This is specific to Docker For Mac. The kubernetes that I run on my machine is the one coming with docker for mac and not minikube.
Finally in my code i use both in a list
"localhost:9092, docker.for.mac.localhost:19092"
I use typeSafe config, so that in prod, this is erased by the env variable. When the env variable is not specified, this is what is used. When i run my micro-service from Intellij localhost:9092 is used. That’s because In that case, i am in the same network as my kafka/zookeeper in my machine. However when I run the same micro-service from docker/kube docker.for.mac.localhost:19092 is used.
Answers to the side questions I had
Yes. Docker for Mac use HyperKit as a lightweight virtual machine, running a linux on it, and Docker Engine is ran on it. The Docker for MAC Kubernetes extension is basically about running kubernetes cluster Services/infrastructure as containers in the docker daemon. Docker for Mac vs. Docker Toolbox . In other words, the host is hyperkit and not osx. But as the above doc explain, Docker for Mac implementation is all about making it appear to the user as if there were no virtualization involve between OSX and Docker.
Connecting to the host using loopback address is an issue that has not been solved. I'm not even sure that it works perfect even if the host is Linux. Not sure, might have been resolved at this point. Nonetheless, it would require to run an image by stating that the container or the pod in the case of kube are on the host network. But in docker for mac, that functionality will never work based on my readings online. Hence the solution of using docker.for.mac.localhost or host.docker.internal, that Docker for Mac did set up to refer to the mac host and not the hyperkit host.
host.docker.internal and docker.for.mac.localhost are one of the same and the late recommendation at this point is host.docker.internal. This being said, this address did not originally work for me because my Kafka Set up was not good. It is worth readying #criket_007 link to understand that well http://rmoff.net/2018/08/02/kafka-listeners-explained.
Following #MaatDeamon approach I only did the following and it worked for me.
advertised.listeners=PLAINTEXT://:9092
And in your application config properties
localhost:9092,host.docker.internal:9092
Related
I am trying to use Colima to run an apache-php docker container. My uni provides docker images derived from upstream ones configured for our course using docker-compose.
The container works as it should but I can't get its Xdebug to connect to my PhpStorm.
This is what it says in the Xdebug log:
Creating socket for 'host.docker.internal:9003', poll success, but error: Operation now in progress (29).
This tells me absolutely nothing.
The setup is admittedly quite complex (x86 Apache ran via QEMU in Docker in Linux VM in macOS on ARM CPU) but I can do nc host.docker.internal 9003 from any docker container, so I have no idea why Xdebug isn't able to reach my host. (Only works when the IDE is running and on no other ports, so it's definitely connecting to PhpStorm.)
Any idea what could be going on here?
On Colina, the IP address is hard coded to "192.168.5.2", so setting xdebug.client_host=192.168.5.2 should do the trick. There is now also an alias for it, called host.lima.internal.
As per this documentation page.
The problem is the uni's docker-compose.yml which configured the container with:
extra_hosts:
- "host.docker.internal:host-gateway"
and apparently that can break host.docker.internal in some situations: https://github.com/docker/for-linux/issues/264#issuecomment-759737542
The solution is to remove those two lines.
I have been using docker for some time now on my Synology NAS and it works just fine but I need something a little more powerful to run Plex Media Server.
I have installed Photon OS (on my ESXi Host) running Docker and enabled SSH. I have setup a static IP using the following network file configs:
5-Main.link
[Match]
MACAddress=00:00:00:00:00:00 (example)
[Link]
Description=Main
Name=Main
5-Main.network
[Match] Name=Main
[Network] Gateway=192.168.1.1
Address=192.168.1.3/24
DNS=192.168.1.1
DHCP=no
[DHCP] UseDNS=false
I have installed Portainer fine and that works on the 192.168.1.3:9443 address.
What I am trying to achive like I had done successfully on the NAS is to grant a container its own IP address. I created a docker network using the following command:
docker network create --driver=macvlan --gateway=192.168.1.1 --subnet=192.168.1.0/24 --ip-range=192.168.1.0/24 -o parent=Main LAN
This creates the network fine but when you attach it to a container, give the container a MAC address and an IP address within the same range (192.168.1.50) it does not ping out of the container or from the host (or another device) in. curl ipino.io also does not work from inside of the container. These exact steps works perfectly on the Synology NAS (with selecting the NAS' correct NIC etc)
I have tried using Debian OS instead but that also has the same problem. I have tried accepting all traffic in the IP tables but no effect also. On Debian OS, I have also allowed the NIC to be DHCP instead and tried using a static IP within the container but still no effect. I am running out of ideas. I feel this would be something simple but I have searched high and low now but I am coming to a dead end. Any advice would be greatly appreciated.
I am learning docker network. I created a simple docker-compose file that starts two tomcat containers:
version: '3'
services:
tomcat-server-1:
container_name: tomcat-server-1
image: .../apache-tomcat-10.0:1.0
ports:
- "8001:8080"
tomcat-server-2:
container_name: tomcat-server-2
image: .../apache-tomcat-10.0:1.0
After I start the containers with docker-compose up I can see that tomcat-server-1 responses on http://localhost:8001. At the first glance, the tomcat-server-2 is not available from localhost. That great, this is what I need.
When I inspect the two running containers I can see that they use the following internal IPs:
tomcat-server-1: 172.18.0.2
tomcat-server-2: 172.18.0.3
I see that the tomcat-server-1 is available from the host machine via http://172.18.0.2:8080 as well.
Then the following surprised me:
The tomcat-server-2 is also available from the host machine vie http://172.18.0.3:8080 despite port mapping is not defined for this container in the docker-compose.yml file.
What I would like to reach is the following:
The two tomcat servers must see each other in the internal docker network via hostnames.
Tomcat must be available from the host machine ONLY if the port mapping is defined in the docker-compose file, eg.: "8001:8080".
If no port mapping definition then the container could NOT be unavailable. Either from localhost or its internal IP, eg.: 172.18.0.3.
I have tried to use different network configurations like the bridge, none, and host mode. No success.
Of course, the host mode can not work because both tomcat containers use internally the same port 8080. So if I am correct then only bridge or none mode that I can consider.
Is that possible to configure the docker network this way?
That would be great to solve this via only the docker-compose file without any external docker, iptable, etc. manipulation.
Without additional firewalling setup, you can't prevent a Linux-native host from reaching the container-private IP addresses.
That having been said, the container-private IP addresses are extremely limited. You can't reach them from other hosts. If Docker is running in a Linux VM (as the Docker Desktop application provides on MacOS or Windows) then the host outside the VM can't reach them either. In most cases I would recommend against looking up the container-private IP addresses up at all since they're not especially useful.
I wouldn't worry about this case too much. If your security requirements need you to prevent non-Docker host processes from contacting the containers, then you probably also have pretty strict controls over what's actually running on the host and who can log in; you shouldn't have unexpected host processes that might be trying to connect to the containers.
I am currently developing docker containers using visual studio, and these container images are supposed to run in a kubernetes cluster that I am also running locally.
Currently, the docker container that is running via visual studio is not being deployed to a kubernetes cluster, but for some reason am I able to ping the kubernetes pod's ip address from the docker container, but for which I don't quite understand; should they not be separated, and not be able to reach each other?
And it cannot be located on the kubernetes dashboard?
And since they are connected, why can't I use the kubernetes service to connect to my pod from my docker container?
The docker container is capable of pinging the cluster IP, meaning that it is reachable.
nslookup the service is not able to resolve the hostname.
So, as I already stated in the comment:
When Docker is installed, a default bridge network named docker0 is
created. Each new Docker container is automatically attached to this
network, unless a custom network is specified.
Thats mean you are able to ping containers by their respective IP. But you are not able to resolve DNS names of cluster objects - you VM know nothing about internal cluster DNS server.
Few option what you can do:
1) explicitly add record of cluster DNS to /etc/hosts inside VM
2) add a record to /etc/resolv.conf with nameserver and search inside VM. See one of my answers related to DNS resolution on stack: nslookup does not resolve Kubernetes.default
3)use dnsmasq as described in Configuring your Linux host to resolve a local Kubernetes cluster’s service URLs article. Btw I highly recommend you read it from the beginning till the end. It greatly describes how to work with DNS and what workaround you can use.
Hope it helps.
I'm trying to use Apache Kafka, i.e. a version usable in connection with docker (wurstmeister/kafka-docker), in a Virtual Machine and connect to the kafka broker in the dockers over the host system of the VM.
I will describe the setup in more detail:
My host system is an Ubuntu 64 bit 14.04.3 LTS (Kernel 3.13) running on an "usual" computer. I have a complete and complex structure of various docker-containers interacting with each other. In order to not disturb, or better said, to encapsulate this whole structure, it is not an option to run the docker images directly on the host system. Another reason for that is the need for different python libraries on the host, which interfere with the python-lib version required by docker-compose (which is used to start the different docker images).
Therefore the desired solution should be to setup a virtual machine over VirtualBox (guest system: Ubuntu 16.04.1 LTS) and to completely run the docker environment in this VM. This has the distinct advantage that the VM itself can be configured exactly according to the requirements of the docker structure.
As mentioned above one of the docker images provides kafka and zookeeper functionality to use for communication and messaging. That means the .yaml file, which sets up a container running this image, forwards all necessary ports of kafka and zookeeper to the host system of the docker environment (which is the guest system of the virtual box VM). In order to also make the docker environment visible in the host system, I forwarded all the ports over network settings in VirtualBox (network -> adapter NAT -> advanced -> port forwarding). The behaviour of the system is as follows:
When I run the docker-environment (including kafka) I can connect, consume and produce from the VM-kafka, which uses the recommended standard kafka shell scripts (producer & consumer api).
When I run a Kafka and zookeeper server ON THE VM guest system, I can connect from outside the VM (host), produce and consume over producer & consumer api.
When I run Kafka in the docker environment I can connect from the host system outside the VM, meaning I can see all topics, get infos about topics, while also seeing some debug output from kafka and zookeeper in the docker.
What is not possible is unfortunately to produce or consume any message from the host system to/from the docker-kafka. From the producer API I get a "Batch expired" exception, the consumer returns with a "ClosedChannelException".
I have done a lot of googling and found a lot of hints how to solve similiar problems. Most of them refer to the advertised.host.name-parameter in kafka-server.properties, which is accessible over KAFKA_ADVERTISED_HOST_NAME in .yaml. https://stackoverflow.com/a/35374057/3770602 e.g. refers to that parameter when both the errors mentioned above occur. Unfortunately none of the scenarios feature both docker AND VM functionality.
Further more trying to modify this parameter does not have an affect at all. Although I am not very familiar with docker and kafka, I can see a problem here, as the kafka consumer and producer would get the local IP of the docker environment, which is 10.0.2.15 in the NAT case, to use as broker. But unfortunately this IP is not visibile from outside the VM. Consequently the solution would probably be a changed setup, where one should use the bridged mode in VirtualBox networking. Strange thing here is, that a bridged connection (of course) leads to an own IP of the VM over DHCP, which then leads to a non-accessible docker-kafka both from the VM AND the host system. This last behaviour seems to be very awkward to me.
Concluding my question would be, if somebody has some experiences with this or a similiar scenario and can tell me, how to configure docker-kafka, VirtualBox VM and host system. I have really tried a lot of different settings for the consumer and producer call with no success. Of course some docker, kafka or both docker & kafka experts are welcome to answer as well.
If you need additional information or you can provide some hints, please let me know.
Thanks in advance