Hi I’m using Spotify/Kafka and am running it with
docker run —name ka -p 9092:9092 -p 2181:2181 —env ADVERTISED_HOST=localhost —env ADVERTISED_PORT 2181 —net mynet spotify/kafka
I make sure I run my second container using the same net and I can ping the Kafka container using ka.mynet
Also in this second container I downloaded kafka and it’s shell scripts and I’m able to do a
./kafka-topics.sh —zookeeper ka.mynet —list and see the “Test” topic
Now any attempt to produce or consume spits out errors. Producer complains about something to do with not finding a leader.
Other Googling has led me to believe it has something to do with the advertised host.
Ok it seems like the only way to get this to work is to assign my machine's current IP address as the ADVERTISED_HOST env variable.
So if my machine's IP is 192.168.1.11 then:
docker run —name ka -p 9092:9092 -p 2181:2181 —env ADVERTISED_HOST=192.168.1.11 —env ADVERTISED_PORT=9092 —net mynet spotify/kafka
Related
When I start MySQL :
docker run --rm -d -e MYSQL_ROOT_PASSWORD=root -p 3306:3306 -v /Docker/data/matos/mysql:/var/lib/mysql mysql:5.7
And start PHPMyAdmin :
docker run --rm -d -e PMA_HOST=172.17.0.1 phpmyadmin/phpmyadmin:latest
PMA cannot connect to the DB server.
When I try with PMA_HOST=172.17.0.2 (which is the address assigned to the MySQL container), it works.
But :
as MySQL container publishes its 3306 port, I think it should be reachable on 172.17.0.1:3306.
I don't want to use the 172.17.0.2 address because the MySQL container can be assigned another address whenever it restarts
Am I wrong ?
(I know I can handle this with docker-compose, but prefer managing my containers one by one).
(My MySQL container is successfully telnetable from my laptop with telnet 172.17.0.1 3306).
(My docker version : Docker version 20.10.3, build 48d30b5).
Thanks for your help.
Create a new docker network and start both containers with the network
docker network create my-network
docker run --rm -d --network my-network -e MYSQL_ROOT_PASSWORD=root -p 3306:3306 -v /Docker/data/matos/mysql:/var/lib/mysql --name mysql mysql:5.7
docker run --rm -d --network my-network -e PMA_HOST=mysql phpmyadmin/phpmyadmin:latest
Notice in the command that I've given the mysql container a name 'mysql' and used it as the address for phpmyadmin
Just found out the problem.
My ufw was active on my laptop, and did not allow explicitly port 3306.
I managed to communicate between PMA container and MySQL, using 172.17.0.1, either by disabling ufw or adding a rule to explicitly accept port 3306.
Thanks #kidustiliksew for your quick reply, and the opportunity you gave me to test user-defined networks.
maybe it's a good idea to use docker-compose.
Create a docker-compose.yml file and inside declare two services, one web and the other db, then you can reference them through their service names (web, db)
ex: PMA_HOST=db
I need to make a Docker container for a project involving streaming data using Kafka and Zookeeper. Looking around I found this docker image from Spotify, including Kafka and Zookeeper.
How should I include it in my project? Should I include in the Dockerfile the suggested commands, listed below?
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip \`docker-machine active\``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip \`docker-machine active\``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
How about using a docker-compose file?
In your *.yaml you can set-up the services to pull the Kafka and Zookeeper images from Spotify's DockerHub, map ports (e.g. "2181:2181" and "9092:9092" for ZK and Kafka, respectively), set ENV variables, and persist data to a volume so you don't lose your topics and offsets.
The docker daemon is running on an Ubuntu machine. I'm trying to start up a zookeeper ensemble in a swarm. The zookeeper nodes themselves can talk to each other. However, from the host machine, I don't seem to be able to access the published ports.
If I start the container with -
docker run \
-p 2181:2181 \
--env ZOO_MY_ID=1 \
--env ZOO_SERVERS="server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
zookeeper
It works like a charm. On my host machine I can say echo conf | nc localhost 2181 and zookeeper says something back.
However if I do,
docker service create \
-p 2181:2181 \
--env ZOO_MY_ID=1 \
--env ZOO_SERVERS="server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
zookeeper
and run the same command echo conf | nc localhost 2181,
it just gets stuck. I don't even get a new prompt on my terminal.
This works just as expected on the Docker Playground on the official Zookeeper Docker Hub page. So I expect it should for me too.
But... If I docker exec -it $container sh and then try the command in there, it works again.
Aren't published ports supposed to be accessible even by the host machine for a service?
Is there some trick I'm missing about working with overlay networks?
Try to use docket service create --publish 2181:2181 instead.
I believe the container backing the service is not directly exposed and has to go through the Swarm networking.
Otherwise, inspect your service to check which port are published: docker service inspect <service_name>
Source: documentation
Ok, I am pretty new to Docker world. So this might be a very basic question.
I have a container running in Docker, which is running RabbitMQ. Let's say the name of this container is "Rabbit-container".
RabbitMQ container was started with this command:
docker run -d -t -i --name rmq -p 5672:5672 rabbitmq:3-management
Python script command with 2 args:
python ~/Documents/myscripts/migrate_data.py amqp://rabbit:5672/ ~/Documents/queue/
Now, I am running a Python script from my host machine, which is creating some messages. I want to send these messages to my "Rabbit-container". Hence I want to connect to this container from my host machine (Mac OSX).
Is this even possible? If yes, how?
Please let me know if more details are needed.
So, I solved it by simply mapping the RMQ listening port to host OS:
docker run -d -t -i --name rmq -p 15672:15672 -p 5672:5672 rabbitmq:3-management
I previously had only -p 15672:15672 in my command. This is mapping the Admin UI from Docker container to my host OS. I added -p 5672:5672, which mapped RabbitMQ listening port from Docker container to host OS.
If you're running this container in your local OSX system then you should find your default docker-machine ip address by running:
docker-machine ip default
Then you can change your python script to point to that address and mapped port on <your_docker_machine_ip>:5672.
That happens because docker runs in a virtualization engine on OSX and Windows, so when you map a port to the host, you're actually mapping it to the virtual machine.
You'd need to run the container with port 5672 exposed, perhaps 15672 as well if you want WebUI, and 5671 if you use SSL, or any other port for which you add tcp listener in rabbitmq.
It would be also easier if you had a specific IP and a host name for the rabbitmq container. To do this, you'd need to create your own docker network
docker network create --subnet=172.18.0.0/16 mynet123
After that start the container like so
docker run -d --net mynet123--ip 172.18.0.11 --hostname rmq1 --name rmq_container_name -p 15673:15672 rabbitmq:3-management
note that with rabbitmq:3-management image the port 5672 is (well, was when I used it) already exposed so no need to do that. --name is for container name, and --hostname obviously for host name.
So now, from your host you can connect to rmq1 rabbitmq server.
You said that you have never used docker-machine before, so i assume you are using the Docker Beta for Mac (you should see the docker-icon in the menu bar at the top).
Your docker run command for rabbit is correct. If you now want to connect to rabbit, you have two options:
Wrap your python script in a new container and link it to rabbit:
docker run -it --rm --name migration --link rmq:rabbit -v ~/Documents/myscripts:/app -w /app python:3 python migrate_data.py
Note that we have to link rmq:rabbit, because you name your container rmq but use rabbit in the script.
Execute your python script on your host machine and use localhost:5672
python ~/Documents/myscripts/migrate_data.py amqp://localhost:5672/ ~/Documents/queue/
I found this docker image for Kafka
https://hub.docker.com/r/spotify/kafka/
and I can easily create a docker container using command documented in the link
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`boot2docker ip` --env ADVERTISED_PORT=9092 spotify/kafka
This is good. But I want to configure a "multiple" node Kafka cluster running on a docker swarm.
How can I do that?
Edit 28/11/2017:
Kafka added listener.security.protocol.map to their config. This allows you to set different listener addresses and protocols depending on whether you are inside or outside the cluster, and stops Kafka getting confused by any load balancing or ip translation which occurs in docker. Wurstmeister has a working docker image and example compose file here. I tried this a while back with a few docker machine nodes set up as a swarm and it seems to work.
tbh though I just attach a Kafka image to the overlay network and run the Kafka console commands when ever I want to interact with it now.
Hope that helps
Old Stuff Below
I have been trying this with docker 1.12 using docker swarm mode
create nodes
docker-machine create -d virtualbox master
docker-machine create -d virtualbox worker
master_config=$(docker-machine config master | tr -d '\"')
worker_config=$(docker-machine config worker | tr -d '\"')
master_ip=$(docker-machine ip master)
docker $master_config swarm init --advertise-addr $master_ip --listen-addr $master_ip:2377
worker_token=$(docker $master_config swarm join-token worker -q)
docker $worker_config swarm join --token $worker_token $master_ip:2377
eval $(docker-machine env master)
create the zookeeper service
docker service create --name zookeeper \
--constraint 'node.role == manager' \
-p 2181:2181 \
wurstmeister/zookeeper
create the kafka service
docker service create --name kafka \
--mode global \
-e 'KAFKA_PORT=9092' \
-e 'KAFKA_ADVERTISED_PORT=9092' \
-e 'KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092' \
-e 'KAFKA_ZOOKEEPER_CONNECT=tasks.zookeeper:2181' \
-e "HOSTNAME_COMMAND=ip r | awk '{ ip[\$3] = \$NF } END { print ( ip[\"eth0\"] ) }'" \
--publish '9092:9092' \
wurstmeister/kafka
Though for some reason this will only work from within the ingress or user defined overlay network and the connection will break to Kafka if you try and connect to it through one of the guest machines.
Changing the advertised IP doesn't make things any better...
docker service create --name kafka \
--mode global \
-e 'KAFKA_PORT=9092' \
-e 'KAFKA_ADVERTISED_PORT=9092' \
-e 'KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092' \
-e 'KAFKA_ZOOKEEPER_CONNECT=tasks.zookeeper:2181' \
-e 'KAFKA_LOG_DIRS=/kafka/kafka-logs' \
-e "HOSTNAME_COMMAND=curl 192.168.99.1:5000" \
--publish '9092:9092' \
wurstmeister/kafka
I think the new mesh networking and load balancing in docker might be interfering with the Kafka connection some how....
to get the host container I have a flask app running locally which I curl
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route('/')
def hello_world():
return request.remote_addr
The previous approach raise some questions:
How to specify the IDs for the zookeeper nodes?
How to specify the id of the kafka nodes, and the zookeeper nodes?
#kafka configs
echo "broker.id=${ID}
advertised.host.name=${NAME}
zookeeper.connect=${ZOOKEEPERS}" >> /opt/kafka/config/server.properties
Everything should be resolvable in the overlay network.
Moreover, in the issue Cannot create a Kafka service and publish ports due to rout mesh network there is a comment to don't use the ingress network.
I think the best option is to specify your service by using a docker compose with swarm. I'll edit the answer with an example.
There are 2 concerns to consider: networking and storage.
Since Kafka is stateful service, until cloud native storage is figured out, it is advisable to use global deployment mode. That is each swarm node satisfying constraints will have one kafka container.
Another recommendation is to use host mode for published port.
It's also important to properly set advertised listeners option so that each kafka broker knows which host it's running on. Use swarm service templates to provide real hostname automatically.
Also make sure that published port is different from target port.
kafka:
image: debezium/kafka:0.8
volumes:
- ./kafka:/kafka/data
environment:
- ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- KAFKA_MAX_MESSAGE_BYTES=20000000
- KAFKA_MESSAGE_MAX_BYTES=20000000
- KAFKA_CLEANUP_POLICY=compact
- LISTENERS=PLAINTEXT://:9092
- BROKER_ID=-1
- ADVERTISED_LISTENERS=PLAINTEXT://{{.Node.Hostname}}:11092
depends_on:
- zookeeper
deploy:
mode: global
ports:
- target: 9092
published: 11092
protocol: tcp
mode: host
networks:
- kafka
I can't explain all the options right now, but it's the configuration that works.
set broker.id=-1 in server.properties to allow kafka to auto generate the broker ID. Helpful in Swarm mode.