Force ephemeral port binding in docker-compose - docker

I'm trying to use kafka-docker: https://github.com/wurstmeister/kafka-docker. Docker sets a new port for the kafka brokers every single time I start up the service again. Meaning, the 32781, 32782, and 32783 changes every time.
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c9e2188366cb wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 5 minutes ago Up 5 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp kafka-docker_zookeeper_1
d3db1c6b39aa kafka-docker_kafka "start-kafka.sh" 5 minutes ago Up 5 minutes 0.0.0.0:32781->9092/tcp kafka-docker_kafka_3
3ea8951fef31 kafka-docker_kafka "start-kafka.sh" 5 minutes ago Up 5 minutes 0.0.0.0:32782->9092/tcp kafka-docker_kafka_2
6bf090978b05 kafka-docker_kafka "start-kafka.sh" 5 minutes ago Up 5 minutes 0.0.0.0:32783->9092/tcp kafka-docker_kafka_1
I understand why that needs to happen (thanks to this awesome wiki: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity).
However, is there a way to hack it so that the ephemeral ports that docker compose chooses starts at the same number? For example, if it ALWAYS started with 32781, my life would be a lot easier.
Thanks

If you specify a range of port for kafka container, for example, 32781-32790, it will assign port from the range. I can't guarantee it always will start with 32781, as I didn't find any confirmation regarding it in the documentation, but in my past experience it always starts with the first port in the port range.
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "32781-32790:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock

Related

Why does my docker-compose port config work?

I'm in the process of creating a docker-compose config which maintains:
a node.js server, and
a separate postgres server.
Tutorials emphasise that postgres port 5432 must be exposed or forwarded so that the node container can access it: facilitated in the below docker-compose.yml.
version: "3.7"
services:
db:
container_name: db
image: postgres:alpine
ports:
- "5010:5432"
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: verysecretpass
POSTGRES_DB: pg-dev
server:
container_name: dashboard-api
build: .
volumes:
- .:/server
ports:
- "5000:5000"
This produces the below docker ps output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0d790cd4929e server_server "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:5000->5000/tcp dashboard-api
818296c1fc02 postgres:alpine "docker-entrypoint.s…" 7 minutes ago Up 4 minutes 0.0.0.0:5010->5432/tcp pg
In the above state, node gets ECONN REFUSED when attempting to connect with this url: postgres://postgres:verysecretpass#db:5010/pg-dev
Yet, the same connection string can connect when using 5432 instead of 5010.
In fact, using 5432, connection succeeds even when pg container has no port configuration whatsoever. The below docker ps output reflects no-port-config state in which node container can happily connect:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
76da96c15c05 server_server "docker-entrypoint.s…" 7 seconds ago Up 7 seconds 0.0.0.0:5000->5000/tcp dashboard-api
51c221ac2c54 postgres:alpine "docker-entrypoint.s…" 8 seconds ago Up 7 seconds 5432/tcp db
Why does this work? What am I missing here?
Using:
Docker version 20.10.0, build 7287ab3
docker-compose version 1.27.4, build 40524192
Unless otherwise configured, the services in a docker-compose document are automatically added to a network. There's no need to expose ports in this network.
If you want to expose ports on a container to the outside world, you will need to explicitly map these as you did. This however does not change anything for communication between services in the same network. If you have no reason to access the database from outside the network (e.g. inspect data using a DB tool on your own machine), you don't have to map / expose any ports of the db container.

Unable to create Kafka topics with Kafka and Zookeeper running on Docker

I have Kafka and Zookeeper running on two separate Docker containers:
<private-domain>/wurstmeister-kafka:0.10.1.0-2
<private-domain>/wurstmeister-zookeeper:3.4.9
Both containers seem to be up, but when I try to create Kafka topics by getting in to the first container:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
I get this error:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2020-06-07 03:10:55,293] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
Please notice that I did read other related questions and tried adding arguments to the command, such as -e ZK_HOSTS="localhost:2181". I know of other people working in the environment as mine who were able to run the commands successfully, so I suspect this might be a configuration issue on my side. Can you please guide?
EDIT: Here are the Docker Compose files:
version: '2'
services:
kafka:
image: <private-domain>/wurstmeister-kafka:0.10.1.0-2
container_name: kafka
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: 127.0.0.1:2181
restart:
"unless-stopped"
and
version: '2'
services:
zk:
image: <private-domain>/wurstmeister-zookeeper:3.4.9
container_name: zk
ports:
- "2181:2181"
restart:
"unless-stopped"
and the output of docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bf67a49da57a wurstmeister-kafka:0.10.1.0-2 "start-kafka.sh" 5 months ago Up 29 minutes 0.0.0.0:9092->9092/tcp kafka
ef3e908d82b3 wurstmeister-zookeeper:3.4.9 "/bin/sh -c '/usr/sbin/sshd && bash /usr/bin/start-zk.sh'" 5 months ago Up 29 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp zk
You have two Compose files. Thus, Your containers are on separated networks, and cannot refer each other.
You must add both services in one file, under one services: block, and run only one docker-compose up command
You can find working compose files here across the internet, or you could use minikube / oc with Kafka Helm Charts or Operators, which is how the large companies are testing Kafka in containers.

Trying to get my apache container to respond to localhost:8080

I have a bunch of containers running on my ubuntu laptop.As you can see below, I should be able to type in localhost:8080 in the browser and get some response. Instead I get
ERR_CONNECTION_REFUSED
Any idea why?
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
143fbe3867e3 richb201/apache-with-code:latest "/app-entrypoint.sh …" 19 hours ago Up 19 hours 8080/tcp, 0.0.0.0:8000->8000/tcp, 8443/tcp sub_crud_webserver_1
0c2048ef273e bitnami/mariadb:latest "/entrypoint.sh /run…" 21 hours ago Up 21 hours 3306/tcp sub_crud_mariadb_1
fb30f99f037b bitnami/php-fpm:latest "php-fpm -F --pid /o…" 2 days ago Up 19 hours 9000/tcp html_php-fpm_1
be5155202f43 bitnami/mariadb:latest "/entrypoint.sh /run…" 2 days ago Up 19 hours 3306/tcp html_mariadb_1
docker-compose.yml
version: '3'
services:
webserver:
image: richb201/apache-with-code:latest
ports:
- '8000:8000'
volumes:
- /sub_crud:/var/www/html/sub_crud
environment:
XDEBUG_CONFIG: remote_host='richb201-XPS-13-9370'
mariadb:
image: bitnami/mariadb:latest
environment:
- ALLOW_EMPTY_PASSWORD=yes
volumes:
- mariadb-data:/bitnami
php-fpm:
image: bitnami/php-fpm:latest
volumes:
- /sub_crud:/var/www/html/sub_crud
volumes:
According to your docker-compose that you mentioned in a comment (better put this into your question though) you map port 8000 to port 8000. Hence, any requests going to port 8000 on your local machine are received from port 8000 within your container.
If you want to use port 8080 you have to adapt the docker-compose file accordingly, i.e., 8080:8080. Note here that I assume that the apache server in your richb201/apache-with-code image is running on port 8080.

Unexpected unmapped port 80 in docker container

when i run docker ps i am seeing port 80 unmapped on the apphub-ui container, i am expecting it to be mapped to 4200, which i also see. I dont understand why 80/tcp is present, it is creating connection refused issues in my app.
Here is the docker ps output
docker ps 10s +59%
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
22cb2b9c28d4 swimlane/apphub-ui:latest "/usr/bin/reefer -t …" 20 seconds ago Up 17 seconds 80/tcp, 0.0.0.0:80->4200/tcp apphub_apphub-ui_1
19a78f93bfe7 swimlane/apphub-api:latest "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3000->3000/tcp apphub_apphub-api_1
7b3478e2dcd5 mongo:3.6 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:27017->27017/tcp apphub_mongo_1
My docker-compose file is as follows
version: '3.6'
volumes:
mongo:
services:
mongo:
image: mongo:3.6
ports:
- 27017:27017
volumes:
- mongo:/data/db
apphub-api:
image: swimlane/apphub-api:latest
ports:
- 3000:3000
depends_on:
- mongo
apphub-ui:
image: swimlane/apphub-ui:latest
ports:
- 80:4200
depends_on:
- apphub-api
environment:
- APPHUB_API_HOST=apphub-api
My app runs fine if i change the mapped ports for the ui feature to 4200:4200, and i dont see the stray unmapped 80
I'm having the feeling that you want to map the port 4200 on your host to the port 80 of your container. If that's the case, you should switch the values. i.e.
ports:
- 4200:80

Why can't I connect to my local docker-compose container on Windows 10?

I'm trying to dockerize a Python application, for which I've been following this tutorial. The tutorial is from April 2015 and still uses Docker Machine, which, judging from this answer, is no longer necessary to run Docker containers locally on Windows.
I got it working with Docker Machine before, and was able to see the web app and interact with it. But now I'm trying to get this working without Docker Machine, with Docker version 17.06.0-ce, build 02c1d87, on Windows 10.
Here's the docker-compose.yml:
web:
restart: always
build: ./web
expose:
- "8000"
links:
- postgres:postgres
volumes:
- /usr/src/app/static
env_file: .env
command: /usr/local/bin/gunicorn -w 2 -b :8000 app:app
nginx:
restart: always
build: ./nginx/
ports:
- "80:80"
volumes:
- /www/static
volumes_from:
- web
links:
- web:web
data:
image: postgres:latest
volumes:
- /var/lib/postgresql
command: "true"
postgres:
restart: always
image: postgres:latest
volumes_from:
- data
ports:
- "5432:5432"
I started the containers:
$ docker-compose up -d
Creating polly_data_1 ...
Creating polly_data_1 ... done
Creating polly_postgres_1 ...
Creating polly_postgres_1 ... done
Creating polly_web_1 ...
Creating polly_web_1 ... done
Creating polly_nginx_1 ...
Creating polly_nginx_1 ... done
Then, when I run docker ps, it shows the following three containers running:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b2c1048f3a5 polly_nginx "/usr/sbin/nginx" 4 seconds ago Up 3 seconds 0.0.0.0:80->80/tcp polly_nginx_1
d561ac5b901a polly_web "/usr/local/bin/gu..." 5 seconds ago Up 4 seconds 8000/tcp polly_web_1
ecb029d6ec3a postgres:latest "docker-entrypoint..." 7 seconds ago Up 5 seconds 0.0.0.0:5432->5432/tcp polly_postgres_1
(At this point, navigating to http://localhost:8000/ in Chrome already yields ERR_CONNECTION_REFUSED.)
I then ran the script to set up the database, as per the tutorial (extra //s because I'm using Git Bash on Windows 10):
$ docker-compose run web ///usr/local/bin/python create_db.py
Starting polly_data_1 ...
Starting polly_data_1 ... done
Starting polly_postgres_1 ... done
Now when I run docker ps, it shows the following four containers running:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a129c12f5982 polly_web "//usr/local/bin/p..." 5 seconds ago Up Less than a second 8000/tcp polly_web_run_1
9b2c1048f3a5 polly_nginx "/usr/sbin/nginx" 16 seconds ago Up 15 seconds 0.0.0.0:80->80/tcp polly_nginx_1
d561ac5b901a polly_web "/usr/local/bin/gu..." 17 seconds ago Up 16 seconds 8000/tcp polly_web_1
ecb029d6ec3a postgres:latest "docker-entrypoint..." 19 seconds ago Up 17 seconds 0.0.0.0:5432->5432/tcp polly_postgres_1
And localhost:8000 is still refusing to connect. The web container exposes port 8000, so I don't get why I can't connect to it.
How can I get this working so I can access the web app in the web container locally?
Just change:
expose:
- "8000"
By
ports:
- "8000:8000"
Btw http://localhost:80 is not working?
Regards
Turns out, as suggested by Carlos and 200_OK as part of their answers and comments, it was working as intended - it was running at port 80, not 8000.
Web exposes port 8000 internally inside the container. But that port is not mapped to your host machine port.
I think the problem is in your command. The option is -p, not -b.
web:
restart: always
build: ./web
expose:
- "8000"
links:
- postgres:postgres
volumes:
- /usr/src/app/static
env_file: .env
command: /usr/local/bin/gunicorn -w 2 -p :8000 app:app

Resources