I am using docker-compose to start up some containers which form a cluster of Solr and Zookeeper nodes.
This is my compose file (taken from here):
version: '3.8'
services:
solr1:
image: solr:8.7.0
container_name: solr1
ports:
- "8981:8983"
environment:
- ZK_HOST=zoo1:2181,zoo2:2181,zoo3:2181
networks:
- solr
depends_on:
- zoo1
- zoo2
- zoo3
solr2:
image: solr:8.7.0
container_name: solr2
ports:
- "8982:8983"
environment:
- ZK_HOST=zoo1:2181,zoo2:2181,zoo3:2181
networks:
- solr
depends_on:
- zoo1
- zoo2
- zoo3
solr3:
image: solr:8.7.0
container_name: solr3
ports:
- "8983:8983"
environment:
- ZK_HOST=zoo1:2181,zoo2:2181,zoo3:2181
networks:
- solr
depends_on:
- zoo1
- zoo2
- zoo3
zoo1:
image: zookeeper:3.6.2
container_name: zoo1
restart: always
hostname: zoo1
ports:
- 2181:2181
- 7001:7000
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
networks:
- solr
zoo2:
image: zookeeper:3.6.2
container_name: zoo2
restart: always
hostname: zoo2
ports:
- 2182:2181
- 7002:7000
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
networks:
- solr
zoo3:
image: zookeeper:3.6.2
container_name: zoo3
restart: always
hostname: zoo3
ports:
- 2183:2181
- 7003:7000
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
networks:
- solr
networks:
solr:
If I run docker compose ls, I see that the project is correctly running:
NAME STATUS
solr-cluster running(6)
and this is the output of docker container ls:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d5333378e78c 25b74ae1e488 "docker-entrypoint.s…" 23 hours ago Up 52 minutes 0.0.0.0:8983->8983/tcp, :::8983->8983/tcp solr3
5fac27cd0443 25b74ae1e488 "docker-entrypoint.s…" 23 hours ago Up 52 minutes 0.0.0.0:8982->8983/tcp, :::8982->8983/tcp solr2
db37ef90ca98 25b74ae1e488 "docker-entrypoint.s…" 23 hours ago Up 52 minutes 0.0.0.0:8981->8983/tcp, :::8981->8983/tcp solr1
b8b55694e281 a72350516291 "/docker-entrypoint.…" 23 hours ago Up 52 minutes 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2183->2181/tcp, :::2183->2181/tcp, 0.0.0.0:7003->7000/tcp, :::7003->7000/tcp zoo3
10885eafe4e8 a72350516291 "/docker-entrypoint.…" 23 hours ago Up 52 minutes 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2182->2181/tcp, :::2182->2181/tcp, 0.0.0.0:7002->7000/tcp, :::7002->7000/tcp zoo2
558f8574c036 a72350516291 "/docker-entrypoint.…" 23 hours ago Up 52 minutes 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp, 0.0.0.0:7001->7000/tcp, :::7001->7000/tcp zoo1
If I try to navigate a solr node with my browser using host localhost (e.g. http://localhost:8983/), the container responds correctly with the Solr Admin interface.
However, if I try to access with my browser the same container but from its IP (e.g. http://172.23.0.7:8983/, where 172.23.0.7 is the IP of solr3 container) I get a connection timeout. I also get a connection timeout if I try to ping that IP.
Why does this happen? Shouldn't both localhost and 172.23.0.7 hosts work?
I need to access my container from its IP because the Solr nodes registers to Zookeeper using their IP. So, since I can't access the containers from their IP, I get a ConnectionTimeout exception when I try to connect to them programmatically through the SolrJ APIs:
public static void main(String[] args)
{
List<String> zkNodes = new ArrayList<>(3);
zkNodes.add("localhost:2181");
zkNodes.add("localhost:2182");
zkNodes.add("localhost:2183");
SolrClient solrClient = new CloudSolrClient.Builder(zkNodes, Optional.empty())
.withConnectionTimeout(10000)
.withSocketTimeout(10000)
.build();
try
{
CollectionAdminRequest.listCollections(solrClient);
}
catch (IOException | SolrServerException e)
{
e.printStackTrace();
}
}
This is the exception I get:
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://172.23.0.7:8983/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:370)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:298)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1157)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:918)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:850)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
at org.apache.solr.client.solrj.request.CollectionAdminRequest.listCollections(CollectionAdminRequest.java:2690)
at Main.main(Main.java:28)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 172.23.0.7:8983 [/172.23.0.7] failed: connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 172.23.0.7:8983 [/172.23.0.7] failed: connect timed out
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
... 11 more
Caused by: java.net.SocketTimeoutException: connect timed out
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 21 more
I don't know if it that may help, but anyway I am on a Windows host, so I cannot use the --network host flag for Docker.
The 172.x.x.x IPs are IPs only reachable within the Docker network, that is between the Solr and Zookeeper nodes.
When you connect to Zookeeper from outside, Zookeeper doesn't know there is an "outside", it only knows IP of Solr servers in its network and send them back to you. But these IPs are not reachable from "outside".
One workaround can be to connect to Solr servers directly instead of using Zookeeper (with URLs like localhost:8981 and HttpSolrClient).
Or, you need to tell Solr servers to "present" themselves with the outside IP to Zookeeper.
I think the advertised address for solar instances are the IP's of docker instances and the ports are the default ones instead of the redirected ones. You should register the docker ports instead of the solr ports in the pods.
Maybe the solrcloud part of the solr.xml can help, but not sure...
https://solr.apache.org/guide/8_8/format-of-solr-xml.html
Even though Gaël J's answer solves the problem in the general case, I'll post an answer to explain what needs to be done for Solr.
First, you need to tell Solr what is its hostname, by using the SOLR_HOST environment variable. This is the hostname which Solr will use when it registers itself to Zookeeper and it's also the hostname that Zookeeper will send to its clients when they ask for a Solr URL.
In my case, such clients were:
a SolrJ application which run on the Docker host machine (but outside from Docker);
the other Solr nodes, which may ask to Zookeeper the URL of other nodes
To make the SOLR_HOST understandable by both of these clients, I used the value of services.solrX.container_name as the SOLR_HOST, and I have added that value also to my hosts file on the Docker host machine.
So, this is the docker-compose.yml file:
version: '3.7'
services:
solr1:
image: solr:8.7.0
container_name: solr1
ports:
- "8981:8981"
environment:
- ZK_HOST=zoo1:2181
- SOLR_HOST=solr1
- SOLR_PORT=8981
networks:
- solr
depends_on:
- zoo1
solr2:
image: solr:8.7.0
container_name: solr2
ports:
- "8982:8982"
environment:
- ZK_HOST=zoo1:2181
- SOLR_HOST=solr2
- SOLR_PORT=8982
networks:
- solr
depends_on:
- zoo1
zoo1:
image: zookeeper:3.6.2
container_name: zoo1
restart: unless-stopped
hostname: zoo1
ports:
- 2181:2181
- 7001:7000
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=zoo1:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
networks:
- solr
networks:
solr:
and this is the hosts file:
# local solr cloud cluster
127.0.0.1 solr1
127.0.0.1 solr2
Note also that both solr1 and solr2 point to localhost, so they should have different ports (this is why the SOLR_PORT environment variable is needed; otherwise both nodes would use the default 8983 port).
Related
I am using docker for my sample Spark + Kafka project in windows machine.
I am facing
WARN ClientUtils: Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
[error] (run-main-0) org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
[error] org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
------
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
Below is my docker-compose.yml
version: '2'
services:
test1:
build: test1service/.
depends_on:
- kafka
test2:
build: test2/.
depends_on:
- kafka
- test1
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
extra_hosts:
- "localhost: 127.0.0.1"
kafka:
image: confluentinc/cp-kafka:5.1.0
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
#KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
extra_hosts:
- "localhost: 127.0.0.1"
Below is my sample code in test2 service
val inputStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9092")
.option("subscribe", "test1")
.option("startingOffsets", "earliest")
.load()
docker ps command output is
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c8feb49e12b test1 "/usr/bin/start.sh" About an hour ago Up 50 minutes test1
4535ce246541 test2 "/usr/bin/myservice-…" About an hour ago Up 50 minutes test2
733766f72adb confluentinc/cp-kafka:5.1.0 "/etc/confluent/dock…" About an hour ago Up 51 minutes 0.0.0.0:9092->9092/tcp kafka_1
d915e25cb226 confluentinc/cp-zookeeper:5.1.0 "/etc/confluent/dock…" About an hour ago Up 51 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper_1
Has anyone faced similar issue, how did you resolve?
You try to reach kafka:9092, but docker compose has generated the container kafka_1, which is why there is no name resolution.
Docker gives internal ip to your containers, and use their container name to create an internal DNS on this network (with the embbed dns server)
Your environment variables are not changed by Docker Compose to fit the name it gives to your container.
You should use "container_name: kafka" in your container description to get a static container name.
Only thing you need is network. After that all services will be connected and you can use kafka:9092.
version: '2'
services:
test1:
build: test1service/.
depends_on:
- kafka
networks:
- app-network
test2:
build: test2/.
depends_on:
- kafka
- test1
networks:
- app-network
zookeeper:
image: confluentinc/cp-zookeeper:5.1.0
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
extra_hosts:
- "localhost: 127.0.0.1"
networks:
- app-network
kafka:
image: confluentinc/cp-kafka:5.1.0
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
#KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
extra_hosts:
- "localhost: 127.0.0.1"
networks:
- app-network
networks:
app-network:
driver: bridge
Try with localhost and then the container port:
localhost:9092
Here is an example command for creating a topic named "orders"
docker exec kafka kafka-topics --bootstrap-server localhost:9092 --create --topic orders --partitions 1 --replication-factor 1
I want to make a NiFi cluster in docker over 3 vm's.
I found a docker-compose file that creates a cluster on one node and try to edit this file.
I found out that i need zookeeper, but do i need one zookeeper each instance? and what ports should i open or map in docker?
the docker-compose file i found:
version: "3"
services:
zookeeper:
hostname: zookeeper
container_name: zookeeper
image: zookeeper:3.6.1
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
nifi:
image: apache/nifi:1.11.4
ports:
- 8080 # Unsecured HTTP Web Port
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ELECTION_MAX_WAIT=1 min
I changed the file like this (on each VM the ip is correct )
version: "3"
services:
zookeeper:
hostname: zookeeper
container_name: zookeeper
image: 'zookeeper:3.6.1'
ports:
- 2181
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
nifi:
image: apache/nifi:1.11.4
ports:
- 8080 # Unsecured HTTP Web Port
- 8082
- 9001
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
# - NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ZK_CONNECT_STRING=192.168.2.10:2181,192.168.2.20:2181,192.168.2.30:2181
- NIFI_ELECTION_MAX_WAIT=1 min
- NIFI_CLUSTER_ADDRESS=192.168.2.XX
and in the logs i found this message but cant find any solution
ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I found out that i need zookeeper, but do i need one zookeeper each instance?
No, you can use one zookeeper
and what ports should i open or map in docker?
as I know you need 2888, 3888, 2181 to be open. But only 2181 to communicate with nifi
2888 and 3888 for zookeeper cluster communication.
I installed Kafka on a VM Ubuntu 18.0.4 with following compose file
version: '2'
networks:
kafka-net:
driver: bridge
services:
zookeeper-server:
image: 'bitnami/zookeeper:latest'
networks:
- kafka-net
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka-server1:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9092:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
kafka-server2:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9093:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9093
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
It installed without any problem.
sudo docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39f38caf57cb bitnami/kafka:latest "/entrypoint.sh /run…" 3 hours ago Up 5 minutes 0.0.0.0:9092->9092/tcp kafka_kafka-server1_1
088a703b5b76 bitnami/kafka:latest "/entrypoint.sh /run…" 3 hours ago Up 3 hours 0.0.0.0:9093->9092/tcp kafka_kafka-server2_1
6a754bda47ea bitnami/zookeeper:latest "/entrypoint.sh /run…" 3 hours ago Up 3 hours 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp kafka_zookeeper-server_1
Now, I want to connect to my Kafka on my VM with the following setting:
I test it from localhost with the following
root#ubuntu:~# kafkacat -b 192.168.179.133:9092 -L
Metadata for all topics (from broker -1: 192.168.179.133:9092/bootstrap):
1 brokers:
broker 1001 at localhost:9092
0 topics:
But in my windows 10 I can not connect to 192.168.179.133:9092 with Conduktor
As you see it returns error.
Test ZK is OK but Test kafka Connectivity raise the error !
You should change KAFKA_CFG_ADVERTISED_LISTENERS if your conductor is not installed in the same machine as Kafka cluster installed.
It should be like this for kafka-server1:
KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.179.33:9092
and kafka-server2:
KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.179.33:9093
Note: You should consider to add both kafka servers in conductor for redundancy.
You can check this for more information.
I've tried the 'with docker' docs here but it's not working from the localhost:7000, localhost:8081, or any other port I use. What am I missing?
REDIS_PORT=6379
### Redis ################################################
redis:
container_name: redis
hostname: redis
build: ./redis
volumes:
- ${DATA_PATH_HOST}/redis:/data
ports:
- "${REDIS_PORT}:6379"
networks:
- backend
### REDISCOMMANDER ################################################
redis-commander:
container_name: rediscommander
hostname: redis-commander
image: rediscommander/redis-commander:latest
restart: always
environment:
- REDIS_HOSTS=local:redis:6379
ports:
- "7000:80"
networks:
- frontend
- backend
depends_on:
- redis
Docker ps gives me:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
042c9a2e918a rediscommander/redis-commander:latest "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8081/tcp, 0.0.0.0:7000->80/tcp rediscommander
86bc8c1ca5ff laradock_redis "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:6379->6379/tcp redis
Docker logs rediscommander gives me:
$ docker logs rediscommander
Creating custom redis-commander config '/redis-commander/config/local-production.json'.
Parsing 1 REDIS_HOSTS into custom redis-commander config '/redis-commander/config/local-production.json'.
node ./bin/redis-commander
Using scan instead of keys
No Save: false
listening on 0.0.0.0:8081
access with browser at http://127.0.0.1:8081
Redis Connection redis:6379 using Redis DB #0
Redis commader is listening on port 8081 in container. That is why you should change port binding to
ports:
- "7000:8081"
in redis commander block and access it via localhost:7000.
From docker-compose I got this yml:
version: '2'
services:
zookeeper:
container_name: zookeeper
image: confluentinc/cp-zookeeper:3.1.1
ports:
- "2080:2080"
environment:
- ZOOKEEPER_CLIENT_PORT=2080
- ZOOKEEPER_TICK_TIME=2000
kafka:
container_name: kafka
image: confluentinc/cp-kafka:3.1.1
ports:
- "9092:9092"
environment:
- KAFKA_CREATE_TOPICS=Topic1:1
- KAFKA_ZOOKEEPER_CONNECT=192.168.99.100:2080
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.99.100:9092
depends_on:
- zookeeper
schema-registry:
container_name: schema-registry
image: confluentinc/cp-schema-registry:3.1.1
ports:
- "8081:8081"
environment:
- SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=192.168.99.100:2080
- SCHEMA_REGISTRY_HOST_NAME=localhost
depends_on:
- zookeeper
- kafka
When I stand up this docker the console output ends with:
schema-registry | Error while running kafka-ready.
schema-registry | org.apache.kafka.common.errors.TimeoutException: Timed out waiting for Kafka to create /brokers/ids in Zookeeper. timeout (ms) = 40000
schema-registry exited with code 1
It seems like kafka never connect Zookeper or something like that, does anyone knows why this is happening?
Does changing
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=192.168.99.100:2080
into
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2080
help?
Additionally, KAFKA_ZOOKEEPER_CONNECT=192.168.99.100:2080 should mention zookeeper as well, instead of an IP address. Or, how can you be sure of that IP address?
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.99.100:9092 mentions an IP address you might not be able to guarantee as well. For the latter, that IP address could be changed into kafka.
I also had challenges in getting Kafka and Zookeeper to work in Docker (via Docker Compose). In the end, https://github.com/confluentinc/cp-docker-images/blob/5.0.0-post/examples/kafka-single-node/docker-compose.yml worked for me. You could use that as a source of inspiration.