Docker - Cannot talk to Postgres - docker

I am using docker-compose to orchestrate two services but I am getting an error for one of those. Upon following the logs, this is my output:
2023/02/01 08:34:35 [INFO] version.go:13 versionPrint(): starting Commento
2023/02/01 08:34:35 [INFO] database_connect.go:20 dbConnect(): opening connection to postgres: postgres://postgres:redacted#db:5432/commento?sslmode=disable
2023/02/01 08:34:35 [ERROR] database_connect.go:31 dbConnect(): cannot talk to postgres, retrying in 10 seconds (4 attempts left): dial tcp 172.18.0.2:5432: connect: connection refused
2023/02/01 08:34:45 [INFO] database_connect.go:20 dbConnect(): opening connection to postgres: postgres://postgres:redacted#db:5432/commento?sslmode=disable
2023/02/01 08:34:45 [ERROR] database_connect.go:31 dbConnect(): cannot talk to postgres, retrying in 10 seconds (3 attempts left): pq: unknown authentication response: 10
2023/02/01 08:34:55 [INFO] database_connect.go:20 dbConnect(): opening connection to postgres: postgres://postgres:redacted#db:5432/commento?sslmode=disable
2023/02/01 08:34:55 [ERROR] database_connect.go:31 dbConnect(): cannot talk to postgres, retrying in 10 seconds (2 attempts left): pq: unknown authentication response: 10
2023/02/01 08:35:05 [INFO] database_connect.go:20 dbConnect(): opening connection to postgres: postgres://postgres:redacted#db:5432/commento?sslmode=disable
2023/02/01 08:35:06 [ERROR] database_connect.go:31 dbConnect(): cannot talk to postgres, retrying in 10 seconds (1 attempts left): pq: unknown authentication response: 10
2023/02/01 08:35:16 [INFO] database_connect.go:20 dbConnect(): opening connection to postgres: postgres://postgres:redacted#db:5432/commento?sslmode=disable
2023/02/01 08:35:16 [ERROR] database_connect.go:31 dbConnect(): cannot talk to postgres, retrying in 10 seconds (0 attempts left): pq: unknown authentication response: 10
2023/02/01 08:35:26 [INFO] database_connect.go:20 dbConnect(): opening connection to postgres: postgres://postgres:redacted#db:5432/commento?sslmode=disable
2023/02/01 08:35:26 [ERROR] database_connect.go:35 dbConnect(): cannot talk to postgres, last attempt failed: pq: unknown authentication response: 10
fatal error: pq: unknown authentication response: 10
I haven't tried any solutions yet because I don't know where to start. How do I fix this?
Steps to reproduce:
On a clean Ubuntu 22.04LTS Server, installed docker and docker-compose.
Created a directory for the service and used the following configuration for docker-compose.yml :
version: '3'
services:
server:
image: registry.gitlab.com/commento/commento:SET_VERSION
ports:
- 8080:8080
environment:
COMMENTO_ORIGIN: http://commento.example.com:8080
COMMENTO_PORT: 8080
COMMENTO_POSTGRES: postgres://postgres:postgres#db:5432/commento?sslmode=disable
depends_on:
- db
db:
image: postgres
environment:
POSTGRES_DB: commento
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
volumes:
- postgres_data_volume:/var/lib/postgresql/data
volumes:
postgres_data_volume:
Ran docker-compose up -d and after that I followed the logs for both the services. The service "commento_server_1" produced the output mentioned above.

First, using registry.gitlab.com/commento/commento:SET_VERSION leads to the bellow error:
Pulling server (registry.gitlab.com/commento/commento:SET_VERSION)...
ERROR: manifest for registry.gitlab.com/commento/commento:SET_VERSION not found: manifest unknown: manifest unknown
because there is no release SET_VERSION of commento.
So you must have done something different. Please, next time you ask a question, make sure the instructions you provide to reproduce your issue are minimal and complete. See https://stackoverflow.com/help/minimal-reproducible-example
Anyway, I was able to reproduce your issue switching from commento:SET_VERSION to commento:latest. The issue you are facing is due to the fact that the latest release of commento (at the time of writing) doesn't support the latest release of postgres (at the time of writing).
The solution is to downgrade the postgres version to 13:
version: '3'
services:
db:
image: postgres:13
(keep other lines of your docker-compose.yml as they are).
Also, as future versions of commento may no longer support postgres 13. So when writing a docker-compose.yml, I would recommend always specifying the versions of all services:
services:
server:
image: registry.gitlab.com/commento/commento:v1.4.0

Related

Running ELK on docker, Kibana says: Unable to retrieve version information from Elasticsearch nodes

I was referring to example given in the elasticsearch documentation for starting elastic stack (elastic and kibana) on docker using docker compose. It gives example of docker compose version 2.2 file. So, I tried to convert it to docker compose version 3.8 file. Also, it creates three elastic nodes and has security enabled. I want to keep it minimal to start with. So I tried to turn off security and also reduce the number of elastic nodes to 2. This is how my current compose file looks like:
version: "3.8"
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-amd64
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
environment:
- node.name=es01
- cluster.name=docker-cluster
- cluster.initial_master_nodes=es01
- bootstrap.memory_lock=true
- xpack.security.enabled=false
deploy:
resources:
limits:
memory: 1g
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
# [
# "CMD-SHELL",
# # "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
# ]
# Changed to:
test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
- es01
image: docker.elastic.co/kibana/kibana:8.0.0-amd64
volumes:
- kibanadata:/usr/share/kibana/data
ports:
- 5601:5601
environment:
- SERVERNAME=kibana
- ELASTICSEARCH_HOSTS=https://localhost:9200
deploy:
resources:
limits:
memory: 1g
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
volumes:
esdata01:
driver: local
kibanadata:
driver: local
Then, I tried to run it:
docker stack deploy -c docker-compose.nosec.noenv.yml elk
Creating network elk_default
Creating service elk_es01
Creating service elk_kibana
When I tried to check their status, it displayed following:
$ docker container list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3dcd08134e38 docker.elastic.co/kibana/kibana:8.0.0-amd64 "/bin/tini -- /usr/l…" 3 minutes ago Up 3 minutes (health: starting) 5601/tcp elk_kibana.1.ng8aspz9krfnejfpsnqzl2sci
7b548a43c45c docker.elastic.co/elasticsearch/elasticsearch:8.0.0-amd64 "/bin/tini -- /usr/l…" 3 minutes ago Up 3 minutes (healthy) 9200/tcp, 9300/tcp elk_es01.1.d9a107j6wkz42shti3n6kpfmx
I noticed that kibana's status gets stuck at (health: starting). When I checked Kibana's logs with command docker service logs -f elk_kibana, it had following WARN and ERROR lines:
[WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200
It seems that kibana is not able to connect with Elasticsearch, but why? Is it because of disabling of security and that we cannot have security disabled?
PS-1: Earlier, when I set elasticsearch host as follows in kibana's environment in the docker compose file:
ELASTICSEARCH_HOSTS=https://es01:9200 # that is 'es01' instead of `localhost`
it gave me following error:
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND es01
So, after checking this question, I changed es01 to localhost as specified earlier (that is in complete docker compose file content before PS-1.)
PS-2: Replacing localhost with 192.168.0.104 gives following error
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 192.168.0.104:9200
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. write EPROTO 140274197346240:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
Try this :
ELASTICSEARCH_HOSTS=http://es01:9200
I don't know why it can run in my PC, since Elasticsearch is supossed use SSL. But in your case using http working just fine.

docker-compose up is failing to start one of the container service

I have defined 3 services in the docker-compose.yaml file. Out of which 2 services (my-app_my-app_1 & my-app_mongodb_1) are getting started automatically when firing this command docker-compose -f docker-compose.yaml up. But failing to start one of the service (my-app_mongo-express_1). Just to add, I can start the failed container successfully again by executing docker start my-app_mongo-express_1 separately.
Contents of file - docker-compose.yaml:
→ cat docker-compose.yaml
version: '3'
services:
my-app:
image: maryo/my-app:1.2
ports:
- 3000:3000
mongodb:
image: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
volumes:
- mongo-data:/data/db
mongo-express:
image: mongo-express
ports:
- 8080:8081
environment:
- ME_CONFIG_MONGODB_ADMINUSERNAME=admin
- ME_CONFIG_MONGODB_ADMINPASSWORD=password
- ME_CONFIG_MONGODB_SERVER=mongodb
volumes:
mongo-data:
driver: local
Output of docker-compose ps
→ docker-compose ps
NAME COMMAND SERVICE STATUS PORTS
my-app_mongo-express_1 "tini -- /docker-ent…" mongo-express exited (0)
my-app_mongodb_1 "docker-entrypoint.s…" mongodb running 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp
my-app_my-app_1 "docker-entrypoint.s…" my-app running 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
Docker logs for mongo-express container:
→ docker logs my-app_mongo-express_1
Welcome to mongo-express
------------------------
(node:8) [MONGODB DRIVER] Warning: Current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Could not connect to database using connectionString: mongodb://admin:password#mongodb:27017/"
(node:8) UnhandledPromiseRejectionWarning: MongoNetworkError: failed to connect to server [mongodb:27017] on first connect [Error: connect ECONNREFUSED 172.25.0.4:27017
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16) {
name: 'MongoNetworkError'
}]
at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:441:11)
at Pool.emit (events.js:314:20)
at /node_modules/mongodb/lib/core/connection/pool.js:564:14
at /node_modules/mongodb/lib/core/connection/pool.js:1000:11
at /node_modules/mongodb/lib/core/connection/connect.js:32:7
at callback (/node_modules/mongodb/lib/core/connection/connect.js:289:5)
at Socket.<anonymous> (/node_modules/mongodb/lib/core/connection/connect.js:319:7)
at Object.onceWrapper (events.js:421:26)
at Socket.emit (events.js:314:20)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
(node:8) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:8) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Works fine if I start that container separately:
→ docker start my-app_mongo-express_1
my-app_mongo-express_1
→ docker-compose ps
NAME COMMAND SERVICE STATUS PORTS
my-app_mongo-express_1 "tini -- /docker-ent…" mongo-express running 0.0.0.0:8080->8081/tcp, :::8080->8081/tcp
my-app_mongodb_1 "docker-entrypoint.s…" mongodb running 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp
my-app_my-app_1 "docker-entrypoint.s…" my-app running 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
What am I missing? Why I am not able to start all the containers together using docker-compose up?
You can use the depends_on option to control the order in which your defined services startup.
In this specific case, the mongo-express service has a dependency on the mongodb service, and so if the mongo-express service is started before mongodb, it will fail to connect:
Could not connect to database using connectionString
This is why starting the mongo-express service manually succeeds (because mongodb is already running). However, note the following caveat from the documentation which you may still need to address:
However, for startup Compose does not wait until a container is “ready” (whatever that means for your particular application) - only until it’s running . . . To handle this, design your application to attempt to re-establish a connection to the database after a failure. If the application retries the connection, it can eventually connect to the database.

Hyperledger Fabric - Unable to Start Network with Four Kafka and Three Zookeeper Ensemble

I'm trying to setup a network of 2 organizations each having two peers. A 3rd organisation having 2 orderer nodes with kakfa-zookeeper ensemble with 4 kafka and 3 zookeeper nodes.
Below is the relevant part of my crypto-config.yaml file:
OrdererOrgs:
- Name: Orderer
Domain: ordererOrg.example.com
Template:
Count: 2
Below is the relevant part of my configtx.yaml file:
- &OrdererOrg
Name: OrdererOrg
ID: OrdererMSP
MSPDir: crypto-config/ordererOrganizations/ordererOrg.example.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Writers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Admins:
Type: Signature
Rule: "OR('OrdererMSP.admin')"
.................
Orderer: &OrdererDefaults
OrdererType: kafka
Addresses:
- orderer0.ordererOrg.example.com:7050
- orderer1.ordererOrg.example.com:7040
BatchTimeout: 2s
BatchSize:
MaxMessageCount: 10
AbsoluteMaxBytes: 99 MB
PreferredMaxBytes: 512 KB
Kafka:
Brokers:
- kafka0.ordererOrg.example.com:9092
- kafka1.ordererOrg.example.com:9092
- kafka2.ordererOrg.example.com:9092
- kafka3.ordererOrg.example.com:9092
...............
Below is the relevant part of my Docker base file:
zookeeper:
image: hyperledger/fabric-zookeeper
environment:
- ZOO_SERVERS=server.1=zookeeper0.ordererOrg.example.com:2888:3888 server.2=zookeeper1.ordererOrg.example.com:2888:3888 server.3=zookeeper2.ordererOrg.example.com:2888:3888
restart: always
kafka:
image: hyperledger/fabric-kafka
restart: always
environment:
- KAFKA_MESSAGE_MAX_BYTES=103809024 # 99 * 1024 * 1024 B
- KAFKA_REPLICA_FETCH_MAX_BYTES=103809024 # 99 * 1024 * 1024 B
- KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
- KAFKA_MIN_INSYNC_REPLICAS=2
- KAFKA_DEFAULT_REPLICATION_FACTOR=3
- KAFKA_ZOOKEEPER_CONNECT=zookeeper0.ordererOrg.example.com:2181,zookeeper1.ordererOrg.example.com:2181,zookeeper2.ordererOrg.example.com:2181
Below is the relevant part of my Docker Compose file:
zookeeper0.ordererOrg. example.com:
container_name: zookeeper0.ordererOrg.example.com
extends:
file: base/kafka-base.yaml
service: zookeeper
environment:
- ZOO_MY_ID=1
ports:
- '2181:2181'
- '2888:2888'
- '3888:3888'
networks:
- byfn
kafka0.ordererOrg.example.com:
container_name: kafka0.ordererOrgvodworks.example.com
extends:
file: base/kafka-base.yaml
service: kafka
depends_on:
- zookeeper0.ordererOrg.example.com
- zookeeper1.ordererOrg.example.com
- zookeeper2.ordererOrg.example.com
environment:
- KAFKA_BROKER_ID=0
ports:
- '9092:9092'
- '9093:9093'
networks:
- byfn
-----------------------
Note: The same structure is being followed for:
- zookeeper1.ordererOrg. example.com
- zookeeper2.ordererOrg. example.com
And
- kafka1.ordererOrg.example.com
- kafka2.ordererOrg.example.com
- kafka3.ordererOrg.example.com
When I run the network start command I get the following error messages:
✖ Starting business network definition. This may take a minute...
Error: Error trying to start business network. Error: No valid
responses from any peers. Response from attempted peer comms was an
error: Error: REQUEST_TIMEOUT
And when I run the same network start command again, I get the following:
✖ Starting business network definition. This may take a minute...
Error: Error trying to start business network. Error: No valid
responses from any peers. Response from attempted peer comms was an
error: Error: chaincode registration failed: timeout expired while
starting chaincode tt_poc:0.0.1 for transaction
And images files are also not being created against the chaincode (BNA file) as you can see the ccenv containers and orderer logs in the image below:
And I get the following logs as well on console after peer channel create command, though channel gets created successfully:
2019-03-25 15:20:34.567 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and rderer connections initialized
2019-03-25 15:20:34.956 UTC [cli.common] readBlock -> INFO 002 Got status: &{SERVICE_UNAVAILABLE}
I tried to provide maximum information but still please let me know if you require logs of any other container as well. Thanks for your time.
I finally able to resolve this issue. There was nothing wrong with these YAML configurations. The issue was with the docker configurations that It was lacking in resources and the strange thing is that I didn't get any resources related error in any container logs file. So, I just increased CPUs and Memory settings in the docker advanced configurations like below:
And after these configurational changes, my network started successfully and working properly.
Thanks to my colleague #Rafiq who help me in sorting out this issue.

Docker/zookeeper Will not attempt to authenticate using SASL

Good Day,
I wanted to test the config store which is built using spring boot. The instruction given to me is run the project using docker-compose.yml files. I'm new with this,I've tired to execute but while running those commands on iMAC terminal I'm facing the following exception.
platform-config-store | 2018-03-05 11:55:12.167 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#22bbbe6
platform-config-store | 2018-03-05 11:55:12.286 INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
platform-config-store | 2018-03-05 11:55:12.314 WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
platform-config-store | java.net.ConnectException: Connection refused
platform-config-store | at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_144]
platform-config-store | at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_144]
platform-config-store | at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
platform-config-store | at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
platform-config-store |
platform-config-store | 2018-03-05 11:55:13.422 INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
platform-config-store | 2018-03-05 11:55:13.424 WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
I've googled this problem and on some posts it was mentioned that zookeeper client server is not available that's why this error is occurring. So for this I've configured the zookeeper local instance on my machine and made changes in docker-compose.yml file. Instead of getting the image from docker, I tried to get it from local machine. It didn't work and faced the same issue.
Also some of them posted that this related to the firewall. I've verified and firewall's turned off.
Following is the docker-compose file I'm executing.
docker-compose.yml
version: "3.0"
services:
zookeeper:
container_name: zookeeper
image: docker.*****.net/zookeeper
#image: zookeeper // tired to connect with local zookeeper instance
ports:
- 2181:2181
postgres:
container_name: postgres
image: postgres
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=p3rmission
redis:
container_name: redis
image: redis
ports:
- 6379:6379
Could anyone please guide me, what I'm missing here. Help will be appreciated. Thanks

Dockerized Spring Cloud Stream services with Kafka broker unable to connect to Zookeeper

I'm testing a sample spring cloud stream application (running on a Ubuntu linux machine) with one source and one sink services. All my services are docker-containerized and I would like to use kafka as message broker.
Below the relevant parts of the docker-compose.yml:
zookeeper:
image: confluent/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.9.0.0-1
container_name: kafka
ports:
- "9092:9092"
links:
- zookeeper:zk
environment:
- KAFKA_ADVERTISED_HOST_NAME=192.168.33.101
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_DELETE_TOPIC_ENABLE=true
- KAFKA_LOG_RETENTION_HOURS=1
- KAFKA_MESSAGE_MAX_BYTES=10000000
- KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
- KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
- KAFKA_NUM_PARTITIONS=2
- KAFKA_DELETE_RETENTION_MS=1000
.
.
.
# not shown: eureka service registry, spring cloud config service, etc.
myapp-service-test-source:
container_name: myapp-service-test-source
image: myapp-h2020/myapp-service-test-source:0.0.1
environment:
SERVICE_REGISTRY_HOST: 192.168.33.101
SERVICE_REGISTRY_PORT: 8761
ports:
- 8081:8080
.
.
.
Here the relevant part of application.yml for my service-test-source service:
spring:
cloud:
stream:
defaultBinder: kafka
bindings:
output:
destination: messages
content-type: application/json
kafka:
binder:
brokers: ${SERVICE_REGISTRY_HOST:192.168.33.101}
zkNodes: ${SERVICE_REGISTRY_HOST:192.168.33.101}
defaultZkPort: 2181
defaultBrokerPort: 9092
The problem is the following, if I launch the docker-compose above, in the test-source container log I notice that the service fails to connect to zookeeper, giving a repeated set of Connection refused error, and finishing with a ZkTimeoutException which makes the service terminate (see below).
The strange fact is that, if instead of running my source (and sink) test services as docker containers I run them as jar files via maven mvn spring-boot:run <etc...> the services work fine and are able to exchange messages via kafka. (note that kafka, zookeeper, etc. are still running as docker containers).
.
.
.
*** THE FOLLOWING REPEATED n TIMES ***
2017-02-14 14:40:09.164 INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-14 14:40:09.166 WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
.
.
.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
Any idea what the problem might be?
edit:
I discovered that in the "jar" execution logs the test-source service tries to connect to zookeeper through the IP 127.0.0.1, as can be seen from the log snipped below:
2017-02-15 14:24:04.159 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:04.159 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:04.178 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-02-15 14:24:04.201 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15a421fd9ec000a, negotiated timeout = 10000
2017-02-15 14:24:05.870 INFO 10348 --- [ main] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#72ba68e3
2017-02-15 14:24:05.882 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:05.883 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
This explains why everything works on the jar execution but not the docker one (the zookeeper container exports its 2181 port to the host machine, so it's visible as localhost for the service process when running directly on the host machine), but doesn't solve the problem: Apparently the spring cloud stream kafka configuration is ignoring the property spring.cloud.stream.kafka.binder.zkNodes as set in the application.yml (note that if I log the value of such environment variable from the service, I see the correct value of 192.168.33.101 that I hardcoded there for debugging purposes).
You have set the defaultBinder to be rabbit while trying to use the Kafka binder configuration. Do you have both rabbit and kafka binders in the classpath of your application? In that case, you can enable here
zookeeper:
image: wurstmeister/zookeeper
container_name: 'zookeeper'
ports:
- 2181:2181
--------------------- kafka --------------------------------
kafka:
image: wurstmeister/kafka
container_name: 'kafka'
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CREATE_TOPICS=kafka_docker_topic:1:1
ports:
- 9092:9092
depends_on:
- zookeeper
spring:
profiles: dev
cloud:
stream:
defaultBinder: kafka
kafka:
binder:
brokers: kafka # i added brokers and zkNodes property
zkNodes: zookeeper #
bindings:
input:
destination: message
content-type: application/json

Resources