Hello everyone while trying to set up kafka with ssl certs with bitnami image i m getting following error No available authentication scheme for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings..
I have used self signed certficates for authentication.I am attaching my docker-compose.yml for reference.
services:
zookeeper:
image: bitnami/zookeeper:latest
ports:
- "2181:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: bitnami/kafka:latest
ports:
- "9093:9093"
environment:
KAFKA_CFG_LISTENERS: SSL://:9093
ALLOW_PLAINTEXT_LISTENER: yes
KAFKA_CFG_ADVERTISED_LISTENERS: SSL://kafka:9093
KAFKA_CFG_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: SSL
KAFKA_ZOOKEEPER_PROTOCOL: PLAINTEXT
KAFKA_CFG_SSL_KEYSTORE_LOCATION: /bitnami/kafka/config/certs/kafka.keystore.jks
KAFKA_CFG_SSL_KEYSTORE_PASSWORD: kafkakeystorepass
KAFKA_CFG_SSL_TRUSTSTORE_LOCATION: /bitnami/kafka/config/certs/kafka.truststore.jks
KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD: kafkatruststorepass
volumes:
- "./secrets/kafka.broker.keystore.jks:/bitnami/kafka/config/certs/kafka.keystore.jks"
- "./secrets/kafka.broker.truststore.jks:/bitnami/kafka/config/certs/kafka.truststore.jks"
Related
i am trying to dockerize my web application. i am running a apache webserver + mariadb and redis server as you can see in my docker-compose file combined with an nginx proxy to use local domains and ssl.
everything works fine as long is i use the container names to connect to mysql / redis. I dont want to change all localhosts in my code to the mysql / redis container names.
Is there a way to keep "localhost" as Host instead of the containers name?
version: "3.5"
services:
nginx-proxy:
image: jwilder/nginx-proxy
container_name: portal-proxy
networks:
- portal
ports:
- "80:80"
- "443:443"
volumes:
- ./certs:/etc/nginx/certs
- /var/run/docker.sock:/tmp/docker.sock:ro
portal:
image: portal:latest
container_name: portal-webserver
networks:
- portal
volumes:
- ./portal:/var/www/html/portal
links:
- db
restart: always
environment:
VIRTUAL_HOST: portal.dev
db:
image: mariadb:latest
container_name: portal-db
networks:
- portal
ports:
- "3306:3306"
restart: always
environment:
MYSQL_DATABASE: portal
MYSQL_USER: www-data
MYSQL_PASSWORD: www-data
MYSQL_ROOT_PASSWORD: asdf1234
volumes:
- ./db:/docker-entrypoint-initdb.d
- ./db:/var/lib/mysql
redis:
image: redis:latest
container_name: portal-redis
environment:
- ALLOW_EMPTY_PASSWORD=yes
networks:
- portal
ports:
- "6379:6379"
networks:
portal:
name: portal
Use a common hostname (staging.docker.host) on all containers, that resolves to the docker host's ip 1.2.3.4.
So adding this to containers:
extra_hosts:
- "staging.docker.host:1.2.3.4"
and use that name (staging.docker.host) in all you connection endpoints.
On you local machine you also add (staging.docker.host) to your /etc/hosts or C:\Windows\System32\drivers\etc\hosts with localhost 127.0.0.1 staging.docker.host.
I'm first time Traefik user and I successfully configured this docker compose setup for Jira with Traefik and Let's Encrypt Cert.
My problem is that Jira must be able to connect to his self. Their are some Jira Services like Gadgets that loads it's data via JavaScript from via his own address over http. This typ of service does not work for me. Their is a support documents that describes this problems and also shows solutions for this. But I don't know how to setup this up correctly with Traefik/Docker. https://confluence.atlassian.com/jirakb/how-to-fix-gadget-titles-showing-as-__msg_gadget-813697086.html
Your help would be great. Thanks a lot!
version: '3'
services:
reverse-proxy:
image: traefik # The official Traefik docker image
command: --docker # Enables the web UI and tells Traefik to listen to docker --api
ports:
- "80:80" # The HTTP port
- "443:443" # The HTTPS port
- "8081:8080" # The Web UI (enabled by --api)
hostname: traefik
restart: unless-stopped
domainname: ${DOMAINNAME}
networks:
- frontend
- backend
labels:
- "traefik.enable=false"
- "traefik.frontend.rule=Host:traefik.${DOMAINNAME}"
volumes:
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
- /etc/compose/traefik:/etc/traefik
- /etc/compose/shared:/shared
jira:
image: dchevell/jira-software:${JIRAVERSION}
ports:
- 8080:8080
networks:
- backend
restart: unless-stopped
volumes:
- /data/files/jira/data:/var/atlassian/application-data/jira
environment:
- JVM_MAXIMUM_MEMORY=2048m
- JVM_MINIMUM_MEMORY=768m
- CATALINA_CONNECTOR_PROXYNAME=jira.${DOMAINNAME}
- CATALINA_CONNECTOR_PROXYPORT=443
- CATALINA_CONNECTOR_SCHEME=https
- CATALINA_CONNECTOR_SECURE=true
depends_on:
- jira-postgresql
links:
- "jira-postgresql:database"
labels:
- "traefik.enable=true"
- "traefik.backend=jira"
- "traefik.frontend.rule=Host:jira.${DOMAINNAME}"
- "traefik.port=8080"
jira-postgresql:
image: postgres:9.6.11-alpine
networks:
- backend
ports:
- 5432:5432
restart: unless-stopped
volumes:
- /data/index/postgresql/data/:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=jira
- POSTGRES_USER=jira
- POSTGRES_DB=jira
labels:
- "traefik.enable=false"
# Portainer
portainer:
image: portainer/portainer
container_name: portainer
restart: always
ports:
- 9000:9000
command: -H unix:///var/run/docker.sock
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./etc-portainer/data:/data
environment:
TZ: ${TZ}
labels:
- "traefik.enable=false"
networks:
frontend:
external:
name: frontend
backend:
driver: bridge
Configuration I got working with apps over secure - not super intuitive, but it looks like it accepts redirects secure traffic properly. I've got mine using acme on godaddy for certs, and it appears to be functioning properly over https with a forced recirect:
Forced redirect for reference:
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
And the dockerfile that I made to get things deployed properly:
version: '3'
services:
jira:
image: dchevell/jira-software:8.1.0
deploy:
restart_policy:
condition: on-failure
labels:
- traefik.frontend.rule=Host:jira.mydomain.com
- traefik.enable=true
- traefik.port=8080
ports:
- "8080"
networks:
- traefik-pub
- jiranet
environment:
- CATALINA_CONNECTOR_PROXYNAME=jira.mydomain.com
- CATALINA_CONNECTOR_PROXYPORT=443
- CATALINA_CONNECTOR_SCHEME=https
- CATALINA_CONNECTOR_SECURE=true
jira-postgresql:
image: postgres:11.2-alpine
networks:
- jiranet
ports:
- "5432"
volumes:
- jira-postgres-data:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=supersecret
- POSTGRES_USER=secret_user
- POSTGRES_DB=jira_db
labels:
- "traefik.enable=false"
volumes:
jira-postgres-data:
networks:
traefik-pub:
external: true
jiranet:
driver: overlay
This still required manual configuration of the database - I may one day take the time to build my own jira dockerfile that accepts the database config already, but with this one working, I don't see much point in pre-configuring the database connection when it's 20 seconds of extra work vs. rebuilding a dockerfile that I haven't written myself.
I'm trying to setup Kafka in a docker container for local development. My docker-compose.yml looks as follows:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181"
hostname: zookeeper
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092"
hostname: kafka
environment:
KAFKA_CREATE_TOPICS: "UploadFile:1:1,GetFile:1:1,TrackUpload:1:1,GetEmailContent:1:1" # topic:partition:replicas
KAFKA_ADVERTISED_HOST_NAME: kafka # docker-machine ip
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
Unfortunately my node app running on my localhost (not in docker) cannot connect to it. I used the url 'kafka:9092' and even 'localhost:9092'. Nothing works. Any idea what's happening?
Expose the host port 9092 for kafka service & you should be able to connect via "localhost:9092" from the app or host machine.
....
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
ports:
- "9092:9092"
....
I'm trying to use microservices Spring Boot with Kafka, but my Spring Boot containers can not connect to the Kafka container.
docker-compose.yml:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
restart: always
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka
container_name: kafka
restart: always
ports:
- 9092:9092
depends_on:
- zookeeper
links:
- zookeeper:zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
consumer:
image: consumer
container_name: consumer
depends_on:
- kafka
restart: always
ports:
- 8084:8080
depends_on:
- kafka
links:
- kafka:kafka
producer:
image: producer
container_name: producer
depends_on:
- kafka
restart: always
ports:
- 8085:8080
depends_on:
- kafka
links:
- kafka:kafka
application.properties in Consumer:
spring.kafka.consumer.bootstrap-servers=kafka:9092
spring.kafka.consumer.group-id=WorkUnitApp
spring.kafka.consumer.topic=kafka_topic
application.properties in Producer:
spring.kafka.producer.bootstrap-servers=kafka:9092
But if I run the Kafka in a container and the Spring Boot microservices locally it works.
application.properties in Consumer:
spring.kafka.consumer.bootstrap-servers=0.0.0.0:9092
spring.kafka.consumer.group-id=WorkUnitApp
spring.kafka.consumer.topic=kafka_topic
application.properties in Producer:
spring.kafka.producer.bootstrap-servers=0.0.0.0:9092
What's the problem, why does the links from the docker not work ?
p.s. 0.0.0.0 because mac os
Edited
I added in docker-compose.yml environments to kafka but it still does not work either
- KAFKA_ADVERTISED_PORT=9092
You need to advertise your Kafka broker as kafka, which is the effective hostname for all linking containers (i.e. the hostname that the client needs to connect to from the Kafka protocol perspective, and so kafka:9092 is correct, not 0.0.0.0):
kafka:
...
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
I have written docker-compose.yml file to create the following containers:
Confluent-Zookeeper
Confluent-Kafka
Confluent-Schema Registry
I want a single docker-compose file to spun up the necessary containers, expose required ports and interconnect the dependent containers. The goal is to have
I am using the official confluent images from Docker Hub.
My docker-compose file looks like this:
zookeeper:
image: confluent/zookeeper
container_name: confluent-zookeeper
hostname: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "2181:2181"
kafka:
environment:
KAFKA_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
image: confluent/kafka
container_name: confluent-kafka
hostname: kafka
links:
- zookeeper
ports:
- "9092:9092"
schema-registry:
image: confluent/schema-registry
container_name: confluent-schema_registry
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
SCHEMA_REGISTRY_HOSTNAME: schema-registry
SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081
SCHEMA_REGISTRY_DEBUG: 'true'
SCHEMA_REGISTRY_KAFKASTORE_TOPIC_REPLICATION_FACTOR: '1'
links:
- kafka
- zookeeper
ports:
- "8081:8081"
Now when I run docker-compose up, all these containers will be created and launched. But Schema Registry container exits immediately. docker logs gives the following output:
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
[2017-05-17 06:06:33,415] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)
org.apache.kafka.common.config.ConfigException: Only plaintext and SSL Kafka endpoints are supported and none are configured.
at io.confluent.kafka.schemaregistry.storage.KafkaStore.getBrokerEndpoints(KafkaStore.java:254)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.<init>(KafkaStore.java:111)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:136)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:53)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
at io.confluent.rest.Application.createServer(Application.java:117)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)
I searched for this issue but nothing helped. I tried various other configurations like providing KAFKA_ADVERTISED_HOSTNAME, changing SCHEMA_REGISTRY_LISTENERS value, etc. but no luck.
Can anybody point out the exact configuration issue why Schema Registry container is failing?
Those are old and deprecated docker images. Use the latest supported docker images from confluentinc https://hub.docker.com/u/confluentinc/
You can find a full compose file here - confluentinc/cp-docker-images
You're missing the hostname (hostname: schema-registry) entry in the failing container. By default Docker will populate a container's /etc/hosts with the linked containers' aliases and names, plus the hostname of self.
The question is old, though it might be helpful to leave a solution that worked for me. I am using docker-compose:
version: '3.3'
services:
zookeeper:
image: confluent/zookeeper:3.4.6-cp1
hostname: "zookeeper"
networks:
- test-net
ports:
- 2181:2181
environment:
zk_id: "1"
kafka:
image: confluent/kafka:0.10.0.0-cp1
hostname: "kafka"
depends_on:
- zookeeper
networks:
- test-net
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: "kafka"
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_BROKER_ID: "0"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
schema-registry:
image: confluent/schema-registry:3.0.0
hostname: "schema-registry"
depends_on:
- kafka
- zookeeper
networks:
- test-net
ports:
- 8081:8081
environment:
SR_HOSTNAME: schema-registry
SR_LISTENERS: http://schema-registry:8081
SR_DEBUG: 'true'
SR_KAFKASTORE_TOPIC_REPLICATION_FACTOR: '1'
SR_KAFKASTORE_TOPIC_SERVERS: PLAINTEXT://kafka:9092
networks:
test-net:
driver: bridge`