Increasing number of brokers in kafka using KAFKA_ADVERTISED_HOST_NAME - docker

Question

I'm new to Kafka and I'm trying to run a Kafka service on my local machine and use it to transfer some data from one .NET project to another.
I'm using docker-compose.yml file to create two docker containers for zookeeper and Kafka from wurstmeister.
In Kafka definition in environment variables there is KAFKA_ADVERTISED_HOST_NAME which I set to 127.0.0.1.
In docker hub of wurstmeister/kafka says I quote
"modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host IP if you want to run multiple brokers.)".
When I set my topic to have more than 1 replica and more than 1 partition I get this message
"Error while executing topic command : Replication factor: 2 larger than available brokers: 1.".
What is the right IP address to define in KAFKA_ADVERTISED_HOST_NAME which will allow me to get more than 1 broker?
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock

Answer 1

Note - Running multiple brokers on one machine does not provide true fault tolerance.
use it to transfer some data from one .NET project to another
You only need one broker for this.
First, suggest reading https://github.com/wurstmeister/kafka-docker/wiki/Connectivity
KAFKA_ADVERTISED_HOST_NAME is deprecated. Don't use it.
It has been replaced by KAFKA_ADVERTISED_LISTENERS, which can both contain ${DOCKER_HOST_IP:-127.0.0.1} (or your host IP from ifconfig) since this is the IP your client will use (from the host).
Since brokers are clients of themselves, you also need to advertise the container names. And, if you want to containerize your client, those addresses would also be used by your app.
Beyond Kafka networking configs, KAFKA_BROKER_ID needs to be different between each, and your error is suggesting you need to override all the other replication factor broker configs.
All in all.
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper
kafka-1:
image: wurstmeister/kafka
depends_on: [zookeeper]
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9092,PLAINTEXT_INTERNAL://kafka-1:29092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
kafka-2:
image: wurstmeister/kafka
ports:
- "9093:9093"
environment:
KAFKA_BROKER_ID: 2 # unique
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9093,PLAINTEXT_INTERNAL://kafka-2:29093
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,PLAINTEXT_INTERNAL://:29093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_DEFAULT_REPLICATION_FACTOR: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
depends_on: # ensure this joins the other
- zookeeper
- kafka-1
I'm trying to run a Kafka service on my local machine
Set bootstrap.servers="localhost:9092,localhost:9093"
Further reading - Connect to Kafka running in Docker
Example usage
Note that topic creation doesn't seem to automatically work. You should ideally be using your application to actually create / check topic existence.
$ kcat -b localhost:9093 -L
Metadata for all topics (from broker -1: localhost:9093/bootstrap):
2 brokers:
broker 2 at 127.0.0.1:9093
broker 1 at 127.0.0.1:9092 (controller)
0 topics:
producing some data... and consuming it
$ kcat -b localhost:9093 -t test -o earilest
% Auto-selecting Consumer mode (use -P or -C to override)
hello
world
sample
data
of
no
importance

Answer 2

it should be ip address of your machine.
Linux: use ifconfig command and your IP will be inet <IP_ADDRESS>
in windows you will get this by using ipconfig