Elasticsearch Unreachable: [http://kafka:9200/][Manticore::SocketException] Connection refused - docker

Question

I am trying to run Spring application with logstash, Elastic Search, Kafka and Kibana.
[main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://kafka:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://kafka:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
The above error is being repeated under logstash container
docker-compose.yml
version: '3.7'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
#restart: always
networks:
- tweetapp-network
kafka:
image: wurstmeister/kafka
container_name: kafka
#restart: always
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "tweetapp-logs:1:1, Tweets:1:1"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- tweetapp-network
mongodb:
image: mongo
container_name: mongodb
# restart: always
ports:
- "27017:27017"
# volumes:
# - mongodb-volume:/data/db
networks:
- tweetapp-network
springboot:
image: tweetapp
#restart: always
ports:
- "8080:8080"
depends_on:
- mongodb
- kafka
- elasticsearch
- logstash
- kibana
networks:
- tweetapp-network
logstash:
image: logstash:7.7.0
container_name: logstash
hostname: logstash
ports:
- "9600:9600"
volumes:
- .\logstash:/usr/share/logstash/pipeline/
links:
- elasticsearch:elasticsearch
depends_on:
- elasticsearch
networks:
- tweetapp-network
elasticsearch:
image: elasticsearch:7.7.0
container_name: elasticsearch
hostname: elasticsearch
ports:
- "9200:9200"
environment:
- discovery.type=single-node
networks:
- tweetapp-network
kibana:
image: kibana:7.7.0
container_name: kibana
hostname: kibana
ports:
- "5601:5601"
links:
- elasticsearch:elasticsearch
depends_on:
- elasticsearch
networks:
- tweetapp-network
# Networks to be created to facilitate communication between containers
networks:
tweetapp-network:
I made sure that Elastic Search is working http://localhost:9200/ , for this URL I get JSON output.
logstash.config
input {
kafka {
bootstrap_servers => "kafka:9092"
topics => ["tweetapp-logs"]
}
}
filter {
grok {
match => [ "message", "%{GREEDYDATA}" ]
}
}
output {
elasticsearch {
hosts => ["kafka:9200"]
index => "tweetapp"
workers => 1
}
}
I am new to docker-compose, Elastic search and Kafka. Any help will be appreciated.

Answer 1

Seems like you are unlucky.
Replace Elasticsearch instead of kafka in host name
Try this:
input { kafka { bootstrap_servers => "kafka:9092" topics => ["tweetapp-logs"] } } filter { grok { match => [ "message", "%{GREEDYDATA}" ] } } output { elasticsearch { hosts => ["elasticsearch:9200"] index => "tweetapp" workers => 1 } }