Docker-compose file builds same images twice - docker

Question

I am trying to build images for my app. However, when I run "docker-compose up" command, it builds some of the containers twice. I couldn't figure the reason of it. I think the tags cause this kind of situation, but I couldn't figure where 'latest' tag come from.
Here it is my docker-compose.yml:
version: '3.2'
services:
elasticsearch:
build:
context: elasticsearch/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./elasticsearch/config/elasticsearch.yml
target: /usr/share/elasticsearch/config/elasticsearch.yml
read_only: true
- type: volume
source: elasticsearch
target: /usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_PASSWORD: changeme
# Use single node discovery in order to disable production mode and avoid bootstrap checks.
# see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
discovery.type: single-node
networks:
- elk
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./logstash/config/logstash.yml
target: /usr/share/logstash/config/logstash.yml
read_only: true
- type: bind
source: ./logstash/pipeline
target: /usr/share/logstash/pipeline
read_only: true
ports:
- "5044:5044"
- "5000:5000/tcp"
- "5000:5000/udp"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
kibana:
build:
context: kibana/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./kibana/config/kibana.yml
target: /usr/share/kibana/config/kibana.yml
read_only: true
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
zookeeper:
image: 'bitnami/zookeeper:latest'
container_name: zookeeper
ports:
- "2181:2181"
networks:
- elk
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:latest'
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9093:9093"
networks:
- elk
environment:
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
ALLOW_PLAINTEXT_LISTENER: 'yes'
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_LISTENERS: CLIENT://:9092,EXTERNAL://:9093
KAFKA_CFG_ADVERTISED_LISTENERS: CLIENT://kafka:9092,EXTERNAL://localhost:9093
KAFKA_INTER_BROKER_LISTENER_NAME: CLIENT
links:
- logstash
app:
container_name: "ml-pipeline"
build: .
ports:
- "7000:7000"
- "5001:5001"
depends_on:
- kafka
- elasticsearch
- logstash
networks:
- elk
links:
- kafka
networks:
elk:
driver: bridge
volumes:
elasticsearch:
And output of this is:
As you can see there are duplicate images. How can I solve it ?

Answer 1

Actually there is nothing that indicates that docker-compose built the images twice. Your screenshot shows that the images have multiple tag names. But without further context it's hard to say how this happened and how docker-compose was involved in this.
One possible cause for this:
the pre-built images from docker.elastic.co were downloaded by docker pull docker.elastic.co/... or another docker run command
docker-compose up was looking for images named twitter-stream-dl-docker_* and since it couldn't find them triggered a docker-compose build
docker-compose build built the images - but using the docker build cache it could re-use all layers of the existing docker.elastic.co/... images which must have been built from the same source
the new built images resulted in the same final images which were then tagged with the name expected by docker-compose, i.e. twitter-stream-dl-docker_*
If you want to force a new local built either:
build without using the cache: docker-compose build --no-cache
delete the downloaded images: docker rmi docker.elastic.co/...

Answer 2

All 3 ELK containers have a build context with a Dockerfile that by default only consists of a FROM line. In the Dockerfiles you could add additional plugins.
part of your docker-compose.yml:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
logstash/Dockerfile:
ARG ELK_VERSION
# https://github.com/elastic/logstash-docker
FROM docker.elastic.co/logstash/logstash:${ELK_VERSION}
# Add your logstash plugins setup here
# Example: RUN logstash-plugin install logstash-filter-json
docker-compose pulls the Image docker.elastic.co/logstash/logstash:${ELK_VERSION} and builds it's own version version twitter-stream-dl-docker_XXX. Since the build doesn't do anything it simply tags the old image with the new tag so they have the same Image ID.
In case you're wondering your folder's name is twitter-stream-dl-docker so the images have that tag (or you used docker-compose -p twitter-stream-dl-docker).
I hope that clears things up, but feel free to ask anything that's ambigious.