Multiple docker-compose file with different context path - docker

I have 2 docker-compose files that I need to run together, the locations of the files are like
/home/project1/docker-compose.yml
and
/home/project2/docker-compose.yml
so clearly both the services should have different contextpath
But when I run below docker compose command
docker-compose -f /home/project1/docker-compose.yml -f /home/project2/docker-compose.yml config
I get to see, Both the service are getting the same context path
app:
build:
context: /home/project1
dockerfile: Dockerfile
app2:
build:
context: /home/project1
dockerfile: Dockerfile
How can I resolve this issue I want both my services to have their own project path ie.
app service should have context path /home/project1
and
app2 service should have context path /home/project2

They way I found could run multiple services is:
1. First of all create images for all the services with the help of docker build command
ex: in my case it is a java application with maven build tool, so command I used:
mvn clean package docker:build -Denv=$ENV -DskipTests
2. After all the images built, create a common docker-compose file which will look like this
version: '3'
services:
service1:
image: <service1-image>
ports:
- "8100:8080"
environment:
- TZ=Asia/Kolkata
- MYSQL_USER=root
- MYSQL_PASSWORD=unroot
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
service2:
image: <service2-image>
ports:
- "8101:8080"
environment:
- TZ=Asia/Kolkata
- MYSQL_USER=root
- MYSQL_PASSWORD=unroot
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
service3:
image: <service3-image>
environment:
- TZ=Asia/Kolkata
- MYSQL_USER=root
- MYSQL_PASSWORD=unroot
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
networks:
default:
external:
name: dev
The way I have mentioned mysql root, password, you can also add environment variables.
3.Then run below command to run all the services
docker-compose -f docker-compose.yml up
or run below command to run specific service
docker-compose -f docker-compose.yml up service1
This is working fine for me, this will require 1 time setup but after that it will be very easy and fast.

Option 1:
Use absolute paths for the contexts in both docker-compose files
Option 2:
Create a docker-compose.override.yml with the absolute paths:
version: "3"
services:
service1:
build:
context: /home/project1
service2:
build:
context: /home/project2
and include it in the docker-compose command:
docker-compose -f /home/project1/docker-compose.yml -f /home/project2/docker-compose.yml -f /home/docker-compose.override.yml config
On linux, to avoid hard-coding of the base path in the docker-compose.override.yml, you can use PWD environment variable:
services:
service1:
build:
context: ${PWD}/project1
service2:
build:
context: ${PWD}/project2

It seems that when using multiple docker-compose files the context is taken from the location of the first file. It is explained in detail in this issue
https://github.com/docker/compose/issues/3874

Related

Docker-compose file build context explanation

I am using a docker-compose file that is working but I have poor understanding of the file
In the following file how can I interpret
build:
context:
ulimits:
sysctl:
version: '3.1'
services:
zabbix-server:
container_name: zabbix-server
build:
context: zabbix-server-mysql
image: zabbix:ubuntu-5.4.4-custom
ulimits:
nproc: 65535
nofile:
soft: 20000
hard: 40000
sysctls:
- net.ipv4.ip_local_port_range=1024 65000
- net.ipv4.conf.all.accept_redirects=0
- net.ipv4.conf.all.secure_redirects=0
- net.ipv4.conf.all.send_redirects=0
build and context
ulimits 1 ulimits 2
sysctl 1 sysctl 2
and many many other articles

Docker ERROR: only one instance of "host" network is allowed [duplicate]

This question already has answers here:
How to run docker containers in host network mode using docker-compose?
(2 answers)
Closed 2 years ago.
I am trying to run a container where I need to user network driver as "host" instead of "bridge". I am running it on Centos machine and my docker-compose.yml is
version: '3.4'
services:
testContainer:
build:
context: .
args:
HADOOP_VERSION: 2.6.0
HIVE_VERSION: 1.1.0
image: testcontainer
container_name: testcontainer
hostname: testcontainer
ports:
- 9200:9200
- 9300:9300
- 5601:5601
- 9001:9001
ulimits:
memlock:
soft: -1
hard: -1
networks:
- elknet
networks:
elknet:
driver: host
But i am getting the following error when I fire "docker-compose up" :
ERROR: only one instance of "host" network is allowed
Can anyone please suggest how can I use host network using docker-compose.yml.
Also note that if I use network_host as suggested by #larsks, I am still getting error
version: '3.4'
services:
testContainer:
build:
context: .
args:
HADOOP_VERSION: 2.6.0
HIVE_VERSION: 1.1.0
image: testcontainer
container_name: testcontainer
hostname: testcontainer
ports:
- 9200:9200
- 9300:9300
- 5601:5601
- 9001:9001
ulimits:
memlock:
soft: -1
hard: -1
network_mode: host
I am getting following error
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services: 'testContainer'
Get rid of the networks section in your docker-compose.yml, and add a network_mode directive to your service definition:
services:
testContainer:
build:
context: .
args:
HADOOP_VERSION: 2.6.0
HIVE_VERSION: 1.1.0
image: testcontainer
container_name: testcontainer
hostname: testcontainer
ports:
- 9200:9200
- 9300:9300
- 5601:5601
- 9001:9001
ulimits:
memlock:
soft: -1
hard: -1
network_mode: host

Unable connect from one container to another

I have a Dockerfile and docker-compose.yml file set up, but not sure if they are correct and I am unable to run it without an error.
My Dockerfile is:
FROM golang:1.14-alpine
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN go get
RUN go run server.go
and my compose.yml is:
version: "3.5"
services:
elasticsearch:
container_name: "elasticsearch"
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
ports:
- 9200:9200
gqlgen:
container_name: "gqlgen"
build: ./
restart: "on-failure"
ports:
- "8080:8080"
depends_on:
- elasticsearch
This is how the root of my folder looks like:
I tried to run: docker-compose up from the root directory and this is what I get:
panic: Get "http://127.0.0.1:9200/": dial tcp 127.0.0.1:9200: connect: connection refused
I think I am doing my setup wrong.
UPDATE:
Based on suggestions and more stuff that I read online, I changed my DOCKERFILE as:
FROM golang:1.14-alpine
RUN mkdir /app
ADD . /app
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o server .
CMD ["./server"]
and compose file:
version: "3.5"
services:
elasticsearch:
container_name: "elasticsearch"
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
environment:
- node.name=elasticsearch
- cluster.name=es-docker-cluster
- bootstrap.memory_lock=true
- cluster.initial_master_nodes=elasticsearch
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
golang:
container_name: "golang"
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: unless-stopped
depends_on:
- elasticsearch
and it builds correctly now.
But same issue with running docker-compose up.
panic: Get "http://elasticsearch:9200/": dial tcp 172.18.0.2:9200: connect: connection refused
You have a problem because you address Elasticsearch incorrectly.
Inside docker container 127.0.0.1 refers to the container itself, so your app is trying to find Elasticsearch where there isn't one.
The correct way to reference one docker container from another is by using docker container name. So in your case, it would be using name: elasticsearch.
Edit:
There is another issue with your configuration.
You miss some vital elements of Elasticsearch configuration.
Here you have snippet with minimal configuration for a single node Elasticsearch cluster.
services:
elasticsearch:
container_name: "elasticsearch"
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
environment:
- node.name=elasticsearch
- cluster.name=es-docker-cluster
- bootstrap.memory_lock=true
- cluster.initial_master_nodes=elasticsearch
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
All I have written before is still valid. After modifying docker-compose your last version which refers to Elasticsearch via http://elasticsearch:9200 should work fine.
Edit:
As #David Maze pointed out there is a third issue in your example.
Instead of RUN go run server.go you should have CMD go run server.go.
What you are doing is running your app during your build, when you want to run your app inside the container.
The more conventional approach would be to build app, instead of copying the source, copying the binary to the container and running the binary inside the container.
Here you have some information about that: https://medium.com/travis-on-docker/multi-stage-docker-builds-for-creating-tiny-go-images-e0e1867efe5a
So the above action to replace localhost with elasticsearch is correct.
But this it should happen only when you are starting your up with docker-compose.
Do not attempt to call elasticsearch from your IDE using elasticearch instead of host.
I suggest to make the host of elasticsearch configurable and for local configuration keep localhost, but you can override it in docker-compose file.
version: "3.5"
services:
elasticsearch:
container_name: "elasticsearch"
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
environment:
- node.name=elasticsearch
- cluster.name=es-docker-cluster
- bootstrap.memory_lock=true
- cluster.initial_master_nodes=elasticsearch
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
golang:
container_name: "golang"
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: unless-stopped
depends_on:
- elasticsearch
environment:
- ELASTICSEARCH_HOST: elasticsearch
Where ELASTICSEARCH_HOST is a variable that you use in your project

Using netcat to wait neo to be ready within the same container

So consider the following... If we had a docker container called app, this app contained an instance of neo4j database. Within the Dockerfile under the CMD we reference entrypoint.sh. From this script we are running the following ...
end="$((SECONDS+60))"
while true; do
nc -z localhost 7687 && break
[[ "${SECONDS}" -ge "${end}" ]] && exit 1
sleep 1
done
The question is why does netcat not see the neo, even though it is booting. I have confirmed this by commenting out the CMD line within the dockerfile and checking that we can get to it though a browser.
If there is only one container enclosing neo and netcat running within the entrypoint.sh will it even have access to neo when it comes up or would netcat need to be in a seperate container all together ?
My docker-compose.yaml is below...
version: "2.1"
services:
app:
build:
context: .
container_name: neo4j-ingestion
expose:
- "7474"
- "7687"
ports:
- "7687:7687"
- "7474:7474"
environment:
MEMORY_LOCK: "true"
DB_START_DELAY: "10"
PROCESSORS: "2"
DATA_INGEST_FOLDER: /ingest/pending
NEO4J_AUTH: "neo4j/password"
NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
NEO4J_dbms_memory_heap_maxSize: "4G"
NEO4J_HOSTNAME: "0.0.0.0"
NEO4J_USERNAME: "neo4j"
NEO4J_PASSWORD: "password"
PROCESS_NAME: "neo4j-ingest"
INGESTION_TYPE: "incremental"
SLEEP_PERIOD: "20"
CREATE_SCHEMA: "false"
NEO4J_dbms_security_procedures_unrestricted: "apoc.*"
IS_DELTA: "1"
volumes:
- neodata:/data
- "./test-integration/incremental:/ingest"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
neodata:
driver: local

docker compose file is invalid ,additional properties not allowed tty

I really need help on this error I don't understand why I get this error. Thanks
docker -v
Docker version 1.13.1, build 092cba3
docker-compose -v
docker-compose version 1.11.1, build 7c5d5e4
this is my dockerfile
version: '2.0'
services:
arcgis-server:
container_name: "arcgis-server"
image: "arcgis-server:10.4.1"
volumes:
- "./license:/license"
- "./arcgisserver:/arcgis/server/usr/directories"
- "./config-store:/arcgis/server/usr/config-store"
build:
context: .
dockerfile: "Dockerfile"
ulimits:
nproc: 25059
nofile:
soft: 65535
hard: 65535
ports:
- "127.0.0.1:6080:6080"
- "127.0.0.1:6443:6443"
- "4001:4001"
- "4002:4002"
- "4004:4004"
stdin_open: true
tty: true
here is the error
docker-compose build
ERROR: The Compose file './docker-compose.yml' is invalid because:
Additional properties are not allowed ('tty' was unexpected)
You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version ("2.0", "2.1", "3.0") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
Actually I test on the old machine it worked fine. I would appreciate your helps. Thanks again!!!!
tty needs to be defined as a setting on your service, not at the top level. Yaml files are space sensitive, so removing the leading spaces puts the setting at the top level where it's not valid. Use the following syntax to fix it:
version: '2.0'
services:
arcgis-server:
container_name: "arcgis-server"
image: "arcgis-server:10.4.1"
volumes:
- "./license:/license"
- "./arcgisserver:/arcgis/server/usr/directories"
- "./config-store:/arcgis/server/usr/config-store"
build:
context: .
dockerfile: "Dockerfile"
ulimits:
nproc: 25059
nofile:
soft: 65535
hard: 65535
ports:
- "127.0.0.1:6080:6080"
- "127.0.0.1:6443:6443"
- "4001:4001"
- "4002:4002"
- "4004:4004"
stdin_open: true
tty: true

Resources