Docker compose 3.1, communication between two docker-compose.yml - docker

For past 3 days I have been trying to connect two docker containers generated by two separate docker-compose files.
I tried a lot of approaches none seem to be working. Currently after adding network to compose files, service with added network doesn't start. My goal is to access endpoint from another container.
Endpoint-API compose file:
version: "3.1"
networks:
test:
services:
mariadb:
image: mariadb:10.1
container_name: list-rest-api-mariadb
working_dir: /application
volumes:
- .:/application
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=list-api
- MYSQL_USER=list-api
- MYSQL_PASSWORD=root
ports:
- "8003:3306"
webserver:
image: nginx:stable
container_name: list-rest-api-webserver
working_dir: /application
volumes:
- .:/application
- ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
ports:
- "8005:80"
networks:
- test
php-fpm:
build: docker/php-fpm
container_name: list-rest-api-php-fpm
working_dir: /application
volumes:
- .:/application
- ./docker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini
Consumer compose file:
version: "3.1"
networks:
test:
services:
consumer-mariadb:
image: mariadb:10.1
container_name: consumer-service-mariadb
working_dir: /consumer
volumes:
- .:/consumer
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=consumer
- MYSQL_USER=consumer
- MYSQL_PASSWORD=root
ports:
- "8006:3306"
consumer-webserver:
image: nginx:stable
container_name: consumer-service-webserver
working_dir: /consumer
volumes:
- .:/consumer
- ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
ports:
- "8011:80"
networks:
- test
consumer-php-fpm:
build: docker/php-fpm
container_name: consumer-service-php-fpm
working_dir: /consumer
volumes:
- .:/consumer
- ./docker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini
Could someone tell me best way of accessing API container from within consumer? I'm loosing my mind on this one.

Your two
networks:
test:
don't refer to the same network, but they are independent and don't talk to each other.
What you could do is having an external and pre-existing network that both compose file can talk and share. You can do that with docker network create, and then refer to that in your compose files with
networks:
default:
external:
name: some-external-network-here

Related

Persist nifi data and volume

I want to make my nifi data volume and configuration persist means even if I delete container and docker compose up again I would like to keep what I built so far in my nifi. I try to mount volumes as follows in my docker compose file in volumes section nevertheless it doesn't work and my nifi processors are not saved. How can I do it correctly? Below my docker-compose.yaml file.
version: "3.7"
services:
nifi:
image: koroslak/nifi:latest
container_name: nifi
restart: always
environment:
- NIFI_HOME=/opt/nifi/nifi-current
- NIFI_LOG_DIR=/opt/nifi/nifi-current/logs
- NIFI_PID_DIR=/opt/nifi/nifi-current/run
- NIFI_BASE_DIR=/opt/nifi
- NIFI_WEB_HTTP_PORT=8080
ports:
- 9000:8080
depends_on:
- openldap
volumes:
- ./volume/nifi-current/state:/opt/nifi/nifi-current/state
- ./volume/database/database_repository:/opt/nifi/nifi-current/repositories/database_repository
- ./volume/flow_storage/flowfile_repository:/opt/nifi/nifi-current/repositories/flowfile_repository
- ./volume/nifi-current/content_repository:/opt/nifi/nifi-current/repositories/content_repository
- ./volume/nifi-current/provenance_repository:/opt/nifi/nifi-current/repositories/provenance_repository
- ./volume/log:/opt/nifi/nifi-current/logs
#- ./volume/conf:/opt/nifi/nifi-current/conf
postgres:
image: koroslak/postgres:latest
container_name: postgres
restart: always
environment:
- POSTGRES_PASSWORD=secret123
ports:
- 6000:5432
volumes:
- postgres:/var/lib/postgresql/data
pgadmin:
container_name: pgadmin
image: dpage/pgadmin4:4.18
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=admin
- PGADMIN_DEFAULT_PASSWORD=admin
ports:
- 8090:80
metabase:
container_name: metabase
image: metabase/metabase:v0.34.2
restart: always
environment:
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabase
MB_DB_PORT: 5432
MB_DB_USER: metabase_admin
MB_DB_PASS: secret123
MB_DB_HOST: postgres
ports:
- 3000:3000
depends_on:
- postgres
openldap:
image: osixia/openldap:1.3.0
container_name: openldap
restart: always
ports:
- 38999:389
# Mocked source systems
jira-api:
image: danielgtaylor/apisprout:latest
container_name: jira-api
restart: always
ports:
- 8000:8000
command: https://raw.githubusercontent.com/mvrabel/nifi-postgres-metabase/master/api_examples/jira-api.json
pipedrive-api:
image: danielgtaylor/apisprout:latest
container_name: pipedrive-api
restart: always
ports:
- 8100:8000
command: https://raw.githubusercontent.com/mvrabel/nifi-postgres-metabase/master/api_examples/pipedrive-api.yaml
restcountries-api:
image: danielgtaylor/apisprout:latest
container_name: restcountries-api
restart: always
ports:
- 8200:8000
command: https://raw.githubusercontent.com/mvrabel/nifi-postgres-metabase/master/api_examples/restcountries-api.json
volumes:
postgres:
nifi:
openldap:
metabase:
pgadmin:
Using Registry you can achieve that all changes you are doing or your nifi are committed to git. I.e. if you change some processor configuration, it will be reflected in your git repo.
As for flow files, you may need to fix volumes mappings.

Docker-compose doesn't persist volumes

I'm having a problem persisting data with docker-compose.
I want my service chatmysql to persist data I put inside a database, but everytime i run docker-compose down it all vanishes.
I checked directory /var/lib/docker/volumes to see if it stores data there when containers are running and the volume was completely empty.
I didn't have that issue when I was running containers with docker run command so I guess its fault of my docker-compose.yaml file. Can someone help me?
I'm running this on Ubuntu 20.04.
version: '3'
services:
chatmysql:
image: mysql/mysql-server
container_name: chatmysql
hostname: db
user: root
networks:
- chatnet
ports:
- 3307:3306
volumes:
- chatmysqlvolume:/lib/var/mysql
chatbackend:
depends_on:
- chatmysql
build:
context: backend/src
container_name: chatbackend
hostname: backend
networks:
- chatnet
ports:
- 8080:8080
environment:
- MYSQLUSERNAME=${MYSQLUSERNAME:-user}
- MYSQLPASSWORD=${MYSQLPASSWORD:?database password not set}
- MYSQLHOST=${MYSQLHOST:-db}
- MYSQLPORT=${MYSQLPORT:-3306}
- MYSQLDBNAME=${MYSQLDBNAME:-test}
restart: always
deploy:
restart_policy:
condition: on-failure
chatfrontend:
build: frontend
container_name: chatfrontend
hostname: front
networks:
- chatnet
ports:
- 3000:3000
volumes:
chatmysqlvolume:
networks:
chatnet:
driver: bridge
You need to change the mounted volume, try this :
version: '3.7'
services:
chatmysql:
image: mysql/mysql-server
container_name: chatmysql
hostname: db
user: root
networks:
- chatnet
ports:
- 3307:3306
volumes:
- chatmysqlvolume:/var/lib/mysql
chatbackend:
depends_on:
- chatmysql
build:
context: backend/src
container_name: chatbackend
hostname: backend
networks:
- chatnet
ports:
- 8080:8080
environment:
- MYSQLUSERNAME=${MYSQLUSERNAME:-user}
- MYSQLPASSWORD=${MYSQLPASSWORD:?database password not set}
- MYSQLHOST=${MYSQLHOST:-db}
- MYSQLPORT=${MYSQLPORT:-3306}
- MYSQLDBNAME=${MYSQLDBNAME:-test}
restart: always
deploy:
restart_policy:
condition: on-failure
chatfrontend:
build: frontend
container_name: chatfrontend
hostname: front
networks:
- chatnet
ports:
- 3000:3000
volumes:
chatmysqlvolume:
networks:
chatnet:
driver: bridge

docker-compose with volumes - no files in project directory in continers

I am trying to change my docker-compose to use volumes but my /application dir in containers is empty.
When i have config without volumes everything works fine:
volumes:
- .:/application
But when i use
volumes:
- code:/application
volumes:
code:
i get empty /application in containers.
Full docker-compose file:
version: "3.9"
services:
mariadb:
image: mariadb:10.5
container_name: youtube-playlist-mariadb
working_dir: /application
networks:
- backend
volumes:
- /var/lib/mysql/data:/var/lib/mysql
- /var/lib/mysql/logs:/var/log/mysqld.log
- /var/docker/mariadb/conf:/etc/mysql
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=database
- MYSQL_USER=root
- MYSQL_PASSWORD=password
ports:
- "3003:3306"
web:
image: nginx:alpine
container_name: youtube-playlist-web
working_dir: /application
networks:
- frontend
- backend
volumes:
- code:/application
- ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
ports:
- "8080:80"
php:
build: docker/php-fpm
container_name: youtube-playlist-php
working_dir: /application
networks:
- frontend
- backend
volumes:
- code:/application
node:
image: node:12.22.1
container_name: youtube-playlist-node
working_dir: /application
networks:
- frontend
- backend
volumes:
- code:/application
networks:
frontend:
backend:
volumes:
code:
x-mutagen:
sync:
defaults:
ignore:
vcs: true
code:
alpha: "."
beta: "volume://code"
mode: "two-way-resolved"
Edit: added mutagen config
Edit:
SOLUTION
I added this config to docker-compose
volumes:
code:
driver: local
driver_opts:
type: none
device: $PWD
o: bind
when you do what you are doing, you are just creating a volume where the content of the /application is going to be stored.
This is different to a bind-mount as you are doing in the first example (the one you say it's working). In this case, you are binding a directory (.) to /application.

How to control services start-up sequence in docker-compose.yml file?

Here is my docker-compose.yml file
version: '3'
services:
mongodb:
image: hedge-mongo:1.0
container_name: mongodb
ports:
- "27017:27017"
networks:
- appNetwork
service-A:
image: service-A:1.0
container_name: service-A
ports:
- "3030:3030"
networks:
- appNetwork
depends_on:
- mongodb
service-B:
image: service-B:1.0
container_name: service-B
ports:
- "4200:4200"
networks:
- appNetwork
service-C:
image: service-C:1.0
container_name: service-C
ports:
- "8082:8082"
networks:
- appNetwork
networks:
appNetwork:
external: true
In above file service-A and service-C are trying to connect to MongoDB
Issue:
In above file service-A and service-C are getting connected to MongoDB before MongoDB is up
so, I want to run service-A and service-C after MongoDB is up completely.
I tried adding depends-on for service-A and service-C
service-A:
image: service-A:1.0
container_name: service-A
ports:
- "3030:3030"
networks:
- appNetwork
depends_on:
- mongodb
service-C:
image: service-C:1.0
container_name: service-C
ports:
- "8082:8082"
networks:
- appNetwork
depends_on:
- mongodb
still, it did not work out sometimes it works sometimes it doesn't so it's not reliable.
I want to make sure my other services are up only after MongoDB is up.
that is a race condition, you're services is trying to start at the same time, thats why you got a result of sometimes it success and sometimes its not.. here an example on how to solve the race condition..
example 1
example 2

Upgrade docker-compose to version 3

This is how my docker-compose.yml file looks like. As you can see, there is a nginx server, a mongoDB, the main application and for testing a nightwatch and selenium container.
nginx:
container_name: 'nginx'
image: 'nginx:1.11'
restart: 'always'
ports:
- '80:80'
- '443:443'
volumes:
- '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
- '/opt/nginx/conf/nginx.conf:/etc/nginx/nginx.conf:ro'
- '/etc/letsencrypt:/etc/letsencrypt'
- '/opt/nginx/www:/var/www:ro'
links:
- 'app'
nightwatch:
container_name: nightwatch
image: 'registry.example.com/project/core:nightwatch'
links:
- selenium
stdin_open: true
tty: true
selenium:
container_name: selenium
image: selenium/standalone-chrome
ports:
- 4444:4444
links:
- app
db:
container_name: db
image: 'mongo:3.4'
restart: 'always'
volumes:
- '/opt/mongo/project/prod:/data/db'
app:
container_name: app
image: 'registry.example.de/project/core:latest'
restart: always
links:
- 'db'
environment:
- ROOT_URL=https://example.com
- MONGO_URL=mongodb://db/db
You can also see, that I'm still using version 1 and I want to upgrade to current version (3). And there are a few problems for converting the file.
For example I do not understand the network option which should be used instead of the deprecated link to get access to containers.
In the nightwatch container I'm running a script like
module.exports = {
'start application': function(browser) {
browser
.url('http://app') // <--
.waitForElementVisible('body', 10000)
.getTitle(function(result) {
this.assert.equal(typeof result, 'string')
})
},
}
So I need to get access to the app container via nightwatch, which also needs selenium. The main app needs of course the db.
I need some help converting to version 3:
version: '3'
services:
nginx:
...
nightwatch:
...
This should be enough:
version: "3"
services:
nginx:
container_name: 'nginx'
image: 'nginx:1.11'
restart: 'always'
ports:
- '80:80'
- '443:443'
volumes:
- '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
- '/opt/nginx/conf/nginx.conf:/etc/nginx/nginx.conf:ro'
- '/etc/letsencrypt:/etc/letsencrypt'
- '/opt/nginx/www:/var/www:ro'
links:
- app
nightwatch:
container_name: nightwatch
image: 'registry.example.com/project/core:nightwatch'
links:
- selenium
stdin_open: true
tty: true
selenium:
container_name: selenium
image: selenium/standalone-chrome
ports:
- 4444:4444
links:
- app
db:
container_name: db
image: 'mongo:3.4'
restart: 'always'
volumes:
- '/opt/mongo/project/prod:/data/db'
app:
container_name: app
image: 'registry.example.de/project/core:latest'
restart: always
links:
- db
environment:
- ROOT_URL=https://example.com
- MONGO_URL=mongodb://db/db
You don't need to configure extra networks than the default provided by compose.
Linking containers is still configured as so. Try this:
version: "3"
services:
abc:
image: ubuntu
command: tail -f /dev/null
cde:
image: busybox
command: ping abc
links:
- abc

Resources