Make container X to wait n seconds untill localstack queues are created - docker

I'm trying to create a docker-compose file to run localstack (for sqs), 1 mysql databases and 2 services together.
The problem I'm trying to deal with is that the services starts to build and run before the queues are created (which I don't want).
Is there a way to make the services sleep ? I've tried to use health check but it didn't make a difference.
Here's how the file looks:
version: "3.8"
services:
localstack:
container_name: "DGT-localstack_main"
image: localstack/localstack
ports:
- "4566:4566" # LocalStack Gateway
- "4510-4559:4510-4559" # external services port range
- "53:53" # DNS config (only required for Pro)
- "53:53/udp" # DNS config (only required for Pro)
- "443:443" # LocalStack HTTPS Gateway (only required for Pro)
environment:
- DEBUG=${DEBUG-}
- PERSISTENCE=${PERSISTENCE-}
- LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
- LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY-} # only required for Pro
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
networks:
- localstack_network
awslocal_cli:
image: amazon/aws-cli
depends_on:
- localstack
entrypoint: /bin/sh -c
networks:
- localstack_network
command: >
'
echo "########### Creating profile ###########"
aws configure set aws_access_key_id ignore
aws configure set aws_secret_access_key ignore
aws configure set region eu-north-1
echo "########### Creating SQS ###########"
aws sqs create-queue --endpoint-url=http://localstack:4566 --queue-name=FIRST_QUEUE
aws sqs create-queue --endpoint-url=http://localstack:4566 --queue-name=SECOND_QUEUE
echo "########### Listing SQS ###########"
aws sqs list-queues --endpoint-url=http://localstack:4566
'
db:
container_name: db
image: mysql:8.0.28
command: --lower_case_table_names=1
ports:
- "3308:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=maindb
volumes:
- ./db_config/core/data:/var/lib/mysql
networks:
- localhost_network
api:
container_name: Api
image: api:1.0
build:
context: blabla
dockerfile: blabla
ports:
- blabla
env_file: ./Server/common.env
environment:
- blabla
restart: on-failure
depends_on:
- core
networks:
- localhost_network
core:
container_name: Core
image: core:1.0
build:
context: blabla
dockerfile: blabla
ports:
- "5115:80"
env_file: .blabla
environment:
- blabla
restart: on-failure
depends_on:
- localstack
- awslocal_cli
- db
networks:
- localstack_network
- localhost_network
networks:
localstack_network:
localhost_network:
I'm sorry for the bad indentation (they are right indented in my file)

Use the depends_on function.
If you want container B to start after container A you would need to create you docker-compose like this
A:
image: ...
other_settings: ...
B:
image: ...
other_settings: ...
depends_on: A
This would ensure that container B is only started if container A is already up and running.
If there is a URL to check if the queue is already created you could also try this:
https://docs.docker.com/compose/startup-order/

You can try to wrap starting command in script wait-for-it.sh. Complete example is hire: https://docs.docker.com/compose/startup-order/

Related

Trying to connect all my docker containers to a separate MariaDB container

So, I've setup several container apps that use MariaDB as their db backend, using docker-compose.
Containers are setup as needed and therefore MariaDB gets installed each time on every container that uses the db.
For example, I have some containers (PHPMyAdmin, NGiNX-PM, etc.) that use MariaDB, and they, in turn, have a version of it installed within their container. I also have a separate container (MariaDB) that I would rather have shared amongst the other containered apps and, thereby, I'd only have to maintain one version of the db.
I've searched for a solution, but no luck. Needless to say, I'm a noob at docker.
The only thing I can come up with is that all the apps need to be installed through the same docker-compose.yaml file to use the same db? That would make for a very long file if I had many containers running, and I'd prefer to have a directory per app and all the app's contents available in this one location.
I'm sure there is a way, I just haven't been able to figure it out.
So this is what I've tried:
The following setup is what I've tried but I am unable to get it to work:
(/docker/apps/mariadb/mariadb.yml)
version: '3.9'
networks:
NET:
external: true
services:
#############################################################################################
# MariaDB (docker-compose -f mariadb.yml up -d) #
#############################################################################################
mariadb:
image: jsurf/rpi-mariadb:latest
restart: unless-stopped
environment:
- TZ=${TIMEZONE}
- MYSQL_DATABASE=dockerApps
- MYSQL_USER=root
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
volumes:
- $HOME/docker/apps/mariadb/db:/var/lib/mysql
expose:
- '3306'
networks:
- NET
(/docker/apps/nginxpm/nginxpm.yml)
version: '3.9'
networks:
NET:
external: true
services:
#############################################################################################
# NGiNX Proxy Manager (docker-compose -f nginxpm.yml up -d) #
#############################################################################################
nginxpm:
container_name: NGiNX_Proxy_Manager
image: 'jc21/nginx-proxy-manager:latest'
ports:
- '80:80'
- '81:81'
- '443:443'
volumes:
- ./config.json:/app/config/production.json
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
networks:
- NET
depends_on:
- mariadb
(/docker/apps/phpmyadmin/phpmyadmin.yml)
version: "3.9"
networks:
NET:
external: true
services:
#############################################################################################
# phpMyAdmin (docker-compose up -d -OR- docker-compose -f phpmyadmin.yml up -d) #
#############################################################################################
phpmyadmin:
image: phpmyadmin:latest
container_name: phpMyAdmin
restart: unless-stopped
environment:
PMA_HOST: mariadb
PMA_USER: root
PMA_PASSWORD: ${MYSQL_PASSWORD}
volumes:
# Must add ServerName directive to end of file "ServerName 127.0.0.1"
- $HOME/docker/apps/phpmyadmin/apache2.conf:/etc/apache2/apache2.conf
ports:
- '8004:80'
networks:
- NET
Any help in this matter is greatly appreciated.
Ok, so after some more reading and testing, I've found the answer to my issue. I was assuming that "depends_on" was supposed to connect the containers, somehow. Not true!
I found that "external_links" is the correct way of connecting them.
So, my final docker-compose file looks like this:
(/docker/apps/nginxpm/nginxpm.yml)
version: '3.9'
networks:
NET:
external: true
services:
#############################################################################################
# NGiNX Proxy Manager (docker-compose -f nginxpm.yml up -d) #
#############################################################################################
nginxpm:
container_name: NGiNX_Proxy_Manager
image: 'jc21/nginx-proxy-manager:latest'
ports:
- '80:80'
- '81:81'
- '443:443'
volumes:
- ./config.json:/app/config/production.json
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
networks:
- NET
external_links:
- mariadb

Running Multiple Microservices with multiple docker compose with NATS

I am new to microservices and I have a project to setup multiple microservies, The project is setup like this.
Every nest js application has
API Application exposed to a port
database
docker-compose file responsible which creates the containers for each microservice.
Now what I am doing is to have
Nest JS MICROSERVICE APP 1
- API exposed to port 5000
- Postgres database working on 5432
- NATS running on 4222
NEST JS APP MICROSERVICE 2
- API exposed to port 5001
- Postgres database working on 5433
- NATS not running on 4222 as it is already occupied. If I change the port how I am gonna use the same message broker on both services.
The problem is I wanted to use the same NATS message broker on the second microservice and all the newly created microservice. my docker-compose file for NEST JS APP 1 is as follows.
version: '3.9'
services:
api:
container_name: nest_app_1
image: nest_app_1
build:
dockerfile: Dockerfile
context: .
ports:
- 127.0.0.1:5000:5000
env_file:
- .env
depends_on:
- db
- nats
networks:
- main
db:
container_name: postgres
image: postgres:latest
ports:
- 127.0.0.1:5432:5432
volumes:
- ./data:/var/lib/postgresql/data
env_file:
- .env
networks:
- main
pgadmin:
container_name: pgadmin
image: dpage/pgadmin4
ports:
- 127.0.0.1:8080:80
env_file:
- .env
networks:
- main
nats:
image: nats-streaming:latest
entrypoint:
- /nats-streaming-server
- -cid
- main_cluster
ports:
- "127.0.0.1:4222:4222"
- "127.0.0.1:6222:6222"
- "127.0.0.1:8222:8222"
restart: always
tty: true
networks:
- main
networks:
main:
driver: bridge
Second NEST JS microservice docker-compose is as follows
version: '3.9'
services:
api:
container_name: nest_app_2
image: nest_app_2
build:
dockerfile: Dockerfile
context: .
ports:
- 127.0.0.1:5001:5001
env_file:
- .env
depends_on:
- app_db_2
networks:
- main
app_db_2:
container_name: postgres_2
image: postgres:latest
ports:
- 127.0.0.1:5433:5432
volumes:
- ./data:/var/lib/postgresql/data
env_file:
- .env
networks:
- main
pgadmin:
container_name: pgadmin_2
image: dpage/pgadmin4
ports:
- 127.0.0.1:8081:80
env_file:
- .env
networks:
- main
nats:
image: nats-streaming:latest
entrypoint:
- /nats-streaming-server
- -cid
- main_cluster
ports:
- "127.0.0.1:4222:4222"
restart: always
tty: true
networks:
- main
networks:
main:
driver: bridge
Now I want to use NATS to communicate between both apps. So if I publish message from microservice 1 and I subscribe that message to microservice 2 and so on.
yes, sure the host ports are occupied if you link it through the Host network stack. You can only have one service linked to ip:port
It looks like you trying to start two NATS instances and let them join the same NATS cluster. But maybe you need two instances for development. You just want to see messages passing through it.
Option 1: just put everything in one compose and use depends_on and the same NATS node for both services
Option 2: Use a separate compose stack to provision your NATS infrastructure and use
extrnal_links.
Option 3: Define custom network for NATS cluster where every NATS container get's own iP.
But I would start with 1.

understanding Docker compose mongodb and rails application

there is ruby on rails application which uses mongodb and postgresql databases. When I run it locally everything works fine, however when I try to open in a remote container, it throws error message
2021-03-14T20:22:27.985+0000 Failed: error connecting to db server: no reachable servers
the docker-compose.yml file defines following services:
redis mongodb db rails
I start remote containers with following command:
docker-compose build - build successful
docker-compose up -d - containers are up and running
when I connect to the rails container and try to do
bundle exec rake aws:restore_db
error mentioned above is thrown. I don't know what is wrong here. The mongodb container is up and running.
the docker-compose.yml is mentioned below:
version: '3.4'
services:
redis:
image: redis:5.0.5
mongodb:
image: mongo:3.6.13
volumes:
- mongo-data:/data/db
db:
image: postgres:11.3
volumes:
- db-data:/var/lib/postgresql/data
rails:
build: .
image: proj:latest
depends_on:
- db
- mongodb
- redis
volumes:
- .:/proj
ports:
- "3000:3000"
tty: true
stdin_open: true
env_file:
- .env/development.env
volumes:
db-data:
mongo-data:
this is how I start all four remote containers:
$ docker-compose up -d
Starting proj_db_1 ... done
Starting proj_redis_1 ... done
Starting proj_mongodb_1 ... done
Starting proj_rails_1 ... done
please help me to understand how remote containers should interact with each other.
Your configuration should point to the services by name and not to a port on localhost. For example, if you ware connecting to redis as localhost:6380 or 127.0.0.1:6380, now you need to use redis:6380
If this is still not helping, you can try to add links between containers in order the names given to them as services to be resolved. So the file will look something like this:
version: '3.4'
services:
redis:
image: redis:5.0.5
networks:
- front-end
links:
- "mongodb:mongodb"
- "db:db"
- "rails:rails"
mongodb:
image: mongo:3.6.13
volumes:
- mongo-data:/data/db
networks:
- front-end
links:
- "redis:redis"
- "db:db"
- "rails:rails"
db:
image: postgres:11.3
volumes:
- db-data:/var/lib/postgresql/data
networks:
- front-end
links:
- "redis:redis"
- "mongodb:mongodb"
- "rails:rails"
rails:
build: .
image: proj:latest
depends_on:
- db
- mongodb
- redis
volumes:
- .:/proj
ports:
- "3000:3000"
tty: true
stdin_open: true
env_file:
- .env/development.env
networks:
- front-end
links:
- "redis:redis"
- "mongodb:mongodb"
- "db:db"
volumes:
db-data:
mongo-data:
networks:
front-end:
The links will allow for a hostnames to be defined in the containers.
The link flag is legacy, and in new versions of docker-engine it's not required for user defined networks. Also, the links will be ignored in case of docker swarm deployment. However since there are sill old installations of Docker and docker-compose, this is one thing to try in troubleshooting.

How to deploy on AWS a Rails webpack_dev_server Docker service?

I have a simple Rails/React app that works with Docker with 3 services:
'database' for postgres
'web' for Rails
'webpack_dev_server' for react
In AWS I've created:
* built a custom image for nginx,
* set s3 to hold ecs configs.
* a production cluster,
* private repositories for the 'web' and nginx, tagged both images and pushed to the repositories
* create 4 ec2 instances, 2 for the web and 2 for react
Now I'm ready to create task definitions but I'm not sure how to handle webpack_dev_server (React).
Can we build the image with the same dockerfile as the the web?
For the task definition, should it look like the web as well?
Here's the docker-compose.yml file that works.
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- .:/usr/src/app
- gem_cache:/gems
env_file:
- .env/development/database
- .env/development/web
environment:
- WEBPACKER_DEV_SERVER_HOST=webpack_dev_server
- DOCKERIZED=true
webpack_dev_server:
build: .
command: ./bin/webpack-dev-server
ports:
- 3035:3035
volumes:
- .:/usr/src/app
- gem_cache:/gems
env_file:
- .env/development/web
- .env/development/database
environment:
- WEBPACK_DEV_SERVER=0.0.0.0
database:
image: postgres
env_file:
- .env/development/database
volumes:
- db_data:/var/lib/postgresql/data
volumes:
db_data:
gem_cache:

How to run Docker container in it's own network

Today I switched from "Docker Toolbox" to "Docker for Mac", because Docker now has finally write-access to my User directory (which doesn't worked with "Docker Toolbox") - Yay!
But this change also includes that all containers now running under my localhost and not under Docker's IP as before (e.g. 192.168.99.100).
Since my localhost listens to various ports by default (80, 443, ...) and I don't want to always add new created ports, that doesn't conflict with the standard one's, to my local dev domains (e.g. example.dev:8443), I wonder how to run my containers as before.
I read about network configs and tried a lot of things (creating a new host network, exposing ports with an IP in front of it, ...), but didn't got it working.
What kind of config do I need to run my app container with the IP 192.168.99.100? Thats my docker-compose.yml so far.
version: '2'
services:
app:
build:
context: .
dockerfile: Dockerfile
depends_on:
- mysql
- redis
- memcached
ports:
- 80:80
- 443:443
- 22:22
- 3000:3000
- 3001:3001
volumes:
- ./app/:/app/
- /tmp/debug/:/tmp/debug/
- ./:/docker/
volumes_from:
- storage
# cap and privileged needed for slowlog
cap_add:
- SYS_PTRACE
privileged: true
env_file:
- etc/environment.yml
- etc/environment.development.yml
mysql:
build:
context: docker/mysql/
dockerfile: MariaDB-10
ports:
- 3306:3306
volumes_from:
- storage
volumes:
- ./data/mysql:/var/lib/mysql
- /tmp/debug/:/tmp/debug/
env_file:
- etc/environment.yml
- etc/environment.development.yml
redis:
build: docker/redis/
volumes_from:
- storage
env_file:
- etc/environment.yml
- etc/environment.development.yml
memcached:
build: docker/memcached/
volumes_from:
- storage
env_file:
- etc/environment.yml
- etc/environment.development.yml
storage:
build: docker/storage/
volumes:
- /storage
You need to declare "networks:" for each of your services:
e.g.
version: '2'
services:
app:
image: xxxx:xxx
ports:
- "80:80"
networks:
- my-network
mysql:
image: xxxx:xxx
networks:
- my-network
networks:
my-network:
driver: bridge
Then from side your app configuration, you can use "mysql" as the hostname of database server.
You can define a network in your compose file, then add any services to the network.
https://docs.docker.com/compose/networking/
But I would suggest you just use different ports now that you are running natively. I.e. 8080:80

Resources