Exposing a Docker database service only on the internal network with Traefik - docker

Let's say I defined two services "frontend" and "db" in my docker-compose.yml which are deployed to a Docker swarm, i.e. they may also run in different stacks. With this setup Traefik automatically generates the frontend and backend for each stack which is fine.
Now I have another Docker container running temporarily in a Jenkins pipeline which shall be able to access the db service in a specific stack. My first idea was to expose the db service by adding it to the cluster-global-net network so that Traefik can generate a frontend route to the bakend. This basically works.
But I'd like to hide the database service from "the public" while still being able to connect another Docker container to it via its stack or service name using the internal "default" network.
Can this be done somehow?
version: '3.6'
networks:
default: {}
cluster-global-net:
external: true
services:
frontend:
image: frontend_image
ports:
- 8080
networks:
- cluster-global-net
- default
deploy:
labels:
traefik.port: 8080
traefik.docker.network: cluster-global-net
traefik.backend.loadbalancer.swarm: 'true'
traefik.backend.loadbalancer.stickiness: 'true'
replicas: 1
restart_policy:
condition: any
db:
image: db_image
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=false
- MYSQL_DATABASE=db_schema
- MYSQL_USER=db_user
- MYSQL_PASSWORD=db_pass
ports:
- 3306
volumes:
- db_volume:/var/lib/mysql
networks:
- default
restart: on-failure
deploy:
labels:
traefik.port: 3306
traefik.docker.network: default

What you need is a network on which both of them are deployed, but that it's not visible from anyone else.
To do such, create a network , add it to your db service and frontend, and also to your temporary service. And indeed, remove traefik label on db because they are not needed anymore here.
EG :
...
networks:
default: {}
cluster-global-net:
external: true
db-net:
external: true
services:
frontend:
image: frontend_image
networks:
- cluster-global-net
- default
- db-net
deploy:
...
db:
image: db_image
...
networks:
- default
- db-net
restart: on-failure
#no labels
docker network create db-net
docker stack deploy -c <mycompose.yml> <myfront>
docker service create --network db-net <myTemporaryImage> <temporaryService>
Then, the temporaryService as well as the frontend can reach the db through db:3306
BTW : you don't need to open the port for the frontend, since traefik will access it internally (trafik.port).
EDIT : new exemple with network created from compose file.
...
networks:
default: {}
cluster-global-net:
external: true
db-net: {}
services:
frontend:
image: frontend_image
networks:
- cluster-global-net
- default
- db-net
deploy:
...
db:
image: db_image
...
networks:
- default
- db-net
restart: on-failure
#no labels
docker stack deploy -c <mycompose.yml> someStackName
docker service create --network someStackName_db-net <myTemporaryImage> <temporaryService>

Related

docker nginx reverse proxy 503 Service Temporarily Unavailable

I want to use nginx as reverse proxy for my remote home automation access.
My infrastructure yaml looks like follows:
# /infrastructure/docker-compose.yaml
version: '3'
services:
proxy:
image: jwilder/nginx-proxy:alpine
container_name: proxy
networks:
- raspberry_network
ports:
- 80:80
- 443:443
environment:
- ENABLE_IPV6=true
- DEFAULT_HOST=${RASPBERRY_IP}
volumes:
- ./proxy/conf.d:/etc/nginx/conf.d
- ./proxy/vhost.d:/etc/nginx/vhost.d
- ./proxy/html:/usr/share/nginx/html
- ./proxy/certs:/etc/nginx/certs
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/tmp/docker.sock:ro
restart: always
networks:
raspberry_network:
My yaml containing the app configuration looks like this:
# /apps/docker-compose.yaml
version: '3'
services:
homeassistant:
container_name: home-assistant
image: homeassistant/raspberrypi4-homeassistant:stable
volumes:
- ./homeassistant:/config
- /etc/localtime:/etc/localtime:ro
environment:
- 'TZ=Europe/Berlin'
- 'VIRTUAL_HOST=${HOMEASSISTANT_VIRTUAL_HOST}'
- 'VIRTUAL_PORT=8123'
deploy:
resources:
limits:
memory: 250M
restart: unless-stopped
networks:
- infrastructure_raspberry_network
ports:
- '8123:8123'
networks:
infrastructure_raspberry_network:
external: true
Via portainer I validated that both containers are contected to the same network. However, when accessing my local IP of the raspberry pi 192.168.0.10 I am receiving "503 Service Temporarily Unavailable".
Of course when I try accessing my app via the virtual host domain xxx.xxx.de it neither works.
Any idea what the issue might be? Or any ideas how to further debug this?
You need to specify the correct VIRTUAL_HOST in the backends environment variable and make sure that they're on the same network (or docker bridge network)
Make sure that any containers that specify VIRTUAL_HOST are running before the nginx-proxy container runs. With docker-compose, this can be achieved by adding to depends_on config of the nginx-proxy container

How can 1 microservice communicate with an instance of elasticsearch in another microservice?

I have currently 2 microservices running. One of them has elasticsearch up and running as a container and the data has been pushed to elasticsearch. The other microservice needs to hit an endpoint that fethces the data in elasticsearch and display that data in the browser. The data can be seen in kibana but not when the other microservice hits that endpoint. What could I be doing wrong?
I have tried adding the name of the alias of elasticsearch in the other microservice .env file but still does not seem to communicate with elasticsearch.
------microservice 1-----
version: '3.5'
services:
micro1_php:
environment:
SERVICE_NAME: micro1-app
DB_CONNECTION: local
ELASTIC_HOST: mic_elasticsearch
ELASTIC_PORT: 9200
networks:
- default
- proxynet
mic_elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.3
environment:
discovery.type: single-node
volumes:
- "./storage/es-data:/usr/share/elasticsearch/data" # to be persistant across docker-compose restarts
networks:
default: {}
proxynet:
aliases:
- micro.elasticsearch
ports:
- "9200:9200"
networks:
proxynet:
name: custom_network
external: true
------microservice 2-----
version: '3.5'
services:
micro2_php:
environment:
SERVICE_NAME: micro2-app
DB_CONNECTION: local
networks:
- default
- proxynet
networks:
proxynet:
name: custom_network
external: true

Docker mis-forwarding ports

I have several domains sharing one public IP (EC2 instance). My setup is like this:
/home/ubuntu contains docker-compose.yml:
version: '3'
services:
nginx-proxy:
image: "jwilder/nginx-proxy"
container_name: nginx-proxy
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
ports:
- "80:80"
restart: "always"
This creates a network named ubuntu_default which will allow other compose instances to join. The nginx-proxy image creates reverse proxies for these other compose instances so that you can visit example.com and be routed to the appropriate UI within the appropriate compose instance.
/home/ubuntu/example.com/project-1 contains a docker-compose.yml like:
version: '3'
services:
db:
build: "./db" # mongo
volumes:
- "./data:/data/db"
restart: "always"
api:
build: "./api" # a node backend
ports:
- "9005:9005"
restart: "always"
depends_on:
- db
ui:
build: "./ui" # a react front end
ports:
- "8005:8005"
restart: "always"
environment:
- VIRTUAL_HOST=project-1.example.com # this tells nginx-proxy which domain to proxy
- VIRTUAL_PORT=8005 # this tells nginx-proxy which port to proxy
networks:
default:
external:
name: ubuntu_default
/home/ubuntu/testing.com/project-2 contains a docker-compose.yml like:
version: '3'
services:
db:
build: "./db" # postgres
volumes:
- "./data:/var/lib/postgresql/data"
restart: "always"
api:
build: "./api" # a python backend
ports:
- "9000:9000"
restart: "always"
depends_on:
- db
ui:
build: "./ui" # a react front end
ports:
- "8000:8000"
restart: "always"
environment:
- VIRTUAL_HOST=testing.com,www.testing.com # tells nginx-proxy which domains to proxy
- VIRTUAL_PORT=8000 # tells nginx-proxy which port to proxy
networks:
default:
external:
name: ubuntu_default
So basically:
project-1.example.com:80 forwards to the UI running on :8005
project-1.example.com:80/api forwards to the API running on :9005
testing.com forwards to the UI running on :8000
testing.com/api forwards to the API running on :9000
...and that all works perfectly as long as I only run one at a time. The moment I start both Compose instances, the /api urls start clashing. I can sit on one of them and refresh repeatedly and sometimes I'll see the one for example.com/api and sometimes I'll see the one for testing.com/api.
I have no idea whats going on at this point. Maybe the premise I'm working against is fundamentally flawed but it seems like an intended use of Docker/Compose. I'm open to suggestions to accomplish the same otherwise.
Docker containers communicate using DNS lookups on their network. If multiple containers have the same alias on the same network, it will round robin load balance between the containers with each network connection. If you don't want containers to talk to each other, then you don't want them on the same docker network. The good news is you solve this by using more than one network, and not putting the api and db server on the frontend proxy network:
version: '3'
services:
db:
build: "./db" # postgres
volumes:
- "./data:/var/lib/postgresql/data"
restart: "always"
api:
build: "./api" # a python backend
ports:
- "9000:9000"
restart: "always"
depends_on:
- db
ui:
build: "./ui" # a react front end
ports:
- "8000:8000"
restart: "always"
networks:
- default
- proxy
environment:
- VIRTUAL_HOST=testing.com,www.testing.com # tells nginx-proxy which domains to proxy
- VIRTUAL_PORT=8000 # tells nginx-proxy which port to proxy
networks:
proxy:
external:
name: ubuntu_default
If you do not override the default network, docker will create one for your compose project and use it for any containers not assigned to another network.

Kibana can't reach elasticsearch

Using docker-compose v3 and deploying to a swarm:
version: '3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.4.1
deploy:
replicas: 1
ports:
- "9200:9200"
tty: true
kibana:
image: docker.elastic.co/kibana/kibana:5.4.1
deploy:
mode: global
ports:
- "5601:5601"
depends_on:
- elasticsearch
tty: true
I see this in the kibana service log:
Unable to revive connection: http://elasticsearch:9200/
Elasticsearch service is running and can be reached.
Swarm consists of 3 nodes.
What am I missing?
Update:
I turns out that if I try to access kibana on the same swarm node where elasticsearch is running, it works. All other nodes either have a network problem or cannot resolve the elasticsearch name.
I found the reason, and the solution.
My swarm is running on AWS - All nodes are placed in the same security group and I assumed all ports were open internally in that security group. That's not the case.
I explicitly configured the security group to allow inbound traffic as per dockers routing mesh specs here: https://docs.docker.com/engine/swarm/ingress/
Docker-compose by default generates a network and puts all services within it. But I do not know if it changes in docker swarm. To define it you can do this.
version: '3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.4.1
deploy:
replicas: 1
ports:
- "9200:9200"
tty: true
networks:
- some-name
kibana:
image: docker.elastic.co/kibana/kibana:5.4.1
deploy:
mode: global
ports:
- "5601:5601"
links:
- elasticsearch
depends_on:
- elasticsearch
tty: true
networks:
- some-name
networks:
some-name:
driver: overlay
I hope it serves you, I will wait for news.

Docker created with docker-compose not visible from outside server

I've created my docker-compose file with 3 dockerfile attached. Everything is working but currently I'd like to expose outside port 8000.
This is not happening. The host is unreachable :(
What's wrong with this?
version: '3'
services:
elastic:
build: ./elastic
ports:
- 5500:80
tty: true
networks:
- default
api:
build: ./api
ports:
- 5000:80
depends_on:
- elastic
tty: true
networks:
- default
web:
build: ./web
ports:
- 8000:80
depends_on:
- api
tty: true
networks:
- outside
- default
networks:
outside:
external:
name: docker_gwbridge
I had a similar issue with an app running on port other than 80/443. I deployed the app on AWS EC2, and the host could not be reached. In order to make it visible, I had to add an inbound rule in "Security Groups" of EC2 instance, which exposed other ports (8000 in my case).

Resources