docker port mapping using docker-gen and letsencrypt-companion - docker

i have several flask applications which i want to run on a server as separate docker containers. on the server i already have several websites running with a reverse proxy and the letsencrypt-nginx-proxy-companion. unfortunately i can't get the containers to run. I think it is because of the port mapping. When I start the containers on port 80, I get the following error message "[ERROR] Can't connect to ('', 80)" from gunicorn. On all other ports it starts successfully, but then I can't access it from outside.
what am I doing wrong?
docker-compose.yml
version: '3'
services:
db:
image: "mysql/mysql-server:5.7"
env_file: .env-mysql
restart: always
app:
build: .
env_file: .env
expose:
- "8001"
environment:
- VIRTUAL_HOST:example.com
- VIRTUAL_PORT:'8001'
- LETSENCRYPT_HOST:example.com
- LETSENCRYPT_EMAIL:foo#example.com
links:
- db:dbserver
restart: always
networks:
default:
external:
name: nginx-proxy
Dockerfile
FROM python:3.6-alpine
ARG CONTAINER_USER='flask-user'
ENV FLASK_APP run.py
ENV FLASK_CONFIG docker
RUN adduser -D ${CONTAINER_USER}
USER ${CONTAINER_USER}
WORKDIR /home/${CONTAINER_USER}
COPY requirements requirements
RUN python -m venv venv
RUN venv/bin/pip install -r requirements/docker.txt
COPY app app
COPY migrations migrations
COPY run.py config.py entrypoint.sh ./
# runtime configuration
EXPOSE 8001
ENTRYPOINT ["./entrypoint.sh"]
entrypoint.sh
#!/bin/sh
source venv/bin/activate
flask deploy
exec gunicorn -b :8001 --access-logfile - --error-logfile - run:app
reverse-proxy/docker-compose.yml
version: '3'
services:
nginx:
image: nginx
labels:
com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
container_name: nginx
restart: always
ports:
- "80:80"
- "443:443"
volumes:
- /srv/www/nginx-proxy/conf.d:/etc/nginx/conf.d
- /srv/www/nginx-proxy/vhost.d:/etc/nginx/vhost.d
- /srv/www/nginx-proxy/html:/usr/share/nginx/html
- /srv/www/nginx-proxy/certs:/etc/nginx/certs:ro
nginx-gen:
image: jwilder/docker-gen
command: -notify-sighup nginx -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
container_name: nginx-gen
restart: always
volumes:
- /srv/www/nginx-proxy/conf.d:/etc/nginx/conf.d
- /srv/www/nginx-proxy/vhost.d:/etc/nginx/vhost.d
- /srv/www/nginx-proxy/html:/usr/share/nginx/html
- /srv/www/nginx-proxy/certs:/etc/nginx/certs:ro
- /var/run/docker.sock:/tmp/docker.sock:ro
- /srv/www/nginx-proxy/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
nginx-letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: nginx-letsencrypt
restart: always
volumes:
- /srv/www/nginx-proxy/conf.d:/etc/nginx/conf.d
- /srv/www/nginx-proxy/vhost.d:/etc/nginx/vhost.d
- /srv/www/nginx-proxy/html:/usr/share/nginx/html
- /srv/www/nginx-proxy/certs:/etc/nginx/certs:rw
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
NGINX_DOCKER_GEN_CONTAINER: "nginx-gen"
NGINX_PROXY_CONTAINER: "nginx"
DEBUG: "true"
networks:
default:
external:
name: nginx-proxy

Related

Celery worker don't download tasks from RabbitMQ after restart all containers in docker-compose

I migrate from Redis to RabbitMQ.
I start my project in docker-compose, I got problem with Celery tasks after I migrate from Redis to RabbitMQ broker.
Celery don't download old tasks when I reload all containers.
Celery got simple logs without downloading old tasks from RabbitMQ.
[2022-10-27 21:44:39,263: INFO/MainProcess] Connected to amqp://admin:**#rabbitmq:5672//
[2022-10-27 21:44:39,293: INFO/MainProcess] mingle: searching for neighbors
[2022-10-27 21:44:40,349: INFO/MainProcess] mingle: all alone
[2022-10-27 21:44:40,414: INFO/MainProcess] celery#29441ac7ffed ready.
docker-compose.yaml
version: "2.2"
services:
nginx:
build:
context: ./nginx
dockerfile: Dockerfile
container_name: nginx
restart: always
ports:
- ${PUB_PORT}:80
volumes:
- static_volume:/var/www/static
- ./backend/mediafiles:/var/www/media
depends_on:
- django
django:
build:
context: ./backend
dockerfile: Dockerfile.prod
container_name: backend
restart: always
env_file:
- ./.env.prod
environment:
- IS_DOCKER=True
- DJANGO_SETTINGS_MODULE=core.settings.production
volumes:
- static_volume:/django/staticfiles
- ./backend/mediafiles:/django/mediafiles
- ./backend:/django # only for local development
depends_on:
postgres:
condition: service_healthy
aiogram:
build:
context: ./telegram_bot
dockerfile: Dockerfile
container_name: telegram_bot
restart: always # crash: not found token
command: ["python", "main.py"]
volumes:
- ./backend/mediafiles:/bot/mediafiles
env_file:
- ./.env.prod
environment:
- IS_DOCKER=True
depends_on:
- django
postgres:
image: postgres:13.0-alpine
container_name: project_db
restart: always
volumes:
- postgres_volume:/var/lib/postgresql/data
depends_on:
- redis
ports:
- 54321:5432
environment:
- POSTGRES_DB=${POSTGRES_DB}
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
healthcheck:
test: ["CMD","pg_isready", "--username=${POSTGRES_USER}","-d", "{POSTGRES_DB}"]
redis:
build: ./redis
ports:
- ${REDIS_PORT}:6379
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD}
volumes:
- ./redis/redis.conf/:/usr/local/etc/redis.conf
- ./redis/data:/usr/local/redis/data
- ./redis/redis.log:/usr/local/redis/redis.log
restart: always
container_name: redis
# celery worker
celery:
container_name: celery
restart: always
build:
context: ./backend
dockerfile: Dockerfile.celery.prod
command: celery -A core worker -l info
environment:
- DJANGO_SETTINGS_MODULE=core.settings.production
env_file:
- ./.env.prod
depends_on:
- django
- redis
- postgres
- rabbitmq
# message broker for celery
rabbitmq:
container_name: rabbitmq
restart: always
image: rabbitmq:3.9-alpine
volumes:
- "./rabbitmq-data:/var/lib/rabbitmq"
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
ports:
- "5672:5672"
- "15672:15672"
volumes:
postgres_volume:
static_volume:
redis_data:
Dockerfile.celery.prod
FROM python:3.8.5
WORKDIR /django
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Copy only requirements to cache them in docker layer
RUN pip install --upgrade pip
COPY ./requirements.txt /django/
RUN pip install -r requirements.txt
COPY . .
I tried to run delayed tasks with Celery worker after reload all containers in docker-compose including RabbitMQ too.

docker-composer up -d getting error postgres: Bind for 0.0.0.0:5432 failed: port is already allocated

I am on the Mac with docker install version 2.0.0.3 (31259)
docker-compose up -d
Removing ab-insight_postgres_1
Starting ab-insight_data_1 ... done
Recreating 31d36fb9c48a_ab-insight_postgres_1 ... error
ERROR: for 31d36fb9c48a_ab-insight_postgres_1 Cannot start service postgres: b'driver failed programming external connectivity on endpoint ab-insight_postgres_1 (5ed1c634dd3a43c2cd988ff7f14b5c1f3cde848e375c2915cf92420f819e21ac): Error starting userland proxy: Bind for 0.0.0.0:5432 failed: port is already allocated'
ERROR: for postgres Cannot start service postgres: b'driver failed programming external connectivity on endpoint ab-insight_postgres_1 (5ed1c634dd3a43c2cd988ff7f14b5c1f3cde848e375c2915cf92420f819e21ac): Error starting userland proxy: Bind for 0.0.0.0:5432 failed: port is already allocated'
ERROR: Encountered errors while bringing up the project.
Here is my docker-compose.yml
version: '2'
services:
web:
restart: always
build: ./web
expose:
- "8000"
volumes:
- /home/flask/app/web
command: /usr/local/bin/gunicorn -w 2 -b :8000 project:app
depends_on:
- postgres
nginx:
restart: always
build: ./nginx
ports:
- "80:80"
volumes:
- /www/static
volumes_from:
- web
depends_on:
- web
data:
image: postgres:11
volumes:
- /var/lib/postgresql
command: "true"
postgres:
restart: always
build: ./postgresql
volumes_from:
- data
expose:
- "5432"
and here is my Dockerfile
FROM python:3.6.1
MAINTAINER Ka So <kanel.soeng#kso.com>
# Create the group and user to be used in this container
RUN groupadd flaskgroup && useradd -m -g flaskgroup -s /bin/bash flask
# Create the working directory (and set it as the working directory)
RUN mkdir -p /home/flask/app/web
WORKDIR /home/flask/app/web
# Install the package dependencies (this step is separated
# from copying all the source code to avoid having to
# re-install all python packages defined in requirements.txt
# whenever any source code change is made)
COPY requirements.txt /home/flask/app/web
RUN pip install --no-cache-dir -r requirements.txt
# Copy the source code into the container
COPY . /home/flask/app/web
RUN chown -R flask:flaskgroup /home/flask
USER flask
run docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
This is happening due to postges running locally on your machine on the same port you have mentioned in your docker-compose.yml for postges service.
Either stop the sevice running on your local machine.(not recommended)
Or use other port to map to 5432 port of docker. To do so replace the
expose
-5432
in postgresa service with the following code
ports:
- "5433:5432"
The whole docker compose file will look like:
version: '2'
services:
web:
restart: always
build: ./web
expose:
- "8000"
volumes:
- /home/flask/app/web
command: /usr/local/bin/gunicorn -w 2 -b :8000 project:app
depends_on:
- postgres
nginx:
restart: always
build: ./nginx
ports:
- "80:80"
volumes:
- /www/static
volumes_from:
- web
depends_on:
- web
data:
image: postgres:11
volumes:
- /var/lib/postgresql
command: "true"
postgres:
restart: always
build: ./postgresql
volumes_from:
- data
ports:
- "5433:5432"

Live debugging node application running in docker

I have application built in React running on Docker. I am looking for a way to debug it. I am using Visual Studio Code. Here is my Docker file and Docker-compose file
FROM node:boron
ARG build_env
RUN mkdir /usr/share/unicode && cd /usr/share/unicode && wget ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt
COPY package.json /tmp/package.json
RUN cd /tmp && npm install
COPY ./shim/RelayDefaultNetworkLayer.js /tmp/node_modules/react-relay/lib/RelayDefaultNetworkLayer.js
COPY ./shim/buildRQL.js /tmp/node_modules/react-relay/lib/buildRQL.js
RUN mkdir -p /var/www && cp -a /tmp/node_modules /var/www/
WORKDIR /var/www
COPY . ./
RUN if [ "$build_env" != "development" ]; then npm run build-webpack && npm run gulp; fi
EXPOSE 8080
CMD ["npm", "run", "--debug=5858 prod"]
My docker-compose file looks like
version: '2'
services:
nginx:
container_name: nginx
image: openroad/nginx
build:
context: nginx
ports:
- "80:80"
volumes:
- ./nginx/nginx.development.conf:/etc/nginx/nginx.conf
networks:
- orion-network
graphql:
container_name: graphql
image: openroad/graphql
build:
context: integration_api
volumes:
- ./integration_api:/var/www
environment:
- NODE_ENV=development
command: npm run dev
working_dir: /var/www
networks:
orion-network:
ipv4_address: 172.16.238.10
pegasus:
container_name: pegasus
image: openroad/pegasus
build:
context: pegasus
args:
build_env: development
expose:
- "3000"
volumes:
- ./pegasus:/var/www/public
environment:
- NODE_ENV=development
command: npm run dev
working_dir: /var/www/public
extra_hosts:
- "local.pegasus.com:192.168.99.100"
networks:
orion-network:
ipv4_address: 172.16.238.11
frontend:
container_name: orion-frontend
image: openroad/orion-frontend
build:
context: orion-frontend
args:
build_env: development
expose:
- "3000"
ports:
- "5858:5858"
volumes:
- ./orion-frontend:/var/www/public
environment:
- NODE_ENV=development
command: npm run --debug=5858 dev
working_dir: /var/www/public
networks:
orion-network:
ipv4_address: 172.16.238.12
admin:
container_name: orion-admin
image: openroad/orion-admin
build:
context: orion-admin
args:
build_env: development
expose:
- "3000"
volumes:
- ./orion-admin:/var/www/
environment:
- NODE_ENV=development
command: npm run dev
working_dir: /var/www/
networks:
orion-network:
ipv4_address: 172.16.238.13
uploads:
container_name: orion-uploads
image: openroad/orion-uploads
build:
context: orion-uploads
volumes:
- ./orion-uploads:/var/www/
working_dir: /var/www/
networks:
orion-network:
ipv4_address: 172.16.238.14
dashboard:
container_name: orion-dashboard
image: openroad/orion-dashboard
build:
context: orion-dashboard
args:
build_env: development
volumes:
- ./orion-dashboard/src:/var/www/src
- ./orion-dashboard/package.json:/var/www/package.json
- ./orion-dashboard/webpack.config.babel.js:/var/www/webpack.config.babel.js
- ./orion-dashboard/node_modules:/var/www/node_modules
- ./orion-dashboard/data/babelRelayPlugin.js:/var/www/data/babelRelayPlugin.js
working_dir: /var/www
environment:
- NODE_ENV=development
- GRAPHQLURL=http://172.16.238.10:8080/graphql
- PORT=8080
command: npm run dev
networks:
orion-network:
ipv4_address: 172.16.238.15
networks:
orion-network:
driver: bridge
driver_opts:
com.docker.network.bridge.enable_ip_masquerade: "true"
ipam:
driver: default
config:
- subnet: 172.16.238.0/24
gateway: 172.16.238.1
I wanted ability to debug application running under orion-frontend container. I tried various option without any success. I tried https://codefresh.io/docker-tutorial/debug_node_in_docker/ and https://blog.docker.com/2016/07/live-debugging-docker/ already.
I may be wrong about the command syntax for npm run (didn't find this command in the npm docs), but you may need to separate the --debug=5858 and prod args, like this:
CMD ["npm", "run", "--debug=5858", "prod"]

Why Dockerfile doesn't run multiple commands

I want use Docker run my project(react+nodejs+mongodb),
Dockerfile:
FROM node:8.9-alpine
ENV NODE_ENV production
WORKDIR /usr/src/app
COPY ["package.json", "package-lock.json*", "npm-shrinkwrap.json*", "./"]
RUN npm install --production --silent && mv node_modules ../
COPY . .
CMD nohup sh -c 'npm start && node ./server/server.js'
docker-compose.yml:
version: '2.1'
services:
chat:
image: chat
container_name: chat
build: .
environment:
NODE_ENV: production
ports:
- "3000:3000"
- "8080:8080"
volumes:
- ./:/usr/src/app
links:
- mongo
mongo:
container_name: mongo
image: mongo
ports:
- "27017:27017"
run docker-compose up --build, the 3000 port is worked, but the 8080 port dies
localhost:3000
localhost:8080
I would suggest create a container for the server and have it seperate from the "chat" container. Its best to have each container do one thing and one thing only (almost like the philosophy behind unix commands)
In any case here is some modifications that I would make to the compose file.
version: '2.1'
services:
chat:
image: chat
container_name: chat
build: .
environment:
NODE_ENV: production
ports:
- "3000:3000"
- "8080:8080"
volumes:
- ./:/usr/src/app
links:
- mongo
mongo:
container_name: mongo
image: mongo
# You don't need to expose this port to the outside world. Because you linked the two containers the chat app
# will be able to connect to mongodb using hostname mongodb inside the container network.
# ports:
# - "27017:27017"
Btw what happens if you run:
$ docker-compose down
and then
$ docker-compose up
$ docker ps
can you see the ports exposed in docker ps output?
your chat service depends on mongo so you also need to have this in your chat
depends_on:
- mongo
This docker-compose file works for me. Note that i am saving the data from the database to a local directory. You should add this directory to gitignore.
version: "3.2"
services:
mongo:
container_name: mongo
image: mongo:latest
environment:
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=password
- NODE_ENV=production
ports:
- "28017:27017"
expose:
- 28017 # you can connect to this mongodb with studio3t
volumes:
- ./mongodb-data:/data/db
restart: always
networks:
- docker-network
express:
container_name: express
environment:
- NODE_ENV=development
restart: always
build:
context: .
args:
buildno: 1
expose:
- 3000
ports:
- "3000:3000"
links:
- mongo # link this service to the database service
depends_on:
- mongo
command: "npm start" # override the default command to use nodemon in dev
networks:
- docker-network
networks:
docker-network:
driver: bridge
You may also find that using node you have to wait for the mongodb container to be ready before you can connect to the database.

Multiple services with different ports and the same domain using jwilder/nginx-proxy

I have some services in docker-compose:
version: "3"
services:
site:
volumes:
- .:/app
build:
dockerfile: Dockerfile.dev
context: docker
ports:
- "80:80"
webpack:
image: node:6.12.0
ports:
- "8080:8080"
volumes:
- .:/app
working_dir: /app
command: bash -c "yarn install; yarn run gulp server"
db:
image: mysql:5.7.20
ports:
- "3306:3306"
environment:
MYSQL_DATABASE: ${DB_NAME}
MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
And I can connect to exposed ports of services:
Site -- localhost:80
Webpack -- localhost:8080
MySQL: -- localhost:3306
How can I use nginx-proxy to expose multiple ports of different servers on the same domain (?):
Site -- example.dev:80
Webpack -- example.dev:8080
MySQL: -- example.dev:3306
This works:
services:
nginx-proxy:
image: jwilder/nginx-proxy
ports:
- "80:80"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
site:
volumes:
- .:/app
build:
dockerfile: Dockerfile.dev
context: docker
expose:
- 80
environment:
VIRTUAL_HOST: ${VIRTUAL_HOST}
But this is not:
services:
nginx-proxy:
image: jwilder/nginx-proxy
ports:
- "80:80"
- "8080:8080"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
site:
volumes:
- .:/app
build:
dockerfile: Dockerfile.dev
context: docker
expose:
- 80
environment:
VIRTUAL_HOST: ${VIRTUAL_HOST}
webpack:
image: node:6.12.0
expose:
- 8080
environment:
VIRTUAL_HOST: ${VIRTUAL_HOST}
VIRTUAL_PORT: 8080
volumes:
- .:/app
working_dir: /app
command: bash -c "yarn install; yarn run gulp server"
What am I do wrong? How can I solve this problem?
//Sorry for my worst English. Hope you'll understand me
Update:
This is just an example. In the future I'll make proxy as external network and will connect services to it. And I wont to run two docker-compose "files" on the same host (VPS). Purpose: production and test versions on the same host, that use same ports BUT different domains. For example:
example.com -- Web Site
example.com:81 -- PhpMyAdmin
test.example.com -- Web Site for testing
test.example.com:81 -- PhpMyAdmin for testing

Resources