I'm trying to generate multiple instances of the same git repository with a generic docker-compose.yml file and multiple .env files.
For this somewhere in the code I generate a temporary folder which contains:
.env:
APP_PORT="3000"
APP_NAME="app-name"
REPO_NAME="repo-name"
docker-compose.yml
version: '3.6'
services:
web-app:
image: golang:alpine
environment:
- APP_PORT
- APP_NAME
- REDIS_HOST=db-app
ports:
- ${APP_PORT}:${APP_PORT}
volumes:
- /opt/docker/repositories/${REPO_NAME}:/app
command: sh -c "cd /app && go run ./"
db-app:
image: redis:alpine
then running docker-compose config in this directory gives me the following output :
services:
db-app:
image: redis:alpine
web-app:
command: sh -c "cd /app && go run ./"
environment:
APP_NAME: app-name
APP_PORT: '3000'
REDIS_HOST: db-app
image: golang:alpine
ports:
- published: 3000
target: 3000
volumes:
- /opt/docker/repositories/repo-name:/app:rw
version: '3.6'
This did not only interpolate env variables, it also changed some fields such as ports with published and target, and a :rw at the end of my volume.
This is all done in Go, and when I try to unmarshal the output into a Go struct with yaml fields, it is not recognized as a valid docker-compose file because of the ports field (which is supposed to be an array of strings).
How can I make it so docker-compose config only replaces the ${APP_PORT} with its value and not add these extra unwanted fields ?
Reading the source code, I found this in the config types:
def legacy_repr(self):
return normalize_port_dict(self.repr())
Which is the representation you need. So I searched for legacy_repr in the source code and found this:
if 'ports' in service_dict:
service_dict['ports'] = [
p.legacy_repr() if p.external_ip or version < VERSION else p
for p in service_dict['ports']
]
So apparently, to trigger the use of the legacy representation, you either need to have an external IP address or need to do something with the version. I tried to downgrade the docker-compose.yaml file version but it didn't change anything (maybe it's the docker-compose CLI's version instead).
Reading the spec of the docker-compose config file, in the ports section, you can specify the IP address in the short syntax:
[HOST:]CONTAINER[/PROTOCOL] where:
HOST is [IP:](port | range)
CONTAINER is port | range
PROTOCOL to restrict port to specified protocol. tcp and udp values are defined by the specification, Compose implementations MAY offer support for platform-specific protocol names.
So a solution is to replace ${APP_PORT}:${APP_PORT} by:
0.0.0.0:${APP_PORT}:${APP_PORT}
By setting the external IP address to 0.0.0.0 you are not restricting anything and you force the use of the legacy representation.
Related
I am trying to use docker volume for the first time and I am having a hard time getting the container to share files with the host machine (Ubuntu). I can see the files my code is writing inside the container using docker exec but none of the files are in the volume under /var/lib/docker/volumes.
My DockerFile
FROM node:16-alpine
RUN apk add dumb-init
RUN addgroup gp && adduser -S appuser -G gp
RUN mkdir -p /usr/src/app/logs
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . /usr/src/app/
RUN chown -R appuser:gp /usr/src/app/logs/
USER appuser
My docker-compole.yml
version: "3.6"
services:
my-service:
user: appuser
container_name: demou
build:
context: .
image: "myService"
working_dir: /usr/src/app
ports:
- 8080:8080 #
environment:
- NODE_VERSION=16
volumes:
- /logs:/logs/:rw
command: sh -c "dumb-init node src/server.js"
networks:
- Snet
# restart: always
volumes:
logs:
# driver: local
name: "logs"
networks:
Snet:
name: "Snetwork"
server.js doesn't do anything besides writing a helloworld.txt file to the logs directory. when I run the app in the container,I dont see any errors or even warning. It's just the logs are not available on the host machine where docker keeps its volumes. What I missing here?
Thanks
The compose file uses a bind mount (indicated by the leading / before logs:
...
services:
my-service:
...
volumes:
- /logs:/logs/:rw
# ^ this slash makes the mount a bind mount
...
We actually want to use a named volume by removing the leading /:
...
services:
my-service:
...
volumes:
- logs:/logs/:rw
# ^ no slash, will be interpreted as named volume
# referencing the named volume "logs" defined below
...
volumes:
logs:
# driver: local
name: "logs"
...
For more details, please refer to the relevant docker-compose file documentation.
As an aside: I had problems starting the docker-compose.yml file due to an invalid reference format. The image name must not include uppercase letters. So I had to change it to my-service. Even then, I was not able to build the my-service image due to missing files.
Here is a full docker-compose.yml that reproduces the desired behaviour, I used an alpine with a simple script to write to the volume:
version: "3.6"
services:
my-service:
image: alpine:3.14.3
working_dir: /logs
volumes:
- logs:/logs/:rw
command: sh -c 'echo "Hello from alpine" > log.txt'
volumes:
logs:
name: logs
You hint that you're trying to actually read the logs that come out, reasonably enough. For this use case you should use a Docker bind mount and not a named volume.
Where you specify
volumes:
- /logs:/logs:rw
The first part (starting with a slash) is an absolute path on the host; if you ls / on the host system, outside a container, you should see the logs directory there. The second part is a path inside the container, which doesn't match what you've indicated in the Dockerfile. If you change it to
volumes:
- ./logs:/usr/src/app/logs:rw
# ^^ ^^^^^^^^^^^^
making it a relative path on the host side and the intended directory on the container side, then you will be able to directly read the logs in a subdirectory of the directory containing the docker-compose.yml file. You can delete the volumes: block at the end of the file.
(For completeness, if the left-hand side of a volumes: entry doesn't contain a slash at all, it refers to a named volume specified in the top-level volumes: block; see also #Turing85's answer.)
Permissions-wise, the container process must run as the same numeric user ID that owns the log directory. Any other directories that the container writes to must also have the same numeric owner. It doesn't matter if the code in the image is owned by root (in fact, it's better, because it prevents the code from being accidentally overwritten).
user: 1000 # matches host uid; try running `id -u`
volumes: # or `ls -lnd logs`
- ./logs:/usr/src/app/logs
Also consider setting your application to log to stdout, instead of a file. That avoids this problem, and you can use docker logs to read the log output. In more involved container environments like Kubernetes, there are standard ways to collect logs-to-stdout from containers, but it's much trickier to collect logs-to-files.
I see lots of questions around setting/changing the COMPOSE_PROJECT_NAME or PROJECT_NAME using ENV variables.
I'm fine with the default project name, but I would like to reference it in my compose file.
version: "3.7"
services:
app:
build: DockerFile
container_name: app
volumes:
- ./:/var/app
networks:
- the-net
npm:
image: ${project_name}_app
volumes:
- ./:/var/app
depends_on:
- app
entrypoint: [ 'npm' ]
networks:
- the-net
npm here is arbitrary , hopefully the fact that could be run as its own container or in other ways does not distract from the questions.
is it possible to reference the project name with out setting it manually or first?
Unfortunately it is not possible.
As alluded to, you can create a .env file and populate it with COMPOSE_PROJECT_NAME=my_name, but the config option does not present itself in your environment by default.
Unfortunately the env substitution in docker-compose is fairly limited, meaning we cannot use the available PWD env variable and greedy match it at all
$ cd ~
$ pwd
/home/tqid
$ echo "Base Dir: ${PWD##*/}"
Base Dir: tqid
When we use this reference, compose has issues:
$ docker-compose up -d
ERROR: Invalid interpolation format for "image" option in service "demo": "${PWD##*/}"
It's probably better to be explicit anyway, the COMPOSE_PROJECT_NAME is based on your dir, and if someone clones to a new folder then it gets out of whack, including the .env file in source control would provide a re-usable and consistent place to reference the name
https://docs.docker.com/compose/reference/envvars/#compose_project_name
using the same image as another container was what I was after ... reuse the image and change the entry point.
Specify the same build: options for both containers.
This seems inefficient, in that it will trigger the build sequence twice and docker images will list both of them. However, the way Docker's layer caching works, if identical RUN commands are run on identical input images, the resulting layer will simply be reused, and the two final images will have the same image ID; they will literally be the same image with two names.
The context I've run into this the most is with a Python application where the same code base is used for a Django or Flask Web server, plus a Celery worker. The Docker-level setup is fairly language-independent, though: specify the same build: for both containers, and override the command: for the container(s) that need to do a non-default task.
version: '3.8'
services:
app:
build: .
ports: ['3000:3000']
environment:
REDIS_HOST: redis
worker:
build: . # <-- same as app
command: npm run worker # <-- overrides Dockerfile CMD
environment:
REDIS_HOST: redis
redis:
image: redis
It is also valid to specify build: and image: together in the docker-compose.yml file; this specifies the name of the image that will be built. It's frequently useful to explicitly specify this because you will need to point at a specific Docker Hub or other registry location to push the built image. If you do this, then you'll know the image name and don't need to depend on the context name.
version: '3.8'
services:
app:
build: .
image: registry.example.com/my/app:${TAG:-latest}
worker:
image: registry.example.com/my/app:${TAG:-latest}
command: npm run worker
You will need to manually docker-compose build in this setup. Compose's workflow doesn't have a way to specify that one container's build must run before a different container can start.
I cannot get mariadb to use another port other than 3306 when running it in a docker container using a docker-compose file.
I have already read the mariadb/docker documentation, searched online and conducted my own experiments.
docker-compose file:
version: '3.1'
services:
db:
image: mariadb
restart: always
environment:
- MYSQL_ROOT_PASSWORD=mypassword
- MYSQL_TCP_PORT=33030
- MYSQL_UNIX_PORT=33020
ports:
- "33020:33020"
Dockerfile:
FROM: mariadb: 10.3.14
COPY mydbscript.sql /docker-entrypoint-initdb.d/
EXPOSE 33020
CMD ["mysqld"]
It never uses port 33020. It still uses port 3306. How can I pass the port dynamically via the docker-compose file at run-time?
You need to replace the default my.cnf to specify a custom port for MariaDB/MySQL:
cd /where/your/docker-compose.yml/located
docker run -it mariadb cat /etc/mysql/my.cnf > my.cnf
# use any text editor your like to open my.cnf, search for "port = 3306"
# and replace it to the port you like to have.
Configure your docker-compose.yml like this:
version: '3.1'
services:
db:
image: mariadb
restart: always
volumes:
- type: bind
source: ./my.cnf
target: /etc/mysql/my.cnf
environment:
- MYSQL_ROOT_PASSWORD=mypassword
# add your other configurations here
The container image is statically bound to :3306. If you wish to change this, you'll need to build a new image and configure the database to run elsewhere.
However, Docker permits you to map (publish) this as a different port :33020.
The correct way to do this is to:
docker-compose MYSQL_TCP_PORT=3306
docker-compose ports: - "33020:3306"
Dockerfile EXPOSE 3306 (unchanged)
Containers (internally) will correctly reference :3306 but externally (from the host) the database will be exposed on :33020.
NB Within docker-compose (network), other containers must continue to reference the database on port :3306.
#DazWilkin, #philip-tzou, of course it's possible!
How to set the port without config file is even explained in the dockerhub-page of mariadb. (https://hub.docker.com/_/mariadb) #Software just did the mistake of using '=' instead of ':' in the docker-compose.yml. I did it the first time too because I copied the environment veriables from a docker run bash file.
This docker-compose.yml (with .env File) works for me to set both, internal and external port of my mariaDB service:
version: "3.9"
services:
database:
image: mariadb:10.8
container_name: ${db_containername}
environment:
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD: yes
MARIADB_DATABASE: ${db_database}
MARIADB_USER: ${db_user}
MARIADB_PASSWORD: ${db_password}
MYSQL_TCP_PORT: ${db_port_internal}
MARIADB_AUTO_UPGRADE: 1
MYSQL_UNIX_PORT: /run/mysqld/mysqld.sock
MARIADB_MYSQL_LOCALHOST_USER: true
restart: always
ports:
- '${db_port_external}:${db_port_internal}'
expose:
- ${db_port_external}
volumes:
- 'database_data:/var/lib/mysql'
This is how you can set a different port for your mariadb inside the container
1. Create a my.cnf file inside the same directory as you dockerfile
write this inside the my.cnf file
[mysqld]
port = 33020
2. Add the cnf file to the dockerfile & edit the EXPOSE
Add this line of code to your docker file
COPY my.cnf /etc/mysql/my.cnf
And make sure to change the external exposed port to the one you want to use e.i
EXPOSE 33020
3. Make sure to change the port in the docker-compose.yml file
ports:
- "33020:33020"
You can now connect to your database in either the terminal using the docker exec -it {databasename} mysql -u root -p or in something like mysql workbench by setting the ip: localhost and the port to 33020
Hope this helps.
I am unable to change the port that Swagger uses in docker compose. It works fine with regular docker, I simply set the -p argument on the run command. It seems that I should just need to set the ports field in the docker-compose file. But no matter what I try it just runs on 8080.
I am using the latest versions of docker and docker-compose. The docker image is called swaggerapi/swagger-ui. I have attempted setting the ports field for the container. Also tried setting the url variable in the swagger definition file. Tried changing the expose port. I tried with the docker-compose run command which lets you start an individual service and has the -p argument. Still nothing.
Ideally I should use this to build and run:
sudo docker-compose up --build --force-recreate
My compose file:
version: '3'
services:
swagger:
build: swagger
network_mode: "host"
ports:
- "8081:8080"
env_file: .env
environment:
- SWAGGER_JSON=/swagger.json
volumes:
data:
driver: "local"
And the docker file for the swagger service:
FROM swaggerapi/swagger-ui
EXPOSE 8081
COPY swagger.json /swagger.json
ENV SWAGGER_JSON "/swagger.json"
No matter what I do it wont change ports.
Just change the port in your docker-compose file
swagger:
build: swagger
network_mode: "host"
ports:
- "8081:"**Port which you want to expose**"
env_file: .env
environment:
- SWAGGER_JSON=/swagger.json
How to be with orphan images when you have 2 independent projects and you want them to work at the same time or at least to build running docker-compose up -d without --remove-orphans flag when images are already built for another project.
docker compose file1:
version: '2'
services:
applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project1
nginx:
build: ./images/nginx
image: project1/nginx:latest
ports:
- "80:80"
volumes_from:
- applications
networks:
appnet:
aliases:
- project1.app
- admin.project1.app
php:
image: project1/php:latest
ports:
- "7778:7778"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- applications
networks:
- appnet
mysql:
image: project1/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- mysqldata:/var/lib/mysql
networks:
- appnet
ports:
- "33066:3306"
workspace:
image: project1/workspace:latest
build:
context: ./images/workspace
volumes_from:
- applications
working_dir: /var/www/vhosts/project1
networks:
- appnet
networks:
appnet:
driver: "bridge"
volumes:
mysqldata:
driver: "local"
the second docker compose file:
version: '2'
services:
project2_applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project2
project2_nginx:
build: ./images/nginx
image: project2/nginx:latest
ports:
- "8080:80"
volumes_from:
- project2_applications
networks:
project2_appnet:
aliases:
- project2.app
- admin.project2.app
project2_php:
image: project2/php:latest
ports:
- "7777:7777"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- project2_applications
networks:
- project2_appnet
project2_mysql:
image: project2/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- project2_mysqldata:/var/lib/mysql
networks:
- project2_appnet
ports:
- "33067:3306"
project2_workspace:
image: project2/workspace:latest
build:
context: ./images/workspace
volumes_from:
- project2_applications
working_dir: /var/www/vhosts/videosite
networks:
- project2_appnet
networks:
project2_appnet:
driver: "bridge"
volumes:
project2_mysqldata:
driver: "local"
And now when I have already built project1 and trying to run docker-compose up -d for the second project I see warning:
WARNING: Found orphan containers (docker_workspace_1, docker_nginx_1, docker_php_1, docker_mysql_1, docker_memcached_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
I have a supposition that it's because container names for project1 should be more specific and I need to add some prefixes like I'm doing for project2, but project1 is in use by many other developers and I do not want to change it.
Is there any way to turn off orphan check?
And the second thing: is just a warning message but for some reason, after it appearing compose is failing with error:
ERROR: Encountered errors while bringing up the project.
And to make it work I need to run docker-compose up -d --remove-orphans
Compose uses the project name (which defaults to the basename of the project directory) internally to isolate projects from each other. The project name is used to create unique identifiers for all of the project's containers and other resources. For example, if your project name is myapp and it includes two services db and web, then Compose starts containers named myapp_db_1 and myapp_web_1 respectively.
You get the "Found orphan containers" warning because docker-compose detects some containers which belong to another project with the same name.
To prevent different projects from interfering with each other (and suppress the warning) you can set a custom project name by using any of the following options:
The -p command line option.
COMPOSE_PROJECT_NAME environment variable. This environment variable can also be set via an environment file (.env in the current working directory by default).
Top-level name element in the Compose file. Note: if you pass multiple files to docker-compose via the -f option, then the value from the last file will be used.
docker-compose takes the name of the directory it is in as the default project name.
You can set a different project name by using -p or --project-name.
https://docs.docker.com/compose/reference/#use--p-to-specify-a-project-name
I had a similar problem because my projects all had the docker/docker-compose.yml structure.
To build on other answers, I create a .env file with my docker compose projects. I have a number of projects that all use the docker directory but are different projects.
To use docker-compose -p is a bit error prone, so creating .env file in the same directory as the docker-compose.yml:
-rw-rw-r-- 1 auser auser 1692 Aug 22 20:34 docker-compose.yml
-rw-rw-r-- 1 auser auser 31 Aug 22 20:44 .env
alleviates the necessary overhead of remembering -p.
In the .env file, I can now set the COMPOSE_PROJECT_NAME variable:
COMPOSE_PROJECT_NAME=myproject
On running:
docker-compose up -d
the COMPOSE_PROJECT_NAME is substituted without the use of -p.
Reference:
https://docs.docker.com/compose/env-file/
docker-compose up --remove-orphans
you can run this command to clean orphan containers. As specified in the warning
If the orphaned containers are expected and not intended to remove, you can set COMPOSE_IGNORE_ORPHANS variable to true.
Consise but just right away working source is here.
One option is to put it as a line into .env file next to docker-compose.yml like this:
COMPOSE_IGNORE_ORPHANS=True
Another option is pass or set it as an environment variable.
sh:
COMPOSE_IGNORE_ORPHANS=True docker-compose up -d
or
export COMPOSE_IGNORE_ORPHANS=True
docker-compose up -d
cmd:
SET COMPOSE_IGNORE_ORPHANS=True&& docker-compose up -d
powershell:
$env:COMPOSE_IGNORE_ORPHANS = 'True'; & docker-compose up -d
TL;DR
You can also add a unique name: myproject to each of your compose files.
My journey
In case this helps anybody else scrounging around to find help for the above issue (This is in support of the already good comments here):
I have several config files in the same directory
redis.yml
mariadb.yml
...
and I kept getting the same error about orphan containers when I ran
docker-compose -f <one of my configs>.yml up
as of now you can simply put each yml file into a separate project. This is simply done using the command like parameter "-p my_project_name" as has already been mentioned before. BUT the name must be in all lowercase!
This got me a little closer but I also kept forgetting that to bring the docker container down using docker-compose I needed to include that parameter as well.
For example to start the container:
docker-compose -p myproject-d redis.yml up -d
and to destroy the container
docker-compose -p myproject-d redis.yml down
Today I found that I can simply add the name: bit into the yml config. Here is an example for redis:
version: '3.9'
name: redis
services:
redis_0:
...
Now I can simply start the container with the following and don't have to worry about project names again:
docker-compose -f redis.yml <up/down>
This happens when your docker-compose file has got updated. I received similar error on Docker startup and found out that another team member updated the docker-compose.yml as part of cleanup.
To fix this, I deleted the docker group using the Delete button in Docker Desktop and started it again. This fixed the error for me.
As a complement for the existing answers, if you're using docker-compose with the -f option, to my surprise docker-compose will use the name of the parent folder of the first file passed via -f as the project name.
For example, assuming the following folder structure:
/
└── Users/
└── papb/
├── a.yml
└── foo/
└── b.yml
If you're in /Users and run docker-compose -f papb/a.yml -f papb/foo/b.yml:
The project name will be inferred as papb
Any relative paths you have in both files will be resolved against /Users/papb
If you're in /Users and run docker-compose -f papb/foo/b.yml -f papb/a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo
If you're in /Users/papb and run docker-compose -f foo/b.yml -f a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo