I have the following projects structure on my machine filesystem:
../
├── angular_front_end/
│ └── docker-compose.yml
│ └── Dockerfile
├── node_back_end_service/
│ └── docker-compose.yml
│ └── Dockerfile
└── php_back_end_service/
└── docker-compose.yml
└── Dockerfile
The thing is, I don't want to go through each one and do docker-compose up, it's horrible to maintain.
Is there a way to unite them all under one command somehow?
Also, can I run all of them under one container, like the back-end container in the screenshot below?
Thanks a lot!
You can create a single docker-compose.yml file at the root of this directory hierarchy that launches everything.
version: '3.8'
services:
frontend:
# Builds Dockerfile within that directory,
# can only reference files within this directory
build: angular_front_end
ports: ['3000:3000']
node:
build: node_back_end
php:
build: php_back_end
To the extent that these services require databases or other Docker resources, they all need to be duplicated in this top-level docker-compose.yml file.
In principle it's possible to reuse your existing Compose files, but there are two big issues you'll run into. You need to consistently use multiple docker-compose -f options, every time you run a docker-compose command; with your setup this will quickly become unwieldy (even with just three services). Also, all filesystem paths are interpreted relative to the first -f option's path so a declaration like build: . won't point at the right place.
Related
I'm new to docker and I get a strange error when building my first containers. I'm using WSL2 if that matters.
Here is the part that's causing it :
# MySQL Service
mysql:
image: mysql:8
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: usr_data
volumes:
- ./.docker/mysql/my.cnf:/etc/mysql/conf.d/my.cnf
# - ./.docker/mysql/ohmy.cnf:/etc/mysql/conf.d/my.cnf
- ./.docker/test/test.txt:/tmp/test/test.txt
- mysqldata:/var/lib/mysql
healthcheck:
test: mysqladmin ping -h 127.0.0.1 -u root --password=MYSQL_ROOT_PASSWORD
interval: 5s
retries: 10
Both files my.cnf and ohmy.cnf exist and have the same content.
When I use docker-compose up -d I get the error :
ERROR: for mysql Cannot create container for service mysql: not a directory
When I uncomment the ohmy.cnf line and comment the my.cnf line I get no errors and it builds just fine. It also works great with the little test.txt I made.
I fail to see the difference between the two, and while it may work with my little workaround, I'd like to understand what is causing the error in the first place.
Thank you for your time.
Edit :
Here's my ./.docker
./.docker
├── mysql
│ ├── db
│ │ └── db.sql
│ ├── my.cnf
│ └── ohmy.cnf
├── nginx
│ └── conf.d
│ └── php.conf
├── php
│ └── Dockerfile
└── test
└── test.txt
6 directories, 6 files
In WSL 2 I updated docker compose to use $PWD instead of a relative path for volumes:
volumes:
- $PWD/logs:/var/log/docker:delegated
Delete the volumes from /var/lib/docker/volumes/[your-volumes]. Run the below commands
docker volume prune
Remove all unused local volumes. Unused local volumes are those which are not referenced by any containers.
docker system prune
Remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes.
restarting my laptop helped in my special case, the other commands did not
I have my project architecture like this:
.
├── app/
├── context/
│ ├── Dockerfile
│ ├── .dockerignore
│ └── php.ini
├── database/
├── http/
├── composer.json
└── docker-compose.yml
and in docker-compose.yml I have the following configuration:
version: '3.8'
services:
app:
container_name: "ERP"
restart: always
build:
context: ./context
dockerfile: Dockerfile
stdin_open: true
tty: true
ports:
- '8000:80'
links:
- db_server
volumes:
- .:/usr/src/app
working_dir: /usr/src/app
db_server:
container_name: "db_server"
image: 'mysql:8.0'
ports:
- '3306:3306'
But when I set the Dockerfile content to set up the application with docker-compose up, having Dockerfile content as this:
FROM ubuntu:20.04
WORKDIR /usr/src/app
RUN cat composer.json
It says "No such file or directory composer.json". Why?
UPDATE
I managed to solve the problem based on ENTRYPOINT configuration ..
as far I understand - I'm new to docker - , the ENTRYPOINT defines an entry script running at the beginning of a container , thus , starting that script will be definitely in the run time of the container and after the initializations specified by docker-compost.yml file .. so the contents of the actual context will be available to the script to see it in the time it runs ..
Thank you all for your answers .
That's because you define the context to be "./context", so you are stuck into this folder, where composer.json isn't.
Use "." for the context. and context/Dockerfile for the dockerfile.
Then mounting '.' will mount the whole directory, and not only the ./context one.
The build process to create an image occurs before the runtime process that takes that image and runs a container. The compose file includes a build section to allow you to easily build the image before running it, but all of the other parts of the compose file define runtime configurations like volume mounts and networking.
At build time, you do not get to specify the volume sources, at most you can define a required volume target in the image with a VOLUME step. (Note if you do that, future RUN steps within the Dockerfile may have issues modifying that directory since many build tools mount an anonymous volume as you've requested, but only capture the changes to the container filesystem, not the volume filesystem).
If you need the contents of a file or directory in your image, you must perform a COPY or ADD step in the Dockerfile to copy from the build context (typically imported as . or the current directory) into the image.
Key build.context define a path to a directory containing a Dockerfile. This is a context of building image, and during build process docker doesn't have access to composer.json file (it is out of the context).
Command RUN runs command during build. If you want to run it when container is starting, you should use CMD.
FROM ubuntu:20.04
WORKDIR /usr/src/app
CMD cat composer.json
Currently I have a project (repo) in Gitlab which is an angular app. I'm using Gitlab CI/CD to build, test, release and deploy. Releasing will build a new docker image pushing it to the Gitlab registry and after deploying it on NGinx in a docker container on my Digital Ocean droplet. This works fine.
Let's say I want to add a backend to it like the MEAN stack so I would have 2 containers running using a docker-compose file.
container 1 - Angular
container 2 - Node.js, Express.js and MongoDB
The 2 gitlab projects (repo's) will have to be build separately when a change occurs (own Dockerfile and gitlab-ci.yml file) but deployed together using the docker-compose file.
Where do I manage/put the docker-compose file?
I hope my explanation is clear and if I'm assuming correctly.
Thanks in advance.
According to your comment I understand you'd be interested in adopting a monorepo configuration.
In this case, for the question
Where do I manage/put the docker-compose file?
you could just put the docker-compose.yml file at the root of your GitLab CI project, which would lead to a directory structure like this:
monorepo-project/
├── backend/
│ ├── Dockerfile
│ ├── .dockerignore
│ └── src/
├── frontend/
│ ├── Dockerfile
│ ├── .dockerignore
│ └── src/
├── docker-compose.yml
├── .git/
├── .gitignore
└── .gitlab-ci.yml
As pointed out in https://docs.gitlab.com/ee/user/packages/workflows/monorepo.html (the original version of this page, deleted by this commit, is still available at this URL), you can tweak your configuration using the changes: key, so that if just one part of the project changes (e.g., the frontend), then the CI behaves accordingly.
Further pointers
For more examples, see e.g. this article in Medium which specifically relies on Docker, or that blog article which takes advantage of the needs: key.
Finally, the semantics of the GitLab CI YAML conf file is well-documented in https://docs.gitlab.com/ee/ci/yaml/ (to be bookmarked!).
For example, I have the next structure of project:
.
├── docker-compose.yml
├── library
└── _services
└──_service1
| └── Dockerfile
├──_service2
| └── Dockerfile
└──_service3
└── Dockerfile
How can I copy library into each service? Or is it exist a better way to create services images with library package?
You can't copy files that are in a parent directory of where you Dockerfile is.
Of course you don't want to copy your library content into each service directory but you can.
Create a distinct Dockerfile for each service at the top level.
Eg:
docker-compose.yml
library
Dockerfile.service1
Dockerfile.service2
Dockerfile.service3
then each dockerfile can COPY the library in.
If your library is a fundamental part of your services, you can simply create an image for it and make it the base image for your services.
Eg:
base
library
Dockerfile
services
Dockerfile.service1
Dockerfile.service2
Dockerfile.service3
with Dockerfile
FROM alpine:3.7
COPY library/...
docker build -t base .
and a Dockerfile.serviceN
FROM base
Generally, I find it better to not include building of Dockerfile in the docker compose. You build your services when needed, push them to an image repository (eg: quay.io, docker.io) and your docker compose file pulls them in at deploy time.
One way of maintaining is single image and having Docker Volumes . You can also specify the volume in docker-compose.yml and use the shared data.
Volumes in compose : https://docs.docker.com/compose/compose-file/#volumes.
Volumes in Docker : https://docs.docker.com/storage/volumes/.
My current project consists of a mongo server, a rabbitmq server and a dotnet core service. It is structured as follows:
.
├── project1.docker-compose.yml #multiple docker-compose files for all projects
├── .dockerignore
├── Util/
| └── some common code across all projects
└── Project1/ #there are multiple projects at the same level with the same structure
├── .docker/
| ├── mongodb
| | └──Dockerfile
| └── rabbitmq
| └──Dockerfile
├── BusinessLogicClasses/
| └── some classes that contain my business logic
└── DotNetCoreService/
├── my service code
└── .docker
└──Dockerfile
Right now I am able to use docker-compose command to build the images for mongodb, rabbitmq and the dot net core succesfully. The docker-compose.yml sits at the home directory level because my different projects (in this case Project1) references code found under the Util directory. Therefore I need to be able to provide a context that is above both directories so that I can use COPY operations on the Dockerfile.
My basic project1.docker-compose.yml is as follows (I excluded not important parts)
version: '3'
services:
rabbitmq:
build:
context: Project1/.docker/rabbitmq/
mongodb:
build:
context: Project1/.docker/mongodb/
dotnetcoreservice:
build:
context: ./
dockerfile: Project1/DotNetCoreService/.docker/Dockerfile
As can be seen, the context for the dotnetcoreservice is at the home directory level. Therefore my Dockerfile for that specific image needs to target the full paths from the context as follows:
#escape=`
FROM microsoft/dotnet:2.0-sdk AS build
WORKDIR /app
COPY Project1/ ./Project1/
COPY Util/ ./Util/
RUN dotnet build Project1/DotNetCoreService/
This dockerfile works succesfully when invoked via the docker-compose command at the home directory level, however when invoked via the docker build .\Project1\DotNetCoreService\.docker\ command it fails with the following message:
COPY failed: stat
/var/lib/docker/tmp/docker-builder241915396/Project1: no
such file or directory
I think this is a matter of the actual context because the docker build instruction automatically sets the context to where the Dockerfile is. I would like to be able to use this same directory structure to create images both with the docker-compose build as well as with the docker build instructions.
Is this somehow possible?
Use flag -f to set custom path
Example: docker build --rm -t my-app -f path/to/dockerfile .
May 2022: The new releases of Dockerfile 1.4 and Buildx v0.8+ come with the ability to define multiple build contexts.
This means you can use files from different local directories as part of your build.
Dockerfiles now Support Multiple Build Contexts
Tõnis Tiigi
Multiple Projects
Probably the most requested use case for named contexts capability is the possibility to use multiple local source directories.
If your project contains multiple components that need to be built together, it’s sometimes tricky to load them with a single build context where everything needs to be contained in one directory.
There’s a variety of issues:
every component needs to be accessed by their full path,
you can only have one .dockerignore file,
or maybe you’d like each component to have its own Dockerfile.
If your project has the following layout:
project
├── app1
│ ├── .dockerignore
│ ├── src
├── app2
│ ├── .dockerignore
│ ├── src
├── Dockerfile
…with this Dockerfile:
#syntax=docker/dockerfile:1.4
FROM … AS build1
COPY –from=app1 . /src
FROM … AS build2
COPY –from=app2 . /src
FROM …
COPY –from=build1 /out/app1 /bin/
COPY –from=build2 /out/app2 /bin/
…you can invoke your build with docker buildx build –build-context app1=app1/src –build-context app2=app2/src .. Both of the source directories are exposed separately to the Dockerfile and can be accessed by their respective names.
This also allows you to access files that are outside of your main project’s source code.
Normally when you’re inside the Dockerfile, you’re not allowed to access files outside of your build context by using the ../ parent selector for security reasons.
But as all build contexts are passed directly from the client, you’re now able to use --build-context othersource=../../path/to/other/project to avoid this limitation.