My idea is to have a single docker-compose file that I can configure with --profile db_as_container flag depending on whether I want to have cloud database (MongoDB atlas) or local database-as-container.
Here's my docker-compose.yml:
version: '3'
services:
app:
image: '${APP_NAME}:${TAG:-latest}'
build: .
ports:
- '${PORT}:${PORT}'
mongo_db:
image: mongo
container_name: mongo_db_container
ports:
- "27017:27017"
profiles:
- db_as_container
My docker compose up shell script (dc-up.sh) deduces whether my DB_CONNECTION_STRING is cloud type or local container type and calls appropriate up command.
TAG=${TAG} docker-compose --profile db_as_container up -d --build
vs.
TAG=${TAG} docker-compose up -d --build
And this works locally and does not complain about using profiles.
Problem is when my Gitlab CI runner runs my build script (build-and-push.sh):
TAG=${TAG} docker-compose build
It produces this error:
The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.mongo_db: 'profiles'
What am I doing wrong here?
Actually the issue was not the version of the compose file specification, but rather docker-compose itself as i've found out here
Support for service profiles was added in docker-compose version 1.28
Updating from 1.262 to 1.28.6 solved my issue.
Only Docker Compose v3 has profiles option. I'm guessing your CI compose version is a lower one.
You will need to upgrade it to version 3.
Related
I have a few microservices. Jenkins builds these projects, creates docker images and publishes them to the artifactory.
I have another project for automation testing which is using these docker images.
We have a docker-compose file that has all the configuration of all microservice images.
Following is sample docker-compose
version: "1.0"
services:
my-service:
image: .../my-service:1.0.0-SNAPSHOT-1
container_name: 'my-service'
restart: always
volumes:
...
ports:
...
...
all these are working fine.
Now to update the image then I have to manually change the image tag (1.0.0-SNAPSHOT-2) in docker-compose.
This is an issue because this involves human intervention. Is there any way to pull the newest docker image without any change in docker-compose?
NOTE - I cannot create images with the latest tag. Getting issue when publishing image with the same name in the artifactory (unauthorized: The client does not have permission for manifest: Not enough permissions to delete/overwrite artifact).
Well What actually you can do is, use environment variables substitutions in cli commands (envsubst). Let me explain an escenario as example.
First in the docker-compose.yaml you define an environmet variable, as a tag of the container
version: "3"
services:
my-service:
image: .../my-service:$TAG
container_name: 'my-service'
restart: always
volumes:
...
ports:
...
...
Second, with cli command (or terminal) you define an environment variable, with you version. This part is important because here you add your version tag to the container (and you can execute bash commands to extract some id, or last git commit or what ever you want to execute as tag for the container, i give you some ideas)
export TAG=1.0.0-SNAPSHOT-1
export TAG="$(bash /path/to/script/tag.sh)"
export TAG="$(git log --format="%H" -n 1)"
And the third part and last one is for execute "envsubst" and then execute docker-compose.yaml to deploy your container. Note the pipe |, very important for execution.
envsubst < docker-compose.yaml | docker-compose up -d
link to envsubst
I use this format to deploy tagged containers in kubernetes, but the idea must be the same with docker compose.
envsubst < deployment.yaml | kubectl apply -f -
And change version to 3 in the docker-compose.yaml. Good luck
I am new in docker and docker-compose. I am working on a project which uses both docker and docker compose. After adding a line in the key "volumes" of docker-compose.yml. I did docker restart
docker restart service1
and the volume was automatically recognized. Below part of the docker-compose.yml
services:
service1:
build:
context: .
dockerfile: Dockerfile.service1
container_name: service1_name
hostname: service1_name
volumes:
- /etc/teste/service1/conf.d/:/usr/share/logstash/pipeline/ #I added this line
My question Why docker restart recognizes modifications in docker-compose.yml file, where can I find this "setting"?
Working with docker-compose the scenario should be the following:
Create your docker-compose.yml
docker-compose up # this will build everything and run your services
If you make any changes to your compose file, stating docker-compose build 'yourservicename' will rebuild that specific service.
The command "docker restart" as you can see in the help page is for restarting containers
I am trying to start a docker image (https://hub.docker.com/r/parrotstream/hbase/)
on Windows 10 with
docker-compose -p parrot up
but I get this error:
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
Executing the command in the directory with the docker image in it does not work either.
I am new to using Docker and I am unsure how to start the container. According to the Docker Hub page of the image, this is all I have to do. Am I missing something?
Thanks
Edit:
As pointed out by the replies, I've downloaded the folder from github, including the docker-compose.yml. I am currently getting an error because of my permission.
ERROR: for hbase Cannot start service hbase: driver failed programming external connectivity on endpoint hbase (5fb66c3b2b0d3092edce09f03cc803cc3ea447c07a1a2135271238de626458c6): Error starting userland proxy: Bind for 0.0.0.0:8080: unexpected error Permission denied
ERROR: for hbase Cannot start service hbase: driver failed programming external connectivity on endpoint hbase (5fb66c3b2b0d3092edce09f03cc803cc3ea447c07a1a2135271238de626458c6): Error starting userland proxy: Bind for 0.0.0.0:8080: unexpected error Permission denied
ERROR: Encountered errors while bringing up the project.
Do I have a wrong configuration in docker?
The actual docker-compose.yml that you are looking for may be the one hosted in their github repo found here.
version: '3'
services:
hbase:
container_name: hbase
build:
context: .
dockerfile: Dockerfile
image: parrotstream/hbase:latest
external_links:
- hadoop
- zookeeper
ports:
- 8080:8080
- 8085:8085
- 9090:9090
- 9095:9095
- 60000:60000
- 60010:60010
- 60020:60020
- 60030:60030
networks:
default:
external:
name: parrot_default
By default, docker-compose tries to read the configuration from a file named docker-compose.yml within you current working directory. You could override this behavior with docker-compose -f <anotherfile.yml>.
Options:
-f, --file FILE Specify an alternate compose file
(default: docker-compose.yml)
Yes, command needs a compose file and the readme assumes that you have a docker-compose.yml in the directory where you execute the command.
You can find one in the linked repository from DockerHub parrot-stream/docker-hbase
You need to create a docker-compose file as follows
# docker-compose.yml
version: '2'
services:
parrot:
image: parrotstream/hbase
then you can create a build and run is using
docker-compose build parrot # build image
docker-compose up parrot # run
I am new to Docker and have docker-compose.yml which is containing many services and iI need to start one particular service. I have docker-compose.yml file with information:
version: '2'
services:
postgres:
image: ${ARTIFACTORY_URL}/datahub/postgres:${BUILD_NUMBER}
restart: "no"
volumes:
- /etc/passwd:/etc/passwd
volumes_from:
- libs
depends_on:
- libs
setup:
image: ${ARTIFACTORY_URL}/setup:${B_N}
restart: "no"
volumes:
- ${HOME}:/usr/local/
I am able to call docker-compose.yml file using command:
docker-compose -f docker-compose.yml up -d --no-build
But I need to start "setup service" in docker-compose file:
How can I do this?
It's very easy:
docker compose up <service-name>
In your case:
docker compose -f docker-compose.yml up setup -d
To stop the service, then you don't need to specify the service name:
docker compose down
will do.
Little side note: if you are in the directory where the docker-compose.yml file is located, then docker-compose will use it implicitly, there's no need to add it as a parameter.
You need to provide it in the following situations:
the file is not in your current directory
the file name is different from the default one, eg. myconfig.yml
As far as I understand your question, you have multiple services in docker-compose but want to deploy only one.
docker-compose should be used for multi-container Docker applications. From official docs :
Compose is a tool for defining and running multi-container Docker
applications.
IMHO, you should run your service image separately with docker run command.
PS: If you are asking about recreating only the container whose image is changed among the multiple services in your docker-compose file, then docker-compose handles that for you.
What is the difference between docker-compose build and docker build?
Suppose in a dockerized project path there is a docker-compose.yml file:
docker-compose build
And
docker build
docker-compose can be considered a wrapper around the docker CLI (in fact it is another implementation in python as said in the comments) in order to gain time and avoid 500 characters-long lines (and also start multiple containers at the same time). It uses a file called docker-compose.yml in order to retrieve parameters.
You can find the reference for the docker-compose file format here.
So basically docker-compose build will read your docker-compose.yml, look for all services containing the build: statement and run a docker build for each one.
Each build: can specify a Dockerfile, a context and args to pass to docker.
To conclude with an example docker-compose.yml file :
version: '3.2'
services:
database:
image: mariadb
restart: always
volumes:
- ./.data/sql:/var/lib/mysql
web:
build:
dockerfile: Dockerfile-alpine
context: ./web
ports:
- 8099:80
depends_on:
- database
When calling docker-compose build, only the web target will need an image to be built. The docker build command would look like :
docker build -t web_myproject -f Dockerfile-alpine ./web
docker-compose build will build the services in the docker-compose.yml file.
https://docs.docker.com/compose/reference/build/
docker build will build the image defined by Dockerfile.
https://docs.docker.com/engine/reference/commandline/build/
Basically, docker-compose is a better way to use docker than just a docker command.
If the question here is if docker-compose build command, will build a zip kind of thing containing multiple images, which otherwise would have been built separately with usual Dockerfile, then the thinking is wrong.
Docker-compose build, will build individual images, by going into individual service entry in docker-compose.yml.
With docker images, command, we can see all the individual images being saved as well.
The real magic is docker-compose up.
This one will basically create a network of interconnected containers, that can talk to each other with name of container similar to a hostname.
Adding to the first answer...
You can give the image name and container name under the service definition.
e.g. for the service called 'web' in the below docker-compose example, you can give the image name and container name explicitly, so that docker does not have to use the defaults.
Otherwise the image name that docker will use will be the concatenation of the folder (Directory) and the service name. e.g. myprojectdir_web
So it is better to explicitly put the desired image name that will be generated when docker build command is executed.
e.g.
image: mywebserviceImage
container_name: my-webServiceImage-Container
example docker-compose.yml file :
version: '3.2'
services:
web:
build:
dockerfile: Dockerfile-alpine
context: ./web
ports:
- 8099:80
image: mywebserviceImage
container_name: my-webServiceImage-Container
depends_on:
- database
Few additional words about the difference between docker build and docker-compose build.
Both have an option for building images using an existing image as a cache of layers.
with docker build, the option is --cache-from <image>
with docker-composer, there is a tag cache_from in the build section.
Unfortunately, up until now, at this level, images made by one are not compatible with the other as a cache of layers (Ids are not compatible).
However, docker-compose v1.25.0 (2019-11-18), introduces an experimental feature COMPOSE_DOCKER_CLI_BUILD so that docker-compose uses native docker builder (therefore, images made by docker build can be used as a cache of layers for docker-compose build)