How do you specify shared memory size in a Docker Compose configuration file?
I have an application that requires at least 2gb of shared memory, which I'd normally specify during docker run using the --shm-size 2g option.
According to the Docker Compose documentation, support for shm-size was added in version 2.3. However, if I run docker-compose with the trivial configuration:
version: "3"
services:
app_test:
build:
context: .
shm_size: '2gb'
dockerfile: Dockerfile.test
it fails with the error:
ERROR: The Compose file './docker-compose.test.yml' is invalid because:
services.app_test.build contains unsupported option: 'shm_size'
I'm using docker-compose version 1.24.1, which should be the most recent release that supports all modern file versions, including support for shm-size. Why is it giving me this error? What am I doing wrong?
Related
https://greenbone.github.io/docs/latest/22.4/container/index.html
After running docker-compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition pull I see these errors:
Unsupported config option for services: 'notus-data'
Unsupported config option for volumes: 'scap_data_vol'
Note that this command return random errors messages, when I run this second time I see:
Unsupported config option for volumes: 'data_objects_vol'
Unsupported config option for services: 'scap-data'
Docker version 20.10.22, build 3a2c30b
docker-compose version 1.25.0
The currently used docker-compose version doesn't match the following requirements described in the linked documentation:
docker-compose version 1.29.0 or newer is required
I met the following issue that compose file
version: '3'
services:
minkowski:
build:
context: .
dockerfile: DockerfileGPU
volumes:
- "../:/app:rw"
- "${DATA_PATH}:/app/data:rw"
working_dir: /app
tty: true
stdin_open: true
network_mode: "host"
runtime: nvidia
results in
ERROR: The Compose file './docker/compose-gpu.yaml' is invalid because:
services.minkowski.build contains unsupported option: 'runtime'
I have docker version 20.10.21 and docker-compose 1.25.0. Do you have any idea why that happens?
I tried use different versions. Running
sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
works fine
The runtime: option isn't supported in Compose file version 3; it is only in version 2. More broadly, current versions of Compose support both Compose file versions 2 and 3, and it's okay to use either. Version 3's options are more oriented towards the Swarm orchestrator and for some single-host-specific options like this you need to use version 2.
version: '2.4' # not 3.x
services:
...:
runtime: nvidia
The newer Compose Specification also supports runtime: but support for this is inconsistent across Compose versions. The Compose 1.25.0 you mention will not support it. That doesn't suggest specific values for version:, and I might label a file as version: '4.0' if you're using Compose Specification specific functionality.
I managed to fix the issue by installing the newer docker compose. With docker compose 2.x it works just fine, without further changes
Directory/ file hierarchy:
root/docker-compose.yaml
root/docker-compose.override.yml
root/Dockerfile
root/External/docker-compose.yaml
root/External/docker-compose.override.yml
root/External/Dockerfile
docker-compose.yaml in External directory has definiton of service_b and docker-compose.yaml in root directory has defintioon of service_a.
I want to include docker-compose.yaml from External directory to docker-compose.yaml in root directory, so docker-compose.yaml in root directory looks like:
version: "3.7"
services:
service_a:
image: service_a
build:
context: .
target: service_a
service_b:
extends:
file: "External/docker-compose.yaml"
service: service_b
Issue: file "root/External/docker-compose.override.yml" is ignored...
Why? How to fix it?
There are four versions of the Compose file format, depending on the tools you have available. Docker has recently started distributing a rewritten Compose as a plugin for the docker binary and this supports a newer Compose Specification. If docker-compose --version tells you it's a 1.x version, then your tool only supports version 2 (slightly more suited to single-host installations) and version 3 (slightly more suited to Swarm installations) (and the very old version 1, for which the Compose tool version 2 drops support).
Service-level extends: is not supported for Compose file version 3, only for version 2 and the Specification format.
Compose file version
Version 1
Version 2
Version 3
Specification
Supports extends:
no
YES
no
YES
Works in docker-compose v1
YES
YES
YES
no
Works in docker compose v2
no
YES
YES
YES
The easiest workaround is to change your Compose file version back to version 2. There's no particular downside to doing this, unless you're also trying to deploy using Docker Swarm and need resource constraints there.
version: "2.4"
I'm setting up a new instance of PyCharm, and would like to set up a Python interpreter using docker-compose, but PyCharm doesn't seem to like my docker-compose version.
First of all, in Build, Execution, Deployment > Docker > Tools, it can't figure out the version of docker-compose I'm using, because docker-compose no longer supports the '-v' option.
Then, when I go to add a new Python interpreter of type Docker Compose, I get parse errors from my docker-compose file. It seems like it thinks the file is using 2.x syntax and is confused by the new syntax options.
Error while parsing "/Users/_______/projects/________/docker-compose.yml": Cannot deserialize value of type 'java.lang.String' from Object value (token 'JsonToken.START_OBJECT') at [Source: UNKNOWN; line -1; column: -1] (through reference chain: java.util.LinkedHashMap["x-service-defaults"]->com.intellij.docker.compose.configuration.beans.v1.DockerComposeServiceV1["build"])
A stripped down version of the docker-compose file which still has this error is below:
version: '3.4'
x-service-defaults:
&service-defaults
build:
context: .
dockerfile: Dockerfile
services:
core:
<<: *service-defaults
command: /bin/bash -c "python /ci/healthcheck.py && python manage.py runserver 0.0.0.0:8000"
volumes:
- ./ci/:/ci
- static:/usr/src/app/static
postgres:
image: postgres:12.6
volumes:
- .pgdata/:/var/lib/postgresql/data
- ./docker-postgresql-multiple-databases:/docker-entrypoint-initdb.d
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
static:
The error seems to suggest that it doesn't understand the fact that the build property in x-service-defaults is an object rather than a string, which is 3.x syntax.
Even worse, if I try modifying the docker-compose.yml to change the aforementioned build property to just the string '.', then I don't get an error, but the list of services is wrong.
I expect to be given the options of core or postgres, since those are the options listed under services but instead it lists all the top level options: networks, services, volumes, and x-service-defaults.
It's worth noting that if I run docker-compose from the terminal, then it parses the file fine and runs my containers exactly like I expect.
Versions
I installed PyCharm from the JetBrains website, and it assures me that it has no updates. It's on version 2021.1.3, build #PY-211.7628.24.
I installed Docker Desktop for Mac using the Homebrew Docker Cask (and not using any of the formulae). It also says no new updates. It's version 3.5.2 (66501), which means Docker Engine 20.10.7 and Compose version v2.0.0-beta.6 (which is also what docker-compose --version reports).
It turns out that the answer is that PyCharm does not currently support docker-compose version 2, because it is still in beta. I simply should still be using docker-compose version 1.29.2.
As for why I was using docker-compose version 2, apparently the Docker cask on Homebrew defaults to enabling the "Use Docker Compose V2" Experimental Feature. I was able to disable this in my Docker Desktop settings.
My idea is to have a single docker-compose file that I can configure with --profile db_as_container flag depending on whether I want to have cloud database (MongoDB atlas) or local database-as-container.
Here's my docker-compose.yml:
version: '3'
services:
app:
image: '${APP_NAME}:${TAG:-latest}'
build: .
ports:
- '${PORT}:${PORT}'
mongo_db:
image: mongo
container_name: mongo_db_container
ports:
- "27017:27017"
profiles:
- db_as_container
My docker compose up shell script (dc-up.sh) deduces whether my DB_CONNECTION_STRING is cloud type or local container type and calls appropriate up command.
TAG=${TAG} docker-compose --profile db_as_container up -d --build
vs.
TAG=${TAG} docker-compose up -d --build
And this works locally and does not complain about using profiles.
Problem is when my Gitlab CI runner runs my build script (build-and-push.sh):
TAG=${TAG} docker-compose build
It produces this error:
The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.mongo_db: 'profiles'
What am I doing wrong here?
Actually the issue was not the version of the compose file specification, but rather docker-compose itself as i've found out here
Support for service profiles was added in docker-compose version 1.28
Updating from 1.262 to 1.28.6 solved my issue.
Only Docker Compose v3 has profiles option. I'm guessing your CI compose version is a lower one.
You will need to upgrade it to version 3.