I met the following issue that compose file
version: '3'
services:
minkowski:
build:
context: .
dockerfile: DockerfileGPU
volumes:
- "../:/app:rw"
- "${DATA_PATH}:/app/data:rw"
working_dir: /app
tty: true
stdin_open: true
network_mode: "host"
runtime: nvidia
results in
ERROR: The Compose file './docker/compose-gpu.yaml' is invalid because:
services.minkowski.build contains unsupported option: 'runtime'
I have docker version 20.10.21 and docker-compose 1.25.0. Do you have any idea why that happens?
I tried use different versions. Running
sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
works fine
The runtime: option isn't supported in Compose file version 3; it is only in version 2. More broadly, current versions of Compose support both Compose file versions 2 and 3, and it's okay to use either. Version 3's options are more oriented towards the Swarm orchestrator and for some single-host-specific options like this you need to use version 2.
version: '2.4' # not 3.x
services:
...:
runtime: nvidia
The newer Compose Specification also supports runtime: but support for this is inconsistent across Compose versions. The Compose 1.25.0 you mention will not support it. That doesn't suggest specific values for version:, and I might label a file as version: '4.0' if you're using Compose Specification specific functionality.
I managed to fix the issue by installing the newer docker compose. With docker compose 2.x it works just fine, without further changes
Related
Directory/ file hierarchy:
root/docker-compose.yaml
root/docker-compose.override.yml
root/Dockerfile
root/External/docker-compose.yaml
root/External/docker-compose.override.yml
root/External/Dockerfile
docker-compose.yaml in External directory has definiton of service_b and docker-compose.yaml in root directory has defintioon of service_a.
I want to include docker-compose.yaml from External directory to docker-compose.yaml in root directory, so docker-compose.yaml in root directory looks like:
version: "3.7"
services:
service_a:
image: service_a
build:
context: .
target: service_a
service_b:
extends:
file: "External/docker-compose.yaml"
service: service_b
Issue: file "root/External/docker-compose.override.yml" is ignored...
Why? How to fix it?
There are four versions of the Compose file format, depending on the tools you have available. Docker has recently started distributing a rewritten Compose as a plugin for the docker binary and this supports a newer Compose Specification. If docker-compose --version tells you it's a 1.x version, then your tool only supports version 2 (slightly more suited to single-host installations) and version 3 (slightly more suited to Swarm installations) (and the very old version 1, for which the Compose tool version 2 drops support).
Service-level extends: is not supported for Compose file version 3, only for version 2 and the Specification format.
Compose file version
Version 1
Version 2
Version 3
Specification
Supports extends:
no
YES
no
YES
Works in docker-compose v1
YES
YES
YES
no
Works in docker compose v2
no
YES
YES
YES
The easiest workaround is to change your Compose file version back to version 2. There's no particular downside to doing this, unless you're also trying to deploy using Docker Swarm and need resource constraints there.
version: "2.4"
I'm setting up a new instance of PyCharm, and would like to set up a Python interpreter using docker-compose, but PyCharm doesn't seem to like my docker-compose version.
First of all, in Build, Execution, Deployment > Docker > Tools, it can't figure out the version of docker-compose I'm using, because docker-compose no longer supports the '-v' option.
Then, when I go to add a new Python interpreter of type Docker Compose, I get parse errors from my docker-compose file. It seems like it thinks the file is using 2.x syntax and is confused by the new syntax options.
Error while parsing "/Users/_______/projects/________/docker-compose.yml": Cannot deserialize value of type 'java.lang.String' from Object value (token 'JsonToken.START_OBJECT') at [Source: UNKNOWN; line -1; column: -1] (through reference chain: java.util.LinkedHashMap["x-service-defaults"]->com.intellij.docker.compose.configuration.beans.v1.DockerComposeServiceV1["build"])
A stripped down version of the docker-compose file which still has this error is below:
version: '3.4'
x-service-defaults:
&service-defaults
build:
context: .
dockerfile: Dockerfile
services:
core:
<<: *service-defaults
command: /bin/bash -c "python /ci/healthcheck.py && python manage.py runserver 0.0.0.0:8000"
volumes:
- ./ci/:/ci
- static:/usr/src/app/static
postgres:
image: postgres:12.6
volumes:
- .pgdata/:/var/lib/postgresql/data
- ./docker-postgresql-multiple-databases:/docker-entrypoint-initdb.d
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
static:
The error seems to suggest that it doesn't understand the fact that the build property in x-service-defaults is an object rather than a string, which is 3.x syntax.
Even worse, if I try modifying the docker-compose.yml to change the aforementioned build property to just the string '.', then I don't get an error, but the list of services is wrong.
I expect to be given the options of core or postgres, since those are the options listed under services but instead it lists all the top level options: networks, services, volumes, and x-service-defaults.
It's worth noting that if I run docker-compose from the terminal, then it parses the file fine and runs my containers exactly like I expect.
Versions
I installed PyCharm from the JetBrains website, and it assures me that it has no updates. It's on version 2021.1.3, build #PY-211.7628.24.
I installed Docker Desktop for Mac using the Homebrew Docker Cask (and not using any of the formulae). It also says no new updates. It's version 3.5.2 (66501), which means Docker Engine 20.10.7 and Compose version v2.0.0-beta.6 (which is also what docker-compose --version reports).
It turns out that the answer is that PyCharm does not currently support docker-compose version 2, because it is still in beta. I simply should still be using docker-compose version 1.29.2.
As for why I was using docker-compose version 2, apparently the Docker cask on Homebrew defaults to enabling the "Use Docker Compose V2" Experimental Feature. I was able to disable this in my Docker Desktop settings.
this is my docker-compose.yaml:
version: '2.3'
services:
nvidia-smi-test:
runtime: nvidia
image: nvidia/cuda:9.0-base
environment:
- NVIDIA_VISIBLE_DEVICES=all
when I run docker-compose up it will appear:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
Unsupported config option for services.nvidia-smi-test: 'runtime'
but if I don't use docker-compose just use command docker run --runtime=nvidia nvidia/cuda:9.0-base nvidia-smi
there is no error ，
I don't know exactly what went wrong。Below is version of docker and docker-compose:
Docker version 19.03.5, build 633a0ea838
docker-compose version 1.17.0, build ac53b73
I would be very grateful if everyone could make some comments！
I found the problem, this is because the docker-compose version is not right, Docker Compose must be version 1.19.0 or higher.
I just uninstall the docker-compose (version 1.17.0) and install docker-compose(version 1.21.2) everything goes well!
How do you specify shared memory size in a Docker Compose configuration file?
I have an application that requires at least 2gb of shared memory, which I'd normally specify during docker run using the --shm-size 2g option.
According to the Docker Compose documentation, support for shm-size was added in version 2.3. However, if I run docker-compose with the trivial configuration:
version: "3"
services:
app_test:
build:
context: .
shm_size: '2gb'
dockerfile: Dockerfile.test
it fails with the error:
ERROR: The Compose file './docker-compose.test.yml' is invalid because:
services.app_test.build contains unsupported option: 'shm_size'
I'm using docker-compose version 1.24.1, which should be the most recent release that supports all modern file versions, including support for shm-size. Why is it giving me this error? What am I doing wrong?
I would like to run 2 docker images with docker-compose.
one image should run with nvidia-docker and the other with docker.
I've seen this post use nvidia-docker-compose launch a container, but exited soon
but this is not working for me(not even running only one image)...
any idea would be great.
UPDATE : please check nvidia-docker 2 and its support of docker-compose first
https://github.com/NVIDIA/nvidia-docker/wiki/Frequently-Asked-Questions#do-you-support-docker-compose
(I'd first suggest adding the nvidia-docker tag).
If you look at the nvidia-docker-compose code here it only generates a specific docker-file for docker-compose after a query of the nvidia configuration on localhost:3476.
You can also make by hand this docker-compose file as they turn out to be quite simple, follow this example, replace 375.66 with your nvidia driver version and put as many /dev/nvidia[n] lines as you have graphic cards (did not try to put services on separate GPUs but go for it !):
services:
exampleservice0:
devices:
- /dev/nvidia0
- /dev/nvidia1
- /dev/nvidiactl
- /dev/nvidia-uvm
- /dev/nvidia-uvm-tools
environment:
- EXAMPLE_ENV_VARIABLE=example
image: company/image
volumes:
- ./disk:/disk
- nvidia_driver_375.66:/usr/local/nvidia:ro
version: '2'
volumes:
media: null
nvidia_driver_375.66:
external: true
Then just run this hand-made docker-compose file with a classic docker-compose command.
Maybe you can then compose with non nvidia dockers by skipping the nvidia specific stuff in the other services.
Additionally to the accepted answer, here's my approach, a bit shorter.
I needed to use the old version of docker-compose (2.3) because of the required runtime: nvidia (won't necessarily work with version: 3 - see this). Setting NVIDIA_VISIBLE_DEVICES=all will make all the GPUs visible.
version: '2.3'
services:
your-service-name:
runtime: nvidia
environment:
- NVIDIA_VISIBLE_DEVICES=all
# ...your stuff
My example is available here.
Tested on NVIDIA Docker 2.5.0, Docker CE 19.03.13 and NVIDIA-SMI 418.152.00 and CUDA 10.1 on Debian 10.