As you know may noticed, Docker changed the name of compose from docker-compose to docker compose
I have a Makefile that calls docker-compose
run:
docker-compose up --build
However I want to make my Makefile portable, I was wondering if it is possible to the Makefile first tries if docker-compose exists, if not, uses docker compose
Is it possible?
If you want it to be the most portable then you'd implement it in the shell, something like:
run:
test -n "$$(command -v docker-compose)" \
&& docker-compose up --build \
|| docker compose up --build
If you're willing to use make-specific features you can do something a bit fancier such as:
ifeq ($(shell command -v docker-compose;),)
COMPOSE := docker compose
else
COMPOSE := docker-compose
endif
run:
$(COMPOSE) up --build
Related
We are trying to store the container names in my Makefile but I see below error when executing the build, someone please advise. Thanks.
.PHONY: metadata
metadata: .env1
docker pull IMAGE_NAME
docker run $IMAGE_NAME;
ID:= $(shell docker ps --format '{{.Names}}')
#echo ${ID}
docker cp ${ID}:/app/.env .env2
Container names are not shown in below "ID" Variable when executing the makefile from Jenkins
ID:=
/bin/sh: ID:=: command not found
There are a couple of things you can do in terms of pure Docker mechanics to simplify this.
You can specify an alternate command when you docker run an image: anything after the image name is taken as the image to run. For instance, you can cat the file as the main container command, and replace everything you have above as:
.PHONY: getmetadata
getmetadata: .env2
.env2: .env1
docker run --rm \
-e "ARTIFACTORY_USER=${ARTIFACTORY_CREDENTIALS_USR}" \
-e "ARTIFACTORY_PASSWORD=${ARTIFACTORY_CREDENTIALS_PSW}" \
--env-file .env1 \
"${ARTIFACTDATA_IMAGE_NAME}" \
cat /app/.env \
> $#
(It is usually better to avoid docker cp, docker exec, and other imperative-type commands; it is fairly inexpensive and better practice to run a new container when you need to.)
If you can't do this, you can docker run --name your choice of names, and then use that container name in the docker cp option.
.PHONY: getmetadata
getmetadata: .env2
.env2: .env1
docker run --name getmetadata ...
docker cp getmetadata:/app/.env $#
docker stop getmetadata
docker rm getmetadata
If you really can't avoid this at all, each line of the Makefile runs in a separate shell. On the one hand this means you need to join together lines if you want variables from one line to be visible in a later line; on the other, it means you have normal shell functionality available and don't need to use the GNU Make $(shell ...) extension (which evaluates when the Makefile is loaded and not when you're running the command).
.PHONY: getmetadata
getmetadata: .env2
.env2: .env1
# Note here:
# $$ escapes $ for the shell
# Multiple shell commands joined together with && \
# Beyond that, pure Bourne shell syntax
ID=$$(docker run -d ...) && \
echo "$$ID" && \
docker cp "$$ID:/app/.env" "$#"
I am relatively new to using docker-compose and am running a stack with the following command
docker-compose \
--project-name version-12 \
-f installation/docker-compose-common.yml \
-f installation/docker-compose-erpnext.yml \
--project-directory installation \
up -d
now, with the non-default docker-compose.yml files I can't manage to have docker-compose stop, docker-compose ps to work. I have tried to use the -f, or --project-name flags but couldn't make it happen.
Can anyone kindly advise how to make this work in such a scenario?
You need to repeat all of the docker-compose options for every command you need to run.
There are two ways around this. One is to write a shell script wrapper that invokes this command:
#!/bin/sh
# I am `docker-compose-erpnext.sh`
# Run me with any normal `docker-compose` options
exec docker-compose \
--project-name version-12 \
-f installation/docker-compose-common.yml \
-f installation/docker-compose-erpnext.yml \
--project-directory installation \
"$#"
Docker Compose also supports environment variables for most of its settings; many of these in turn can also be included in a .env file. You can't specify --project-directory this way, but it's documented to default to the directory of the Compose file.
export COMPOSE_PROJECT_NAME=version-12
export COMPOSE_FILE=installation/docker-compose-common.yml:installation/docker-compose-erpnext.yml
docker-compose up -d
docker-compose ps
You can put these two settings in a file name .env in the directory from which you're running docker-compose (not the installation subdirectory); but if you have multiple deployments you're trying to manage, you can't specify an alternate name for the file (there is neither a CLI option nor an environment variable setting for it).
It's CentOS 7, already installed vi and vim in my CentOS and I can use them. I run docker in CentOS, when I excute this line below:
docker exec -it mysolr /bin/bash
I cannot use vi/vim in the solr container:
bash: vim: command not found
Why is that and how do I fix it so I can use vi/vim to edit file in docker container?
A typical Docker image contains a minimal set of libraries and utilities to run one specific program. Additionally, Docker container filesystems are not long-lived: it is extremely routine to delete and recreate a container, for instance to use a newer version of a base image.
The upshot of this is that you never want to directly edit files in a Docker container, and most images aren't set up with "rich" editing tools. (BusyBox contains a minimal vi and so most Alpine-based images will too.) If you make some change, it will be lost as soon as you delete the container. (Similarly, you usually can install vim or emacs or whatever, but it will get lost as soon as the container is deleted: installing software in a running container isn't usually a best practice.)
There are two good ways to deal with this, depending on what kind of file it is.
If the file is part of the application, like a source file, edit, debug, and test it outside of Docker space. Once you're convinced it's right (by running unit tests and by running the program locally), docker build a new image with it, and docker run a new container with the new image.
ed config.py
pytest
docker build -t imagename .
docker run -d -p ... --name containername imagename
...
ed config.py
pytest
docker build -t imagename .
docker stop containername
docker run -d -p ... --name containername imagename
If the file is configuration that needs to be injected when the application starts, the docker run -v option is a good way to push it in. You can directly edit the config file on your host, but you'll probably need to restart (or delete and recreate) the container for it to notice.
ed config.txt
docker run \
-v $PWD/config.txt:/etc/whatever/config.txt \
--name containername -p ... \
imagename
...
ed config.txt
docker stop containername
docker rm containername
docker run ... imagename
I recently found out about Podman (https://podman.io). Having a way to use Linux fork processes instead of a Daemon and not having to run using root just got my attention.
But I'm very used to orchestrate the containers running on my machine (in production we use kubernetes) using docker-compose. And I truly like it.
So I'm trying to replace docker-compose. I will try to keep docker-compose and using podman as an alias to docker as Podman uses the same syntax as docker:
alias docker=podman
Will it work? Can you suggest any other tool? I really intend to keep my docker-compose.yml file, if possible.
Yes, that is doable now, check podman-compose, this is one way of doing it, another way is to convert the docker-compose yaml file to a kubernetes deployment using Kompose. there is a blog post from Jérôme Petazzoni #jpetazzo: from docker-compose to kubernetes deployment
Update 6 May 2022 : Podman now supports Docker Compose v2.2 and higher (see Podman 4.1.0 release notes)
Old answer:
Running docker-compose with Podman as a normal user (rootless)
Requirement: Podman version >= 3.2.1 (released in June 2021)
Install the executable docker-compose
curl -sL -o ~/docker-compose https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)
chmod 755 ~/docker-compose
Alternatively you could also run docker-compose in a container image (see below).
Run
systemctl --user start podman.socket
Set the environment variable DOCKER_HOST
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
Run
~/docker-compose up -d
Running docker-compose with Podman as root
Requirement: Podman version >= 3.0 (released in February 2021)
Follow the same procedure but remove the flag --user
systemctl start podman.socket
Running docker-compose in a container image
Use the container image docker.io/docker/compose to run
docker-compose
podman \
run \
--rm \
--detach \
--env DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock \
--security-opt label=disable \
--volume $XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock \
--volume $(pwd):$(pwd) \
--workdir $(pwd) \
docker.io/docker/compose \
--verbose \
up -d
(the flag --verbose is optional)
The same command with short command-line options on a single line:
podman run --rm -d -e DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock --security-opt label=disable -v $XDG_RUNTIME_DIR/podman/podman.sock:$XDG_RUNTIME_DIR/podman/podman.sock -v $(pwd):$(pwd) -w $(pwd) docker.io/docker/compose --verbose up -d
Regarding SELINUX: Runnng Podman with SELINUX is preferable from a security point-of-view, but I didn't get it to work on a Fedora 34 computer so I disabled SELINUX by adding the command-line option
--security-opt label=disable
Troubleshooting tips
Test the Docker REST API
A minimal check to see that the Docker REST API is working:
$ curl -H "Content-Type: application/json" \
--unix-socket $XDG_RUNTIME_DIR/podman/podman.sock \
http://localhost/_ping
OK$
Avoid short container image names
If any of your docker-compose.yaml or Dockerfile files contain a short container image name, for instance
$ grep image: docker-compose.yaml
image: mysql:8.0.19
$
$ grep FROM Dockerfile
FROM python:3.9
$
edit the files to use the whole container image name instead
$ grep image: docker-compose.yaml
image: docker.io/library/mysql:8.0.19
$
$ grep FROM Dockerfile
FROM docker.io/library/python:3.9
$
Most often short names have been used to reference DockerHub Official Images
(a catalogue) so a good guess would be to prepend the container image name with docker.io/library/
There are currently many different container image registries, not just DockerHub (docker.io). Writing the whole container image name is thus a good practice. Podman might complain otherwise depending on how Podman is configured.
Rootless users can't bind to ports below 1024
If for instance
$ grep -A1 ports: docker-compose.yml
ports:
- 80:80
$
edit docker-compose.yaml so that the host port number >= 1024, for instance 8080
$ grep -A1 ports: docker-compose.yml
ports:
- 8080:80
$
An alternative solution is to adjust net.ipv4.ip_unprivileged_port_start with sysctl (see Shortcomings of Rootless Podman)
In case Systemd is missing
Most Linux distributions use Systemd where you would preferably start the Podman service (providing the REST API) by "starting" the Podman socket
systemctl --user start podman.socket
or
systemctl start podman.socket
but in case Systemd is missing you could also start the Podman service directly
podman system service --time 0 unix:/some/path/podman.sock
Systemd gives the extra benefit that the Podman service is started on demand with Systemd socket activation and stops after some time of inactivity.
Caveat: Swarm functionality is missing
A difference to Docker is that the functionality relating to Swarm is not supported when using docker-compose with Podman.
References:
https://www.redhat.com/sysadmin/podman-docker-compose
https://github.com/containers/podman/discussions/10644#discussioncomment-857897
Ensure Podman is installed on your machine.
You can install Podman Compose in a terminal with the following command:
pip3 install https://github.com/containers/podman-compose/archive/devel.tar.gz
cd into the directory your docker-compose file is located in
Run podman-compose up
See the following link for a decent introduction.
There are scope of services which are defined in docker-compose.yml. These services have been started. I need to rebuild only one of these and start it without up other services.
I run the following commands:
docker-compose up -d # run all services
docker-compose stop nginx # stop only one. but it is still running !!!
docker-compose build --no-cache nginx
docker-compose up -d --no-deps # link nginx to other services
At the end I get the old nginx container.
Docker-compose doesn't kill all running containers!
docker-compose up
$ docker-compose up -d --no-deps --build <service_name>
--no-deps - Don't start linked services.
--build - Build images before starting containers.
With docker-compose 1.19 up
docker-compose up --build --force-recreate --no-deps [-d] [<service_name>..]
Without one or more service_name arguments all images will be built if missing and all containers will be recreated.
From the help menu
Options:
-d, --detach Detached mode: Run containers in the background,
print new container names. Incompatible with
--abort-on-container-exit.
--no-deps Don't start linked services.
--force-recreate Recreate containers even if their configuration
and image haven't changed.
--build Build images before starting containers.
Without cache
To force a rebuild to ignore cached layers, we have to first build a new image
docker-compose build --no-cache [<service_name>..]
From the help menu
Options:
--force-rm Always remove intermediate containers.
-m, --memory MEM Set memory limit for the build container.
--no-cache Do not use cache when building the image.
--no-rm Do not remove intermediate containers after a successful build.
Then recreate the container
docker-compose up --force-recreate --no-deps [-d] [<service_name>..]
This should fix your problem:
docker-compose ps # lists all services (id, name)
docker-compose stop <id/name> #this will stop only the selected container
docker-compose rm <id/name> # this will remove the docker container permanently
docker-compose up # builds/rebuilds all not already built container
As #HarlemSquirrel posted, it is the best and I think the correct solution.
But, to answer the OP specific problem, it should be something like the following command, as he doesn't want to recreate ALL services in the docker-compose.yml file, but only the nginx one:
docker-compose up -d --force-recreate --no-deps --build nginx
Options description:
Options:
-d Detached mode: Run containers in the background,
print new container names. Incompatible with
--abort-on-container-exit.
--force-recreate Recreate containers even if their configuration
and image haven't changed.
--build Build images before starting containers.
--no-deps Don't start linked services.
Maybe these steps are not quite correct, but I do like this:
stop docker compose: $ docker-compose down
WARNING: The following prune -a will delete all images, you may not want this as it could effect other projects. you can read more here
remove the container: $ docker system prune -a
start docker compose: $ docker-compose up -d
docker-compose stop nginx # stop if running
docker-compose rm -f nginx # remove without confirmation
docker-compose build nginx # build
docker-compose up -d nginx # create and start in background
Removing container with rm is essential. Without removing, Docker will start old container.
For me it only fetched new dependencies from Docker Hub with both --no-cache and --pull (which are available for docker-compose build.
# other steps before rebuild
docker-compose build --no-cache --pull nginx # rebuild nginx
# other steps after rebuild, e.g. up (see other answers)
The problem is:
$ docker-compose stop nginx
didn't work (you said it is still running). If you are going to rebuild it anyway, you can try killing it:
$ docker-compose kill nginx
If it still doesn't work, try to stop it with docker directly:
$ docker stop nginx
or delete it
$ docker rm -f nginx
If that still doesn't work, check your version of docker, you might want to upgrade.
It might be a bug, you could check if one matches your system/version. Here are a couple, for ex:
https://github.com/docker/docker/issues/10589
https://github.com/docker/docker/issues/12738
As a workaround, you could try to kill the process.
$ ps aux | grep docker
$ kill 225654 # example process id
Simply use :
docker-compose build [yml_service_name]
Replace [yml_service_name] with your service name in docker-compose.yml file. You can use docker-compose restart to make sure changes are effected. You can use --no-cache to ignore the cache.
You can use:
docker-compose build
And if you are using a docker profile:
docker-compose --profile profile_name build
Only:
$ docker-compose restart [yml_service_name]