How to run any commands in docker volumes? - docker

After couple of days testing and working on docker (i am in general trying to migrate from vagrant to docker) i encountered a huge problem which i am not sure how or where to fix it.
docker-compose.yml
version: "3"
services:
server:
build: .
volumes:
- ./:/var/www/dev
links:
- database_dev
- database_testing
- database_dev_2
- mail
- redis
ports:
- "80:8080"
tty: true
#the rest are only images of database redis and mailhog with ports
Dockerfile
example_1
FROM ubuntu:latest
LABEL Yamen Nassif
SHELL ["/bin/bash", "-c"]
RUN apt-get install vim mc net-tools iputils-ping zip curl git -y
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
RUN cd /var/www/dev
RUN composer install
Dockerfile
example_2
....
RUN apt-get install apache2 openssl php7.2 php7.2-common libapache2-mod-php7.2 php7.2-fpm php7.2-mysql php7.2-curl php7.2-dom php7.2-zip php7.2-gd php7.2-json php7.2-opcache php7.2-xml php7.2-cli php7.2-intl php7.2-mbstring php7.2-redis -y
# basically 2 files with just rooting to /var/www/dev
COPY docker/config/vhosts /etc/apache2/sites-available/
RUN service apache2 restart
....
now the example_1 composer.json file/directory not found
and example_2 apache will says the root dir is not found
file/directory = /var/www/dev
i guess its because its a volume and it wont be up until the container is fully up because if i launch the container without the prev commands which will lead to an error i can then login to the container and execute the commands from command line without anyerror
HOW TO FIX THIS ?

In your first Dockerfile, use the COPY directive to copy your application into the image before you do things like RUN composer install. It'd look something like
FROM php:7.0-cli
COPY . /usr/src/app
WORKDIR /usr/src/app
RUN composer install
(cribbed from the php image documentation; that image may not have composer preinstalled).
In both Dockerfiles, remember that each RUN command creates a new empty container, runs its command, and cleans up after itself. That means commands like RUN cd ... have no effect, and you can't start a service in the background in one RUN command and have it available later; it will get stopped before the Dockerfile moves on to the next line.
In the second Dockerfile, commands like service or systemctl or initctl just don't work in Docker and you shouldn't try to use them. Standard practice is to start the server process as a foreground process when the container launches via a default CMD directive. The flip side of this is that, since the server won't start until docker run time, your volume will be available at that point. I might RUN mkdir in the Dockerfile just to be sure it exists.

The problem seems the execution order. At image build time /var/www/dev is available. When you start a container from that image the container /var/www/dev is overwritten with your local mount.
If you need no access from your host, the you can simple skip the extra volume.
In case you want use it in other containers to, the you should work with symlinks.

Related

Dockerfile not copying file to image - no such file or folder

What I am trying to achieve:
copy a redis.config template to my docker image
read .env variables content and replace the template variables references (such as passwords, ports etc.) with values from .env
start the redis-server with the prepared config file
This way, I can have multiple redis instances setup for local dev, staging and production environments.
I have the following folder structure:
/redis
--.env
--Dockerfile
--redis.conf
This is the Dockerfile:
FROM redis:latest
COPY redis.conf ./
RUN apt-get update
RUN apt-get -y install gettext
RUN envsubst < redis.conf > redisconf
EXPOSE $REDIS_PORT
CMD ["redis-server redis.conf"]
When I go to the redis folder and run docker build -t redis-test . everything builds as expected, but when I do docker run -dp 6379:6379 redis-test afterwards the container crashes with the following error:
Fatal error, can't open config file '/data/redis-server redis.conf': No such file or directory
It seems that the redis.conf file from my folder is not getting correctly copied to my image? But the envsubst runs as expected so it seems that the file is there and the .env variables get overwriten as expected?
What am I doing wrong?
The immediate error is that you've explicitly put the CMD as a single word, so it is interpreted as an executable filename containing a space rather than an executable and a parameter. Split this into two words:
CMD ["redis-server", "redis.conf"]
There's a larger and more complex problem around when envsubst gets run. You're RUNning it as part of the image build, but that means it happens before the container is run and the environment variables are known.
I'd generally address this by writing a simple entrypoint wrapper script. This runs as the main container process, so after the Docker-level container setup happens, and it can see all of the container environment variables. It can run envsubst or whatever other first-time setup is required, and then run exec "$#" to invoke the normal container command.
#!/bin/sh
envsubst < redis.conf.tmpl > redis.conf
exec "$#"
Make this script executable on the host (chmod +x entrypoint.sh), COPY it into your image, and make that the ENTRYPOINT.
ROM redis:latest
COPY redis.conf.tmpl entrypoint.sh ./
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get -y install gettext
ENTRYPOINT ["./entrypoint.sh"]
CMD ["redis-server", "redis.conf"]

Can `docker-compose` commands by executed from within a Docker container?

Is it possible to run docker-compose commands from with a Docker container? As an example, I am trying to install https://datahubproject.io/docs/quickstart/ FROM within a Docker container that is built using the Dockerfile shown below. The Dockerfile creates a Linux container with the prerequisites the datahubproject.io project needs (Python) and clones the repository code to a Docker container. I then want to be able to execute the Docker compose scripts from the repository code (that is cloned to the newly built Docker container) to create the Docker containers needed to run the datahubproject.io project. This is not a docker commit question.
To try this, I have the following docker-compose.yml script:
version: '3.9'
# This is the docker configuration script
services:
datahub:
# run the commands in the Dockerfile (found in this directory)
build: .
# we need tty set to true to keep the container running after the build
tty: true
...and a Dockerfile (to setup a Linux environment with the requirements needed for datahubproject.io quickstart):
FROM debian:bullseye
ENV DEBIAN_FRONTEND noninteractive
# install some of the basics our environment will need
RUN apt-get update && apt-get install -y \
git \
docker \
pip \
python3-venv
# clone the GitHub code
RUN git clone https://github.com/kuhlaid/datahub.git --branch master --single-branch
RUN python3 -m venv venv
# # the `source` command needs the bash shell
SHELL ["/bin/bash", "-c"]
RUN source venv/bin/activate
RUN python3 -m pip install --upgrade pip wheel setuptools
RUN python3 -m pip install --upgrade acryl-datahub
CMD ["datahub version"]
CMD ["./datahub/docker/quickstart.sh"]
I run docker compose up from a command line where these two scripts are located to run the Dockerfile and create the start container that will be used to install the datahubproject.io project.
I receive this error:
datahub-datahub-1 | Quickstarting DataHub: version head
datahub-datahub-1 | Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
datahub-datahub-1 | No Datahub Neo4j volume found, starting with elasticsearch as graph service
datahub-datahub-1 | ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?
I do not know if what I am trying to do is even possible with Docker. Any suggestions to make this work? - thank you
Can docker-compose commands by executed from within a Docker container?
Yes. A command like any other.
Is it possible to run docker-compose commands from with a Docker container?
Yes.
Any suggestions to make this work?
Like with docker on the host. Either run docker daemon or connect to one with DOCKER_HOST. DIND is relevant https://hub.docker.com/_/docker .
The answer seems to be to modify the docker-compose.yml script to contain two additional settings:
version: '3.9'
# This is the docker configuration script
services:
datahub:
# run the commands in the Dockerfile (found in this directory)
build: .
# we need tty set to true to keep the container running after the build
tty: true
# ---------- adding the following two settings seems to fixes the issue of the `CMD ["./datahub/docker/quickstart.sh"]` failing in the Dockerfile
stdin_open: true
volumes:
- /var/run/docker.sock:/var/run/docker.sock

docker-compose not producting "No Such File or Directory" when files exist in container

I have a simple Dockerfile
FROM python:3.8-slim-buster
RUN apt-get update && apt-get install
RUN apt-get install -y \
curl \
gcc \
make \
python3-psycopg2 \
postgresql-client \
libpq-dev
RUN mkdir -p /var/www/myapp
WORKDIR /var/www/myapp
COPY . /var/www/myapp
RUN chmod 700 ./scripts/*.sh
And an associated docker-compose file
version: "3"
volumes:
postgresdata:
services:
myapp:
image: ralston3/myapp_api:prod-latest
tty: true
command: /bin/bash -c "/var/www/myapp/scripts/myscript.sh && echo 'hello world'"
ports:
- 8000:8000
volumes:
- .:/var/www/myapp
environment:
SOME_ENV_VARS=SOME_VARIABLE
# ... more here
depends_on:
- redis
- postgresql
# ... other docker services defined below
When I run docker-compose up via:
docker-compose up -f /path/to/docker-compose.yml up
My myapp container/service fails with myapp_myapp_1 exited with code 127 with another error mentioning myapp_1 | /bin/sh: 1: /var/www/myapp/scripts/myscript.sh: not found
Further, if I exec into the myapp container via docker exec -it {CONTAINER_ID} /bin/bash I can clearly see that all of my files are there. I can literally run the /var/www/myapp/scripts/myscript.sh and it works fine.
However, there seems to be some issue with docker-compose (which could totally be my mistake). But I'm just confused as to how I can exec into the container and clearly see the files there. But docker-compose exists with 127 saying "No such file or directory".
You are bind mounting the current directory into "/var/www/myapp" so it may be that your local directory is "hiding/overwriting" the container directory. Try removing the volumes declaration for you myapp service and if that works then you know it is the bind mount causing the issue.
Unrelated to your question, but a problem you will also encounter: you're installing Python a second time, above and beyond the version pre-installed in the python Docker image.
Either switch to debian:buster as base image, or don't bother installing antyhign with apt-get and instead just pip install your dependencies like psycopg.
See https://pythonspeed.com/articles/official-python-docker-image/ for explanation why you don't need to do this.
in my case there were 2 stages: builder and runner.
I was getting an executable in builder and running that exe using the alpine image in runner.
My mistake here was that I didn't use the alpine version for the builder. Ex. I used golang:1.20 but when I used golang:1.20-alpine the problem went away.
Make sure you use the correct version and tag!

Running composer install in docker container

I have a docker-compose.yml script which looks like this:
version: '2'
services:
php:
build: ./docker/php
volumes:
- .:/var/www/website
The DockerFile located in ./docker/php looks like this:
FROM php:fpm
RUN php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
RUN php composer-setup.php
RUN php -r "unlink('composer-setup.php');"
RUN mv composer.phar /usr/local/bin/composer
RUN composer update -d /var/www/website
Eventho this always fails with the error
[RuntimeException]
Invalid working directory specified, /var/www/website does not exist.
When I remove the RUN composer update line and enter the container, the directory does exist and contains my project code.
Please tell me if I am doing anything wrong OR if I'm doing the composer update on a wrong place
RUN ... lines are run when the image is being built.
Volumes are attached to the container. You have at least two options here:
use COPY command to, well, copy your app code to the image so that all commands after that command will have access to it. (Do not push the image to any public Docker repo as it will contain your source that you probably don't want to leak)
install composer dependencies with command run on your container (CMD or ENTRYPOINT in Dockerfile or command option in docker-compose)
You are mounting your local volume over your build directory so anything you built in '/var/www/website' will be mounted over by your local volume when the container runs.

How can I write Dockerfile for Yesod? "RUN yesod init -n myApp -d postgresql" didn't work as expected

I tried to make a simple application with Yesod and PostgreSQL using Docker Compose but RUN yesod init -n myApp -d postgresql didn't seem to work as expected.
I defined Dockerfile and docker-compose.yml as below:
Dockerfile:
FROM shuny/ghc-7.8.4:latest
MAINTAINER shuny
# Create default config
RUN cabal update
# Add stackage remote repo
RUN sed -i 's/^remote-repo: [a-zA-Z0-9_\/:.]*$/remote-repo: stackage:http:\/\/www.stackage.org\/lts/g' /root/.cabal/config
# Update packages
RUN cabal update
# Generate locale otherwise happy (because of tf-random) will fail
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
RUN echo $LANG
# Install build tools for yesod
RUN cabal install alex happy yesod-bin
# Install library for yesod-postgres
RUN apt-get update && apt-get install -y libpq-dev
RUN mkdir /code
WORKDIR /code
RUN yesod init -n myApp -d postgresql
WORKDIR /code/myApp
RUN cabal sandbox init && cabal install --only-dependencies --max-backjumps=-1 --reorder-goals
RUN cabal configure && cabal build && cabal install
ADD . /code
WORKDIR /code
# ADD settings.yml /code/myApp/config/
docker-compose.yml:
database:
image: postgres
ports:
- "5432"
web:
build: .
tty: true
command: yesod devel
volumes:
- .:/code/
ports:
- "3000:3000"
links:
- database
and docker-compose build returned as below:
Step 0 : FROM shuny/ghc-7.8.4:latest
...
Step 17 : WORKDIR /code
---> Running in bf99d0aca48c
---> 37c3c94338d7
Removing intermediate container bf99d0aca48c
Successfully built 37c3c94338d7
but when I check like this:
$docker-compose run web /bin/bash
root#0fe5fb1a3b20:/code# ls
root#0fe5fb1a3b20:/code#
it showed nothing while this commands seem to work as expected:
docker run -ti 37c3c94338d7
root#31e94428de37:/code# ls
docker-compose.yml Dockerfile myApp settings.yml
root#31e94428de37:/code# ls myApp/
app config Handler Model.hs Settings.hs test
Application.hs dist Import myApp.cabal static
cabal.sandbox.config Foundation.hs Import.hs Settings templates
How can I fix it?
I really appliciate any feedback, thank you.
You are doing strange things with volumes and the ADD instruction.
First you build your application inside the image:
RUN yesod init -n myApp -d postgresql
WORKDIR /code/myApp
RUN cabal sandbox init && cabal install --only-dependencies --max-backjumps=-1 --reorder-goals
RUN cabal configure && cabal build && cabal install
Then you add the content of the folder that contains the Dockerfile in the /code folder of the image. I guess this step is useless.
ADD . /code
Then, if you run a container without -volume option, everything works fine
docker run -ti 37c3c94338d7
But in your docker-compose.yml file, you specified a volume option that overides the /code folder in the container with the folder that contains the docker-compose.yml file on the host machine. Therefore, you don't have the content generated during the build of your image anymore.
There are two possibilities:
Don't use the volume instruction in the docker-compose.yml file
Put the content of the /code/myApp/ folder of the image inside the ./myApp folder of the host.
It depends on why you want to use the volume option in docker-compose.yml.
I don't really know what is your goal. But if what you are trying to do is to access to the files built inside the container from the host machine, maybe this should do what you are looking for:
Remove the build steps from your Dockerfile
Run a shell inside a "web" container: docker-compose run web bash
Launch the build commands
So you will have built your application while the volume was mounted and will see the files on the host machine.
Exit the shell
Launch Docker Compose normally
If you just want to be able to backup the content of the /code/myApp/ folder, maybe you should omit the path on the host machine from the volume section of docker-compose.yml.
volumes:
- /code/
And follow this section of the documentation
I hope it helps

Resources