Good afternoon
I'm a new user of Docker and I have a question please:
I created a container named "container_worker" which running a Python script to create some data.
The data created is stored in the container in a file named "data".
I want to copy this "data" file to my host to be able to use it for another purpose.
I saw it's possible to do it with the "docker cp" command but I want to do it directly in my Dockerfile or my Docker-compose file.
Here are my files:
Dockerfile:
FROM archlinux
RUN pacman-db-upgrade \
&& pacman -Syyu --noconfirm \
&& pacman -S python --noconfirm \
&& pacman -S python-pip --noconfirm \
&& pip install requests
COPY /worker/script.py .
CMD python3 vmtracer.py >> data
docker-compose.yml
version: "3"
services:
worker:
image: image_worker
container_name: container_worker
build:
dockerfile: ./worker/Dockerfile
Thank you very much.
You can bind/mount volume from your local machine to container and put the output to that shared location.
Use command docker volume create my-vol , to create volume.
Use command docker volume ls ,to list your volumes.
Use parameter volumes: {your-volume-name} in docker-compose file to use the created volume.
Refer the link for more : https://docs.docker.com/storage/volumes/
Related
I'm deploying an application with a Dockerfile and docker-compose. It loads a model from an AWS bucket to run the application. When the containers get restarted (not intentionally but because of the cloud provider), it loads again the model from AWS. What I would like to achive is storing the model on a persistent volume. In case of a restart, I would like to check whether the volume exists and is not empty and if so run a different docker-compose file which has a different bash command, not loading the model from AWS again.
This is part of my docker-compose.yml:
rasa-server:
image: rasa-bot:latest
working_dir: /app
build:
context: ./
dockerfile: Dockerfile
volumes:
- ./models /app/models
command: bash -c "rasa run --model model.tar.gz --remote-storage aws --endpoints endpoints.yml --credentials credentials.yml --enable-api --cors \"*\" --debug --port 5006"
In case of a restart the command would look like this:
command: bash -c "rasa run --model model.tar.gz --endpoints endpoints.yml --credentials credentials.yml --enable-api --cors \"*\" --debug --port 5006"
Note that this
--remote-storage aws
was removed.
This is my Dockerfile:
FROM python:3.7.7-stretch AS BASE
RUN apt-get update \
&& apt-get --assume-yes --no-install-recommends install \
build-essential \
curl \
git \
jq \
libgomp1 \
vim
WORKDIR /app
RUN pip install --no-cache-dir --upgrade pip
RUN pip install rasa==3.1
RUN pip3 install boto3
ADD . .
I know that I can use this:
docker volume ls
to list volumes. But I do not know how to wrap this in a if condition to check whether
- ./models /app/models
exists and is not empty and if it is not empty run a second docker-compose.yml which contains the second modified bash command.
I would accomplish this by making the main container command actually be a script that looks to see if the file exists and optionally fills in the command line argument.
#!/bin/sh
MODEL_EXISTS=$(test -f /app/models/model.tar.gz && echo yes)
exec rasa run \
--model model.tar.gz \
${MODEL_EXISTS:---remote-storage aws} \
--endpoints endpoints.yml \
...
The first line uses the test(1) shell command to see if the file already exists, and sets the variable MODEL_EXISTS to yes if it exists and empty if it does not. Then in the command, there is a shell parameter expansion: if the variable MODEL_EXISTS is unset or empty :- then expand and split the text --remote-storage aws. (This approach inspired by BashFAQ/050.)
In your Dockerfile, COPY this script into your image and make it be the default CMD. It needs to be executable like any other script (run chmod +x on your host and commit that change to source control); since it is executable and begins with a "shebang" line, you do not need to explicitly name the shell when you run it.
...
COPY rasa-run ./
CMD ["./rasa-run"]
In your Compose file, you do not need to override the command:, change the working_dir: from what the Dockerfile sets, or change from a couple of Compose-provided defaults. You should be able to reduce this to
version: '3.8'
services:
rasa-server:
build: .
volumes:
- ./models:/app/models
More generally for this class of question, I might suggest:
Prefer setting a Dockerfile default CMD to a Compose command: override; and
Write out non-trivial logic in a script and run that script as the main container command; don't write complicated conditionals in an inline CMD or command:.
You could have an if statement in your bash command to use AWS or not depending on the result you get from docker volume ls
using -f name= you can filter based on the volume name and then you can check if it's not null and run a different command.
Note that this command is just an example and I have no idea if it works or not as I don't use bash everyday.
command: bash -c "
VOLUME = docker volume ls -f name=FOO
if [ -z "$VOLUME" ];
then
rasa run --model model.tar.gz --remote-storage aws --endpoints endpoints.yml --credentials credentials.yml --enable-api --cors \"*\" --debug --port 5006
else
rasa run --model model.tar.gz --endpoints endpoints.yml --credentials credentials.yml --enable-api --cors \"*\" --debug --port 5006
fi
"
I have a simple Dockerfile
FROM python:3.8-slim-buster
RUN apt-get update && apt-get install
RUN apt-get install -y \
curl \
gcc \
make \
python3-psycopg2 \
postgresql-client \
libpq-dev
RUN mkdir -p /var/www/myapp
WORKDIR /var/www/myapp
COPY . /var/www/myapp
RUN chmod 700 ./scripts/*.sh
And an associated docker-compose file
version: "3"
volumes:
postgresdata:
services:
myapp:
image: ralston3/myapp_api:prod-latest
tty: true
command: /bin/bash -c "/var/www/myapp/scripts/myscript.sh && echo 'hello world'"
ports:
- 8000:8000
volumes:
- .:/var/www/myapp
environment:
SOME_ENV_VARS=SOME_VARIABLE
# ... more here
depends_on:
- redis
- postgresql
# ... other docker services defined below
When I run docker-compose up via:
docker-compose up -f /path/to/docker-compose.yml up
My myapp container/service fails with myapp_myapp_1 exited with code 127 with another error mentioning myapp_1 | /bin/sh: 1: /var/www/myapp/scripts/myscript.sh: not found
Further, if I exec into the myapp container via docker exec -it {CONTAINER_ID} /bin/bash I can clearly see that all of my files are there. I can literally run the /var/www/myapp/scripts/myscript.sh and it works fine.
However, there seems to be some issue with docker-compose (which could totally be my mistake). But I'm just confused as to how I can exec into the container and clearly see the files there. But docker-compose exists with 127 saying "No such file or directory".
You are bind mounting the current directory into "/var/www/myapp" so it may be that your local directory is "hiding/overwriting" the container directory. Try removing the volumes declaration for you myapp service and if that works then you know it is the bind mount causing the issue.
Unrelated to your question, but a problem you will also encounter: you're installing Python a second time, above and beyond the version pre-installed in the python Docker image.
Either switch to debian:buster as base image, or don't bother installing antyhign with apt-get and instead just pip install your dependencies like psycopg.
See https://pythonspeed.com/articles/official-python-docker-image/ for explanation why you don't need to do this.
in my case there were 2 stages: builder and runner.
I was getting an executable in builder and running that exe using the alpine image in runner.
My mistake here was that I didn't use the alpine version for the builder. Ex. I used golang:1.20 but when I used golang:1.20-alpine the problem went away.
Make sure you use the correct version and tag!
I have an existing docker-compose file
version: '3.6'
services:
verdaccio:
restart: always
image: verdaccio/verdaccio
container_name: verdaccio
ports:
- 4873:4873
volumes:
- conf:/verdaccio/conf
- storage:/verdaccio/storage
- plugins:/verdaccio/plugins
environment:
- VERDACCIO_PROTOCOL=https
networks:
default:
external:
name: registry
I would like to use an DockerFile instead of docker-compose as it will be more easy to deploy DockerFile on an Azure container registry.
I have tried many solution posted on blogs and others but nothing worked as I needed.
How can I create simple DockerFile from the above docker-compose file?
You can't. Many of the Docker Compose options (and the equivalent docker run options) can only be set when you start a container. In your example, the restart policy, published ports, mounted volumes, network configuration, and overriding the container name are all runtime-only options.
If you built a Docker image matching this, the most you could add in is setting that one ENV variable, and COPYing in the configuration files and plugins rather than storing them in named volumes. The majority of that docker-compose.yml would still be required.
If you want to put conf, storage and plugins files/folders into image, you can just copy them:
FROM verdaccio/verdaccio
WORKDIR /verdaccio
COPY conf conf
COPY storage storage
COPY plugins plugins
but if you need to keep files/and folder changes, then you should keep them as it is now.
docker-compose uses an existing image.
If what you want is to create a custom image and use it with your docker-compose set up this is perfectly possible.
create your Dockerfile - example here: https://docs.docker.com/get-started/part2/
build an "image" from your Dockerfile: docker build -f /path/to/Dockerfile -t saurabh_rai/myapp:1.0 this returns an image ID something like 12abef12
login to your dockerhub account (saurabh_rai) and create a repo for the image to be pushed to (myapp)
docker push saurabh_rai/myapp:1.0 - will push your image to hub.docker.com repo for your user to the myapp repo. You may need to perform docker login for this to work and enter your username/password as usual at the command line.
Update your docker-compose.yaml file to use your image saurabh_rai/myapp:1.0
example docker-compose.yaml:
version: '3.6'
services:
verdaccio:
restart: always
container_name: verdaccio
image: saurabh_rai/myapp:1.0
ports:
- 4873:4873
volumes:
- conf:/verdaccio/conf
- storage:/verdaccio/storage
- plugins:/verdaccio/plugins
environment:
- VERDACCIO_PROTOCOL=https
network:
- registry
I have solve this issue by using an existing verdaccio DockerFile given below.
FROM node:12.16.2-alpine as builder
ENV NODE_ENV=production \
VERDACCIO_BUILD_REGISTRY=https://registry.verdaccio.org
RUN apk --no-cache add openssl ca-certificates wget && \
apk --no-cache add g++ gcc libgcc libstdc++ linux-headers make python && \
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
wget -q https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.25-r0/glibc-2.25-r0.apk && \
apk add glibc-2.25-r0.apk
WORKDIR /opt/verdaccio-build
COPY . .
RUN yarn config set registry $VERDACCIO_BUILD_REGISTRY && \
yarn install --production=false && \
yarn lint && \
yarn code:docker-build && \
yarn cache clean && \
yarn install --production=true
FROM node:12.16.2-alpine
LABEL maintainer="https://github.com/verdaccio/verdaccio"
ENV VERDACCIO_APPDIR=/opt/verdaccio \
VERDACCIO_USER_NAME=verdaccio \
VERDACCIO_USER_UID=10001 \
VERDACCIO_PORT=4873 \
VERDACCIO_PROTOCOL=http
ENV PATH=$VERDACCIO_APPDIR/docker-bin:$PATH \
HOME=$VERDACCIO_APPDIR
WORKDIR $VERDACCIO_APPDIR
RUN apk --no-cache add openssl dumb-init
RUN mkdir -p /verdaccio/storage /verdaccio/plugins /verdaccio/conf
COPY --from=builder /opt/verdaccio-build .
ADD conf/docker.yaml /verdaccio/conf/config.yaml
RUN adduser -u $VERDACCIO_USER_UID -S -D -h $VERDACCIO_APPDIR -g "$VERDACCIO_USER_NAME user" -s /sbin/nologin $VERDACCIO_USER_NAME && \
chmod -R +x $VERDACCIO_APPDIR/bin $VERDACCIO_APPDIR/docker-bin && \
chown -R $VERDACCIO_USER_UID:root /verdaccio/storage && \
chmod -R g=u /verdaccio/storage /etc/passwd
USER $VERDACCIO_USER_UID
EXPOSE $VERDACCIO_PORT
VOLUME /verdaccio/storage
ENTRYPOINT ["uid_entrypoint"]
CMD $VERDACCIO_APPDIR/bin/verdaccio --config /verdaccio/conf/config.yaml --listen $VERDACCIO_PROTOCOL://0.0.0.0:$VERDACCIO_PORT
By making few changes to the DockerFile I was able to build and push my docker image to azure container registry and deploy to an app service.
#Giga Kokaia, #Rob Evans, #Aman - Thank you for the suggestions it became more easy to think.
I'm using an Apache / MySql Docker-compose set up which is all good. However the issue comes when, as this is for local development, the web container points to a local folder, for which I need Apache to have permissions to.
Using
RUN mkdir /www \
&& chown -R apache:apache /www
VOLUME ["/www"]
is fine if I run the Apache dockerfile by itself or if I run it in docker-compose without specifying a volume. But this means that I can't point that volume at a local directory, in this scenario "www" exists inside the container but doesn't map to the host machine. If I specify a volume inside the docker-compose file then it maps as expected but doesn't allow me to CHOWN the folder / files (even if I exec into the container)
Below is a proof of concept, I'm running on Windows 10 / Docker Desktop Community Version 2.0.0.0-win81 (29211)
EDIT (commented exposing the port, built the dockerfile from docker-compose and changed the port to 80 from 81)
EDIT (I've updated the following files, see bottom, I'm leaving these for posterity)
docker-compose.yml
version: '3.2'
services:
web:
restart: always
build:
context: .
ports:
- 80:80
volumes:
- ./:/www
Dockerfile
FROM centos:centos6 as stage1
RUN yum -y update && yum clean all \
&& yum --setopt=tsflags=nodocs install -y yum-utils \
httpd \
php
FROM stage1 as stage2
RUN mkdir /www \
&& chown -R apache:apache /www
#VOLUME ["/www"]
#EXPOSE 80
ENTRYPOINT ["/usr/sbin/httpd", "-D", "FOREGROUND"]
UPDATED Proof of concept files
Docker-compose.yml
version: '3.2'
services:
web:
build:
context: .
ports:
- 80:80
volumes:
- ./:/www
Dockerfile
FROM centos:centos6
RUN yum -y update && yum clean all \
&& yum --setopt=tsflags=nodocs install -y yum-utils \
httpd \
php
COPY ./entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
entrypoint.sh
#!/bin/bash
set -e #exit straight away if there's an issue
chown -R apache:apache /www
# Apache
/usr/sbin/httpd -D FOREGROUND
Docker for Windows uses a CIFS/Samba network file share to bind-mount host files into the Linux VM running docker. That is always done as root:root so all bind-mount files/dirs will always show that when seen from inside container. This is a known limitation of the way docker shares these files between the OS's.
Workarounds:
In many cases, this isn't an issue. The host files are shared into the container world-readable, so local app development while running in the container is fine. For cache files, user uploads, etc. just be sure they are written into a container path that isn't to the host-bind mount, so they stay in Linux where you can control the perms.
If needed, for development only, run the app in the container as root if it needs write permissions to host OS files. You can override this at runtime: e.g. docker run -u root or user:root in docker-compose.yml
For working with database files, don't bind-mount them, but use named volumes to keep the files in the Linux VM. You can always use docker cp to copy files in and out of volumes for a quick backup.
You're using
RUN mkdir /www \
&& chown -R apache:apache /www
Prior to docker-compose mapping the local . directory to www.
You need to create a file entrypoint.sh or similar. Give it a shbang. And inside that you should run chown -R apache:apache /www. You do not need the mkdir as that's created by docker compose volume config ./:/www.
After that command in your entrypoint.sh file you should add in what you currently have for your entrypoint /usr/sbin/httpd -D FOREGROUND.
Then finally you of course need to set your new entrypoint to use the entrypoint.sh file ENTRYPOINT ["/entrypoint.sh"]
I have an aws/appium test project I want to run in docker. I have a bash script that runs in the container which downloads a file from S3 and creates a zip of my project.
The Dockerfile:
FROM maven:3.3.9
RUN apt-get update && \
apt-get -y install python && \
apt-get -y install python-pip && \
pip install awscli
RUN export PATH=$PATH:/usr/local/bin
There's a docker compose file, the command runs a bash script:
version: '2'
volumes:
maven_cache: ~
services:
application: &application
build: .
tmpfs:
- /tmp:rw,nodev,noexec,nosuid
volumes:
- ./:/app
- maven_cache:/root/.m2/repository
working_dir: /app
command: ./aws-upload.sh
This is the beginning of the ./aws-upload.sh bash script. It prepares the files I need for uploading later:
#!/usr/bin/env bash
mvn clean package -DskipTests=true
aws s3 cp s3://<bucket-name>/app.apk $(pwd)
cp target/zip-with-dependencies.zip $(pwd)
I only want the above files to exist within the container, however they appear locally also. Is there something in my docker-compose file that isn't configured correctly?
Thanks
In your compose file you are defining a volume ./:/app which maps the host folder where the compose file is located to the containers app folder. If you execute your bash script in the app folder it will also make the files it is creating available on the host.
If you want to avoid this either remove the volume mapping (in case you don't need it) or execute the script in another folder which is not mapped to your host.
This is normal. When you declared the following inside the composefile:
volumes:
- ./:/app
This means mount the current host directory onto /app inside the container. This will effectivelty keep the current directory and the /app folder inside the container in sync.
Thus if the aws-upload.sh script creates files in /app, they will also show next to the compose file.