I'm trying to dockerize some services for development on my machine and wondering how docker run --volume=.. works. For example, if I do something like
docker run --volume=/path/to/data:/data [...]
will /path/to/data be (re)created locally only if it doesn't exist? Is the initial data copied from the container's image?
Links to relevant documentation would be appreciated.
The --volume option is described in the docker run reference docs, which forwards you on to the dedicated Managed data in containers docs, which then forwards you on to the Bind mounts docs.
There, it says:
If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v will create the endpoint for you. It is always created as a directory.
Example:
docker run -it -v ${PWD}/scripts:/code/scripts myimage bash
Yes, the directory on the host FS will be created only if it does not already exist.
The same time, Docker will not copy anything from the image into bind-mounted volume, so the mount path will appear as empty directory inside the container. Whatever was in the image will be hidden.
If you need original data to be copied over, you need to implement this functionality yourself. Fortunately, it is pretty easy thing to do.
Among the last steps in Dockerfile, move or copy the original directory elsewhere. E.g. RUN mv /data /original-data
Add a custom script to the image, which would serve as entrypoint, and will copy the data needed into the mounted volume (see code example below). Dockerfile directive: ADD entrypoint.sh /entrypoint.sh
Add ENTRYPOINT directive, to instruct Docker to invoke your script as a part of container initialization: ENTRYPOINT ['/entrypoint.sh']
The script entrypoint.sh could look like following (simplified example):
#!/bin/bash
set -e
SOURCE_DIR=/original-data
TARGET_DIR=/data
if [ $(find $TARGET_DIR -maxdepth 0 -type d -empty) 2>/dev/null) ]; then
cp -r --preserve-all $SOURCE_DIR/* $TARGET_DIR/
fi
# continue Docker container initialization, execute CMD
exec $#
If there's already some entrypoint script in your image, you can just add appropriate logic to it.
Related
I want to make sure I understand correctly docker: when i build an image from the current directory I run:
docker build -t imgfile .
What happens when i change the content of a file in the directory AFTER the image is built? From what i've tried it seems it changes the content of the docker image also dynamically.
I thought the docker image was like a zip file that could only be changed with docker commands or logging into the image and running commands.
The dockerfile is :
FROM lambci/lambda:build-python3.8
WORKDIR /var/task
EXPOSE 8000
RUN echo 'export PS1="\[\e[36m\]zappashell>\[\e[m\] "' >> /root/.bashrc
CMD ["bash"]
And the docker run command is :
docker run -ti -p 8000:8000 -e AWS_PROFILE=zappa -v "$(pwd):/var/task" -v ~/.aws/:/root/.aws --rm zappa-docker-image
Thank you
Best,
Your docker run command isn't really running your image at all. The docker run -v $(pwd):/var/task syntax overwrites what was in /var/task in the image with a bind mount to the current directory on the host. So when you edit a file on your host, the container has the same host directory (and not the content from the image) and you see the changes inside the container as well.
You're right that the image is immutable. The image you show doesn't really contain anything, beyond a .bashrc file that won't usually be used. You can try running the image without the -v options to see:
docker run --rm zappa-docker-image ls -al
# just shows `.` and `..` directories
I'd recommend making sure you COPY your application into the image, setting its CMD to actually run the application, and removing the -v option that overwrites its main directory. If your goal is to run host code against host files with host supporting data like your AWS credentials, you're not really getting much benefit from introducing Docker in between your application and every single file it uses.
I am building my first docker image. I am a beginner
It is a simple python http server. This is my DockerFile
FROM python:3.8.0-slim
WORKDIR /src
COPY src/ .
CMD [ "python", "-m", "http.server", "--cgi", "8000"]
I have a config folder in /src with some config files.
I named the image "my-server"
I create a container with
docker run -d \
--name "my-server" \
-p 8000:8000 \
-v /dockerdata/appdata/my-server/config/:/src/config \
--restart unless-stopped \
my-server
the issue is /dockerdata/appdata/my-server/config/ is empty on my host.
I see this done on all docker images on dockerhub I use and the mounted volumes are not empty for these images.
How do they do it?
Their startup sequence explicitly copies source files into the volume, or otherwise creates them. A Docker mount always replaces the content in the image with the content of whatever's being mounted; there is no way to mount the container content to the host.
(The one exception to this "always" is, if you're using native Docker, and you're mounting a named volume, and the named volume is empty, then content from the image is copied into the volume first; but the content is never ever updated, it only works for named volumes and not other kinds of mounts, and it doesn't work on other environments like Kubernetes. I would not rely on this approach.)
If the configuration is a single file, this isn't a huge imposition. You probably already need to distribute artifacts like a docker-compose.yml file separately from the image, so distributing a default configuration isn't much more. If defaults are compiled into your application and an empty configuration is valid, this also simplifies things. Another helpful approach could be to have a search path for configuration files, and read both a "user" and "system" configuration.
If you do need to copy files out to a host directory or other mount point, I would generally do this with an entrypoint wrapper script. You will need to keep a copy of the configuration in the image somewhere that's not the actual config directory so that you can copy it when it doesn't exist. The script can be fairly straightforward:
#!/bin/sh
# Copy the default configuration if it doesn't exist
if [ ! -f config/config.yml ]; then
cp default-config/config.yml config
fi
# Run the main container command
exec "$#"
You may need to do some shuffling in your Dockerfile; the important thing is to make this script be the ENTRYPOINT but leave the CMD unchanged.
# Save the "normal" config away; the entrypoint script will create
# the "real" config if one isn't mounted
RUN mv config default-config \
&& mkdir config
# Launch the server via the entrypoint wrapper
ENTRYPOINT ["./entrypoint.sh"] # must be JSON array syntax
CMD ["python", "-m", "http.server", "--cgi", "8000"] # unchanged
This is expected, bind mount to a container directory will result to the content in the directory to be obscured. If you mount to a named volume, the directory’s contents are copied into the volume.
docker run -d \
--name "my-server" \
-p 8000:8000 \
-v myvol:/src/config \
--restart unless-stopped \
my-server
Now if you run docker run -it -v myvol:/config --rm busybox ls /config you will see the copied content.
I am new to docker, I'm dynamically creating a file which is in docker container and want to copy that local machine at the same time, please let me know how it is possible through volumes.
For now, I have to use the below command again and again to check the file data :
docker cp source destination
How it can be done through volumes, the file format will be in .csv or .xlsx? I mean what should I write the command in docker files so that it can copy the file
What you need is volume. You have to add your current directory as a volume to the docker container when you first create the container so that they are the same folder. By doing this, you'll be able to sync the files in that folder automatically. But I'm assuming you're using docker for development environment.
This is how I run my container.
docker run -d -it --name {container_name} --volume $PWD:{directory_in_container} --entrypoint /bin/bash {image_name}
In addition to your run command, you have to add --volume $PWD:{directory_in_container} to your run script.
If you have a problem again, just add more detail to your question.
Things you can add might be your Dockerfile, and how you first run your container.
Trying to copy files from the container to the local first
So, I have a custom Dockerfile, RUN mkdir /test1 && touch /test1/1.txt and then I build my image and I have created an empty folder in local path /root/test1
and docker run -d --name container1 -v /root/test1:/test1 Image:1
I tried to copy files from containers to the local folder, and I wanted to use it later on. but it is taking my local folder as a preceding and making my container empty.
Could you please someone help me here?
For example, I have built my own custom Jenkins file, for the first time while launching it I need to copy all the configurations and changes locally from the container, and later if wanted to delete my container and launch it again don't need to configure from the scratch.
Thanks,
The relatively new --mount flag replaces the -v/--volume mount. It's easier to understand (syntactically) and is also more verbose (see https://docs.docker.com/storage/volumes/).
You can mount and copy with:
docker run -i \
--rm \
--mount type=bind,source="$(pwd)"/root/test1,target=/test1 \
/bin/bash << COMMANDS
cp <files> /test1
COMMANDS
where you need to adjust the cp command to your needs. I'm not sure if you need the "$(pwd)" part.
Off the top, without testing to confirm, i think it is
docker cp container1:/path/on/container/filename /path/on/hostmachine/
EDIT: Yes that should work. Also "container1" is used here because that was the container's name provided in the example
In general it works like this
container to host
docker cp containername:/containerpath/ /hostpath/
host to container
docker cp /hostpath/ containername:/containerpath/
This question shows how to copy files out of a stopped container. This requires that I know the full path to the file including its file name. I know the directory I want to copy a file out of, but I do not know its file name since that is generated dynamically. How do I list the files in a directory in a stopped Docker container?
The following Docker command works great if the Docker container is running. But, it fails if the Docker container is stopped.
docker exec --privileged MyContainer ls -1 /var/log
Note: The files are not stored in a persistent volume.
This answer to another question shows how to start a stopped container with another command. Here are the commands to list files in a stopped container.
Commit the stopped container to a new image: test_image.
docker commit $CONTAINER_ID test_image
Run the new image in a new container with a shell.
docker run -ti --entrypoint=sh test_image
Run the list file command in the new container.
docker exec --privileged $NEW_CONTAINER_ID ls -1 /var/log
When starting the container is not an option, you can always export your image (a bit overkill but..) and list its contents:
docker export -o dump.tar <container id>
tar -tvf dump.tar
Reference: Baeldung - Exploring a Docker Container’s Filesystem
The command docker diff *CONTAINER* will list the files added, deleted and changed since the Container started.
If a file did not change since the container was started, then you would have to know the contents of the original image that started the container. So, this answer is not ideal but avoids creating an image and running it.
Unlike container-diff, this command does not require first creating a Docker image.
If you want to see a certain file content, I would suggest using docker container cp command. Here is the doc. It works on stopped container. Example:
docker container cp 02b1ef7de80a:/etc/nginx/conf.d/default.conf ./
This way I got the config file that was generated by templating engine during start.
Try using container-diff with the --type=file option. This will compare two images and report the files added, deleted and modified.
If a file did not change since the container was started, then you would have to know the contents of the original image that started the container. So, this answer is not ideal but avoids creating an image and running it.
This tool requires that you first create an image of the stopped Docker container with docker commit.
Here is the command to install it:
curl -LO https://storage.googleapis.com/container-diff/latest/container-diff-linux-amd64 \
&& chmod +x container-diff-linux-amd64 \
&& mkdir -p $HOME/bin \
&& export PATH=$PATH:$HOME/bin \
&& mv container-diff-linux-amd64 $HOME/bin/container-diff
Here is the command to use the utility:
container-diff analyze $IMAGE --type=file
docker container cp <STOPPED_CONTAINER_ID>:<PATH_TO_FILE> -
Notice the "-" at the end of the command.
It actually "copies" the specified file from the stopped container into "stdout". In other words, it just prints the file contents.
Thanks #azat-khadiev for your direction (I don't know why you got "-1 for that answer...)