I am building a docker image from a dockerfile. However I am doing some installs from files that are currently hosted on a NFS share. In regular Centos I mount the drive with mount.nfs, then run the commands to do the install and point to the NFS share as repository for the install files.
Is there any way to do this with dockers? I read a few posts of docker run -v, but I am not ready to run the docker yet, I first need to create the image.
The alternative is copy the whole repository via zip or tar, then unarchive, do the install and then delete files. However I think this will end up in a huge image.
You'll need experimental software (as of writing) for doing this.
First of all, you have to create a buildx builder instance:
docker buildx create --name insecure --driver docker-container \
--driver-opt image=moby/buildkit:master \
--buildkitd-flags '--allow-insecure-entitlement security.insecure \
--allow-insecure-entitlement network.host'
As of today, the latest release (v0.9.0) of buildkit doesn't have the --insecure support, so you need master.
You should issue this command as the user which does the build.
Then you'll need to add these to your Dockerfile:
# syntax = docker/dockerfile:experimental
RUN --security=insecure mkdir /nfs && \
mount -t nfs -o nolock -o vers=4 $SERVER_IP:/nfs /nfs && \
ls -la /nfs
Third, you have to do your build with buildx and give the following options (--allow and --builder along with your normal options):
docker buildx build --allow security.insecure,network.host \
--builder insecure \
-t image:tag --file=Dockerfile .
You should then have your NFS server mounted at /nfs.
Be aware that this mount will be present only in the same RUN context, because all those steps run in a different container. The next RUN line will see only an empty /nfs directory.
So you should do everything which needs data from /nfs from that RUN step!
When you are building docker image you have full access to host's file system what means that you should easily write in Dockerfile
ADD /nfs-path/file /path-inside-docker-image/file
You don't need any additional action in docker to do that.
Related
I'm new to Docker, so this may be an obvious question that I'm just not using the right search terms to find an answer to, so my apologies if that is the case.
I'm trying to stand up a new CI/CD Pipeline using a purpose built container. So far, I've been using someone else's container, but I need more control over the available dependencies, so I need my own container. To that end, I've built a container (Ubuntu), and I have a local (host) directory for the dependencies, and another for the project I'm building. Both are connected to the container using Docker Volumes (-v option), like this.
docker run --name buildbox \
-v /projectpath:/home/project/ \
-v /dependencies:/home/libs \
buildImage buildScript.sh
Since this is going to eventually live in a Docker repo and be accessed by a GitLab CI/CD Pipeline, I want to store the dependencies directory in as small of a container as possible that I can push up to the Docker repo alongside my Ubuntu build container. That way I can have the Pipeline pull both containers, map the dependencies container to the build container (--volumes-from), and map the project to be built using the -v option; e.g.:
docker run --name buildbox \
-v /projectpath:/home/project/ \
--volumes-from depend_vol \
buildImage buildScript.sh
Thus, I pull buildImage and depend_vol from the Docker repo, run buildImage while attaching the dependencies container and project directory as volumes, then run the build script (and extract the build artifact when it's done). The reason I want them separate is in case I want to create different build containers that use common libraries, or if I want to create version specific dependency containers without having a full OS stored (I have plans for this).
Now, I could just start a lightweight generic container (like busybox) and copy everything into it, but I was wondering if there was simply a way to attach the volume and then store the contents in the image when the container shuts down. Everything I've seen about making a portable data store / volume starts with all the data already copied into the container.
But I want to take my local host dependencies directory and store it in a container. Is there a straightforward way to do this? Am I missing something obvious?
So this works, if not what I was hoping for, since I'm still doing a lot of file copy (just with tarballs).
# Create a tarball of the files on the host to store, don't store the full path
tar -cvf /home/projectFiles.tar -C /home/projectFiles/ .
# Start a lightweight docker container (busybox) with a volume connection to the host (/home:/backup), then extract the tarball into the container
# cd to the drive root and untar the tarball
docker run --name libraryVolume \
-v /home:/backup \
busybox \
/bin/sh -c \
"cd / && mkdir /projectLibs && tar -xvf /backup/projectFiles.tar -C /projectLibs"
# Don't forget to commit the container image
docker commit libraryVolume
That's it. Then push to the repo.
To use it, pull the repo, then start the data volume:
docker run --name projLib \
-v /projectLibs \
--entrypoint "/bin/sh" \
libraryVolume
Then start the container (projBuild) that is going to reference the data volume (projLib).
docker run --it --name projBuild \
--volumes-from=projLib \
-v /home/mySourceCode:/buildProject \
--entrypoint /buildProject/buildScript.sh \
builderImage
Seems to work.
I want to update some container. For testing, I want to create a copy of the corresponding volume. Set up a new container for this new volume.
Is this as easy as doing cp -r volumeOld volumeNew?
Or do I have to pay attention to something?
To clone docker volumes, you can transfer your files from one volume to another one. For that you have to manually create a new volume and then spin up a container to copy the contents.
Someone has already made a script for that, which you might use: https://github.com/gdiepen/docker-convenience-scripts/blob/master/docker_clone_volume.sh
If not, use the following commands (taken from the script):
# Supplement "old_volume" and "new_volume" for your real volume names
docker volume create --name new_volume
docker container run --rm -it \
-v old_volume:/from \
-v new_volume:/to \
alpine ash -c "cd /from ; cp -av . /to"
On Linux it can be as easy as copying a directory. Docker keeps volumes in /var/lib/docker/volumes/<volume_name>, so you can simply copy contents of the source volume into a directory with another name:
# -p to preserve permissions
sudo cp -rp /var/lib/docker/volumes/source_volume /var/lib/docker/volumes/target_volume
Should you want to copy volumes managed by docker-compose, you'll also need to copy the specific labels when creating the new volume.
Else docker-compose will throw something like Volume already exists but was not created by Docker Compose.
Extending on the solution by MauriceNino, these lines worked for me:
# Supplement "proj1_vol1" and "proj2_vol2" for your real volume names
docker volume inspect proj1_vol1 # Look at labels of old volume
docker volume create \
--label com.docker.compose.project=proj2 \
--label com.docker.compose.version=2.2.1 \
--label com.docker.compose.volume=vol2 \
proj2_vol2
docker container run --rm -it \
-v proj1_vol1:/from \
-v proj2_vol2:/to \
alpine ash -c "cd /from ; cp -av . /to"
Btw, this also seems to be the only way to rename Docker volumes.
In my work I use this script to:
clone the container
clone all its volumes and copy contents from the old volumes to the new ones
run the new container (with an arbitrary new image)
reattach the new volumes to the new container at the same destinations as the old ones
However, the script makes some assumptions about the naming of the volumes, so please read the README instructions before applying it.
I did a docker pull and can list the image that's downloaded. I want to see the contents of this image. Did a search on the net but no straight answer.
If the image contains a shell, you can run an interactive shell container using that image and explore whatever content that image has. If sh is not available, the busybox ash shell might be.
For instance:
docker run -it image_name sh
Or following for images with an entrypoint
docker run -it --entrypoint sh image_name
Or if you want to see how the image was built, meaning the steps in its Dockerfile, you can:
docker image history --no-trunc image_name > image_history
The steps will be logged into the image_history file.
You should not start a container just to see the image contents. For instance, you might want to look for malicious content, not run it. Use "create" instead of "run";
docker create --name="tmp_$$" image:tag
docker export tmp_$$ | tar t
docker rm tmp_$$
The accepted answer here is problematic, because there is no guarantee that an image will have any sort of interactive shell. For example, the drone/drone image contains on a single command /drone, and it has an ENTRYPOINT as well, so this will fail:
$ docker run -it drone/drone sh
FATA[0000] DRONE_HOST is not properly configured
And this will fail:
$ docker run --rm -it --entrypoint sh drone/drone
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"sh\": executable file not found in $PATH".
This is not an uncommon configuration; many minimal images contain only the binaries necessary to support the target service. Fortunately, there are mechanisms for exploring an image filesystem that do not depend on the contents of the image. The easiest is probably the docker export command, which will export a container filesystem as a tar archive. So, start a container (it does not matter if it fails or not):
$ docker run -it drone/drone sh
FATA[0000] DRONE_HOST is not properly configured
Then use docker export to export the filesystem to tar:
$ docker export $(docker ps -lq) | tar tf -
The docker ps -lq there means "give me the id of the most recent docker container". You could replace that with an explicit container name or id.
docker save nginx > nginx.tar
tar -xvf nginx.tar
Following files are present:
manifest.json – Describes filesystem layers and name of json file that has the Container properties.
.json – Container properties
– Each “layerid” directory contains json file describing layer property and filesystem associated with that layer. Docker stores Container images as layers to optimize storage space by reusing layers across images.
https://sreeninet.wordpress.com/2016/06/11/looking-inside-container-images/
OR
you can use dive to view the image content interactively with TUI
https://github.com/wagoodman/dive
EXPLORING DOCKER IMAGE!
Figure out what kind of shell is in there bash or sh...
Inspect the image first: docker inspect name-of-container-or-image
Look for entrypoint or cmd in the JSON return.
Then do: docker run --rm -it --entrypoint=/bin/bash name-of-image
once inside do: ls -lsa or any other shell command like: cd ..
The -it stands for interactive... and TTY. The --rm stands for remove container after run.
If there are no common tools like ls or bash present and you have access to the Dockerfile simple add the common tool as a layer.
example (alpine Linux):
RUN apk add --no-cache bash
And when you don't have access to the Dockerfile then just copy/extract the files from a newly created container and look through them:
docker create <image> # returns container ID the container is never started.
docker cp <container ID>:<source_path> <destination_path>
docker rm <container ID>
cd <destination_path> && ls -lsah
To list the detailed content of an image you have to run docker run --rm image/name ls -alR where --rm means remove as soon as exits form a container.
If you want to list the files in an image without starting a container :
docker create --name listfiles <image name>
docker export listfiles | tar -t
docker rm listfiles
We can try a simpler one as follows:
docker image inspect image_id
This worked in Docker version:
DockerVersion": "18.05.0-ce"
if you want to check the image contents without running it you can do this:
$ sudo bash
...
$ cd /var/lib/docker # default path in most installations
$ find . -iname a_file_inside_the_image.ext
... (will find the base path here)
This works fine with the current default BTRFS storage driver.
Oneliner, no docker run (based on responses above)
IMAGE=your_image docker create --name filelist $IMAGE command && docker export filelist | tar tf - | tree --fromfile . && docker rm filelist
Same, but report tree structure to result.txt
IMAGE=your_image docker create --name filelist $IMAGE command && docker export filelist | tar tf - | tree --noreport --fromfile . | tee result.txt && docker rm filelist
I tried this tool - https://github.com/wagoodman/dive
I found it quite helpful to explore the content of the docker image.
Perhaps this is nota very straight forward approach but this one worked for me.
I had an ECR Repo (Amazon Container Service Repository) whose code i wanted to see.
First we need to save the repo you want to access as a tar file. In my case the command went like - docker save .dkr.ecr.us-east-1.amazonaws.com/<name_of_repo>:image-tag > saved-repo.tar
UNTAR the file using the command - tar -xvf saved-repo.tar. You could see many folders and files
Now try to find the file which contain the code you are looking for (if you know some part of the code)
Command for searching the file - grep -iRl "string you want to search" ./
This will make you reach the file. It can happen that even that file is tarred, so untar it using the command mentioned in step 2.
If you dont know the code you are searching for, you will need to go through all the files that you got after step 2 and this can be bit tiring.
All the Best !
There is a free open source tool called Anchore-CLI that you can use to scan container images. This command will allow you to list all files in a container image
anchore-cli image content myrepo/app:latest files
https://anchore.com/opensource/
EDIT: not available from anchore.com anymore, It's a python program you can install from https://github.com/anchore/anchore-cli
With Docker EE for Windows (17.06.2-ee-6 on Hyper-V Server 2016) all contents of Windows Containers can be examined at C:\ProgramData\docker\windowsfilter\ path of the host OS.
No special mounting needed.
Folder prefix can be found by container id from docker ps -a output.
Is it possible to use docker to expose the binary from one container to another container?
For example, I have 2 containers:
centos6
sles11
I need both of these containers to have similar versions git installed. Unfortunately the sles container does not have the version of git that I need.
I want to spin up a git container like so:
$ cat Dockerfile
FROM ubuntu:14.04
MAINTAINER spuder
RUN apt-get update
RUN apt-get install -yq git
CMD /usr/bin/git
# ENTRYPOINT ['/usr/bin/git']
Then link the centos6 and sles11 containers to the git container so that they both have access to a git binary, without going through the trouble of installing it.
I'm running into the following problems:
You can't link a container to another non running container
I'm not sure if this is how docker containers are supposed to be used.
Looking at the docker documentation, it appears that linked containers have shared environment variables and ports, but not necessarily access to each others entrypoints.
How could I link the git container so that the cent and sles containers can access this command? Is this possible?
You could create a dedicated git container and expose the data it downloads as a volume, then share that volume with the other two containers (centos6 and sles11). Volumes are available even when a container is not running.
If you want the other two containers to be able to run git from the dedicated git container, then you'll need to install (or copy) that git binary onto the shared volume.
Note that volumes are not part of an image, so they don't get preserved or exported when you docker save or docker export. They must be backed-up separately.
Example
Dockerfile:
FROM ubuntu
RUN apt-get update; apt-get install -y git
VOLUME /gitdata
WORKDIR /gitdata
CMD git clone https://github.com/metalivedev/isawesome.git
Then run:
$ docker build -t gitimage .
# Create the data container, which automatically clones and exits
$ docker run -v /gitdata --name gitcontainer gitimage
Cloning into 'isawesome'...
# This is just a generic container, but what I do in the shell
# you could do in your centos6 container, for example
$ docker run -it --rm --volumes-from gitcontainer ubuntu /bin/bash
root#e01e351e3ba8:/# cd gitdata/
root#e01e351e3ba8:/gitdata# ls
isawesome
root#e01e351e3ba8:/gitdata# cd isawesome/
root#e01e351e3ba8:/gitdata/isawesome# ls
Dockerfile README.md container.conf dotcloud.yml nginx.conf
My question is related to this question on copying files from containers to hosts; I have a Dockerfile that fetches dependencies, compiles a build artifact from source, and runs an executable. I also want to copy the build artifact (in my case it's a .zip produced by sbt dist in '../target/`, but I think this question also applies to jars, binaries, etc.
docker cp works on containers, not images; do I need to start a container just to get a file out of it? In a script, I tried running /bin/bash in interactive mode in the background, copying the file out, and then killing the container, but this seems kludgey. Is there a better way?
On the other hand, I would like to avoid unpacking a .tar file after running docker save $IMAGENAME just to get one file out (but that seems like the simplest, if slowest, option right now).
I would use docker volumes, e.g.:
docker run -v hostdir:out $IMAGENAME /bin/cp/../blah.zip /out
but I'm running boot2docker in OSX and I don't know how to directly write to my mac host filesystem (read-write volumes are mounting inside my boot2docker VM, which means I can't easily share a script to extract blah.zip from an image with others. Thoughts?
To copy a file from an image, create a temporary container, copy the file from it and then delete it:
id=$(docker create image-name)
docker cp $id:path - > local-tar-file
docker rm -v $id
Unfortunately there doesn't seem to be a way to copy files directly from Docker images. You need to create a container first and then copy the file from the container.
However, if your image contains a cat command (and it will do in many cases), you can do it with a single command:
docker run --rm --entrypoint cat yourimage /path/to/file > path/to/destination
If your image doesn't contain cat, simply create a container and use the docker cp command as suggested in Igor's answer.
docker cp $(docker create --name tc registry.example.com/ansible-base:latest):/home/ansible/.ssh/id_rsa ./hacked_ssh_key && docker rm tc
wanted to supply a one line solution based on pure docker functionality (no bash needed)
edit: container does not even has to be run in this solution
edit2: thanks to #Jonathan Dumaine for --rm so the container will be removed after, i just never tried, because it sounded illogical to copy something from somewhere which has been already removed by the previous command, but i tried it and it works
edit3: due the comments we found out --rm is not working as expected, it does not remove the container because it never runs, so I added functionality to delete the created container afterwards(--name tc=temporary-container)
edit 4: this error appeared, seems like a bug in docker, because t is in a-z and this did not happen a few months before.
Error response from daemon: Invalid container name (t), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed
A much faster option is to copy the file from running container to a mounted volume:
docker run -v $PWD:/opt/mount --rm --entrypoint cp image:version /data/libraries.tgz /opt/mount/libraries.tgz
real 0m0.446s
** VS **
docker run --rm --entrypoint cat image:version /data/libraries.tgz > libraries.tgz
real 0m9.014s
Parent comment already showed how to use cat. You could also use tar in a similar fashion:
docker run yourimage tar -c -C /my/directory subfolder | tar x
Another (short) answer to this problem:
docker run -v $PWD:/opt/mount --rm -ti image:version bash -c "cp /source/file /opt/mount/"
Update - as noted by #Elytscha Smith this only works if your image has bash built in
Not a direct answer to the question details, but in general, once you pulled an image, the image is stored on your system and so are all its files. Depending on the storage driver of the local Docker installation, these files can usually be found in /var/lib/docker/overlay2 (requires root access). overlay2 should be the most common storage driver nowadays, but the path may differ.
The layers associated with an image can be found using $ docker inspect image IMAGE_NAME:TAG, look for a GraphDriver attribute.
At least in my local environment, the following also works to quickly see all layers associated with an image:
docker inspect image IMAGE_NAME:TAG | jq ".[0].GraphDriver.Data"
In one of these diff directories, the wanted file can be found.
So in theory, there's no need to create a temporary container. Ofc this solution is pretty inconvenient.
First pull docker image using docker pull
docker pull <IMG>:<TAG>
Then, create a container using docker create command and store the container id is a variable
img_id=$(docker create <IMG>:<TAG>)
Now, run the docker cp command to copy folders and files from docker container to host
docker cp $img_id:/path/in/container /path/in/host
Once the files/folders are moved, delete the container using docker rm
docker rm -v $img_id
You essentially had the best solution already. Have the container copy out the files for you, and then remove itself when it's complete.
This will copy the files from /inside/container/ to your machine at /path/to/hostdir/.
docker run --rm -v /path/to/hostdir:/mnt/out "$IMAGENAME" /bin/cp -r /inside/container/ /mnt/out/
Update - here's a better version without the tar file:
$id = & docker create image-name
docker cp ${id}:path .
docker rm -v $id
Old answer
PowerShell variant of Igor Bukanov's answer:
$id = & docker create image-name
docker cp ${id}:path - > local-file.tar
docker rm -v $id
I am using boot2docker on MacOS. I can assure you that scripts based on "docker cp" are portable. Because any command is relayed inside boot2docker but then the binary stream is relayed back to the docker command line client running on your mac. So write operations from the docker client are executed inside the server and written back to the executing client instance!
I am sharing a backup script for docker volumes with any docker container I provide and my backup scripts are tested both on linux and MacOS with boot2docker. The backups can be easily exchanged between platforms. Basically I am executing the following command inside my script:
docker run --name=bckp_for_volume --rm --volumes-from jenkins_jenkins_1 -v /Users/github/jenkins/backups:/backup busybox tar cf /backup/JenkinsBackup-2015-07-09-14-26-15.tar /jenkins
Runs a new busybox container and mounts the volume of my jenkins container with the name jenkins_jenkins_1. The whole volume is written to the file backups/JenkinsBackup-2015-07-09-14-26-15.tar
I have already moved archives between the linux container and my mac container without any adjustments to the backup or restore script. If this is what you want you find the whole script an tutorial here: blacklabelops/jenkins
You could bind a local path on the host to a path on the container, and then cp the desired file(s) to that path at the end of your script.
$ docker run -d \
-it \
--name devtest \
--mount type=bind,source="$(pwd)"/target,target=/app \
nginx:latest
Then there is no need to copy afterwards.