I have a following Dockerfile
FROM solr:9
ARG configsets
ADD ${configsets} /opt/solr/server/solr/configsets
RUN mkdir -v -p /var/solr/data && chown solr:solr /var/solr/data
ENTRYPOINT ...
If I build this locally, it works fine. If I build this from ssh inside a CI server, I see that the built image doesn't have /var/solr/data folder created. This mkdir command wasn't present previously (I added it to control folder permissions when a volume is mounted on that path).
The only hypothesis I have, is that docker cached layers from old Dockerfile build, and for whatever reason it fails to understand that new Dockerfile differs from the old one at the point of RUN mkdir part. But why? How can I avoid this? I want to understand robust futureproof solution, and not just adhoc fixing this specific case by deleting cached layers. I don't want to disable caching altogether. I'm largely fine that the dependency image (ie., solr:9) could have a potentially stale cache, it's not an issue to me. Only stale commands in my own Dockerfile are fatally problematic.
Related
I am wondering how it is possible to reproduce the docker commands seen in this docker image. The image copies certain versions of clang and gcc, which is something I wish to do in my own dockerfile. I cannot use the linked docker image, as it contains many commands that are unnecessary for the work I want to do.
The very first command is
ADD file:2cddee716e84c40540a69c48051bd2dcf6cd3bd02a3e399334e97f20a77126ff in /
Further down, there are many similar COPY commands. I wish to reproduce the following command in my own dockerfile:
COPY dir:49371ba683da700cabfad7284da39bd2144aa0c46086c3015a74737d7be6b51e in /compilers/clang/3.4.2
The command copies clang-3.4.2 into the given folder. I am unsure how I can do the same, or even what the hash is/means.
I tried looking, but I couldn't find the Dockerfile used to create the image. There is another way though.
It's quite a large image and I'm on a terrible internet connection, so I haven't tested this myself, but one thing you can do is copy the things you need from the image into a new one of your own like this
FROM cnsun/perses:perses_part_54_name_clang_trunk AS original
FROM ubuntu:latest
COPY --from=original /compilers/clang/3.4.2 /compilers/clang/3.4.2
You can also copy the files from the image to your computer. Then you can copy them from there into new images without referencing the cnsun image:
docker run --rm -v $(pwd):/dest --entrypoint /bin/bash cnsun/perses:perses_part_54_name_clang_trunk -c "cp -r /compilers/clang/3.4.2 /dest"
This will copy the /compilers/clang/3.4.2 directory into the current directory on the host. If your host is Windows, replace $(pwd) with %cd%.
I have a docker container which I use to build software and generate shared libraries in. I would like to use those libraries in another docker container for actually running applications. To do this, I am using the build docker with a mounted volume to have those libraries on the host machine.
My docker file for the RUNTIME container looks like this:
FROM openjdk:8
RUN apt update
ENV LD_LIBRARY_PATH /build/dist/lib
RUN ldconfig
WORKDIR /build
and when I run with the following:
docker run -u $(id -u ${USER}):$(id -g ${USER}) -it -v $(realpath .):/build runtime_docker bash
I do not see any of the libraries from /build/dist/lib in the ldconfig -p cache.
What am I doing wrong?
You need to COPY the libraries into the image before you RUN ldconfig; volumes won't help you here.
Remember that first you run a docker build command. That runs all of the commands in the Dockerfile, without any volumes mounted. Then you take that image and docker run a container from it. Volume mounts only happen when the docker run happens, but the RUN ldconfig has already happened.
In your Dockerfile, you should COPY the files into the image. There's no particular reason to not use the "normal" system directories, since the image has an isolated filesystem.
FROM openjdk:8
# Copy shared-library dependencies in
COPY dist/lib/libsomething.so.1 /usr/lib
RUN ldconfig
# Copy the actual binary to run in and set it as the default container command
COPY dist/bin/something /usr/bin
CMD ["something"]
If your shared libraries are only available at container run-time, the conventional solution (as far as I can tell) would be to include the ldconfig command in a startup script, and use the dockerfile ENTRYPOINT directive to make your runtime container execute this script every time the container runs.
This should achieve your desired behaviour, and (I think) should avoid needing to generate a new container image every time you rebuild your code. This is slightly different from the common Docker use case of generating a new image for every build by running docker build at build-time, but I think it's a perfectly valid use case, and quite compatible with the way Docker works. Docker has historically been used as a CI/CD tool to streamline post-build workflows, but it is increasingly being used for other things, such as the build step itself. This naturally means people are coming up with slightly different ways of using Docker to facilitate various new and different types of workflow.
I'm kind of new to Docker, and I'm running into an issue I can't find a straightforward explanation for. I have a pretty simple Dockerfile that I'm building into an image, but when I deploy the image to Kubernetes as a pod, the results aren't what I expect.
FROM ubuntu:16.04
RUN mkdir workspace
WORKDIR /workspace
COPY . /workspace
CMD ["ls"]
When I check the logs for the deployment, there are no files listed in the /workspace folder, even though the folder itself exists. However, if I change my COPY's destination to a default linux folder like /usr, the files are there as I'd expect. My suspicion is that this has something to do with storage persistence, but since I'm copying the files into the folder when I build my image and the folder persists in the pod, I'm at a loss for why this happens. Any guidance would be greatly appreciated.
I would venture to guess that the ubuntu:... image doesn't have a WORKDIR set to /, and hence your copy command isn't working as expected.
Try changing the run command to be RUN mkdir /workspace and I think you'll see what you expected.
Hope this helps.
New to docker here. I followed the instructions here to make a slim & trim container for my Go Project. I do not fully understand what it's doing though, hopefully someone can enlighten me.
Specifically there are two steps to generating this docker container.
docker run --rm -it -v "$GOPATH":/gopath -v "$(pwd)":/app -e "GOPATH=/gopath" -w /app golang:1.8 sh -c 'CGO_ENABLED=0 go build -a --installsuffix cgo --ldflags="-s" -o hello'
docker build -t myDockerImageName .
The DockerFile itself just contains
FROM iron/base
WORKDIR /app
COPY hello /app/
ENTRYPOINT ["./hello"]
I understand (in a broad sense) that the 1st step is compiling the go program and statically linking the C-dependencies (and doing all this inside an unnamed docker container). The 2nd step just generates the docker image according to the instructions in the DockerFile.
What I don't understand is why the first command starts with docker run (why does it need to be run inside a docker container? Why are we not just generating the Go binary outside of it, and then copying it in?)
And if it's being run inside a docker container, how is binary generated in the docker container being dropped on my local machine's file system?(e.g. why do I need to copy the binary back into the image - as it seems to be doing on line 3 of the DockerFile?)
You're actually using 2 different docker containers, each with a different image. The first container is only around during the compilation... it uses the image golang:1.8. What you're doing is mounting your current working directory into that image and compiling it with the version of GO contained in the image.
The second command builds your custom image that uses the iron/base image as its base. You then copy your built application into that image and run it.
Using a golang container to build the binary is usually done for reproducibility of the build process, i.e.:
it ensures that always the same Go version is used,
compilation takes place in an alway clean environment,
and the build host needs no Go installed at all, or can have a different version,
This way, all parts needed to build the "hello" image can be tracked in a version control system.
However, this example mounts the whole local GOPATH, defeating above purpose. Dependencies must be available to the build container, e.g. by vendoring them. Maybe the author considered vendoring out of scope for his example.
(note: this should be a comment, but my reputation does not allow that)
Let's say I want to run inside a docker container some third party .net core application I don't fully trust.
To simplify, let's assume that application is the simple Hello World console app generated by dotnet new. This is just the 2 files Program.cs and project.json.
Right now I have tried the following approach:
Copy that application into some folder of my host
Create a new container using the microsoft/dotnet image, mounting that folder as a volume, running a specific command for building and running the app:
$ docker run --rm -it --name dotnet \
-v /some/temp/folder/app:/app \
microsoft/dotnet:latest \
/bin/sh -c 'cd /app && dotnet restore && dotnet run'
I was also considering the idea of having a predefined dockerfile with microsoft/dotnet as the base image. It will basically embed the application code, set it as the working dir and run the restore, build and run commands.
FROM microsoft/dotnet:latest
COPY . /app
WORKDIR /app
RUN ["dotnet", "restore"]
RUN ["dotnet", "build"]
ENTRYPOINT ["dotnet", "run"]
I could then copy the predefined dockerfile into the temp folder, build a new image just for that particular application and finally run a new container using that image.
Is the dockerfile approach overkill for simple command line apps? What would be the best practice for running those untrusted applications? (which might be one I completely ignore)
EDIT
Since I will discard the container after it runs and the docker command will be generated by some application, I will probably stay with the first option of just mounting a volume.
I have also found this blog post where they built a similar sanbox environment and ended up following the same mounted volume approach
As far I know, what happens in docker, stays in docker.
When you link a volume (-v) to the image, the process can alter the files in the folder you mounted. But only there. The process cannot follow any symlinks or step out of the mounted folder since it's forbidden for obvious security reasons.
When you don't link anything and copy the application code into the image, it's definitely isolated.
The tcp/udp ports exposition is up to you as well as memory/cpu consumption and you can even isolate the process from internet e.g. like that
Therefore, I don't think that using dockerfile is an overkill and I'd summarize it like this:
When you want to run it once, try it and forget it - use command line if you are ok with typing the nasty command. If you plan to use it more - create a Dockerfile. I don't see much space for declaring "best practice" here, considering it an question of personal preferences.