I'm working with a number of different datasets and Docker. For example, I have a number of files in a folder called /databases like model_1.sql, model_2.sql, model_3.sql etc. I would like my Dockerfile to be configured so that when I copy a file over from my databases folder, it picks out only the most recent file (e.x. model_3.sql) while ignoring the others. Is this possible?
edit: since my question apparently wasn't clear enough, here is the pertinent line -
# Copy our newest version
COPY database/model_3.sql /
Rather than manually typing the 3, I would like to be able to do something along the lines of model_*.sql but not copy all of them, copy the the most recent. So if I eventually add a model_4.sql, I will not need to update my Dockerfile when I run it, it will automatically add model_4 instead of model_3.
Yes you can achieve the same using the --build-args option.
Example: Dockerfile
FROM image:tag
ARG LATEST_FILE #Here I'll parse the LATEST_FILE info as a build argument.
COPY ${LATEST_FILE} #And the value for LATEST_FILE is replaced here.
Docker build command:
docker build -f myDockerfile --build-arg LATEST_FILE=$(ls -t database/model_*.sql | head -1) -t imagename:tag .
$(ls -t database/model_*.sql | head -1) will get you the latest file and will be copied in the COPY step.
Please be aware that parsing ls is only dangerous when the files name contains spaces or some funny characters. If you guarantee that the filenames will not have such characters, then parsing ls is ok.
Related
[EDIT - added clarity]
Here is my current env setup :
$GOPATH = /home/fzd/go
projectDir = /home/fzd/go/src/github.com/fzd/amazingo
amazingo has a go.mod file that lists several (let's say thousands) dependencies.
So far, I used to go build -t bin/amazingo cmd/main.go, but I want to share this with other people and have a build command that is environment-independent. Using go build has the advantage of downloading each dependency once -- and then using those in ${GOPATH}/pkg/mod, which saves time and bandwidth.
I want to build in a multistage docker image, so I go with
> cat /home/fzd/go/src/github.com/fzd/amazingo/Dockerfile
FROM golang:1.17 as builder
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /bin/amazingo cmd/main.go
FROM alpine:latest
COPY --from=builder /bin/amazingo /amazingo
ENTRYPOINT ["/amazingo"]
As you can expect it, the builder is "naked" when I start it, so it has to download all my dependencies when I docker build -t amazingo:0.0.1 . . But it will do so everytime I call it, which can be several times a day.
Fortunately, I already have most of these dependencies on my disk. I would be happy to share these files (that are located in my $GOPATH/pkg/mod) with the builder, and help it build faster on my machine.
So the question is: how can I share my ${GOPATH} (or ${GOPATH}/mod/pkg) with the builder ?
I tried adding the following to the builder
ARG SRC_GOPATH
COPY ${SRC_GOPATH} /go
and call docker build --build-arg SRC_GOPATH=${GOPATH} -o amazingo:0.0.1 ., but it wasn't good enough - I got an error (COPY failed: file not found in build context or excluded by .dockerignore: stat home/fzd/go: file does not exist)
I hope this update brings a bit more clarity to the problem.
=======
I have a project with a go.mod file.
I want to build that project using a multistage docker image.
(this article is a perfect example)
The issue is that I have "lots" of dependencies, and each of them will be downloaded inside my Docker builder stage.
Is there a way to "share" my GOPATH/pkg/mod with the docker build... command (in some ways, having a local cache) ?
Your end goal isn't completely clear, but the way that I use a multistage build would look something like this for a (dirt-simple) go app, assuming that you ultimately want the docker container to run your go app. You will need to get your source into the build container somehow as well - that is not shown here:
FROM golang:1.17.2-alpine3.14 as builder
WORKDIR /my/app/source/dir
RUN go get && go build -o /path/to/my/app/binary
FROM alpine3.14 AS release
# install runtime deps, if any
# create necessary files and folders, if any
COPY --from=builder /path/to/my/app/binary /usr/local/bin
ENTRYPOINT /usr/local/bin/binary --options
In this way, the source of your application and all dependencies will not be present in the released image, only the compiled binary.
Of course you don't have to specify an output path for that, I think it just makes it a little clearer in this example. And of course you can use whatever base image/images you want to - I'm treating this as though you don't need the go runtime on your release image.
I am setting up a docker image, in the dockerfile I have an ADD command where source of the ADD command is a variable.
Dockerfile takes a build argument, I want to use that arg as source of the ADD command.
But ADD command is not expanding the variable and I get an error
Please share any workaround that comes in your mind
FROM ubuntu
ARG source_dir
RUN echo ${source_dir}
ADD ${source_dir} ./ContainerDir
Build command
docker build . -t image --build-arg source_dir=/home/john/Desktop/
data
Error
Step 3/3 : ADD ${source_dir} ./ContainerDir ADD failed: stat /var/lib/docker/tmp/docker-builder311119108/home/john/Desktop/
data: no such file or directory
However, the directory (/home/john/Desktop/
data) exists
From the error message, the variable expanded and complained that you don't have the path in your build context:
stat /var/lib/docker/tmp/docker-builder311119108/a/b/c: no such file or directory
In your example, the build context is . (the current directory) so you need a/b/c in the current directory for this to not error. That also need to not be in any ./.dockerignore file if you have one.
From your second edit:
docker build . -t image --build-arg source_dir=/home/john/Desktop/data
It looks like you are trying to include a directory inside your build from outside of the build context. That is explicitly not allowed in docker builds. All files needed for the ADD and COPY commands need to be included in your context, and the entire content of the context is sent to the build server in the first step, so you want to keep this small (rather than sending the entire home directory). The source is always relative to this context, so /home is looking for ./home since your context is . in the build command.
The fix is to move the data directory to be a sub directory of . where you are building your docker images. You can also switch to COPY since there is no functionality of ADD that you need.
Disclaimer: there are two pieces of over simplification here:
The COPY command can include files from different contexts using the --from option to COPY.
The entire context is sent before the build starts with the classic build command. The newer BuildKit implementation is much more selective about how much and what parts of the context to send.
Following all .NET Core guides basically boils down to a dotnet publish and copying the output of that to /app, then running dotnet myapp.dll.
I have about 40+ (and growing) products running in this setup, and so modifying all dockerfiles with myapp.dll gets quite laborious.
I was wondering if there is some way to find out what the entry dll is during publish? (e.g. with --self-contained the cli generates an arch specific entry file, so you can use that name, but it seems like an unnecessary step given that publish takes longer)
You can create a bash script which will extract project name, and next create a valid path with replacing it in script file.
If you are in solution folder just run: (bash)
PROJECT_NAME=`find ./ -name "*.sln" | head -n 1 | cut -d '/' -f 2 | sed 's/.sln//'`
If you have solution file myapp.sln , this command will return value myapp
Then you pass this value to script:
./runScript.sh "$PROJECT_NAME"
And inside this script:
dotnet "/app/$1.dll"
For dockerfiles you have replace all occurences of eg. {{PROJECT_NAME}} in file to value of variable. Now i don't remember command, but sed is useful for that.
I have a big tar/executable (over 30GB) I COPY/ADD it but this is used only for the installation. Once the application is installed I don't need it anymore.
How can I do? I am trying to use it but:
Everytime I run a build, it takes minutes to define the build context.
I'd like to share this image, if I create a tar with docker save, Is the final version or each layer included in it?
I found some solutions that said I can use RUN wget tar ... && rm tar but I don't want to create webserver for that.
Why isn't possible to mount a volume during build process?! It would be very useful.
Use Docker's multi-stage builds. This mechanism allows you to drop intermediate artifacts and therefore achieve a lightweight image.
Example:
FROM alpine:latest as build
# copy large file
# build
FROM alpine:latest as output
# copy necessary files built in the previous stage
COPY --from=build app /app
Anything built in the build stage will not be included in the final image, unless you explicitly COPY them.
Docs: https://docs.docker.com/develop/develop-images/multistage-build/
This is solvable using 2 different context.
Please follow these steps as mentioned below.
Objective is to create a
docker image that will have you large-build file.
docker image that will have you real codebase/executables.
For this you have to create 2 folders (Build & CodeBase) as follow.
Application<br/>
|---> BUILD <br/>
|======|--->Large-File<br/>
|======|--->Dockerfile<br/>
|--->CodeBase<br/>
|======|--->SRC+Other stuff<br/>
|======|--->Dockerfile<br/>
Build & Codebase both folders will have individual Dockerfile and arrange files accordingly.
Dockerfile(Build)
FROM **Base-Image**
COPY Large-File /tmp/Large-File
Build this and tag it with some name like (base-build-app-image)
#>cd Application <==Application root folder as mentioned above==>
#>docker build -t base-build-app-image BUILD <==path of your build-folder==>
Dockerfile(Codebase)
FROM base-build-app-image
RUN *****
CMD *****
RUN rm -f **/tmp/Large-File**
RUN rm -f **Remove installation files that is not required**
ENTRYPOINT *****
Build this-code-base and base-build-app-image is already in your local docker-repository and your large iso file is not in the current-buid-context
#>cd Application <==Application root folder as mentioned above==>
#>docker build CodeBase <==path of your code-base==>
This time since the context size is only your code base and since this doesn't include that Large file - it will definitely reduce your build time.
You can also take an advance of using docker-compose to do both operations together so you will not have to execute 2 separate commands.
If you need help on preparing this docker-compose file then do let me know in comments.
If anything is not clear then leave a comment or come over chat to fix this issue.
I am wondering if there is an identical command for copying a folder to current directory like it did using the old MS-DOS. Let's say my current directory location is:
/var/www/
I have folders and files at:
/home/hope/subfolder/docs/
/home/hope/subfolder/images/
/home/hope/subfolder/.config
/home/hope/subfolder/readme.txt
I know that the following command:
cp -rT /home/hope/subfolder .
will copy all the files (even dot hidden files) and folders within the "subfolder" folder to the current directory, so the result will be:
/var/www/docs/
/var/www/images/
/var/www/.config
/var/www/readme.txt
Looks like the command to that to copy the source folder to the current location is:
cp -rT /home/hope/subfolder ./subfolder
although this is fine, I find it that sometimes I will make mistakes for complicated folder names for the destination, so is there a way to use a command like:
cp -rT /home/hope/subfolder .
or even like this
cp -rT /home/hope/subfolder /var/www/.
to have the following result:
/var/www/subfolder/docs/
/var/www/subfolder/images/
/var/www/subfolder/.config
/var/www/subfolder/readme.txt
Thank you.
Just omit the -T parameter, as that's what prevents the command from working properly:
cp -r /home/hope/subfolder .
The -T parameter treats the target argument as a file, so no copying will be performed at all if that is actually a directory.
A friendly reminder: virtually all Unix commands have a --help command line argument that is worth trying out in case of a trouble :)
For me the main barrier was the /home part. I needed to copy files from a folder in my home that started with the letter 'a' to my current folder, which was not home. So I used:
cp home/tmp/a* ./
the first line worked for me. While I was trying commands like:
cp ~/home/tmp/a* ./
but this didn't work.