Suppose I have a very nested folder structure with lots of project files:
src
projectA
projectA.csproj
someFile.txt
projectB
projectB.csproj
someFile.txt
projectC
projectC.csproj
someFile.txt
In this case I want my DockerFile to copy over the full folder structure, but only include .csproj files:
src
projectA
projectA.csproj
projectB
projectB.csproj
projectC
projectC.csproj
I can do this for each file line by line, but is there a cleaner way?
COPY src/projectA/projectA.csproj src/projectA/projectA.csproj
COPY src/projectB/projectB.csproj src/projectB/projectB.csproj
COPY src/projectC/projectC.csproj src/projectC/projectC.csproj
I've faced a similar situation and the only solution I've found was to prepare a .tgz file containing what I needed and copy it in the docker image using the ADD directive.
e.g.
this is a run.sh script similar to what I used:
#!/bin/bash
tar cvfz csproj.tgz $( find src -name "*.csproj" )
docker build -t test .
docker run -it --rm test
this is a test Dockerfile:
FROM alpine
RUN mkdir /src
ADD csproj.tgz /src
CMD ls -alR /src
This solution is not very pleasant but it did do what I needed at the time.
The ADD directive (src: https://docs.docker.com/engine/reference/builder/#add) is able to copy files (like the COPY directive) and
If is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed.
Related
I have a dockerfile that copies entire directory to the image and a single file is not copied. The file is located in a subfolder with 2 other files which do get copied.
there is no .dockerignore.
It is a sqlite db file.
this is the dockerfile
FROM mcr.microsoft.com/dotnet/aspnet:6.0
COPY . App/
WORKDIR /App
RUN sed -i 's/TLSv1.2/TLSv1.0/g' /etc/ssl/openssl.cnf
RUN ["chmod", "+x", "/App/entrypoint/entry_script.sh"]
ENTRYPOINT ["/App/entrypoint/entry_script.sh"]
Check if there is this file .dockerignore
This file tells the docker to exclude specific files or folders to be copied to docker image.
I have 3 .NET6 projects in my local 'src' folder as below:
src/A
src/B
src/C
A: web api project, has many nuget dependencies
B: class lib, dependent of A
C: class lib, dependent of A
(Note: of course, project related files including .csproj are present in the local project folders)
My Dockerfile is in project folder A.
I am running the "docker build" command from 'src' folder (i.e. this is the build context) as below:
docker build -f A/Dockerfile -t myapiimage:v1 .
below section of code is from my Dockerfile, where i try to restore nuget dependencies:
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ./A/*.csproj A/
RUN dotnet restore A/A.csproj
this works fine and i understand that .csproj in the image is stored inside /src/A/A.csproj
Am i right?
I want to know whether COPY preserves the source folder structure in the image being built or not?
because after the below 3 lines:
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY *.csproj ./
None of the line works:
RUN dotnet restore A/A.csproj
OR,
RUN dotnet restore A.csproj
OR,
RUN dotnet restore *A.csproj
OR,
RUN dotnet restore src/A/A.csproj
So, where this .csproj files (of all 3 projects) are getting stored in the image?
Your line COPY *.csproj ./ doesn't copy anything, because there are no .csproj files in your src directory on the host machine.
The way I usually do it is a bit clunky. First I do
COPY */*.csproj ./
which copies all the .csproj files into the workdir in the image. Then I do
RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done
which looks at all the .csproj files and creates a directory for them and moves them into that directory.
With that approach, your Dockerfile would look like this
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY */*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done
RUN dotnet restore A/A.csproj
Assume I have the following dir structure:
languages
-en-GB
--page1.json
--page2.json
-fr-FR
--page1.json
--page2.json
Now let's assume I want to copy the folder structure, but only page1.json content:
I've tried this:
COPY ["languages/**/*page1.json", "./"]
Which results in the folders being copied, but no files.
What I want to end up with is
languages
-en-GB
--page1.json
-fr-FR
--page1.json
Copied into my image
I am not sure you can use wildcards to produce the filtered result you are looking for.
I believe there are at least two clean and clear ways to achieve this:
Option 1: Copy everything, and cleanup later:
FROM alpine
WORKDIR /languages
COPY languages .
RUN rm -r **/page2.json
Option 2: Add files you don't want into your .dockerignore
# .dockerignore
languages/**/page*.json
!languages/**/page1.json
Option 3: Copy all to a temporary directory, and copy what you need from inside the container using more flexible tools
FROM alpine
WORKDIR /languages
COPY languages /tmp/langs
RUN cd /tmp/langs ; find -name 'page1.json' -exec cp --parents {} /languages \;
CMD ls -lR /languages
I want to create a Docker container and I wrote image. Everything works great except the COPY command where I got confused. My Dockerfile:
RUN HOME=/home/ros rosdep update
RUN mkdir -p /home/ros/workspace
# Copy the files
COPY $PWD/src/a_file /home/ros/workspace/src
COPY $PWD/src/b_file /home/ros/workspace/src
a_file is a directory like a b_file. When I try to copy these directories into a newly created directory called /home/ros/workspace/src I want a_file and b_file to be both inside /home/ros/workspace/src. Instead of this, I get another src directory /home/ros/workspace/src/src) and the contents of a_file and b_file are inside that directory.
What am I doing wrong?
As mentioned in other answers, $PWDrefers to the image context.
Try to use . instead.
To setup your working directory, use WORKDIR
Also, both a_file and b_file are in src/
All in all, this should work (not tested):
FROM <your-base-image>
WORKDIR /home/ros
RUN rosdep update
RUN mkdir -p workspace
# Copy the files
COPY ./src workspace/src
In your Dockerfile, PWD variable refers to image context (i.e: inside the image).
From COPY documentation:
paths of files and directories will be interpreted as relative to the source of the context of the build.
If src directories are in the root of your build context, your example it will be:
...
COPY src/a_file /home/ros/workspace/src
COPY src/b_file /home/ros/workspace/src
...
Attempting to create a container with microsoft/dotnet:2.1-aspnetcore-runtime. The .net core solution file has multiple projects nested underneath the solution, each with it's own .csproj file. I am attemping to create a more elegant COPY instruction for the sub-projects
The sample available here https://github.com/dotnet/dotnet-docker/tree/master/samples/aspnetapp has a solution file with only one .csproj so creates the Dockerfile thusly:
COPY *.sln .
COPY aspnetapp/*.csproj ./aspnetapp/
RUN dotnet restore
It works this way
COPY my_solution_folder/*.sln .
COPY my_solution_folder/project/*.csproj my_solution_folder/
COPY my_solution_folder/subproject_one/*.csproj subproject_one/
COPY my_solution_folder/subproject_two/*.csproj subproject_two/
COPY my_solution_folder/subproject_three/*.csproj subproject_three/
for a solution folder structure of:
my_solution_folder\my_solution.sln
my_solution_folder\project\my_solution.csproj
my_solution_folder\subproject_one\subproject_one.csproj
my_solution_folder\subproject_two\subproject_two.csproj
my_solution_folder\subproject_three\subproject_three.csproj
but this doesn't (was a random guess)
COPY my_solution_folder/*/*.csproj working_dir_folder/*/
Is there a more elegant solution?
2021: with BuildKit, see ".NET package restore in Docker cached separately from build" from Palec.
2018: Considering that wildcard are not well-supported by COPY (moby issue 15858), you can:
either experiment with adding .dockerignore files in the folder you don't want to copy (while excluding folders you do want): it is cumbersome
or, as shown here, make a tar of all the folders you want
Here is an example, to be adapted in your case:
find .. -name '*.csproj' -o -name 'Finomial.InternalServicesCore.sln' -o -name 'nuget.config' \
| sort | tar cf dotnet-restore.tar -T - 2> /dev/null
With a Dockerfile including:
ADD docker/dotnet-restore.tar ./
The idea is: the archive gets automatically expanded with ADD.
The OP sturmstrike mentions in the comments "Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2)" from Andrew Lock "Sock"
The alternative solution actually uses the wildcard technique I previously dismissed, but with some assumptions about your project structure, a two-stage approach, and a bit of clever bash-work to work around the wildcard limitations.
We take the flat list of csproj files, and move them back to their correct location, nested inside sub-folders of src.
# Copy the main source project files
COPY src/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done
L01nl suggests in the comments an alternative approach that doesn't require compression: "Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files", from Andrew Lock "Sock".
FROM microsoft/aspnetcore-build:2.0.6-2.1.101 AS builder
WORKDIR /sln
COPY ./*.sln ./NuGet.config ./
# Copy the main source project files
COPY src/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done
# Copy the test project files
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done
RUN dotnet restore
# Remainder of build process
This solution is much cleaner than my previous tar-based effort, as it doesn't require any external scripting, just standard docker COPY and RUN commands.
It gets around the wildcard issue by copying across csproj files in the src directory first, moving them to their correct location, and then copying across the test project files.
One other option to consider is using a multi-stage build to prefilter / prep the desired files. This is mentioned on the same moby issue 15858.
For those building on .NET Framework, you can take it a step further and leverage robocopy.
For example:
FROM mcr.microsoft.com/dotnet/framework/sdk:4.8 AS prep
# Gather only artifacts necessary for NuGet restore, retaining directory structure
COPY / /temp/
RUN Invoke-Expression 'robocopy C:/temp C:/nuget /s /ndl /njh /njs *.sln nuget.config *.csproj packages.config'
[...]
# New build stage, independent cache
FROM mcr.microsoft.com/dotnet/framework/sdk:4.8 AS build
# Copy prepped NuGet artifacts, and restore as distinct layer
COPY --from=prep ./nuget ./
RUN nuget restore
# Copy everything else, build, etc
COPY src/ ./src/
RUN msbuild
[...]
The big advantage here is that there are no assumptions made about the structure of your solution. The robocopy '/s' flag will preserve any directory structure for you.
Note the '/ndl /njh /njs' flags are there just to cut down on log noise.
In addition to VonC's answer (which is correct), I am building from a Windows 10 OS and targetting Linux containers. The equivalent to the above answer using Windows and 7z (which I normally have installed anyway) is:
7z a -r -ttar my_project_files.tar .\*.csproj .\*.sln .\*nuget.config
followed by the ADD in the Dockerfile to decompress.
Be aware that after installing 7-zip, you will need to add the installation folder to your environment path to call it in the above fashion.
Looking at the moby issue 15858, you will see the execution of the BASH script to generate the tar file and then the subsequent execution of the Dockerfile using ADD to extract.
Fully automate either with a batch or use the Powershell execution as given in the below example.
Pass PowerShell variables to Docker commands
Qnother solution, maybe a bit slower but all in one
Everything in one file and one command docker build .
I've split my Dockerfile in 2 steps,
First image to tar the *.csproj files
Second image use the tar and setup project
code:
FROM ubuntu:18.04 as tar_files
WORKDIR /tar
COPY . .
RUN find . -name "*.csproj" -print0 | tar -cvf projectfiles.tar --null -T -
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /source
# copy sln
COPY *.sln .
# Copy all the csproj files from previous image
COPY --from=tar_files /tar/projectfiles.tar .
RUN tar -xvf projectfiles.tar
RUN rm projectfiles.tar
RUN dotnet restore
# Remainder of build process
I use this script
COPY SolutionName.sln SolutionName.sln
COPY src/*/*.csproj ./
COPY tests/*/*.csproj ./
RUN cat SolutionName.sln \
| grep "\.csproj" \
| awk '{print $4}' \
| sed -e 's/[",]//g' \
| sed 's#\\#/#g' \
| xargs -I {} sh -c 'mkdir -p $(dirname {}) && mv $(basename {}) $(dirname {})/'
RUN dotnet restore "/src/Service/Service.csproj"
COPY ./src ./src
COPY ./tests ./tests
RUN dotnet build "/src/Service/Service.csproj" -c Release -o /app/build
Copy solution file
Copy project files
(optional) Copy test project files
Make linux magic (scan sln-file for projects and restore directory structure)
Restore packages for service project
Copy sources
(optional) Copy test sources
Build service project
This is working for all Linux containers