How can I exclude python generated files like *.pyc in all subdirectories from the docker image?
I added .dockerignore at the root of the context directory:
# Ignore generated files
*.pyc
Alas docker build ignores this at least for subdirectories and copies the entire directory tree that looks like the following:
/contextdir/
|-- Dockerfile
\-- src/
|-- a.py # this is copied - all right
\-- a.pyc # this should be ignored, but is copied too. Why?
Patterns like *.pyc are matched only at the beginning of the path, or for the files directly below the context directory, but not recursively. To make it work recursively the **/ syntax must be used:
# Ignore generated files
**/*.pyc
The reference at How to create a dockerignore file doesn't put that clear enough.
Finally, I understood that trick. For some reason, I wasn't able to find any mention of this and haven't found any example Dockerfile with such construct, so documenting it here. Was it trivial for everybody else?
The docs for .dockerignore state that the .dockerignore file is interpreted as a list of patterns similar to the file globs of Unix shells. The pattern matching use's Go's filepath matching logic. The docs also cover the recursive pattern:
Beyond Go’s filepath.Match rules, Docker also supports a special wildcard string ** that matches any number of directories (including zero). For example, **/*.go will exclude all files that end with .go that are found in all directories, including the root of the build context.
Related
I have a question about the .dockerignore workflow which I wasn't really able to understand while browsing through the documentation and different internet topics.
Have the following folder structure:
home
|
|- folder_1
|- folder_2
Inside my dockerfile I want to copy the contents of home directory, so I use
COPY ./ /home
Inside .dockerignore I have:
*
!folder_1
!folder_3
I am referring to a non-existent folder - folder_3, which is supposed to be copied, right?
I ran it and it looks like there's no problem with that, thus .dockerignore somehow manages this situation.
If I tried to do the same thing without using .dockerignore, targeting a non-existent directory I would get an error.
If anybody can please clear this workflow, or if a duplicate, please attach some information so I can educate myself.
Thanks in advance!
First of all, .dockerignore works like .gitignore. Inside these files you set the rules on the basis of which files should be added, and which should not.
In your scenario you COPY the whole home directory which consists of folder_1 and folder_2. Your .dockerignore file sets the following rules:
* # ignore all files/directories
!folder_1 # do not ignore folder_1
!folder_3 # do not ignore folder_3
Regardless of whether there is a folder_1 or folder_3 in your local home directory or not, it won't show you any errors, because it just tries to find particular files/directories that are inside .dockerignore. If it finds this file/directory, it applies the rules. If it doesn't find this file/directory, it doesn't do anything with it.
Hope that's a little bit more clear now.
You'll occasionally see reference to a Docker build context. The build has two steps:
The docker build client application creates a tar file of its directory parameter, and sends it in an HTTP request to the Docker daemon.
The Docker daemon unpacks the tar file, finds the Dockerfile in it, and runs it using the file content it was given.
.dockerignore only affects the first step: it keeps docker build from sending the Docker daemon particular files. The .dockerignore file doesn't require there to be a folder_3 directory, it just says that if there is one it shouldn't be excluded. The second step on the Docker daemon side doesn't use .dockerignore at all, and when you COPY . /somewhere it copies the entire build context; that is, whatever was sent in the API request.
There are a couple of practical consequences of this workflow. If you have a very large local directory it can take time to send it to the Docker daemon, and the Docker daemon keeps a duplicate copy of it during the build, so it's often worthwhile to .dockerignore your .git directory and a build tree. This setup is also how docker build works with a Docker daemon on a different system or in a VM, and it's why if you try to COPY a file by name that doesn't exist (COPY folder_3 somewhere) you get an error message referencing a Docker-internal path.
I have a .dockerignore file in which I want to exclude the entire directory /src except all .json files in that /src-Directory.
My current approach looks like this:
/src/*
!/src/**/*.json
I have also tried using /src instead of /src/*, but in both cases is the entire directory excluded and the JSON files are nowhere to be found.
You should change the ignore file as follow:
/src/*
!/src/*.json
!/src/**/*.json
The reason is that !/src/**/*.json searches for json files inside subfolder of /src, and not in /src itself too.
Look at the official documentation if you need further info https://docs.docker.com/engine/reference/builder/#dockerignore-file .
In a Dockerfile, I have
COPY . .
I want to exclude an entire directory, in my case, node_modules directory.
Something like this:
COPY [all but **/node_modules/**] .
Is this possible with Docker?
Create file .dockerignore in your docker build context directory (so in this case, most likely a directory that is a parent to node_modules) with one line in it:
**/node_modules
although you probably just want:
node_modules
Info about dockerignore: https://docs.docker.com/engine/reference/builder/#dockerignore-file
For those who can't use a .dockerignore file (e.g. if you need the file in one COPY but not another):
Yes, but you need multiple COPY instructions. Specifically, you need a COPY for each letter in the filename you wish to exclude.
COPY [^n]* # All files that don't start with 'n'
COPY n[^o]* # All files that start with 'n', but not 'no'
COPY no[^d]* # All files that start with 'no', but not 'nod'
Continuing until you have the full file name, or just the prefix you're reasonably sure won't have any other files.
FOR A ONE LINER SOLUTION, type the following in Command prompt or Terminal at project root.
echo node_modules >> .dockerignore
This command appends "node_modules" in the .dockerignore file. If the .dockerignore does not exist already, it will create a new one. Replace node_modules with the folder you want to exclude.
Warning:
If you are new to Docker ecosystem and/or you already have the .dockerignore file in your project, please take a backup before proceeding.
BONUS: (as pointed out by Joey Baruch)
(To CREATE/OVERWRITE the .dockerignore file via PowerShell, which can be handled by Docker):
>> echo node_modules | Out-File -Encoding UTF8 .dockerignore
Excluding node_modules from current directory
node_modules
Excluding node_modules in any immediate subdirectories
*/node_modules
Here is the official docs
For those using gcloud build:
gcloud build ignores .dockerignore and looks instead for .gcloudignore
Use:
cp .dockerignore .gcloudignore
Source
Adding .dockerignore works for me.
One additional point Those who are trying this solution on Windows , windows will not let you create .dockerignore file (as it doesn't by default allows creating file starting with .)
To create such file starting with . on Windows, include an ending dot also, like : .dockerignore. and hit enter ( provided you have enabled view extension options from folder options )
I used a multi stage build approach since I needed one stage to have access to the file but not another stage so .dockerignore wouldn't work:
FROM ruby AS builder
COPY app/ app/
# Do stuff with app
# remove the stuff you don't want
RUN rm -Rf app/assets
FROM ruby AS publish
# In my real version I needed the absolute path to builder WORKDIR.
# Since I'm copying from the builder stage, app/assets won't exist
# and neither will it be part of the publish image.
COPY --from=builder app app
I want to share a latex document via git with many other people.
Therefore we decided to put all the special sty files, that are not present in everyones latex-installation, into a resources directory. It would be cool, if this dir would be a superdir. of the actual working directory
How exactly can I import those style files?
It is important that even the dependencies of those remote styles are resolved with other remote styles.
You can import a style file (mystyle.sty) into your document in two ways:
If you have it in your path or in the same folder as the .tex file, simply include this line in your preamble: \usepackage{mystyle}
If you have it in a different folder, you can access using its full path as \usepackage{/path/to/folder/mystyle}
That said, if you're not sure if the style file is in everyone's installation, simply include it in the same directory and make sure you do git add mystyle.sty and track it along with the rest of your files (although most likely there won't be any changes to it). There is no need for a parent directory. But if you insist on a different directory, see option 2 above.
It would be better if it were in a subdirectory than in a parent directory, as you can still call the file as \usepackage{subdir/mystyle} and be certain that you are invoking your style file. However, if you escape out to the parent directory, you never know if the other users have a similarly named folder that is not part of your package, which can result in errors.
This probably isn't relevant to you any more, but here is another way to do what you want.
Set up your git repository like this:
mystyle.sty
project/
makefile
project.tex
and put \usepackage{mystyle} in the preamble of project.tex.
Compiling project.tex manually won't work, of course, because mystyle.sty is not in the same directory as project.tex.
However, if makefile contains something along the lines of:
project.pdf: mystyle.sty project.tex
pdflatex project
mystyle.sty: ../mystyle.sty
cp ../$# $#
then running make from within the project directory will cause mystyle.sty to be copied to the correct place before project.tex is (this time successfully) compiled.
This way might seem a little bit over the top, but it does combine the best features of other methods.
If several projects in the same repository require mystyle.sty then having a common mystyle.sty sitting above them all makes more sense than having a copy in each project directory; all these copies would have to be maintained.
The compilation is portable, in the sense that if you gave me your copies of mystyle.sty and project.tex then I would (in theory at least) be able to compile manually without needing to modify the files you gave me.
For example, I would not have to replace \usepackage{/your/path/mystyle} with \usepackage{/my/path/mystyle}.
You can use Makefiles as suggested above. Another option is CMake. I didn't test for parent directories.
If you have the following file structure:
├── CMakeLists.txt
├── cmake
│ └── UseLATEX.cmake
├── img
│ └── logo.jpg
├── lib
│ └── framed.sty
└── main.tex
you should have CMake installed, instructions on CMake resources
UseLATEX.cmake can be downloaded from here
then inside the CMakeLists.txt
╚═$ cat CMakeLists.txt
cmake_minimum_required (VERSION 2.6)
set(PROJECT_NAME_STR myProject)
project(${PROJECT_NAME_STR})
set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
include(UseLATEX)
ADD_LATEX_DOCUMENT(main.tex
IMAGE_DIRS img
DEFAULT_PDF
MANGLE_TARGET_NAMES)
Some example content for main.tex (note the image)
╚═$ cat main.tex
\documentclass{report}
\begin{document}
\begin{center}
\includegraphics[width=300px]{img/logo.jpg}
\end{center}
\end{document}
The lib directory has the *.sty files
You can now compile:
cd /directory/that/has/CMakeLists.txt/
mkdir build
cd build
cmake ..
make
you can then view main.pdf which is in the build directory.
When you use TeX distribution that uses kpathsea, you can use the TEXINPUTS environment variable to specify where TeX is looking for files. The variable needs to be used in the following way.
The paths in TEXINPUTS are separated by :. An empty path will include the default search paths, i.e., just the colon. Two consecutive slashes means that the directory and all sub-directories are searched.
Thus, e.g., to build a file document.pdf which uses files in the current directory, all sub-directories of the resources directory and the default directories, you can use the following Makefile.
document.pdf: document.tex
TEXINPUTS=.:./resources//: pdflatex document.tex
To speed up the filename lookup, you can build a ls-R database using the mktexlsr command.
For all the details on kpathsea take a look at the manual.
You can use latexmk and its facilities
There is a feature documented under Utility subroutines on page 48 here in latexmk which can update TEXINPUTS during a run. If you can consider to use the .latexmkrc file to configure your chain and options, you can add ensure_path() to the file:
Here is an example:
# .latexmkrc
ensure_path('TEXINPUTS', './path/to/something//', '/full/path/to/something/else//')
# [...] Other options goes here.
$pdf_update_method = 3;
$xelatex = 'xelatex -synctex=1 -interaction=nonstopmode -file-line-error %O %S';
$pdf_previewer = 'start "%ProgramFiles%/SumatraPDF/SumatraPDF.exe" %O %S';
$out_dir = 'build/';
Notice the // at the end of a path, This will aid LaTeX to search for files in the specified directory and in all subdirectories.
Please note that while this is an amazing feature, you need to take good care of your naming scheme. If you use the same file name several places, you can run into trouble when importing them with, say \include{somefile}.
Let's say I have this directory structure:
SConstruct
src/
a.cpp
b.cpp
include/
a.h
b.h
in SConstruct I don't want to specify ['src/a.cpp', 'scr/b.cpp'] every time; I'm looking for some way to set the base source directory to 'src'
any hint? I've been looking into the docs but can't find anything useful
A couple of options for you:
First, scons likes to use SConscript files for subdirectories. Put an SConscript in src/ and it can refer to local files (and will generate output in a build subdir as well). You can set up your environment once in the SConstruct. Then you "load" the SConscript from your master SConstruct.
SConscript('src/SConscript')
As your project grows, managing SConscript files in subdirectories is easier than putting everything in the master SConstruct.
Second, here's a similar question / answer that might help -- it uses Glob with a very simple example.
Third, since it's just python, you can make a list of files without the prefix and use a list comprehension to build the real list:
file_sources = [ 'a.c', 'b.c' ]
real_sources = [os.path.join('src', f) for f in file_sources]