Tikz externalize issue when using pdflatex with -output-directory - latex

Question

When using pdflatex with -output-directory I run into issues when externalizing tikz figures. While the .md5 files are created where I expect them the command creating the externalized picture files (.pdf, .log, .dpth) fails. My assumption is that this happens because the pdflatex to create these files does not inherit the -output-directory option and thus fails to create the files in the right spot.
This is a minimal example showing the behavior.
main.tex:
\documentclass{minimal}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=tikz/]
\begin{document}
Test externalize in combination with -output-directory
\tikzsetnextfilename{testpicture}
\begin{tikzpicture}
\node {Node};
\end{tikzpicture}
\end{document}
bash:
mkdir -p build/tikz
pdflatex -output-directory build -shell-escape main.tex
error:
===== 'mode=convert with system call': Invoking 'pdflatex
-shell-escape -halt-on-error -interaction=batchmode -jobname
"tikz/testpicture" "\def\tikzexternalrealjob{main}\input{main}"'
======== This is pdfTeX, Version 3.14159265-2.6-1.40.16
(TeX Live 2015/Debian) (preloaded format=pdflatex)
\write18 enabled.
entering extended mode
! I can't write on file `tikz/testpicture.log'.
resulting directory structure (output of tree):
.
├── build
│   ├── main.aux
│   ├── main.auxlock
│   ├── main.log
│   ├── main.pdf
│   └── tikz
│   └── testpicture.md5
└── main.tex
So again, as far as I understand the log file creation fails due to the lack of a tikz directory in the working directory of the pdflatex command executed by the externalization.
Are my assumptions correct? If they are, how should how I should proceed with this?

Answer 1

As far as I have found out the easiest solution is to create a symlink in your main folder to the tikz folder inside build. Like so: ln -s build/tikz .
This was the solution I found in the documentation of another tikz externalizing library.

Answer 2

tikzexternal does not detect the -output-directory argument. Hence you have to modify the call to the compiler of the externalized images and add this argument by your own.
This also holds true for the inclusion of the image. You have to prefix the image here with the path of the output directory.
Those two points can be seen in the following MWE:
\documentclass{minimal}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=tikz/]
\makeatletter
\tikzset{%
external/system call={%
pdflatex %
\tikzexternalcheckshellescape %
-halt-on-error %
-interaction=batchmode %
-output-directory="build" %
-jobname "\image" %
"\texsource"%
},
/pgf/images/include external/.code={%
\includegraphics{build/#1}%
},%
}
\makeatother
\begin{document}
Test externalize in combination with -output-directory
\tikzsetnextfilename{testpicture}
\begin{tikzpicture}
\node {Node};
\end{tikzpicture}
\end{document}
Some additional notes:
You can find the relevant code in pgf/frontendlayer/tikz/libraries/tikzexternalshared.code.tex
If your tex file is in a sub directory or you use -jobname you have to replace "\texsource" with something like "\string\def\string\tikzexternalrealjob{\tikzexternal#realjob}\string\input{path/to/tex/file/\tikzexternal#realjob}"
Edit
There is a shorter way to do this: Instead of setting external/system call, you can also set external/shell escape={-shell-escape\space-output-directory=build}. This will end up in \tikzexternalcheckshellescape.
For both methods (the original and the shorter one) to work with LuaLaTeX, you need \usepackage{shellesc}.

Answer 3

It could be a question with regards to the permissions as well.
Try with
chmod 777 tikz
right after you create the folder.
You could also try to move your main.tex file into the build folder to see if that changes things.

Answer 4

I found that symlinking as suggested by Artemis is indeed the easiest option, i.e.:
mkdir -p build/fig
ln -sf build/fig .
pdflatex -shell-escape -output-directory=build main.tex
Additionally, if you use \tikzexternalize[mode=list and make], you will also notice that the generated build/main.makefile can't be used as is either, because it uses files both in the base folder and in the build folder. To make it work, again, symlinking seems to be the easiest option, this time the file main.figlist (alternative would be copying files or modifying the makefile). The full process now looks as follows:
# Run once to create list of externalize figures:
mkdir -p build/tikz
ln -sf build/tikz .
pdflatex -output-directory=build main.tex
# Build externalized figures:
ln -sf build/main.figlist .
make -f build/main.makefile
# Run again to include externalized figures into document:
pdflatex -output-directory=build main.tex
# Cleanup:
rm tikz main.figlist
Furthermore, if you encounter problems with other packages you may need to also take into account Phobos' answer to ensure that the build directory is also known while building the individual figures, i.e.:
\tikzexternalize[
prefix=tikz/,
mode=list and make,
shell escape={-shell-escape\space-output-directory=build},
]