I have a tensorflow training script which I want to run using a Docker container (based on the official TF GPU image). Although everything works just fine, running the container with the script is horribly verbose and ugly. The main problem is that my training script allows the user to specify various directories used during training, for input data, logging, generating output, etc. I don't want to have change what my users are used to, so the container needs to be informed of the location of these user-defined directories, so it can mount them. So I end up with something like this:
docker run \
-it --rm --gpus all -d \
--mount type=bind,source=/home/guest/datasets/my-dataset,target=/datasets/my-dataset \
--mount type=bind,source=/home/guest/my-scripts/config.json,target=/config.json \
-v /home/guest/my-scripts/logdir:/logdir \
-v /home/guest/my-scripts/generated:/generated \
train-image \
python train.py \
--data_dir /datasets/my-dataset \
--gpu 0 \
--logdir ./logdir \
--output ./generated \
--config_file ./config.json \
--num_epochs 250 \
--batch_size 128 \
--checkpoint_every 5 \
--generate True \
--resume False
In the above I am mounting a dataset from the host into the container, and also mounting a single config file config.json (which configures the TF model). I specify a logging directory logdir and an output directory generated as volumes. Each of these resources are also passed as parameters to the train.py script.
This is all very ugly, but I can't see another way of doing it. Of course I could put all this in a shell script, and provide command line arguments which set these duplicated values from the outside. But this doesn't seem a nice solution, because if I want to anything else with the container, for example check the logs, I would use the raw docker command.
I suspect this question will likely be tagged as opinion-based, but I've not found a good solution for this that I can recommend to my users.
As user Ron van der Heijden points out, one solution is to use docker-compose in combination with environment variables defined in an .env file. Nice answer.
Related
My input files for the singularity program is not recognized (not found), which I think is due to the directory is not mounted within singularity.
I know that the mounting can be set by this command, but I am not sure which folders to mount.
export SINGULARITY_BIND="/somefolder:/somefolder"
How do I know which folders should be before and after the ":" sign in SINGULARITY_BIND?
I have set:
SINGULARITY_CACHEDIR=/mnt/scratch/username/software
where my singularity is stored.
My complete script:
export SINGULARITY_CACHEDIR=/mnt/scratch/username/software
export SINGULARITY_BIND="/home/username:/mnt/scratch/username"
OUTPUT_DIR="${PWD}/quickstart-output"
INPUT_DIR="${PWD}/quickstart-testdata"
BIN_VERSION="1.4.0"
# Run DeepVariant.
singularity run \
docker://google/deepvariant:"${BIN_VERSION}" \
/opt/deepvariant/bin/run_deepvariant \
--model_type=WGS \ **Replace this string with exactly one of the following [WGS,WES,PACBIO,HYBRID_PACBIO_ILLUMINA]**
--ref="${INPUT_DIR}"/ucsc.hg19.chr20.unittest.fasta \
--reads="${INPUT_DIR}"/NA12878_S1.chr20.10_10p1mb.bam \
--regions "chr20:10,000,000-10,010,000" \
--output_vcf="${OUTPUT_DIR}"/output.vcf.gz \
--output_gvcf="${OUTPUT_DIR}"/output.g.vcf.gz \
--intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" \ **Optional.
--num_shards=1 \ **How many cores the `make_examples` step uses. Change it to the number of CPU cores you have.**
My error:
singularity.clean.sh: line 23: --ref=/home/username/scratch/username/software/quickstart-testdata/ucsc.hg19.chr20.unittest.fasta: No such file or directory
If you want to bind mount your current working directory, you can use --bind $(pwd). If you want to bind mount your home directory, you can use --bind $HOME (note that singularity mounts the home directory by default). See the Singularity documentation for more information.
Based on your INPUT_DIR and OUTPUT_DIR, it seems like you can bind mount your current working directory. To do that, use --bind $(pwd). Note this argument goes before the name of the singularity container.
Just to be safe, also use --pwd $(pwd) to set the working directory in the container as the current working directory on the host.
OUTPUT_DIR="${PWD}/quickstart-output"
INPUT_DIR="${PWD}/quickstart-testdata"
BIN_VERSION="1.4.0"
singularity run --bind $(pwd) --pwd $(pwd) \
docker://google/deepvariant:"${BIN_VERSION}" \
/opt/deepvariant/bin/run_deepvariant \
--model_type=WGS \
--ref="${INPUT_DIR}/ucsc.hg19.chr20.unittest.fasta" \
--reads="${INPUT_DIR}/NA12878_S1.chr20.10_10p1mb.bam" \
--regions "chr20:10,000,000-10,010,000" \
--output_vcf="${OUTPUT_DIR}/output.vcf.gz" \
--output_gvcf="${OUTPUT_DIR}/output.g.vcf.gz" \
--intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" \
--num_shards=1
The syntax of the --bind argument is path-on-host:path-in-container. And using --bind path is shorthand for --bind path:path, which means the source path is mounted as the same path in the container. This can be very useful, because one does not have to rewrite paths and think in terms of the container's directories.
please would you advise :
given a singularity container, how can I copy the files from the local drive to the singularity container ?
I am using a singularity container that is described at :
https://hub.docker.com/r/tobneu/slamdunk
(and from the docker image i have made a singularity image for a SLURM cluster)
I have searched the stackoverflow for answers, however, i have found the answer only to the reverse question i.e. copying the files from the singularity container to the local drive.
https://stackoverflow.com/questions/59736299/transferring-files-from-the-singularity-container-into-the-local-directory
thanks a lot,
bogdan
As mentioned on your previous question, singularity has the -H/--home command line parameter.
singularity exec -H /labs/zzz/data my_image.sif bash -c 'echo "HOME=$HOME";echo "PWD=$PWD"'
# HOME=/labs/zzz/data
# PWD=...
I shall rephrase the question though, as i have noted that the singularity container sees the $HOME folder that i do have on a SLURM cluster (i.e. /home/btanasa).
Please, if I may re-phrase the question : how shall i change the $HOME folder that the singularity container sees, for example /labs/zzz/data ? thanks a million !
Finally, i have got it to work. Shall anyone need to know the answer to the question in the title, it is outlined below by using --bind and --home:
singularity exec \
--bind /local/scratch/btanasa:/output8 \
--home /labs/jlgoldbe/MASSY_data_SLAMseq/the_SAMPLES_MAY2021:/home \
/labs/jlgoldbe/MASSY_data_SLAMseq/the_SAMPLES_MAY2021/SLAMDUNK_SINGULARITY/slamdunk_latest.sif slamdunk all \
-r GRCm38.primary_assembly.genome.fa \
-b 3UTRs_vM14_github_repository.27aug2020.sortdesc.LONG.with.SYMBOLS.to.use.bed \ -o /output8 \
-t 4 \
./8_R1_001.fastq.gz
I can successfully get the full json string with:
docker run --rm -v `pwd`:`pwd` gittools/gitversion-dotnetcore:linux-4.0.0 `pwd` -output json
which outputs to something like:
{
"Major":0,
"Minor":1,
"Patch":0,
"SemVer":"0.1.0-dev-2.1",
.
.
.
"CommitsSinceVersionSource":20,
"CommitsSinceVersionSourcePadded":"0020",
"CommitDate":"2020-05-28"
}
Since I am only interested in SemVer variable I try to use the -showvariable FullSemVer with:
docker run --rm -v `pwd`:`pwd` gittools/gitversion-dotnetcore:linux-4.0.0 `pwd` -output json -showvariable FullSemVer
But it fails with a quite long and nasty error log.
INFO [05/28/20 18:23:12:10] End: Loading version variables from disk cache (Took: 76.31ms)
ERROR [05/28/20 18:23:12:13] An unexpected error occurred:
System.NotImplementedException: The method or operation is not implemented.
I wonder if there is a way to use the -showvariable flag with the gitversion Docker container?
I think the problem is the path argument passed to GitVersion. pwd will give you the working directory on your host, not within the container. GitVersion is unfortunately not aware of the fact that it's executing within a container, so it needs to be provided with the volume directory /repo as the path to calculate a version number for. This is something we should consider changing in version 6.
I also can't remember when -showvariable was implemented, so to be on the safe side you should try with a newer version of our Docker containers. I can also recommend using the alpine container, as it's the smallest one we offer (only 83.9 MB). This works:
docker run \
--rm \
--volume "$(pwd):/repo" \
gittools/gitversion:5.3.4-linux-alpine.3.10-x64-netcoreapp3.1 \
/repo \
-output json \
-showvariable FullSemVer
We have our Flask API in a docker image, we push this docker to a bitbucket repository, then a bitbucket pipeline start deploying.
Everything works as expected, but the compilation of OpenCV is taking in average 15 min.
I would like to know if is there any way to avoid this compilation every time we push to bitbucket. Something like caching.
I have read about cache on bitbucket pipelines but it did not work as I expected.
This is part of my Dockerfile I would like to improve:
RUN mkdir /opt && cd /opt && \
wget -q https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip && \
unzip ${OPENCV_VERSION}.zip && \
rm -rf ${OPENCV_VERSION}.zip && \
mkdir -p /opt/opencv-${OPENCV_VERSION}/build && \
cd /opt/opencv-${OPENCV_VERSION}/build && \
CXX=/usr/bin/clang++ CC=/usr/bin/clang cmake \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_FFMPEG=NO \
-D WITH_IPP=NO \
-D WITH_OPENEXR=NO \
-D WITH_TBB=YES \
-D BUILD_EXAMPLES=NO \
-D BUILD_ANDROID_EXAMPLES=NO \
-D INSTALL_PYTHON_EXAMPLES=NO \
-D BUILD_DOCS=NO \
-D BUILD_opencv_python2=NO \
-D BUILD_opencv_python3=ON \
-D ENABLE_PYTHON3=ON \
-D PYTHON3_EXECUTABLE=/usr/bin/python3 \
.. && \
make VERBOSE=1 -j8 && \
make && \
make install && \
rm -rf /opt/opencv-${OPENCV_VERSION}
I expect some solution like just pointing a pre-compiled version of the OpenCV Api.
I have recently faced this problem and agree that cache doesn't seem to work as expected. However without looking at your entire Dockerfile, it's hard to say. ADD's and COPY's will invalidate the cache so i'd suggest you move this section up to the top if you can before adding any files.
A better solution (if there is no pre-compiled version), is to use the concept of a base image which is what I have done to cut my build time down in half. Basically you build a base image flask-api-base which will install all your packages and compile OpenCV and then your actual final image will pull FROM flask-api-base:latest and build your application specific code. Just remember if the base image changes, you may need to wipe your Bitbucket cache.
I'm unfamiliar with OpenCV but assume that, if there is a binary that you can use, that would be the ideal option.
I'm curious as to why this layer (RUN ...) isn't being cached between builds. It appears that you're cleanly separating the make of OpenCV from other statements in your Dockerfile and so, this RUN should generate a distinct layer that's stable and thus reused across builds.
Does this statement occur after earlier e.g. RUN statements that do change? If so, you may want to reorder this statement and place it earlier in the Dockerfile so that this layer becomes constant. See best practices for the Dockerfile statements that generate layers.
Alternatively, you could make a separate image containing OpenCV and then FROM this image in your code builds. You may do this either using distinct Dockerfiles or multi-stage builds. This way, this image containing the OpenCV build would only be built on (your) demand and reused across subsequent builds.
The solution I used was to create my own image, upload it to Docker hub, and create a new one based on that.
So the first docker image should contain all the basic libraries my system uses.
The second has the environmental variables and the api itself.
I am trying to run kubernetes on coreos. I am using fleet, setup-network-environment, and kube-register to register nodes. However, in my cloud-init file where I write my systemd unit files, the kubelet's unit file won't run this properly:
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 --port=10250 \
--hostname_override=${DEFAULT_IPV4} \
--allow_privileged=true \
--logtostderr=true \
--healthz_bind_address=0.0.0.0
Instead of my public ip, ${DEFAULT_IPV4} results in $default_ipv4, which also doesn't result in the ip. I know --host-name-override should just take a string, and it works when I run this line from command line. There are other unit files where ${ENV_VAR} works fine. Why is it that for the kubelet's unit file, it just breaks?
EDIT 1
/etc/network-environment
LO_IPV4=127.0.0.1
ENS33_IPV4=192.168.195.242
DEFAULT_IPV4=192.168.195.242
ENS34_IPV4=172.22.22.238
EDIT 2
kubelet unit file
- name: kube-kubelet.service
command: start
content: |
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=setup-network-environment.service
After=setup-network-environment.service
[Service]
EnvironmentFile=/etc/network-environment
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet -z /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v0.18.2/bin/linux/amd64/kubelet
ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet
# wait for kubernetes master to be up and ready
ExecStartPre=/opt/bin/wupiao 172.22.22.10 8080
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 \
--port=10250 \
--hostname_override=172.22.22.21 \
--api_servers=172.22.22.10:8080 \
--allow_privileged=true \
--logtostderr=true \
--healthz_bind_address=0.0.0.0 \
--healthz_port=10248
Restart=always
RestartSec=10
The Exec*=command is not a shell command. In my experimenting it was not very good at figuring out where the variable was unless it was by itself. I went and looked at some examples online and they always show the environment variable by itself. So, given a file like /tmp/myfile:
ENV=1.2.3.4
These [Service] definitions won't do what you think:
EnvironmentFile=/tmp/myfile
ExecStart=echo M$ENV
ExecStart=echo $ENV:8080
but, this will work on a line by itself:
EnvironmentFile=/tmp/myfile
ExecStart=echo $ENV
That doesn't help much when trying to pass an argument, like:
EnvironmentFile=/tmp/myfile
ExecStart=echo --myarg=http://$ENV:8080/v2
To accomplish passing the argument I had to put the entire myarg in a string in /tmp/myfile:
ENV="--myarg=http://1.2.3.4:8080/v2"
Finally I could can get my argument passed:
EnvironmentFile=/tmp/myfile
ExecStart=echo $ENV
It would seem the issue was in the version of coreos in the vagrant box. After an update of the vagrant box the environment variable was able to resolve to the proper value.