Snakemake + Docker Volumes - "Missing Rule Exception" - docker

I'm trying to use snakemake with a docker image, but am having trouble with the docker volume. Unfortunately, there are no details on how to use 'singularity-args' to do this.
My snakemake file is:
rule all:
input:
'a/file3.txt'
rule step1:
output:
touch('a/file1.txt')
rule step2:
input:
rules.step1.output[0]
output:
'a/file2.txt'
params:
text = 'this is a test',
path = '/data/file2.txt'
singularity:
"docker://XXX/test"
shell:
"python test.py {params.text} {params.path}"
rule step3:
input:
rules.step2.output[0]
output:
touch('a/file3.txt')
The docker image is basically a python file that writes a string to file (for testing purposes). I'm trying to mount my home directory to the docker /data directory. With docker, I'm able to mount a volume using '-v'.
What is the correct way of doing this with snakemake?
I've tried the following commands (on MacOS and Ubuntu 18.04) and both have failed.
snakemake -s pipeline.py --use-singularity --singularity-args “-B /home/XXX/snakemake/a:/data”
snakemake -s pipeline.py --use-singularity --singularity-args “-B /home/XXX/snakemake/a”
The error message is:
No rule to produce /home/XXX/snakemake/a:/data” (if you use input functions make sure that they don't raise unexpected exceptions).
Am I missing a step?
Thanks in advance!

Just a trivial check... In your command lines you have tilted double quotes (“) instead of the straight ones ("), e.g.:
snakemake -s pipeline.py --use-singularity --singularity-args “-B /home/XXX/snakemake/a”
Maybe you are are copying and pasting from a text editor that uses the tilted quotes? I would use straight quotes as the other type would probably be interpreted in the wrong way.

I was able to get it working on Ubuntu 18.04 with the following command:
SINGULARITY_BINDPATH=“/home/XXX/snakemake/a:/data”; snakemake -s pipeline.py --latency-wait 10 --use-singularity
Unfortunately I wasn’t able to get the flag “--singularity-args” to work. Regardless of using ‘--bind’ or ‘-B’, I got the error “No rule to produce /Users/XXX/Devel/snakemake/a:/data”.
I’m using Snakemake 5.6.0 inside a Python3 virtual environment.
Also, on a side note, I don’t believe the MacOS singularity binary works. It had issues with Snakemake.
This work-around is good enough for now.
UPDATE
While this solution worked, the real solution (typo) was provided by #dariober.

Related

Error in --commit CMD Change in Docker via command line

I am trying to put a spring boot java .jar file into an Image off openJDK and commit the change to it in docker but docker command does not seem to work
I am following this article for steps : https://dzone.com/articles/docker-tutorial-for-beginners-with-java-and-spring
What argument is docker expecting which , I did not give in the command
docker container commit --change='CMD ["java","-jar","/tmp/mytroubleartifact-0.jar"]' upbeat_brahmagupta a-repo-name-of-choice/some-app-name:tagname2
the issue is with the apostrophe ' you should be using apostrophes (in plural - otherwise known as quotation marks) "
This command seems to be working for me
(no changes needed on your behalf):
docker container commit --change="CMD ['java','-jar','/tmp/mytroubleartifact-0.jar']" upbeat_brahmagupta a-repo-name-of-choice/some-app-name:tagname2
though I've seen other people using a single apostrophe for me it also didn't work with your situation. docker docs example also didnt work with just copying the --change part, with both options (-c, --change) and with just one of them, only using double qoutes did the trick, not exactly sure why though. (tried replacing names, making them shorter ¯_(ツ)_/¯)
docker commit --change='CMD ["apachectl", "-DFOREGROUND"]' -c "EXPOSE 80" c3f279d17e0a svendowideit/testimage:version4
thanks to these posts for the working example:
https://adamtheautomator.com/docker-commit/
https://docs.oracle.com/cd/E37670_01/E75728/html/ch04s18.html

Apache PredictionIO - Docker run failed

I have been trying http://predictionio.apache.org/install/install-docker/ this tutorial. I have successfully built Docker image however when I try to run docker run i get the Can't open /etc/predictionio/pio-env.sh error.
docker build -t predictionio/pio pio
docker run -ti predictionio/pio
PS: If I comment out the last line CMD ["sh", "/usr/bin/pio_run"] I can build and run docker image successfully. I can open the file too from docker bash.
I think you need to grant permissions to execute this file. add the following line at the end of your Dockerfile
RUN chmod +x pio_run.sh
also, you might need to change CMD to ENTRYPOINT like following:
ENTRYPOINT ["sh","/usr/bin/pio_run.sh"]
Your output states you are running Windows. Did you use the default command prompt or did you use docker terminal? I had the same error messages in the past on Windows but mysteriously it disappeared after trying the tutorial again. I am not sure what I did different except I might possibly used docker instead of the default command prompt...
Could you also try using docker-compose instead of plain docker commands as described in the tutorial?
Ensure your storage (Postgres, MySQL or ElasticSearch) is running before starting PIO as well.
Just resolved it on my machine.
When you cloned repository on Windows, git converted end of line symbols from Unix-style (\n) to Windows style (\r\n).
You need to open file C:\wherever-you-cloned-pio-repository\predictionio\docker\pio\pio_run and change it back (for e.g. using Visual Studio Code, or Notepad++). Then you need to rebuild the image and it should work.
Also for the future you may want to disable automatic conversion Disable git EOL Conversions

How to run repo from a script inside a container in a jenkins job

I am unable to run repo non-interactively inside a container as part of a freestyle job.
It prompts for the user-name and email. I got round that by doing a git config --global inside the job.
But then it does the color test, and that hangs indefinitely.
Looking at the source code for repo I see this
if os.isatty(0) and os.isatty(1) and not self.manifest.IsMirror:
if opt.config_name or self._ShouldConfigureUser():
self._ConfigureUser()
self._ConfigureColor()
So, I ran the following inside the container:
python -C "import os; print os.isatty(0), os.isatty(1)"
and, sure enough, it printed out True True
Looking at the Jenkins log, it launches the container with --tty specified, and there seems no way to configure that option.
I can't find a bash option to force a script to be run in a non-interactive shell. If I put the above python line in a file and execute it with almost any combination of commands and options, it still prints out True True
The only way I see something different is if I use I/O redirection
bash <a.sh
which prints out False True - i.e. stdin is not a tty, and
bash <a.sh >a.log
which prints False False.
For a complex script, are there any problems using the bash <script approach?
Does anyone know any jenkins magic to prevent docker being launched using --tty?
I know that the --tty is the culprit. I built the container locally and ran the following
$ docker run repotest python -c "import os;print os.isatty(0), os.isatty(1)"
False False
$ docker run --tty repotest python -c "import os;print os.isatty(0), os.isatty(1)"
True True
Running Versions:
repo: 1.12.37 (per Ubuntu 16.04 apt-get)
Jenkins: 2.149
Cloudbees Docker Plugin: 1.7.3
Container base is ubuntu:xenial
I'm using the "Build inside a docker container" option.
To run bash script repo_script.sh "non-interactively", or more exactly speaking without having terminals associated with standard streams, you could run your script simply as
repo_script.sh < /dev/null 2>&1 | cat
assuming you want to see the output the way you would see it running simply as repo_script.sh. By piping the standard output and error to a different process the file descriptor appears as a pipe and not TTY to repo_script.sh. You could also direct output to a file, or even to /dev/null if you do not care about the output:
log_file=/dev/null
repo_script.sh < /dev/null > "${log_file}" 2>&1
Running the script as
bash < repo_script.sh | cat
might would work too, though it is very unorthodox and to my mind hackish way of running a script just to break the association of TTY to the standard input. From script engine point of view, it is different to read a script program from a file than from standard input (which typically, if it is a terminal, is not seekable), so there might be some subtle differences that could possibly bite you in unexpected ways. This way does not as clearly communicate your intention to the next person that need to understand your code, and may lead to partial hair loss in that person due to extraneous head scratching.
There is no need for any bash options, just using the output directions from within the interpreting shell as above described is an easy-to-comprehend, multi-platform compatible standard convention for changing the standard stream associations.
P.S. I think it should be enough for your repo script to just test if the standard input is a TTY. It looks to me like the author of that script did not think deeply enough there. There is simply no use waiting for input if you do not have terminal device associated with standard input, and you could determine that everything needs to run without user interaction from there or stop with an error if that is not possible.

NodeMCU Build in Docker results in "exec format error"

I'm trying the NodeMCU Docker build in Ubuntu 16.04.4 LTS for the first time.
I have read the tagged articles here for Docker and NodeMCU, but don't see this particular error.
"docker run hello-world" has no problems.
I have tried the NodeMCU build command in both forms:
$ docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build
and the explicit path variation:
$ docker run --rm -it -v /home/tim/nodemcu-firmware:/opt/nodemcu-firmware marcelstoer/nodemcu-build
In both cases, I get this error:
standard_init_linux.go:187: exec user process caused "exec format error"
I have searched on this error, and most solutions are related to a missing shebang.
However, I'm not sure what script would need the shebang, or why it would be not working in my case but correct for others.
Has anyone else run across this error?
Speaking without deep technical details, this error means that the kernel can not recognize the format of the executable file, thus, it can not run this file. In your case this error is about the executable file which is started when the container is launched. According to the Cmd entry in the output of docker inspect marcelstoer/nodemcu-build, it is a file /bin/sh, which is an ELF executable.
When Linux can not execute ELF binary and returns such an error (about the file format), it usually is related to the system architecture. More specifically, the image marcelstoer/nodemcu-build contains ELF64 executables (i.e. for amd64 architecture), and your system does not support it (is it i386 or even some flavor of arm?). Running docker run hello-world, however, works fine for you, because hello-world image exists for all architectures supported by Docker.
According to the Dockerfile of marcelstoer/nodemcu-build image, it is built from ubuntu, which exists for different architectures, thus, you may try building the marcelstoer/nodemcu-build image on your system rather than pulling it from the dockerhub.
P.S.: regarding the solution you have linked to your question. This is not about your case (ELF binary), rather it is about a script. In case of script, the executable format is recognized by the shebang (#!) at the very beginning of the file, thus, the script must start with #!, not with the newline. That's why the author got the same error: the kernel could not detect that this is a script and failed to start it. Different (but similar) reasons, same error.

How to run Bazel container images on OSX?

According to the documentation at bazelbuild/rules_docker, it should be possible to work with these container images on OSX, and it also claims that it's possible to do so without docker.
These rules do not require / use Docker for pulling, building, or pushing images. This means:
They can be used to develop Docker containers on Windows / OSX without boot2docker or docker-machine installed.
They do not require root access on your workstation.
How do I do that? Here's a simple rule:
go_image(
name = "helloworld_image",
importpath = "github.com/nictuku/helloworld",
library = ":go_default_library",
visibility = ["//visibility:public"],
)
I can build the image with bazel build :helloworld_image. It produces a tar ball in blaze-bin, but it won't run it:
INFO: Running command line: bazel-bin/helloworld_image
Loaded image ID: sha256:08d312b529d30431c68741fd3a31468a02533f27a8c2c29eedc969dae5a39852
Tagging 08d312b529d30431c68741fd3a31468a02533f27a8c2c29eedc969dae5a39852 as bazel:helloworld_image
standard_init_linux.go:185: exec user process caused "exec format error"
ERROR: Non-zero return code '1' from command: Process exited with status 1.
It's trying to run the linux this is OSX, which is silly.
I also tried doing a "docker load" on the .tar content but it doesn't seem to like that format.
$ docker load -i bazel-bin/helloworld_image-layer.tar
open /var/lib/docker/tmp/docker-import-330829602/app/json: no such file or directory
Help? Thanks!
You are building for your host platform by default so you need to build for the container platform if you want to do that.
Since you are using a go binary, you can do cross compilation by specifying --cpu=k8 on the command line. Ideally we would be able to just say that the docker image needs a linux binary (so no need to specify the --cpu command-line flag) but this is still a work in progress in Bazel.

Resources