EDIT: documentation given by the informatic administration was shitty, old version of singularity, now the order of arguments is different and the problem is solved.
To make my tool more portable, and because I have to use it on a cluster, I have to put my bioinformatics tool at disposal for docker. Tool is located here. The docker hub is 007ptar007/metadbgwas, if you want to experience with it. The Dockerfile is in the repo, and to make it easier to everyone :
FROM ubuntu:latest
ENV DEBIAN_FRONTEND=noninteractive
USER root
COPY ./install_docker.sh ./
RUN chmod +x ./install_docker.sh && sh ./install_docker.sh
ENTRYPOINT ["/MetaDBGWAS/metadbgwas.sh"]
ENV PATH="/MetaDBGWAS/:${PATH}"
And the install_docker.sh script contains :
apt-get update
apt install -y libgatbcore-dev libhdf5-dev libboost-all-dev libpstreams-dev zlib1g-dev g++ cmake git r-base-core
Rscript -e "install.packages(c('ape', 'phangorn'))"
Rscript -e "install.packages('https://raw.githubusercontent.com/sgearle/bugwas/master/build/bugwas_1.0.tar.gz', repos=NULL, type='source')"
git clone --recursive https://github.com/Louis-MG/MetaDBGWAS.git
cd MetaDBGWAS
sed -i "51i#include <limits>" ./REINDEER/blight/robin_hood.h #temporary fix for REINDEER compilation
sh install.sh
The problem :
My tool parses the command line, and needs a verbose (-v, or --verbose) argument. It also needs to reject unknown arguments; anything that isn't used by the tool causes the help message to be printed in the standard output and exits. To use the tool, I need to mount volumes were the data is; using -v /path/to/files:/input option:
singularity run docker://007ptar007/metadbgwas --volumes '/path/to/data:/inputd/:/input' --files /input --strains /input/strains --threads 8 --output ~/output
But my tool sees this as a bad -v option value or the --volume as an unknown option. I can't change this on my tool. How do I solve this conflict ?
You need to put any arguments intended for singularity - such as the volume mounting - before the name of the image you want to run (e.g. the docker image you specify in your command):
singularity run -v '/path/to/data:/input' docker://007ptar007/metadbgwas --files /input --strains /input/strains --threads 8 --output ~/output
I am trying to set up a Docker image (my Dockerfile is available here, sorry for the french README: https://framagit.org/Gwendal/firefox-icedtea-docker) with an old version of Firefox and an old version of Java to run an old Java applet to start a VPN. My image does work and successfully allows me to start the Java applet in Firefox.
Unfortunately, the said applet then tries to run the following command in the container (I've simply removed the --config part from the command as it does not matter here):
INFO: launching '/usr/bin/pkexec sh -c /usr/sbin/openvpn --config ...'
Then the applet exits silently with an error. While investigating, I've tried running a command with pkexec with the same Docker image, and it gives me this result:
$ sudo docker-compose run firefox pkexec /firefox/firefox-sdk/bin/firefox-bin -new-instance
**
ERROR:pkexec.c:719:main: assertion failed: (polkit_unix_process_get_start_time (POLKIT_UNIX_PROCESS (subject)) > 0)
But I don't know polkit at all and cannot understand this error.
EDIT: A more minimal way to reproduce the problem is with this Dockerfile:
FROM ubuntu:16.04
RUN apt-get update \
&& apt-get install -y policykit-1
And then run:
$ sudo docker build -t pkexec-test .
$ sudo docker run pkexec-test pkexec echo Hello
Which leads here again to:
ERROR:pkexec.c:719:main: assertion failed: (polkit_unix_process_get_start_time (POLKIT_UNIX_PROCESS (subject)) > 0)
Should I conclude that pkexec cannot work in a docker container? Or is there any way to make this command work?
Sidenote: I have no control whatsoever on the Java applet that I try to run, it is a horrible and very dated proprietary black box that I am supposed to use at work, for which I have no access to the source code, and that I must use as is.
I have solved my own problem by replacing pkexec by sudo in the docker image, and by allowing passwordless sudo.
Given an ubuntu docker image where a user called developer was created and configured with a USER statement, add these lines:
# Install sudo and make 'developer' a passwordless sudoer
RUN apt-get install sudo
ADD ./developersudo /etc/sudoers.d/developersudo
# Replacing pkexec by sudo
RUN rm /usr/bin/pkexec
RUN ln -s /usr/bin/sudo /usr/bin/pkexec
with the file developersudo containing:
developer ALL=(ALL) NOPASSWD:ALL
This replaces any call to pkexec made in a process running in the container, by a call to sudo without any password prompt, which works nicely.
I have a problem to use the docker rstudio-image rocker/rstudio proposed
on https://www.rocker-project.org/ (docker containers for R). Since I am a beginner with both docker and RStudio, I suspect the problem comes from me and does not deserve a bug report:
I open a proper terminal with 'Docker Quickstart Terminal'
where I run the image with docker run -d -p 8787:8787 -e DISABLE_AUTH=true -v <...>:/home/rstudio/<...> --name rstudio rocker/rstudio
in my browser I then get a nice RStudio instance at the address http://192.168.99.100:8787
but in this instance I can't install several packages such as xml2. I get the message:
Using PKG_CFLAGS=
Using PKG_LIBS=-lxml2
------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libxml-2.0 was not found. Try installing:
* deb: libxml2-dev (Debian, Ubuntu, etc)
* rpm: libxml2-devel (Fedora, CentOS, RHEL)
* csw: libxml2_dev (Solaris)
If libxml-2.0 is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libxml-2.0.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------------------------------------------------
ERROR: configuration failed for package ‘xml2’
* removing ‘/usr/local/lib/R/site-library/xml2’
Warning in install.packages :
installation of package ‘xml2’ had non-zero exit status
I don't know whether xml2 is on the image but the file libxml-2.0.pc does exist on my laptop in the directory /opt/local/lib/pkgconfig and pkg-config is in /opt/local/bin. So I tried linking these pkg paths when running
the image (to see what happen when I play with the image environment
in RStudio), adding options -v
/opt/local/lib/pkgconfig:/home/rstudio/lib/pkgconfig -v
/opt/local/bin:/home/rstudio/bin to the run command. But it doesn't work: for some reason
I don't see the content of lib/pkgconfig in RStudio...
Also the RStudio instance does not accept root/sudo commands so I can't
use tools such as apt-get in the RStudio terminal
so, what's the trick ?
Libraries on your laptop (the host for docker) are not available for docker containers. You should create a custom image with required libraries, create a Dockerfile like this:
FROM rocker/rstudio
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libxml2-dev # add any additional libraries you need
CMD ["/init"]
Above I added the libxml2-dev but you can add as many libraries as you need.
Then build your image using this command (you need to execute below command in directory there you created Dockerfile):
docker build -t my_rstudio:0.1 .
Then you can start your container:
docker run -d -p 8787:8787 -e DISABLE_AUTH=true --name rstudio my_rstudio:0.1
(you can add any additional arguments like -v to above).
I'm facing an unexpected error when running docker build, and I say unexpected because I haven't changed my Dockerfile for a while, and it had worked fine for the last time two weeks ago, but now I'm getting the following error:
failed to create endpoint optimistic_spence on network bridge: failed to add the host (veth9fc3a03) <=> sandbox (veth15abfd6) pair interfaces: operation not supported
In case it is of any help:
Docker version is 18.06.0-ce, build 0ffa8257ec
I don't see any container with docker ps
Systemd returns an active status for the docker process (sudo systemctl status docker)
Build command is: docker build -t user/repo:tag .
Dockerfile looks like:
FROM alpine:3.4
LABEL version="current version"
LABEL description="A nice description."
LABEL maintainer="my#email.com"
RUN apk update && apk add \
gcc \
g++ \
make \
git \
&& git clone https://gitlab.com/user/repo.git \
&& cd repo \
&& make \
&& cp program /bin \
&& rm -r /repo \
&& apk del g++ make git
WORKDIR /tmp
ENTRYPOINT ["program"]
Does anybody understand what is going on? Thank you!
EDIT
When combined with the --network option, the error changes a little, but it won't fix the problem. For example, --network=host gives the following:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
2 errors; 11 distinct packages available
The command '/bin/sh -c apk update && apk add gcc g++ make git && git clone https://gitlab.com/user/repo.git && cd repo && make && cp program /bin && rm -r /repo && apk del g++ make git' returned a non-zero code: 2
Had the same error and systemctl restart docker nor pruning og images & system did not do it for me, I ended up rebooting my computer which seems o have resolved the issue.
It looks that something wrong happened with any docker network bridge and it doesn't let you to create the same because is "zombie".
Try with following steps:
docker network prune, and if it doesn't work, try with:
docker system prune <-- Careful, this also will purge your named volumes contents, i.e, volumes that are not assigned to a container. So, if you have volumes assigned to a container, you should have to re-build/create containers.
/etc/init.d/docker restart
Show me what happens and let's see, actually I need more info about your problem if it doesn't solve it to you.
Two times I have faced this issue and the way to fix it has always been the same. I'm posting in case it can be of any help for somebody:
First, make sure that the DNS server is properly set up (eg. setting DNS to 1.1.1.1).
Second, restarting the docker daemon.
For those using systemd in Linux, systemctl restart did not the job for me. I had to stop and start docker to make it work. After that, I could login and pull images again.
I successfully shelled to a Docker container using:
docker exec -i -t 69f1711a205e bash
Now I need to edit file and I don't have any editors inside:
root#69f1711a205e:/# nano
bash: nano: command not found
root#69f1711a205e:/# pico
bash: pico: command not found
root#69f1711a205e:/# vi
bash: vi: command not found
root#69f1711a205e:/# vim
bash: vim: command not found
root#69f1711a205e:/# emacs
bash: emacs: command not found
root#69f1711a205e:/#
How do I edit files?
As in the comments, there's no default editor set - strange - the $EDITOR environment variable is empty. You can log in into a container with:
docker exec -it <container> bash
And run:
apt-get update
apt-get install vim
Or use the following Dockerfile:
FROM confluent/postgres-bw:0.1
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "vim"]
Docker images are delivered trimmed to the bare minimum - so no editor is installed with the shipped container. That's why there's a need to install it manually.
EDIT
I also encourage you to read my post about the topic.
If you don't want to add an editor just to make a few small changes (e.g., change the Tomcat configuration), you can just use:
docker cp <container>:/path/to/file.ext .
which copies it to your local machine (to your current directory). Then edit the file locally using your favorite editor, and then do a
docker cp file.ext <container>:/path/to/file.ext
to replace the old file.
You can use cat if it's installed, which will most likely be the case if it's not a bare/raw container. It works in a pinch, and ok when copy+pasting to a proper editor locally.
cat > file
# 1. type in your content
# 2. leave a newline at end of file
# 3. ctrl-c / (better: ctrl-d)
cat file
cat will output each line on receiving a newline. Make sure to add a newline for that last line. ctrl-c sends a SIGINT for cat to exit gracefully. From the comments you see that you can also hit ctrl-d to denote end-of-file ("no more input coming").
Another option is something like infilter which injects a process into the container namespace with some ptrace magic: https://github.com/yadutaf/infilter
To keep your Docker images small, don't install unnecessary editors. You can edit the files over SSH from the Docker host to the container:
vim scp://remoteuser#containerip//path/to/document
You can use cat if installed, with the > caracter.
Here is the manipulation :
cat > file_to_edit
#1 Write or Paste you text
#2 don't forget to leave a blank line at the end of file
#3 Ctrl + C to apply configuration
Now you can see the result with the command
cat file
For common edit operations I prefer to install vi (vim-tiny), which uses only 1491 kB or nano which uses 1707 kB.
In other hand vim uses 28.9 MB.
We have to remember that in order for apt-get install to work, we have to do the update the first time, so:
apt-get update
apt-get install vim-tiny
To start the editor in CLI we need to enter vi.
You can open existing file with
cat filename.extension
and copy all the existing text on clipboard.
Then delete old file with
rm filename.extension
or rename old file with
mv old-filename.extension new-filename.extension
Create new file with
cat > new-file.extension
Then paste all text copied on clipboard, press Enter and exit with save by pressing ctrl+z. And voila no need to install any kind of editors.
Sometime you must first run the container with root:
docker exec -ti --user root <container-id> /bin/bash
Then in the container, to install Vim or something else:
apt-get install vim
I use "docker run" (not "docker exec"), and I'm in a restricted zone where we cannot install an editor. But I have an editor on the Docker host.
My workaround is: Bind mount a volume from the Docker host to the container (https://docs.docker.com/engine/reference/run/#/volume-shared-filesystems), and edit the file outside the container. It looks like this:
docker run -v /outside/dir:/container/dir
This is mostly for experimenting, and later I'd change the file when building the image.
After you shelled to the Docker container, just type:
apt-get update
apt-get install nano
You can just edit your file on host and quickly copy it into and run it inside the container. Here is my one-line shortcut to copy and run a Python file:
docker cp main.py my-container:/data/scripts/ ; docker exec -it my-container python /data/scripts/main.py
If you use Windows container and you want change any file, you can get and use Vim in Powershell console easily.
To shelled to the Windows Docker container with PowerShell:
docker exec -it <name> powershell
First install Chocolatey package manager
Invoke-WebRequest https://chocolatey.org/install.ps1 -UseBasicParsing | Invoke-Expression;
Install Vim
choco install vim
Refresh ENVIRONMENTAL VARIABLE
You can just exit and shell back to the container
Go to file location and Vim it vim file.txt
See Stack Overflow question
sed edit file in place
It would be a good option here, if:
To modify a large file, it's impossible to use cat.
Install Vim is not allowed or takes too long.
My situation is using the MySQL 5.7 image when I want to change the my.cnf file, there is no vim, vi, and Vim install takes too long (China Great Firewall). sed is provided in the image, and it's quite simple. My usage is like
sed -i /s/testtobechanged/textwanted/g filename
Use man sed or look for other tutorials for more complex usage.
It is kind of screwy, but in a pinch you can use sed or awk to make small edits or remove text. Be careful with your regex targets of course and be aware that you're likely root on your container and might have to re-adjust permissions.
For example, removing a full line that contains text matching a regex:
awk '!/targetText/' file.txt > temp && mv temp file.txt
(More)
If you can only shell into container with bin/sh (in case bin/bash doesn't work)
and apt or apt-get doesn't work in the container, check whether apk is installed by entering apk in command prompt inside the container.
If yes, you can install nano as follows:
apk add nano
then nano will work as usual
An easy way to edit a few lines would be:
echo "deb http://deb.debian.org/debian stretch main" > sources.list
You can install nano
yum install nano
You can also use a special container which will contain only the command you need: Vim. I chose python-vim. It assumes that the data you want to edit are in a data container built with the following Dockerfile:
FROM debian:jessie
ENV MY_USER_PASS my_user_pass
RUN groupadd --gid 1001 my_user
RUN useradd -ms /bin/bash --home /home/my_user \
-p $(echo "print crypt("${MY_USER_PASS:-password}", "salt")" | perl) \
--uid 1001 --gid 1001 my_user
ADD src /home/my_user/src
RUN chown -R my_user:my_user /home/my_user/src
RUN chmod u+x /home/my_user/src
CMD ["true"]
You will be able to edit your data by mounting a Docker volume (src_volume) which will be shared by your data container (src_data) and the python-vim container.
docker volume create --name src_volume
docker build -t src_data .
docker run -d -v src_volume:/home/my_user/src --name src_data_1 src_data
docker run --rm -it -v src_volume:/src fedeg/python-vim:latest
That way, you do not change your containers. You just use a special container for this work.
First login as root :
docker run -u root -ti bash
Type following commands:
apt-get update &&
apt-get install nano
docker comes up with no editors. so simply install vim, 36MB space don't kill your docker!
Make sure to update the container before trying to install the editor.
apt-get update
apt-get install nano vi