I want to create a shareable image which can be docker loaded into another docker installation directly from docker build
e.g.
Dockerfile:
FROM my-app/env as builder
COPY /my-app /my-app
RUN /my-app/build.sh
FROM ubuntu:20.04
COPY --from=builder /my-app/build/my-app.exe /bin
RUN DEBIAN_FRONTEND=noninteractive apt-get update \
&& apt-get install -y --no-install-recommends \
***some stuff required at runtime*** \
&& apt-get autoclean \
&& apt-get autoremove \
&& ldconfig
ENTRYPOINT ["my-app.exe"]
CMD ["--help"]
Running something like:
/my-app$ docker build -t my-app/exe .
will produce an image which I can use with
$ docker run my-app/exe
If I want to now share this, I need to do:
$ docker save -o my-app.tar my-app/exe
which creates the archive my-app.tar which can be shared with another system running the same arch/OS and used by:
/my-othersystem/my-app$ docker load < my-app.tar
And
$ docker run my-app/exe
now works on the other system.
HOWEVER
This requires building the image into the build system docker repository then saving the image to a file. I don't plan to run the exeutable on the build system so don't want it taking up space.
You can do:
/my-app$ DOCKER_BUILDKIT=1 docker build --output type=tar,dest=out.tar .
But this creates a FS, not an image. I want it to be exported as a docker image compatible with docker load directly, is this possible?
Related
I am trying to build a Docker image from a dockerfile command I received from the previous developer:
bash-5.1$ ls
data_collection demo.py examples requirements.txt start.py
demonstrateur.ipynb Dockerfile README.md serious_game test
bash-5.1$ docker build Dockerfile .
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
I also tried with
bash-5.1$ docker build -t serious-game:0.0.1 -t serious-game:latest Dockerfile .
and already completely reinstalled docker by following this tutorial but it gave the same error.
Here is my Dockerfile content:
bash-5.1$ cat Dockerfile
FROM nvidia/cuda:10.2-base-ubuntu18.04
MAINTAINER me
EXPOSE 5555
EXPOSE 8886
ENV DEBIAN_FRONTEND noninteractive
ENV WD=/home/serious-game/
WORKDIR ${WD}
# Add git and ssh
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get -y install git ssh pkg-config python3-pip python3-opencv
# Dépendances python
COPY requirements.txt /requirements.txt
RUN cd / && \
python3 -m pip install --upgrade pip && \
pip3 install -r requirements.txt
CMD ["start.py"]
If you are trying to build an image from a local Dockerfile, given your current bash location is in the same folder where Dockerfile resides - all you have to do is
docker build .
As written in the docs, Docker uses the file named Dockerfile by default. If you want to specify the file you can use the option --file or -f of the docker build command.
In your case you can just use to solve your problem:
docker build -t serious-game:0.0.1 -t serious-game:latest .
But if you want to specify another file named TestDockerfile (example for testing):
docker build -t serious-game:0.0.1 -t serious-game:latest -f TestDockerfile .
I'm trying to use the docker image of a specific software (picard), and that image is designed to be run interactively, in fact, an already built docker image is provided via Dockerhub:
docker pull broadinstitute/picard
This image works perfectly will the following command:
sudo docker run -i -t -v $PWD:/usr/working broadinstitute/picard
So that within the image one could launch the actual program like:
java -jar /usr/picard/picard.jar [COMMAND] [OPTIONS] ...
What I am trying to accomplish is to execute this image without entering in an interactive shell, simply as:
sudo docker run --rm -v $PWD:/usr/working broadinstitute/picard [COMMAND] [OPTIONS] ...
As far as I understand, this could be done by creating an ENTRYPOINT in the Dockerfile (see it in appendix below), but addind the following line at the bottom of the Dockerfile will not work:
ENTRYPOINT ["java -jar /usr/picard/picard.jar"]
Instead, when I run the image as above, no output is generated, and if a specific command is called (e.g. CreateSequenceDictionary), I get the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"CreateSequenceDictionary\": executable file not found in $PATH": unknown.
What am I missing?
Dockerfile
The dockerfile can be found in the github repo at: https://github.com/broadinstitute/picard/blob/master/Dockerfile. It looks like the following:
FROM openjdk:8
MAINTAINER Broad Institute DSDE <dsde-engineering#broadinstitute.org>
ARG build_command=shadowJar
ARG jar_name=picard.jar
# Install ant, git for building
RUN apt-get update && \
apt-get --no-install-recommends install -y --force-yes \
git \
r-base \
ant && \
apt-get clean autoclean && \
apt-get autoremove -y
# Assumes Dockerfile lives in root of the git repo. Pull source files into container
COPY / /usr/picard/
WORKDIR /usr/picard
# Build the distribution jar, clean up everything else
RUN ./gradlew ${build_command} && \
mv build/libs/${jar_name} picard.jar && \
./gradlew clean && \
rm -rf src && \
rm -rf gradle && \
rm -rf .git && \
rm gradlew && \
rm build.gradle
RUN mkdir /usr/working
WORKDIR /usr/working
The problem is how the ENTRYPOINT is defined.
It should be
ENTRYPOINT ["java", "-jar", "/usr/picard/picard.jar"]
src: https://docs.docker.com/v17.09/engine/reference/builder/#entrypoint
I just learnt about other possible alternative that does not require to modify the original dockerfile, which is overriding the default ENTRYPOINT from the CLI (even if the original image does not define one).
This can be done with the --entrypoint option, but it will only take the name of the file to be executed within the image. If additional arguments are to be used, these must be called after the image name. For example:
docker run --entrypoint="java" $PWD:/usr/working broadinstitute/picard -jar /usr/picard/picard.jar [COMMAND] [OPTIONS]
This blog explains a bit more on the topic.
I have the following Dockerfile:
FROM ubuntu:16.04
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y \
git \
make \
python-pip \
python2.7 \
python2.7-dev \
ssh \
&& apt-get autoremove \
&& apt-get clean
ARG password
ARG username
ENV password $password
ENV username $username
RUN pip install git+http://$username:$password#org.bitbucket.com/scm/do/repo.git
I use the following commands to build the image from this Dockerfile:
docker build -t myimage:v1 --build-arg password="somepassoword" --build-arg username="someuser" .
However, in the build log the username and password that I pass as --build-arg are visible.
Step 8/8 : RUN pip install git+http://$username:$password#org.bitbucket.com/scm/do/repo.git
---> Running in 650d9423b549
Collecting git+http://someuser:somepassword#org.bitbucket.com/scm/do/repo.git
How to hide them? Or is there a different way of passing the credentials in the Dockerfile?
Update
You know, I was focusing on the wrong part of your question. You shouldn't be using a username and password at all. You should be using access keys, which permit read-only access to private repositories.
Once you've created an ssh key and added the public component to your repository, you can then drop the private key into your image:
RUN mkdir -m 700 -p /root/.ssh
COPY my_access_key /root/.ssh/id_rsa
RUN chmod 700 /root/.ssh/id_rsa
And now you can use that key when installing your Python project:
RUN pip install git+ssh://git#bitbucket.org/you/yourproject.repo
(Original answer follows)
You would generally not bake credentials into an image like this. In addition to the problem you've already discovered, it makes your image less useful because you would need to rebuild it every time your credentials changed, or if more than one person wanted to be able to use it.
Credentials are more generally provided at runtime via one of various mechanisms:
Environment variables: you can place your credentials in a file, e.g.:
USERNAME=myname
PASSWORD=secret
And then include that on the docker run command line:
docker run --env-file myenvfile.env ...
The USERNAME and PASSWORD environment variables will be available to processes in your container.
Bind mounts: you can place your credentials in a file, and then expose that file inside your container as a bind mount using the -v option to docker run:
docker run -v /path/to/myfile:/path/inside/container ...
This would expose the file as /path/inside/container inside your container.
Docker secrets: If you're running Docker in swarm mode, you can expose your credentials as docker secrets.
It's worse than that: they're in docker history in perpetuity.
I've done two things here in the past that work:
You can configure pip to use local packages, or to download dependencies ahead of time into "wheel" files. Outside of Docker you can download the package from the private repository, giving the credentials there, and then you can COPY in the resulting .whl file.
pip install wheel
pip wheel --wheel-dir ./wheels git+http://$username:$password#org.bitbucket.com/scm/do/repo.git
docker build .
COPY ./wheels/ ./wheels/
RUN pip install wheels/*.whl
The second is to use a multi-stage Dockerfile where the first stage does all of the installation, and the second doesn't need the credentials. This might look something like
FROM ubuntu:16.04 AS build
RUN apt-get update && ...
...
RUN pip install git+http://$username:$password#org.bitbucket.com/scm/do/repo.git
FROM ubuntu:16.04
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install \
python2.7
COPY --from=build /usr/lib/python2.7/site-packages/ /usr/lib/python2.7/site-packages/
COPY ...
CMD ["./app.py"]
It's worth double-checking in the second case that nothing has gotten leaked into your final image, because the ARG values are still available to the second stage.
For me, I created a bash file call set-up-cred.sh.
Inside set-up-cred.sh
echo $CRED > cred.txt;
Then, in Dockerfile,
RUN bash set-up-cred.sh;
...
RUN rm cred.txt;
This is for hiding echoing credential variables.
I have a dockerfile look like this
FROM ubuntu
MAINTAINER abc <abc.yur#gmail.com>
RUN apt-get update
RUN apt-get install nano
RUN apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:longsleep/golang-backports
RUN apt-get update
RUN apt-get -y install golang-go git
RUN mkdir /work
ENV GOPATH=/work
RUN go get github.com/abc/golang
RUN go build github.com/abc/golang
CMD /golang -addr $ADDR -workers $WORKERS
So I want to build and run container but after the building (docker build .) I can not run this container. So when I am running docker ps -a or docker ps there is not container to run
docker build .
This creates the image and not a container. You need to use
docker images
To get the list of images.
docker ps will show when you run a container using something like below
docker run -d <image>
I'm trying to create a VM with docker and boot2docker. I've made the following Dockerfile, which I'm trying to run through the command line
docker run Dockerfile
Immidiatly it says exactly this:
Unable to find image 'Dockerfile:latest' locally
FATA[0000] Invalid repository name <Dockerfile>, only [a-z0-9_.] are allowed
Dockerfile:
FROM ubuntu:latest
#Oracle Java7 install
RUN apt-get install software-properties-common -y
RUN apt-get update
RUN add-apt-repository -y ppa:webupd8team/java
RUN apt-get update
RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
RUN apt-get install -y oracle-java7-installer
#Jenkins install
RUN wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
RUN sudo echo "deb http://pkg.jenkins-ci.org/debian binary/" >> /etc/apt/sources.list
RUN apt-get update
RUN apt-get install --force-yes -y jenkins
RUN sudo service jenkins start
#Zip support install
RUN apt-get update
RUN apt-get -y install zip
#Unzip hang.zip
RUN unzip -o /var/jenkins/hang.zip -d /var/lib/jenkins/
RUN chown -R jenkins:jenkins /vaR/lib/jenkins
RUN service jenkins restart
EXEC tail -f /etc/passwd
EXPOSE 8080
I am in the directory where the Dockerfile is, when trying to run this command.
Ignore the zip part, as that's for later use
You should run docker build first (which actually uses your Dockerfile):
docker build --tag=imagename .
Or
docker build --tag=imagename -f yourDockerfile .
Then you would use that image tag to docker run it:
docker run imagename
There are tools that can provide this type of feature.
We have achieved using docker compose, though you have to go through
(https://docs.docker.com/compose/overview/)
docker-compose up
but you can also do as work around
$ docker build -t foo . && docker run foo.