I am writing Dockerfile in which trying to download file from s3 to local using aws cli and ADD those files to docker container as below following this page
FROM nrel/energyplus:8.9.0
RUN apt-get update && \
apt-get install -y \
python3 \
python3-pip \
python3-setuptools \
groff \
less \
&& pip3 install --upgrade pip \
&& apt-get clean
RUN pip3 --no-cache-dir install --upgrade awscli
ADD . /var/simdata
WORKDIR /var/simdata
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_REGION=ap-northeast-1
RUN aws s3 cp s3://file/to/path .
RUN mkdir -p /var/simdata
ENTRYPOINT [ "/bin/bash" ]
After running docker build -it <container id>, I expected that I can find the file downloaded from s3 in container, but I could not.
Does anyone know where I can find this file downloaded from s3?
The file should be in /var/simdata.
Since you do not change your work dir before downloading the file from s3, your image uses nrel/energyplus:8.9.0 default work dir.
$ docker pull nrel/energyplus:8.9.0
$ docker inspect --format '{{.ContainerConfig.WorkingDir}}' nrel/energyplus:8.9.0
/var/simdata
A better approach would be download the file to your local from s3 and in Dockerfile copy that file to /var/simdata
COPY file /var/simdata/file
This will give you more control.
Related
I am new to docker. I build a docker image using the below code
FROM ubuntu
ADD requirements.txt .
RUN apt-get update && \
apt-get install -y python3 && \
apt install python3-pip -y && \
apt-get install -y libglib2.0-0 \
libsm6 \
libxrender1 \
libxext6
RUN python3 -m pip install -r requirements.txt
COPY my_code /container/home/user
ENV PYTHONPATH /container/home/user/program_dir_1
RUN apt install -y libgl1
WORKDIR /container/home/user
CMD python3 program_dir_1/program_dir_2/program_dir_3/main.py
Now I have a local dir /home/host/local_dir. I want to write/copy all the file that the program creates during the runtime to this local dir.
I am using the below command to bind the volume
docker run -it --volume /home/host/local_dir:/container/home/user my_docker_image
It is giving me an error that
program_dir_1/program_dir_2/program_dir_3/main.py [Errno 2] No such file or directory
When I run the below command
docker run -it --volume /home/host/local_dir:/container/home/user my_docker_image pwd
It is giving the path to the host dir which I linked. It seems like it is also switching the working directory to the host volume to which I am linking.
Can anyone please help me to understand how to copy all the files and data generated using the working directory of the container to the directory of the host?
PS: I have used the below StackOverflow link and tried to understand but didn't get any success
How to write data to host file system from Docker container # Found one solution but got the error that
docker run -it --rm --volume v_mac:/home/host/local_dir --volume v_mac:/container/home/user my_docker_image cp -r /home/host/local_dir /container/home/user
Docker: Copying files from Docker container to host # This is not much of use as I assume the container should be in a running state. In my mine it exited after completion of the program
I am trying to build a Docker image from a dockerfile command I received from the previous developer:
bash-5.1$ ls
data_collection demo.py examples requirements.txt start.py
demonstrateur.ipynb Dockerfile README.md serious_game test
bash-5.1$ docker build Dockerfile .
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
I also tried with
bash-5.1$ docker build -t serious-game:0.0.1 -t serious-game:latest Dockerfile .
and already completely reinstalled docker by following this tutorial but it gave the same error.
Here is my Dockerfile content:
bash-5.1$ cat Dockerfile
FROM nvidia/cuda:10.2-base-ubuntu18.04
MAINTAINER me
EXPOSE 5555
EXPOSE 8886
ENV DEBIAN_FRONTEND noninteractive
ENV WD=/home/serious-game/
WORKDIR ${WD}
# Add git and ssh
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get -y install git ssh pkg-config python3-pip python3-opencv
# Dépendances python
COPY requirements.txt /requirements.txt
RUN cd / && \
python3 -m pip install --upgrade pip && \
pip3 install -r requirements.txt
CMD ["start.py"]
If you are trying to build an image from a local Dockerfile, given your current bash location is in the same folder where Dockerfile resides - all you have to do is
docker build .
As written in the docs, Docker uses the file named Dockerfile by default. If you want to specify the file you can use the option --file or -f of the docker build command.
In your case you can just use to solve your problem:
docker build -t serious-game:0.0.1 -t serious-game:latest .
But if you want to specify another file named TestDockerfile (example for testing):
docker build -t serious-game:0.0.1 -t serious-game:latest -f TestDockerfile .
This question already has answers here:
Docker: Copying files from Docker container to host
(27 answers)
Closed 1 year ago.
I have a docker file
FROM ubuntu:20.04
################################
### INSTALL Ubuntu build tools and prerequisites
################################
# Install build base
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
build-essential \
git \
subversion \
sharutils \
vim \
asciidoc \
binutils \
bison \
flex \
texinfo \
gawk \
help2man \
intltool \
libelf-dev \
zlib1g-dev \
libncurses5-dev \
ncurses-term \
libssl-dev \
python2.7-dev \
unzip \
wget \
rsync \
gettext \
xsltproc && \
apt-get clean && rm -rf /var/lib/apt/lists/*
ARG FORCE_UNSAFE_CONFIGURE=1
RUN git clone https://git.openwrt.org/openwrt/openwrt.git
WORKDIR /openwrt
RUN ./scripts/feeds update -a && ./scripts/feeds install -a
COPY .config /openwrt/.config
RUN mkdir files
WORKDIR /files
RUN mkdir etc
WORKDIR /etc
RUN mkdir uci-defaults
WORKDIR /uci-defaults
COPY xx_custom /openwrt/files/etc/uci-defaults/xx_custom
WORKDIR /openwrt
RUN make -j 4
RUN ls /openwrt/bin/targets/ramips/mt76x8
WORKDIR /root
CMD ["bash"]
I want to copy all the files inside the folder mt76x8 to the host. I want to that inside the dockerfile so that when I run the docker file I should get the generated files in my host.
How can I do that?
you can use the volume mount to access the docker-generated artifacts on the host machine.
you can also run the command
docker cp to copy the files to the host machine.
if don't want to use the docker command as mention only option is to use the volume.
you can also use docker create once the docker image is ready to create the writable layer and copy data.
You have two choices.
Use docker volumes to map the /openwrt/bin/targets/ramips/mt76x8 folder when you are running the container. i.e. docker run -v {VoluneName}:/openwrt/bin/targets/ramips/mt76x8. All of the files in the mt76x8 folder would be available in the volume folder. If you are using Linux then you will find the docker volumes in /var/lib/docker/volumes/
You can use docker cp command to copy data from container to the host machine. Here is an example
When we use Docker it's very easy push and pull image in a public repository in our https://hub.docker.com but this repository it's free only for public image(only one can be private).
Currently it's possible to execute a reverse engineering of a public image in repository and read the source code of project ?
You can check how an image was created using docker history <image-name> --no-trunc
Update:
Check dive which is a very nice tool that allows you to views image layers.
As yamenk said docker history is the key to this.
As https://github.com/CenturyLinkLabs/dockerfile-from-image is broken, you can use recent
https://hub.docker.com/r/dduvnjak/dockerfile-from-image/
Extract from the site
Note that the script only works against images that exist in your local image
repository (the stuff you see when you type docker images). If you want to
generate a Dockerfile for an image that doesn't exist in your local repo
you'll first need to docker pull it.
For example, you can run it agains itself, to see the code
$ docker run --rm -v /run/docker.sock:/run/docker.sock centurylink/dockerfile-from-image ruby
FROM buildpack-deps:latest
RUN useradd -g users user
RUN apt-get update && apt-get install -y bison procps
RUN apt-get update && apt-get install -y ruby
ADD dir:03090a5fdc5feb8b4f1d6a69214c37b5f6d653f5185cddb6bf7fd71e6ded561c in /usr/src/ruby
WORKDIR /usr/src/ruby
RUN chown -R user:users .
USER user
RUN autoconf && ./configure --disable-install-doc
RUN make -j"$(nproc)"
RUN make check
USER root
RUN apt-get purge -y ruby
RUN make install
RUN echo 'gem: --no-rdoc --no-ri' >> /.gemrc
RUN gem install bundler
ONBUILD ADD . /usr/src/app
ONBUILD WORKDIR /usr/src/app
ONBUILD RUN [ ! -e Gemfile ] || bundle install --system
You can use laniksj/dfimage to reverse engineering of an image.
For example:
# docker run -v /var/run/docker.sock:/var/run/docker.sock laniksj/dfimage <YOUR_IMAGE_ID>
FROM node:12.4.0-alpine
RUN /bin/sh -c apk update
RUN /bin/sh -c apk -Uuv add groff less python py-pip
RUN /bin/sh -c pip install awscli
RUN /bin/sh -c apk --purge -v del py-pip
RUN /bin/sh -c rm /var/cache/apk/*
RUN /bin/sh -c apk add --no-cache curl
ADD dir:4afc740ff29e4a32a34617d2715e5e5dc8740f357254bc6d3f9362bb04af0253 in /app
COPY file:b57abdb61ae72f3a25be67f719b95275da348f9dfb63fb4ff67410a595ae1dfd in /usr/local/bin/
WORKDIR /app
RUN /bin/sh -c npm install
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["node" "app.js"]
dfimage and dockerfile-from-image are broken
dedockify works
imageName=ruby:latest
docker pull $imageName
docker images # -> get imageId
imageId=xxxxxxxxxxxx
# write to Dockerfile
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock mrhavens/dedockify $imageId >Dockerfile
I'm trying to learn Synatxnet. I have it running through Docker. But I really dont know much about either program Synatxnet or Docker. On the Github Sytaxnet page it says
The SyntaxNet models are configured via a combination of run-time
flags (which are easy to change) and a text format TaskSpec protocol
buffer. The spec file used in the demo is in
syntaxnet/models/parsey_mcparseface/context.pbtxt.
How exactly do I find the spec file to edit it?
I compiled SyntaxNet in a Docker container using these Instructions.
FROM java:8
ENV SYNTAXNETDIR=/opt/tensorflow PATH=$PATH:/root/bin
RUN mkdir -p $SYNTAXNETDIR \
&& cd $SYNTAXNETDIR \
&& apt-get update \
&& apt-get install git zlib1g-dev file swig python2.7 python-dev python-pip -y \
&& pip install --upgrade pip \
&& pip install -U protobuf==3.0.0b2 \
&& pip install asciitree \
&& pip install numpy \
&& wget https://github.com/bazelbuild/bazel/releases/download/0.2.2b/bazel-0.2.2b-installer-linux-x86_64.sh \
&& chmod +x bazel-0.2.2b-installer-linux-x86_64.sh \
&& ./bazel-0.2.2b-installer-linux-x86_64.sh --user \
&& git clone --recursive https://github.com/tensorflow/models.git \
&& cd $SYNTAXNETDIR/models/syntaxnet/tensorflow \
&& echo "\n\n\n" | ./configure \
&& apt-get autoremove -y \
&& apt-get clean
RUN cd $SYNTAXNETDIR/models/syntaxnet \
&& bazel test --genrule_strategy=standalone syntaxnet/... util/utf8/...
WORKDIR $SYNTAXNETDIR/models/syntaxnet
CMD [ "sh", "-c", "echo 'Bob brought the pizza to Alice.' | syntaxnet/demo.sh" ]
# COMMANDS to build and run
# ===============================
# mkdir build && cp Dockerfile build/ && cd build
# docker build -t syntaxnet .
# docker run syntaxnet
First, comment out the command line in the dockerfile, then create and cd into an empty directory on your host machine. You can then create a container from the image, mounting a directory in the container to your hard-drive:
docker run -it --rm -v /pwd:/tmp bash
You'll now have a bash session in the container. Copy the spec file into /tmp from /opt/tensorflow/syntaxnet/models/parsey_mcparseface/context.pbtxt (I'm guessing that's where it is given the info you've provided above -- I can't get your dockerfile to build an image so I can't confirm it; you can always run find . -name context.pbtxt from root to find it), and exit the container (ctrl-d or exit).
You now have the file on your host's hd ready to edit, but you really want it in a running container. If the directory it comes from contains only that file, then you can simply mount your host directory at that path in the container. If it contains other things, then you can use a, so called, bootstrap script to move the file from your mounted directory (in the example above, that's tmp) to its home location. Alternatively, you may be able to tell the software where to find the spec file with a flag, but that will take more research.