I'm building a custom rapidsai docker image based on its devel image. Here is the docker file.
FROM rapidsai/rapidsai-dev:0.19-cuda11.0-devel-ubuntu18.04-py3.7
# Defining working directory and adding source code
WORKDIR /usr/src/app
RUN echo "Make sure cuml is installed:"
RUN python -c "import cuml"
But when I built it with this command,
nvidia-docker build . -t test
it returns an error saying:
Step 4/4 : RUN python -c "import cuml"
---> Running in 553d12bf7e68
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cuml'
It seems that it can't recognize cuml library which is already a part of the libraries of the base image. Why it can't import it?
Just fixed it. This will work without any issue
FROM rapidsai/rapidsai-dev:0.19-cuda11.0-devel-ubuntu18.04-py3.7
# Defining working directory and adding source code
WORKDIR /usr/src/app
RUN echo "conda activate rapids" >> ~/.bashrc
SHELL ["/bin/bash", "--login", "-c"]
then install some libraries..and
RUN export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/libcuda.so.1:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
RUN export LD_LIBRARY_PATH=/usr/local/cuda-11.0/compat/libcuda.so.1:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
COPY test_cuml.py .
RUN echo "conda activate rapids" >> ~/.bashrc
SHELL ["/bin/bash", "--login", "-c"]
ENTRYPOINT ["/opt/conda/envs/rapids/bin/python", "test_cuml.py"]
Related
I want to preface this in saying that I am very new to docker and have just got my feet wet with using it. In my Docker file that I run to build the container I install a program that sets some env variables. Here is my Docker file for context.
FROM python:3.8-slim-buster
COPY . /app
RUN apt-get update
RUN apt-get install wget -y
RUN wget http://static.matrix-vision.com/mvIMPACT_Acquire/2.40.0/install_mvGenTL_Acquire.sh
RUN wget http://static.matrix-vision.com/mvIMPACT_Acquire/2.40.0/mvGenTL_Acquire-x86_64_ABI2-2.40.0.tgz
RUN chmod +x ./install_mvGenTL_Acquire.sh
RUN ./install_mvGenTL_Acquire.sh -u
RUN apt-get install -y python3-opencv
RUN pip3 install USSCameraTools
WORKDIR /app
CMD python3 main.py
After executing the build docker command the program "mvGenTL_Acquire.sh" sets env inside the container. I need these variables to be set when executing the run docker command. But when checking the env variables after running the image it is not set. I know I can pass them in directly but would like to use the ones that are set from the install in the build.
Any help would be greatly appreciated, thanks!
For running a bash script when your container is creating:
make an script.sh file:
#!/bin/bash
your commands here
If you are using an alpine image, you must use #!/bin/sh instead of #!/bin/bash on the first line of your bash file.
Now in your Dockerfile copy your bash file in the container and use the ENTRYPOINT instruction for running this file when the container is creating:
.
.
.
COPY script.sh /
RUN chmod +x /script.sh
.
.
.
ENTRYPOINT ["/script.sh"]
Notice that in the ENTRYPOINT instruction use your bash file address in your image.
Now when you create a container, the script.sh file will be executed.
This is my docker file
FROM public.ecr.aws/i7d4o1h8/miniconda3:4.10.3p0
RUN pip install --upgrade pip
COPY condaEnv.yml .
RUN conda env create -f condaEnv.yml python=3.9.7
RUN pip install sagemaker-inference
COPY inference_code.py /opt/ml/code/
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code/
ENV SAGEMAKER_PROGRAM inference_code.py
ENTRYPOINT ["python", "/opt/ml/code/inference_code.py"]
When I run docker build with the command docker build -t docker_name ., it is successful, at the end I see Successfully tagged docker_name:latest
But when I am trying to run the docker image it gives
Traceback (most recent call last):
File "/opt/ml/code/inference_code.py", line 4, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
But in the condaEnv.yml file I have the pandas defined as
name: plato_vrar
channels:
- conda-forge
- defaults
dependencies:
- pandas=1.3.4
- pip=21.2.4
prefix: plato_vrar/
What am I missing here?
In anaconda, creating an environment means only the environment is prepared. You need also to activete it using conda activate <ENV_NAME> then python is correctly linked to the anaconda version rather than the system version. Refer to the conda document for more details: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment
I am trying to run my docker image, but I am receiving the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"python3\": executable file not found in $PATH": unknown.
ERRO[0000] error waiting for container: context canceled
Here is my Dockerfile:
FROM python:3.6-alpine3.7
RUN apk add --no-cache python3-dev \
&& pip3 install --upgrade pip
RUN apk add --no-cache --update \
python3 python3-dev gcc \
gfortran musl-dev
RUN apk add --no-cache libressl-dev musl-dev libffi-dev
RUN python3.6 -m pip install --upgrade pip
RUN apk --no-cache add git
RUN apk add mariadb-dev
WORKDIR /socialworks-api
COPY . /socialworks-api
RUN pip3 --no-cache-dir install -r requirements.txt
ENV PATH="/opt/gtk/bin:$env/development.env"
EXPOSE 5000
ENTRYPOINT ["python3"]
CMD ["app.py"]
I think the issue might be with the ENV PATH which I set. I have also tried setting it to:
ENV PATH="/opt/gtk/bin:${env/development.env}"
But I would receive the following error when building my dockerfile:
Step 11/14 : ENV PATH="/opt/gtk/bin:${env/development.env}"
failed to process "\"/opt/gtk/bin:${env/development.env}\"": missing ':' in substitution
Without setting the environment, my application won't run.
I have also tried running on my dockerfile this command:
RUN export $(grep -v '^#' ./env/development.env | xargs)
It build successfully, but when I enter this command to the terminal:
docker run -it flaskapp
I am receiving an error that it still unable to locate the env variables.
$ docker run -it flaskapp
Traceback (most recent call last):
File "app.py", line 11, in <module>
app.config.from_object("config.Config")
File "/usr/local/lib/python3.6/site-packages/flask/config.py", line 174, in from_object
obj = import_string(obj)
File "/usr/local/lib/python3.6/site-packages/werkzeug/utils.py", line 568, in import_string
__import__(import_name)
File "/socialworks-api/config.py", line 4, in <module>
class Config(object):
File "/socialworks-api/config.py", line 5, in Config
MYSQL_HOST = os.environ['MYSQL_HOST']
File "/usr/local/lib/python3.6/os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'MYSQL_HOST'
In your Dockerfile you specify
ENV PATH="/opt/gtk/bin:$env/development.env"
When you later run python3, the system looks in only these two directories, and when there's not a Python there, you get the error you see. It's probably in /usr/bin/python3, but the directory /usr/bin isn't in $PATH.
The simplest answer here is to delete this line entirely. Your Dockerfile creates neither a /opt/gtk nor a /development.env directory, so there's no files in either path that could be executed. If you do need to install custom software, putting it in a directory that's in the default system path (like /usr/local/bin) is a good approach.
If you do need to keep this line, make sure the existing $PATH stays as part of the value.
ENV PATH="/opt/gtk/bin:$PATH"
(Don't specify ENTRYPOINT ["python"]; combine this into a single CMD ["python3", "app.py"]. I'd encourage you to run docker run --rm -it yourimage /bin/sh to look around the filesystem and run debugging commands like env, but the ENTRYPOINT declaration breaks this use, and having the script file named in the CMD means it's still not the "container-as-command" pattern.)
When I build my docker image and run it using the following commands:
docker build -t iter1 .
docker run -it --rm --name iter1_run iter1
My application runs just fine. However when I try to attach a volume and execute the following command:
docker run -it --rm --name iter_run -v /Users/xxxx/Desktop/Docker_Builds/SingleDocker/xxxxxx:/usr/src/oce -w /usr/src/oce python:3 python oce_test.py
The file oce_test.py cant find Pandas.
Traceback (most recent call last):
File "oce_test.py", line 1, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
The content of my Dockerfile is as follows:
# Docker image
FROM python:3
# Copy requirements
COPY requirements.txt /
# Install Requirements
RUN pip install -r /requirements.txt
# Copy scripts needed for execution
COPY ./xxxx /usr/src/oce
# Establish a working directory
WORKDIR /usr/src/oce
# Execute required script
CMD ["python", "oce_test.py"]
The content of my requirements.txt is as follows:
numpy==1.18.1
pandas==1.0.1
matplotlib==3.1.3
scipy==1.4.1
Python-dateutil==2.8.1
David Maze answered this:
Your docker run command is running a plain python:3 image with no additional packages installed. If you want to use the image from your Dockerfile, but overwriting the application code in the image with arbitrary content from your host, use your image name iter1 instead. (You don't need to repeat the image's WORKDIR or CMD as docker run options.)
I have a Dockerfile in a directory called docker_test. The structure of docker_test is as follows:
M00618927A:docker_test i854319$ ls
Dockerfile hello_world.py
My dockerfile looks like below:
### Dockerfile
# Created by Baktaawar
# Pulling from base Python image
FROM python:3.6.7-alpine3.6
# author of file
LABEL maintainer="Baktawar"
# Set the working directory of the docker image
WORKDIR /docker_test
COPY . /docker_test
# packages that we need
RUN pip --no-cache-dir install numpy pandas jupyter
EXPOSE 8888
ENTRYPOINT ["python"]
CMD ["hello_world.py"]
I run the command
docker build -t dockerfile .
It starts the building process but then gives the following error in not being able to get the numpy etc installed in the image
Sending build context to Docker daemon 4.096kB
Step 1/8 : FROM python:3.6.7-alpine3.6
---> 8f30079779ef
Step 2/8 : LABEL maintainer="Baktawar"
---> Running in 7cf081021b1e
Removing intermediate container 7cf081021b1e
---> 581cf24fa4e6
Step 3/8 : WORKDIR /docker_test
---> Running in 7c58855c4332
Removing intermediate container 7c58855c4332
---> dae70a34626b
Step 4/8 : COPY . /docker_test
---> 432b174b4869
Step 5/8 : RUN pip --no-cache-dir install numpy pandas jupyter
---> Running in 972efa9336ed
Collecting numpy
Downloading https://files.pythonhosted.org/packages/cf/8d/6345b4f32b37945fedc1e027e83970005fc9c699068d2f566b82826515f2/numpy-1.16.2.zip (5.1MB)
Collecting pandas
Downloading https://files.pythonhosted.org/packages/81/fd/b1f17f7dc914047cd1df9d6813b944ee446973baafe8106e4458bfb68884/pandas-0.24.1.tar.gz (11.8MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 357, in get_provider
module = sys.modules[moduleOrReq]
KeyError: 'numpy'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-8c3o0ycd/pandas/setup.py", line 732, in <module>
ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
File "/tmp/pip-install-8c3o0ycd/pandas/setup.py", line 475, in maybe_cythonize
numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1142, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 359, in get_provider
__import__(moduleOrReq)
ModuleNotFoundError: No module named 'numpy'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-8c3o0ycd/pandas/
You are using pip version 18.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip --no-cache-dir install numpy pandas jupyter' returned a non-zero code: 1
How can I get this basic setup done?
You basically need to install the following on alpine, in order to be able to install numpy:
apk --no-cache add musl-dev linux-headers g++
Try the following Dockerfile:
### Dockerfile
# Created by Baktawar
# Pulling from base Python image
FROM python:3.6.7-alpine3.6
# author of file
LABEL maintainer="Baktawar"
# Set the working directory of the docker image
WORKDIR /app
COPY . /app
# Install native libraries, required for numpy
RUN apk --no-cache add musl-dev linux-headers g++
# Upgrade pip
RUN pip install --upgrade pip
# packages that we need
RUN pip install numpy && \
pip install pandas && \
pip install jupyter
EXPOSE 8888
ENTRYPOINT ["python"]
CMD ["hello_world.py"]
You may find this gist interresting:
https://gist.github.com/orenitamar/f29fb15db3b0d13178c1c4dd611adce2
And this package on alpine, is also of interrest I think:
https://pkgs.alpinelinux.org/package/edge/community/x86/py-numpy
Update
In order to tag the image properly, use the syntax:
docker build -f <dockerfile> -t <tagname:tagversion> <buildcontext>
For you, this would be:
docker build -t mypythonimage:0.1 .