I've made the following Dockerfile to build a python application:
FROM python:3.7
WORKDIR /app
# Install python dependencies
ADD requirements.txt /app/
RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt
# Copy sources
ADD . /app
# Run detection
CMD ["detect.py" ]
ENTRYPOINT ["python3"]
The requirements.txt file contains only a few dependencies, including opencv:
opencv-python
opencv-python-headless
filterpy==1.1.0
lap==0.4.0
paho-mqtt==1.5.1
numpy
Pillow
Building the Docker image works perfectly.
When I try to run the image, I got the following error:
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "detect.py", line 6, in <module>
import cv2
File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 5, in <module>
from .cv2 import *
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
Seems like the CV2 dependency is not satisfied.
Is there something I missed building the Dockerfile ?
I have tried to replace opencv-python-headless by python3-opencv but there is no matching distribution found.
libGL.so.1 could be found in libgl1, so you could add next to Dockerfile:
RUN apt update; apt install -y libgl1
Typically, docker images may remove many libraries to make the size small, these dependencies most probably could be already installed in your host system.
So, for me, I usually use dpkg -S in host system to find which package I needed, then install them in container:
shubuntu1#shubuntu1:~$ dpkg -S libGL.so.1
libgl1:amd64: /usr/lib/x86_64-linux-gnu/libGL.so.1
libgl1:amd64: /usr/lib/x86_64-linux-gnu/libGL.so.1.0.0
Related
This is my docker file
FROM public.ecr.aws/i7d4o1h8/miniconda3:4.10.3p0
RUN pip install --upgrade pip
COPY condaEnv.yml .
RUN conda env create -f condaEnv.yml python=3.9.7
RUN pip install sagemaker-inference
COPY inference_code.py /opt/ml/code/
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code/
ENV SAGEMAKER_PROGRAM inference_code.py
ENTRYPOINT ["python", "/opt/ml/code/inference_code.py"]
When I run docker build with the command docker build -t docker_name ., it is successful, at the end I see Successfully tagged docker_name:latest
But when I am trying to run the docker image it gives
Traceback (most recent call last):
File "/opt/ml/code/inference_code.py", line 4, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
But in the condaEnv.yml file I have the pandas defined as
name: plato_vrar
channels:
- conda-forge
- defaults
dependencies:
- pandas=1.3.4
- pip=21.2.4
prefix: plato_vrar/
What am I missing here?
In anaconda, creating an environment means only the environment is prepared. You need also to activete it using conda activate <ENV_NAME> then python is correctly linked to the anaconda version rather than the system version. Refer to the conda document for more details: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment
I have dockerfile that installs from requirements.txt file. When installing librosa lib it pulled many other needed libs, when installing pycparser i got this error:
Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone
may have tampered with them.
torch from https://files.pythonhosted.org/packages/8c/5d/faf0d8ac260c7f1eda7d063001c137da5223be1c137658384d2d45dcd0d5/torch-1.6.0-cp38-cp38-manylinux1_x86_64.whl#sha256=5357873e243bcfa804c32dc341f564e9a4c12addfc9baae4ee857fcc09a0a216 (from -r requirements.txt (line 4)):
Expected sha256 5357873e243bcfa804c32dc341f564e9a4c12addfc9baae4ee857fcc09a0a216
Got eb3c7b3621d64e9d9955ec0546729291338556d4ee8ccbf347169f574816f089
what's the problem with hashes? I did not indicate any hashes or ids in my reqs file.
my requirements file:
flask
pydub
scipy
torch
numpy
librosa
dockerfile:
FROM voice
RUN mkdir -p ./voice_flask/d
WORKDIR /voice_flask/d
COPY . /voice_flask/d
RUN pip install -r requirements.txt
CMD ["python", "server.py"]
I suggest you to install the packages fresh using --no-cache-dir
RUN pip install -r requirements.txt --no-cache-dir
see this Issue
I have a Dockerfile in a directory called docker_test. The structure of docker_test is as follows:
M00618927A:docker_test i854319$ ls
Dockerfile hello_world.py
My dockerfile looks like below:
### Dockerfile
# Created by Baktaawar
# Pulling from base Python image
FROM python:3.6.7-alpine3.6
# author of file
LABEL maintainer="Baktawar"
# Set the working directory of the docker image
WORKDIR /docker_test
COPY . /docker_test
# packages that we need
RUN pip --no-cache-dir install numpy pandas jupyter
EXPOSE 8888
ENTRYPOINT ["python"]
CMD ["hello_world.py"]
I run the command
docker build -t dockerfile .
It starts the building process but then gives the following error in not being able to get the numpy etc installed in the image
Sending build context to Docker daemon 4.096kB
Step 1/8 : FROM python:3.6.7-alpine3.6
---> 8f30079779ef
Step 2/8 : LABEL maintainer="Baktawar"
---> Running in 7cf081021b1e
Removing intermediate container 7cf081021b1e
---> 581cf24fa4e6
Step 3/8 : WORKDIR /docker_test
---> Running in 7c58855c4332
Removing intermediate container 7c58855c4332
---> dae70a34626b
Step 4/8 : COPY . /docker_test
---> 432b174b4869
Step 5/8 : RUN pip --no-cache-dir install numpy pandas jupyter
---> Running in 972efa9336ed
Collecting numpy
Downloading https://files.pythonhosted.org/packages/cf/8d/6345b4f32b37945fedc1e027e83970005fc9c699068d2f566b82826515f2/numpy-1.16.2.zip (5.1MB)
Collecting pandas
Downloading https://files.pythonhosted.org/packages/81/fd/b1f17f7dc914047cd1df9d6813b944ee446973baafe8106e4458bfb68884/pandas-0.24.1.tar.gz (11.8MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 357, in get_provider
module = sys.modules[moduleOrReq]
KeyError: 'numpy'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-8c3o0ycd/pandas/setup.py", line 732, in <module>
ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
File "/tmp/pip-install-8c3o0ycd/pandas/setup.py", line 475, in maybe_cythonize
numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1142, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 359, in get_provider
__import__(moduleOrReq)
ModuleNotFoundError: No module named 'numpy'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-8c3o0ycd/pandas/
You are using pip version 18.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip --no-cache-dir install numpy pandas jupyter' returned a non-zero code: 1
How can I get this basic setup done?
You basically need to install the following on alpine, in order to be able to install numpy:
apk --no-cache add musl-dev linux-headers g++
Try the following Dockerfile:
### Dockerfile
# Created by Baktawar
# Pulling from base Python image
FROM python:3.6.7-alpine3.6
# author of file
LABEL maintainer="Baktawar"
# Set the working directory of the docker image
WORKDIR /app
COPY . /app
# Install native libraries, required for numpy
RUN apk --no-cache add musl-dev linux-headers g++
# Upgrade pip
RUN pip install --upgrade pip
# packages that we need
RUN pip install numpy && \
pip install pandas && \
pip install jupyter
EXPOSE 8888
ENTRYPOINT ["python"]
CMD ["hello_world.py"]
You may find this gist interresting:
https://gist.github.com/orenitamar/f29fb15db3b0d13178c1c4dd611adce2
And this package on alpine, is also of interrest I think:
https://pkgs.alpinelinux.org/package/edge/community/x86/py-numpy
Update
In order to tag the image properly, use the syntax:
docker build -f <dockerfile> -t <tagname:tagversion> <buildcontext>
For you, this would be:
docker build -t mypythonimage:0.1 .
I tried to run the pyspark application.For this first I installed pyspark from pip then pulled openjdk:8 to set the JAVA_HOME variable
Dockerfile :
FROM python:3
ADD my_script.py /
COPY requirements.txt ./
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN pip install --no-cache-dir -r requirements.txt
CMD [ "python", "./my_script.py" ]
my_script.py :
from pyspark import SparkContext
from pyspark import SparkConf
#spark conf
conf1 = SparkConf()
conf1.setMaster("local[*]")
conf1.setAppName('hamza')
print(conf1)
sc = SparkContext(conf = conf1)
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
print(sqlContext)
Requirements.txt :
pyspark
numpy
Getting this error :
C:\Users\hrafiq\Desktop\sample>docker run -it --rm --name data2 my-python-app
<pyspark.conf.SparkConf object at 0x7f4bd933ba58>
/usr/local/lib/python3.7/site-packages/pyspark/bin/spark-class: line 71:
/usr/lib/jvm/java-8-openjdk-amd64//bin/java: No such file or directory
Traceback (most recent call last):
File "./my_script.py", line 14, in <module>
sc = SparkContext(conf = conf1)
File "/usr/local/lib/python3.7/site-packages/pyspark/context.py", line 115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/usr/local/lib/python3.7/site-packages/pyspark/context.py", line 298, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/usr/local/lib/python3.7/site-packages/pyspark/java_gateway.py", line 94, in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
So the question is If is not finding the java file then how will I find the that file? I know it is stored in some virtual hard disk which we dont have any access.
Any help would be appreciated
Thanks
Setting the JAVA_HOME env var is not enough. You need to actually install openjdk inside your docker image.
Your base image (python:3) is itself based on a Debian Stretch image. So you can use apt-get install to fetch the JDK :
FROM python:3
RUN apt-get update && \
apt-get install -y openjdk-8-jdk-headless && \
rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY my_script.py ./
CMD [ "python", "./my_script.py" ]
(In the above I have optimized the layers ordering so that you won't need to re-build the pip install layer when only your script changes)
In running the Docker build (using Jenkins CI), it fails on upgrading pip (last line of the docker file). I need it to upgrade version 8.1.1, as it suggest in the log, as my deploy fails on PIP versions mismatch.
Dockerfile
FROM ubuntu:14.04
FROM python:3.4
# Expose a port for gunicorn to listen on
EXPOSE 8002
# Make a workdir and virtualenv
WORKDIR /opt/documents_api
# Install everything else
ADD . /opt/documents_api
# Set some environment varialbes for PIP installation and db management
ENV CQLENG_ALLOW_SCHEMA_MANAGEMENT="True"
RUN apt-get update
RUN apt-get install -y python3-pip
RUN pip3 install --upgrade pip
Here's the error:
Step 15 : RUN pip3 install --upgrade pip
19:46:00 ---> Running in 84e2bcc850c0
19:46:04 Collecting pip
19:46:04 Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)
19:46:04 Installing collected packages: pip
19:46:04 Found existing installation: pip 7.1.2
19:46:04 Uninstalling pip-7.1.2:
19:46:05 Successfully uninstalled pip-7.1.2
19:46:10 Exception:
19:46:10 Traceback (most recent call last):
19:46:10 File "/usr/local/lib/python3.4/shutil.py", line 424, in _rmtree_safe_fd
19:46:10 os.unlink(name, dir_fd=topfd)
19:46:10 FileNotFoundError: [Errno 2] No such file or directory: 'pip'
19:46:10 You are using pip version 7.1.2, however version 8.1.1 is available.
When you use two FROM directives, docker creates two output images, that's why it's messed up.
First, remove FROM ubuntu:14.04 and don't apt-get update in a Dockerfile, it's a bad practice (your image will be different every time you build, defeating the whole purpose of containers/Docker).
Second, you can check official python images Dockerfile to know which version of pip is installed, for example, python:3.4 (it's already v8.1.1).
Third, there is a special image for you case (external application): python:3.4-onbuild. Your Dockerfile can be reduced to:
FROM python:3.4-onbuild
ENV CQLENG_ALLOW_SCHEMA_MANAGEMENT="True"
EXPOSE 8002
CMD python myapp.py
One last thing, try to use alpine based images, they're much smaller (for python, it's almost 10 time smaller than the ubuntu based).
turns out the host I was running had no outside (internet) access. So the upgrade was failing. We solved it by adding another package to the DTR that had the necessary version in it.
use /usr/bin/ for run pip. Example :
/usr/bin/pip install --upgrade pip
running this command solved the same problem for me (python 3.9):
RUN /usr/local/bin/python -m pip install --upgrade pip