RUN pip install -r requirements.txt not working inside a Dockerfile - docker

I am trying to build a new docker image by adding some python packages to my base docker image. The sudo docker build -t myimage1:cuda10.2 . runs without any issues but I cannot import any of the packages in the new image. I am using sudo docker run --gpus all -it myimage1:cuda10.2 to run the image. Can somebody help me understand what am I missing here?
Dockerfile
FROM nvcr.io/nvidia/rapidsai/rapidsai:cuda10.2-runtime-ubuntu18.04
WORKDIR /rapids/notebooks/
COPY requirements.txt ./
RUN pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
requirements.txt
ujson
This is the trace for docker build
Sending build context to Docker daemon 4.096kB
Step 1/5 : FROM nvcr.io/nvidia/rapidsai/rapidsai:cuda10.2-runtime-ubuntu18.04
---> 98901cabda0a
Step 2/5 : WORKDIR /rapids/notebooks/
---> Using cache
---> 162a9bb732c7
Step 3/5 : COPY requirements.txt ./
---> 7bb48a384987
Step 4/5 : RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt
---> Running in 5f3ca4ed3f93
Collecting pip
Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Uninstalling pip-20.0.2:
Successfully uninstalled pip-20.0.2
Successfully installed pip-20.2.4
Collecting ujson
Downloading ujson-4.0.1-cp38-cp38-manylinux1_x86_64.whl (181 kB)
Installing collected packages: ujson
Successfully installed ujson-4.0.1
Removing intermediate container 5f3ca4ed3f93
---> 58e94a2c4c98
Step 5/5 : COPY . .
---> d5897f4da4ff
Successfully built d5897f4da4ff
Successfully tagged myimage1:cuda10.2
If I run the image and do pip install ujson This is what I get
Collecting ujson
Downloading ujson-4.0.1-cp37-cp37m-manylinux1_x86_64.whl (179 kB)
|████████████████████████████████| 179 kB 947 kB/s
Installing collected packages: ujson
Successfully installed ujson-4.0.1
The only difference is that the ujson package is cp37 while that installed by Dockerfile is cp38? Can somebody explain why the packages installed are for different Python versions?

In the rapids containers there's a virtual environment named rapids where all of the packages are installed and is activated in the default entrypoint for the container. You should change your RUN command to:
RUN source activate rapids && \
pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt

Related

Trying to build a docker image but the build fails on Cloud Build

This is the Dockerfile that I am trying to build using Cloud Build.
FROM ubuntu:latest
LABEL MAINTAINER example
WORKDIR /app
RUN apt-get update \
&& apt-get install -y python3-pip python3-dev \
&& pip3 install --upgrade pip
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5000
CMD ["newrelic-admin","run-program","gunicorn","wsgi:app","--bind","0.0.0.0:5000","--workers","2","--threads","4","--worker-class=gthread","--log-level","info"]
Here is the requirements.txt file
numpy==1.18.2
pyarrow==0.17.0
lightgbm==2.3.1
scikit-learn==0.22.2.post1
pandas==1.0.3
scipy==1.4.1
Flask==2.1.0
tqdm==4.43.0
joblib==0.15.1
newrelic==6.2.0.156
google-cloud-storage==1.33.0
gunicorn==20.1.0
Once the build begins, it gets stuck at pyarrow and returns
Installing build dependencies: finished with status 'error'
[91m error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
How can I fix this?

Docker Image-pip3 Not Found

I'm trying to build a Docker image on Ubuntu 20.04 WSL for Windows 10 and keep running into the following error when Docker gets to the step to run pip3 install:
/bin/sh: 1: pip3: not found
The command '/bin/sh -c pip3 install -r /tmp/requirements.txt' returned a non-zero code: 127
The Dockerfile is:
FROM ubuntu:20.04
COPY bots/art_print.py /bots/
COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "art-print-bot"]
I've uninstalled and reinstalled pip3 and verified that it is there with $ which pip3
/usr/bin/pip3
Any ideas as to why the Docker build is not recognizing pip3?
Looks like you may have an issue with your PATH environment variable. Try changing the pip RUN line to:
RUN python3 -m pip install -r /tmp/requirements.txt

Docker Build - HTTP error 503 Server Error: Backend is unhealthy

Question :- What Backend does this Error refer to when it states - "because of error 503 Server Error: Backend is unhealthy for URL https://files.pythonhosted.org/packages/....
Am using a Dockerfile for the build. Image was built a few hours back with same Dockerfile. Now getting this ERROR -
Entire trace :
dhankar#dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a2_18/a2___DC_API/aa___Docker_RESTAPI$ sudo docker build -t ddrohit/dck1 .
Sending build context to Docker daemon 1.862MB
Step 1/11 : FROM ubuntu:xenial
---> c9d990395902
Step 2/11 : RUN apt-get update
---> Using cache
---> 42f625ed03cb
Step 3/11 : RUN apt-get install -y python3 # install python3
---> Using cache
---> 393d9e7b20ec
Step 4/11 : RUN apt-get install -y python-pip python-dev build-essential
---> Using cache
---> bb956f271bdd
Step 5/11 : COPY ./ /usr/src/rest_api/
---> Using cache
---> 50d0d52e8e22
Step 6/11 : WORKDIR /usr/src/rest_api/
---> Using cache
---> 42e086f6d8c4
Step 7/11 : RUN pip install --upgrade pip
---> Running in d56c0d5cafce
Collecting pip
HTTP error 503 while getting https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl#sha256=86a60a96d85e329962a9e6f6af612cbc11106293dbc83f119802b5bee9874cf3 (from https://pypi.org/simple/pip/)
Could not install requirement pip from https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl#sha256=86a60a96d85e329962a9e6f6af612cbc11106293dbc83f119802b5bee9874cf3 because of error 503 Server Error: Backend is unhealthy for url: https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl
Could not install requirement pip from https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl#sha256=86a60a96d85e329962a9e6f6af612cbc11106293dbc83f119802b5bee9874cf3 because of HTTP error 503 Server Error: Backend is unhealthy for url: https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl for URL https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl#sha256=86a60a96d85e329962a9e6f6af612cbc11106293dbc83f119802b5bee9874cf3 (from https://pypi.org/simple/pip/)
You are using pip version 8.1.1, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip install --upgrade pip' returned a non-zero code: 1
dhankar#dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a2_18/a2___DC_API/aa___Docker_RESTAPI$
My Dockerfile :-
#
FROM ubuntu:xenial
RUN apt-get update
RUN apt-get install -y python3 # install python3
RUN apt-get install -y python-pip python-dev build-essential
#
COPY ./ /usr/src/rest_api/
WORKDIR /usr/src/rest_api/
#
RUN pip install -r /usr/src/rest_api/requirements.txt
#
EXPOSE 8000
#
RUN python ./usr/src/rest_api/manage.py migrate
#
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
#
# docker run -p 8888:8000 --name dck1 ddrohit/dck1
#

Docker multistage build

My dockerfile looks as below :
FROM python:2.7 as builder
RUN pip install pika
RUN pip install requests
RUN pip install simplejson
RUN pip install datetime
RUN pip install grequests
RUN pip install urllib
RUN pip install pandas
COPY Action.py ./Action.py
COPY UtilFunctions.py ./UtilFunctions.py
WORKDIR /app
COPY . .
FROM apline
WORKDIR /app
COPY --from=builder /app /app
CMD [ "python","-u","./Action.py" ]
While building -> sudo docker build --rm -t rule1-test .
Gives following error ->
Step 1 : FROM python:2.7 as builder
Error parsing reference: "python:2.7 as builder" is not a valid repository/tag
Docker version is as below :
Docker version 1.12.6, build 88a4867/1.12.6
Is multistage not supported on this version
I have installed docker on centos machine using
yum install docker
Multi-stage builds are a new feature in Docker 17.05, so you have to update your Docker version to 17.05 or a newer version.

Using confluent-kafka python client in alpine container

I'm trying to run a simple python app that communicates with kafka. I'm looking to use an alpine container for it. Here's my current dockerfile (it's not optimal... just trying to get things working for now).
FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic#live.com)
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update && apk --no-cache add librdkafka
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
COPY api /usr/src/app/api
COPY static /usr/src/app/static
CMD ["python", "api/index.py"]
The requirements file has confluent-kafka in it. The build fails with
OK: 8784 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
so:libcrypto.so.41 (missing):
required by:
librdkafka-0.9.4-r1[so:libcrypto.so.41]
librdkafka-0.9.4-r1[so:libcrypto.so.41]
librdkafka-0.9.4-r1[so:libcrypto.so.41]
so:libssl.so.43 (missing):
required by:
librdkafka-0.9.4-r1[so:libssl.so.43]
librdkafka-0.9.4-r1[so:libssl.so.43]
librdkafka-0.9.4-r1[so:libssl.so.43]
My questions are a) is there a way to get this working without building inside the container? It'd be good enough if I could simply copy the library over to alpine. Or even if I could copy librdkafka over. b) If not, how can I get libssl and libcryto.so working?
here a way to make it work:
FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic#live.com)
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
&& apk upgrade --update-cache --available \
&& apk --no-cache add librdkafka
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
COPY api /usr/src/app/api
COPY static /usr/src/app/static
CMD ["python", "api/index.py"]
Here an explanation:
python:3.6-alpine image are based on alpine linux 3.4: Dockerfile
So you first need to correctly move to edge alpine branch: Edge
This is done by the line:
RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
&& apk upgrade --update-cache --available \
Then you can install librdkafka and the dependencies.
I maintain the ucalgary/python-librdkafka image that extends the official python:3.6-alpine image and installs librdkafka from its source releases. You're welcome to use the image, or take a look at the Dockerfile to see how it's built.
Since you're on edge anyway, you could also just use python3 from there:
% cat Dockerfile
FROM alpine:edge
MAINTAINER Gerd Flaig <stackoverflow#gerd.flaig.name>
RUN apk update && apk add --no-cache python3 librdkafka && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
rm -r /root/.cache
% docker build .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM alpine:edge
edge: Pulling from library/alpine
71c5a0cc58e4: Pull complete
Digest: sha256:99588bc8883c955c157d18fc3eaa4a3c1400c223e6c7cabca5f600a3e9f8d5cd
Status: Downloaded newer image for alpine:edge
---> 8914de95a28d
Step 2/3 : MAINTAINER Gerd Flaig <stackoverflow#gerd.flaig.name>
---> Running in 658b056c4e16
---> a3b2485fabb0
Removing intermediate container 658b056c4e16
Step 3/3 : RUN apk update && apk add --no-cache python3 librdkafka && python3 -m ensurepip && rm -r /usr/lib/python*/ensurepip && pip3 install --upgrade pip setuptools && rm -r /root/.cache
---> Running in d7c3f3b30d89
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.5.0-4449-g8922925e5c [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.5.0-4450-gb7b4122d6f [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 8348 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/16) Installing libressl2.5-libcrypto (2.5.3-r1)
(2/16) Installing libgcc (6.3.0-r3)
(3/16) Installing libressl2.5-libssl (2.5.3-r1)
(4/16) Installing libstdc++ (6.3.0-r3)
(5/16) Installing librdkafka (0.9.4-r1)
(6/16) Installing libbz2 (1.0.6-r5)
(7/16) Installing expat (2.2.0-r0)
(8/16) Installing libffi (3.2.1-r3)
(9/16) Installing gdbm (1.12-r0)
(10/16) Installing xz-libs (5.2.3-r0)
(11/16) Installing ncurses-terminfo-base (6.0-r7)
(12/16) Installing ncurses-terminfo (6.0-r7)
(13/16) Installing ncurses-libs (6.0-r7)
(14/16) Installing readline (6.3.008-r5)
(15/16) Installing sqlite-libs (3.18.0-r0)
(16/16) Installing python3 (3.6.1-r1)
Executing busybox-1.26.2-r0.trigger
OK: 75 MiB in 27 packages
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages
Requirement already satisfied: pip in /usr/lib/python3.6/site-packages
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages
Collecting setuptools
Downloading setuptools-35.0.2-py2.py3-none-any.whl (390kB)
Collecting packaging>=16.8 (from setuptools)
Downloading packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
Downloading appdirs-1.4.3-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
Installing collected packages: pyparsing, six, packaging, appdirs, setuptools
Found existing installation: setuptools 28.8.0
Uninstalling setuptools-28.8.0:
Successfully uninstalled setuptools-28.8.0
Successfully installed appdirs-1.4.3 packaging-16.8 pyparsing-2.2.0 setuptools-35.0.2 six-1.10.0
---> cfb1033ceec0
Removing intermediate container d7c3f3b30d89
Successfully built cfb1033ceec0

Resources