Using confluent-kafka python client in alpine container - docker

I'm trying to run a simple python app that communicates with kafka. I'm looking to use an alpine container for it. Here's my current dockerfile (it's not optimal... just trying to get things working for now).
FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic#live.com)
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update && apk --no-cache add librdkafka
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
COPY api /usr/src/app/api
COPY static /usr/src/app/static
CMD ["python", "api/index.py"]
The requirements file has confluent-kafka in it. The build fails with
OK: 8784 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
so:libcrypto.so.41 (missing):
required by:
librdkafka-0.9.4-r1[so:libcrypto.so.41]
librdkafka-0.9.4-r1[so:libcrypto.so.41]
librdkafka-0.9.4-r1[so:libcrypto.so.41]
so:libssl.so.43 (missing):
required by:
librdkafka-0.9.4-r1[so:libssl.so.43]
librdkafka-0.9.4-r1[so:libssl.so.43]
librdkafka-0.9.4-r1[so:libssl.so.43]
My questions are a) is there a way to get this working without building inside the container? It'd be good enough if I could simply copy the library over to alpine. Or even if I could copy librdkafka over. b) If not, how can I get libssl and libcryto.so working?

here a way to make it work:
FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic#live.com)
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
&& apk upgrade --update-cache --available \
&& apk --no-cache add librdkafka
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
COPY api /usr/src/app/api
COPY static /usr/src/app/static
CMD ["python", "api/index.py"]
Here an explanation:
python:3.6-alpine image are based on alpine linux 3.4: Dockerfile
So you first need to correctly move to edge alpine branch: Edge
This is done by the line:
RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
&& apk upgrade --update-cache --available \
Then you can install librdkafka and the dependencies.

I maintain the ucalgary/python-librdkafka image that extends the official python:3.6-alpine image and installs librdkafka from its source releases. You're welcome to use the image, or take a look at the Dockerfile to see how it's built.

Since you're on edge anyway, you could also just use python3 from there:
% cat Dockerfile
FROM alpine:edge
MAINTAINER Gerd Flaig <stackoverflow#gerd.flaig.name>
RUN apk update && apk add --no-cache python3 librdkafka && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
rm -r /root/.cache
% docker build .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM alpine:edge
edge: Pulling from library/alpine
71c5a0cc58e4: Pull complete
Digest: sha256:99588bc8883c955c157d18fc3eaa4a3c1400c223e6c7cabca5f600a3e9f8d5cd
Status: Downloaded newer image for alpine:edge
---> 8914de95a28d
Step 2/3 : MAINTAINER Gerd Flaig <stackoverflow#gerd.flaig.name>
---> Running in 658b056c4e16
---> a3b2485fabb0
Removing intermediate container 658b056c4e16
Step 3/3 : RUN apk update && apk add --no-cache python3 librdkafka && python3 -m ensurepip && rm -r /usr/lib/python*/ensurepip && pip3 install --upgrade pip setuptools && rm -r /root/.cache
---> Running in d7c3f3b30d89
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.5.0-4449-g8922925e5c [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.5.0-4450-gb7b4122d6f [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 8348 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/16) Installing libressl2.5-libcrypto (2.5.3-r1)
(2/16) Installing libgcc (6.3.0-r3)
(3/16) Installing libressl2.5-libssl (2.5.3-r1)
(4/16) Installing libstdc++ (6.3.0-r3)
(5/16) Installing librdkafka (0.9.4-r1)
(6/16) Installing libbz2 (1.0.6-r5)
(7/16) Installing expat (2.2.0-r0)
(8/16) Installing libffi (3.2.1-r3)
(9/16) Installing gdbm (1.12-r0)
(10/16) Installing xz-libs (5.2.3-r0)
(11/16) Installing ncurses-terminfo-base (6.0-r7)
(12/16) Installing ncurses-terminfo (6.0-r7)
(13/16) Installing ncurses-libs (6.0-r7)
(14/16) Installing readline (6.3.008-r5)
(15/16) Installing sqlite-libs (3.18.0-r0)
(16/16) Installing python3 (3.6.1-r1)
Executing busybox-1.26.2-r0.trigger
OK: 75 MiB in 27 packages
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages
Requirement already satisfied: pip in /usr/lib/python3.6/site-packages
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages
Collecting setuptools
Downloading setuptools-35.0.2-py2.py3-none-any.whl (390kB)
Collecting packaging>=16.8 (from setuptools)
Downloading packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
Downloading appdirs-1.4.3-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
Installing collected packages: pyparsing, six, packaging, appdirs, setuptools
Found existing installation: setuptools 28.8.0
Uninstalling setuptools-28.8.0:
Successfully uninstalled setuptools-28.8.0
Successfully installed appdirs-1.4.3 packaging-16.8 pyparsing-2.2.0 setuptools-35.0.2 six-1.10.0
---> cfb1033ceec0
Removing intermediate container d7c3f3b30d89
Successfully built cfb1033ceec0

Related

dockerfile does not build due to failed size validation

My dockerfile used to build successfully.
I tried to build today (5 days after successful build) with docker build -t fv ., and kept getting the following error:
failed commit on ref "layer-sha256:7a3de07a56633b9096304d02c47f097f3e28ae6c6dd442d1e7c4d26452ecd90a": "layer-sha256:7a3de07a56633b9096304d02c47f097f3e28ae6c6dd442d1e7c4d26452ecd90a" failed size validation: 581433721 != 600361569: failed precondition
any suggestions what this means and how to correct?
my dockerfile is:
FROM rocker/verse
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends build-essential libpq-dev python3.8 python3-pip python3-setuptools python3-dev
RUN pip3 install --upgrade pip
ADD . ./home/rstudio
ADD requirements.txt .
ADD install_packages.r .
# Miniconda and dependencies
RUN cd /tmp/ && \
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda3 && \
/root/miniconda3/condabin/conda install -y python=3.7
ENV PATH=$PATH:/root/miniconda3/bin
#RUN npm install phantomjs-prebuilt --phantomjs_cdnurl=http://cnpmjs.org/downloads
# installing python libraries
RUN pip3 install -r requirements.txt
# installing r libraries
RUN Rscript install_packages.r
another reference I got was:
=> => sha256:7a3de07a56633b9096304d02c47f097f3e28ae6c6dd442d1e7c4d26452ecd90a 580.97MB / 600.36MB 1150.8s
------
> [ 1/10] FROM docker.io/rocker/verse#sha256:3b417b991a32cc8bf9c1fa173ec976a5cc65522a76918df61b5c6cf6261e63a5:
Would this be because of an issue with the base image pulled?
I found that docker build would fail with this error, but it would work if I first pulled the failing image with docker pull <image> and then ran docker build.
this was due to security encryption from my local ip.
when tethering, was able to generate the docker image with non problems
On my side, I got something like below
------
> [1/3] FROM docker.io/library/python#sha256:10fc14aa6ae69f69e4c953cffd9b0964843d8c163950491d2138af891377bc1d:
------
failed commit on ref "layer-sha256:049db2c7eb8a5bd3833cac2f58c6c72b481f1a0288a8b20527529c4970b52762": "layer-sha256:049db2c7eb8a5bd3833cac2f58c6c72b481f1a0288a8b20527529c4970b52762" failed size validation: 311296 != 3056504: failed precondition
On my side, I managed to solve this by disconnecting from a VPN I was connected to.

Trying to build a docker image but the build fails on Cloud Build

This is the Dockerfile that I am trying to build using Cloud Build.
FROM ubuntu:latest
LABEL MAINTAINER example
WORKDIR /app
RUN apt-get update \
&& apt-get install -y python3-pip python3-dev \
&& pip3 install --upgrade pip
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5000
CMD ["newrelic-admin","run-program","gunicorn","wsgi:app","--bind","0.0.0.0:5000","--workers","2","--threads","4","--worker-class=gthread","--log-level","info"]
Here is the requirements.txt file
numpy==1.18.2
pyarrow==0.17.0
lightgbm==2.3.1
scikit-learn==0.22.2.post1
pandas==1.0.3
scipy==1.4.1
Flask==2.1.0
tqdm==4.43.0
joblib==0.15.1
newrelic==6.2.0.156
google-cloud-storage==1.33.0
gunicorn==20.1.0
Once the build begins, it gets stuck at pyarrow and returns
Installing build dependencies: finished with status 'error'
[91m error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
How can I fix this?

RUN pip install -r requirements.txt not working inside a Dockerfile

I am trying to build a new docker image by adding some python packages to my base docker image. The sudo docker build -t myimage1:cuda10.2 . runs without any issues but I cannot import any of the packages in the new image. I am using sudo docker run --gpus all -it myimage1:cuda10.2 to run the image. Can somebody help me understand what am I missing here?
Dockerfile
FROM nvcr.io/nvidia/rapidsai/rapidsai:cuda10.2-runtime-ubuntu18.04
WORKDIR /rapids/notebooks/
COPY requirements.txt ./
RUN pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
requirements.txt
ujson
This is the trace for docker build
Sending build context to Docker daemon 4.096kB
Step 1/5 : FROM nvcr.io/nvidia/rapidsai/rapidsai:cuda10.2-runtime-ubuntu18.04
---> 98901cabda0a
Step 2/5 : WORKDIR /rapids/notebooks/
---> Using cache
---> 162a9bb732c7
Step 3/5 : COPY requirements.txt ./
---> 7bb48a384987
Step 4/5 : RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt
---> Running in 5f3ca4ed3f93
Collecting pip
Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Uninstalling pip-20.0.2:
Successfully uninstalled pip-20.0.2
Successfully installed pip-20.2.4
Collecting ujson
Downloading ujson-4.0.1-cp38-cp38-manylinux1_x86_64.whl (181 kB)
Installing collected packages: ujson
Successfully installed ujson-4.0.1
Removing intermediate container 5f3ca4ed3f93
---> 58e94a2c4c98
Step 5/5 : COPY . .
---> d5897f4da4ff
Successfully built d5897f4da4ff
Successfully tagged myimage1:cuda10.2
If I run the image and do pip install ujson This is what I get
Collecting ujson
Downloading ujson-4.0.1-cp37-cp37m-manylinux1_x86_64.whl (179 kB)
|████████████████████████████████| 179 kB 947 kB/s
Installing collected packages: ujson
Successfully installed ujson-4.0.1
The only difference is that the ujson package is cp37 while that installed by Dockerfile is cp38? Can somebody explain why the packages installed are for different Python versions?
In the rapids containers there's a virtual environment named rapids where all of the packages are installed and is activated in the default entrypoint for the container. You should change your RUN command to:
RUN source activate rapids && \
pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt

No module named 'numpy' while installing scikit-image

I am trying to install scikit-image in alpine docker but for some reason it is keep throwing No module named 'numpy' error
Here is the Dockerfile
COPY requirements.txt requirements.txt
# Installing System Libraries and Python Dependencies
RUN apk update && apk add --no-cache gcc musl-dev make && ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& pip install -r requirements.txt && apk del gcc musl-dev make
And Here is the requirements.txt
numpy
matplotlib
scikit-image
scikit-learn
I have the same problem. I tried to install numpy alone
pip install numpy
then install the requirements file as follows:
pip install -r requirements.txt
This workaround solved my issue. hope it helped!
Since you are using an alpine image and making use of the pip package manager, you need to include in the apk update section as well.
FROM alpine
COPY ./requirements.txt /temp/requirements.txt
# Installing System Libraries and Python Dependencies
WORKDIR '/temp'
RUN apk update && apk add --update py-pip && apk add --no-cache gcc musl-dev make && ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& pip install -r requirements.txt && apk del gcc musl-dev make
Since you are using the alpine as the base image, Make sure to add the other packages as well that requires for your requirement libraries.

Setting up our Rasa/NLU container, error?

I have this file Dockerfile.nlu
FROM chatbot/spacy:latest
WORKDIR /app
COPY nlu ./agent_nlu
RUN python –m rasa_nlu.train --config agent_nlu/config.yml --data agent_nlu/data/ --path agent_nlu/agent --fixed_model_name default
and I get the error below:
]$ sudo docker build -t nlu:latest -f docker/Dockerfile.nlu .
Sending build context to Docker daemon 9.216kB
Step 1/4 : FROM chatbot/spacy:latest
---> 496dc6a38abb
Step 2/4 : WORKDIR /app
---> Using cache
---> 7f02012c8452
Step 3/4 : COPY nlu ./agent_nlu
COPY failed: stat /var/lib/docker/tmp/docker-builder363868051/nlu: no such file or directory
It doesn't look like Docker can find the nlu directory. Are you sure it exists? Are you sure that you are executing the command from the correct directory?
But you also aren't installing Rasa at all or any of it's requirements. Is there a reason you aren't using the pre-built Rasa images? available here with docs here.
Here is a fully functional Docker file pulled from their repo.
FROM python:3.6-slim
ENV RASA_NLU_DOCKER="YES" \
RASA_NLU_HOME=/app \
RASA_NLU_PYTHON_PACKAGES=/usr/local/lib/python3.6/dist-packages
# Run updates, install basics and cleanup
# - build-essential: Compile specific dependencies
# - git-core: Checkout git repos
RUN apt-get update -qq \
&& apt-get install -y --no-install-recommends build-essential git-core openssl libssl-dev libffi6 libffi-dev curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
WORKDIR ${RASA_NLU_HOME}
COPY . ${RASA_NLU_HOME}
# use bash always
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN pip install -r alt_requirements/requirements_spacy_sklearn.txt
RUN pip install -e .
RUN pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.0.0/en_core_web_md-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link en_core_web_md en \
&& pip install https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link de_core_news_sm de
COPY sample_configs/config_spacy.yml ${RASA_NLU_HOME}/config.yml
VOLUME ["/app/projects", "/app/logs", "/app/data"]
EXPOSE 5000
ENTRYPOINT ["./entrypoint.sh"]
CMD ["start", "-c", "config.yml", "--path", "/app/projects"]

Resources