I want to deploy my python project in docker, I wrote lxml>=3.5.0 in the requirments.txt as the project needs lxml. Here is my dockfile:
FROM gliderlabs/alpine:3.3
RUN set -x \
&& buildDeps='\
python-dev \
py-pip \
build-base \
' \
&& apk --update add python py-lxml $buildDeps \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /app
ENV INSTALL_PATH /app
WORKDIR $INSTALL_PATH
COPY requirements-docker.txt ./
RUN pip install -r requirements.txt
COPY . .
RUN apk del --purge $buildDeps
ENTRYPOINT ["celery", "-A", "tasks", "worker", "-l", "info", "-B"]
I got this when I deploy it to docker:
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
Rolling back uninstall of lxml
I though it was because 'python-dev' and 'python-lxml', then I edited the dockfile like this:
WORKDIR $INSTALL_PATH
COPY requirements-docker.txt ./
RUN apt-get build-dev python-lxml
RUN pip install -r requirements.txt
It did not work, and I got another error:
---> Running in 73201a0dcd59
/bin/sh: apt-get: not found
How can I install lxml correctly in docker?
I added RUN apk add --update --no-cache g++ gcc libxslt-dev before RUN pip install -r requirements.txt and it worked.
Accepted answer is not neat and installs redundant packages. Better solution for reducing image size will be:
RUN apk add --no-cache --virtual .build-deps gcc libc-dev libxslt-dev && \
apk add --no-cache libxslt && \
pip install --no-cache-dir lxml>=3.5.0 && \
apk del .build-deps
Result image size will be < 163MB
Since I was using a much more bare-bone image I needed some more libs/apps.
This worked for me:
RUN apk add --update --no-cache g++ gcc libxml2-dev libxslt-dev python-dev libffi-dev openssl-dev make
RUN pip install -r requirements.txt
Since only this answer worked for me and I wanted something light
And I liked this answer, but which didn't work for me at first
I've edited it for myself and got this at the end :
RUN apk add --update --no-cache --virtual .build-deps g++ gcc libxml2-dev libxslt-dev python-dev && \
apk add --no-cache libxslt && \
pip install --no-cache-dir lxml>=3.5.0 && \
apk del .build-deps
The final image is around 110MB, and didn't have anymore any libxml and libslt errors
Do as in
https://hub.docker.com/r/ryanfox1985/docker-couchpotato/builds/boinrrs9dbhnutwjxjw2l8m/
Download the apk and install it
RUN wget http://nl.alpinelinux.org/alpine/edge/main/x86_64/py-lxml-3.4.0-r0.apk -O /var/cache/apk/py-lxml.apk
RUN apk add --allow-untrusted /var/cache/apk/py-lxml.apk
Actually, it's just
RUN apt-get install -y libxslt1-dev
Related
I am trying to build in Bamboo and got this error,
Failed to build pyarrow
21-Sep-2022 06:24:14 ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
21-Sep-2022 06:24:15 The command '/bin/sh -c pip install --upgrade pip && pip install pyarrow' returned a non-zero code: 1
21-Sep-2022 06:24:15 =An error occurred when executing task 'DockerBuild'.
This error occurs only when I add pyarrow or fastparquet in requirements.txt.
This is my requirements.txt file:
requests
urllib3
fastapi
uvicorn[standard]
gunicorn
pytest-cov
prometheus-fastapi-instrumentator
prometheus_client
fastapi-health
python-decouple
ecs-logging
fastapi_health
psycopg2
arrow
anyio
asgiref
certifi
charset-normalizer
click
colorama
h11
idna
python-dotenv
pydantic
sniffio
starlette
typing_extensions
datetime
fastapi_resource_server
sendgrid
PyJWT==2.4.0
bcrypt==3.2.
cryptography==37.0.2
passlib
jose
jira
adal==1.2.7
aiohttp==3.8.1
aiosignal==1.2.0
async-timeout==4.0.2
azure-core==1.25.0
azure-identity==1.10.0
azure-storage-blob==12.13.1
pandas==1.4.4
multidict==6.0.2
numpy==1.23.2
ordered-set==4.1.0
oauthlib==3.2.0
packaging==21.3
python-dateutil==2.8.2
pytz==2022.2.1
requests-oauthlib==1.3.1
six==1.16.0
yarl==1.8.1
Below is my dockerfile:
FROM python:3.10.4-alpine3.15
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.12/main' >> /etc/apk/repositories
RUN apk upgrade && apk add make gcc g++
RUN apk update
RUN apk add libffi-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN apk add --no-cache musl-dev linux-headers g++
RUN pip install --upgrade pip && pip install arrow && pip install pyarrow
RUN pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "1000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
Is this error because of the python image?
I am still learning docker so not sure what went wrong here. Can anyone please help me in understanding this?
I have changed the docker file and built it from source since I came to know that pyarrow wheels are not provided for alpine.
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "5000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0","-m 3000m"]
This is my dockerfile code and my .py code uses splinter for the chromedriver.
FROM python:3.7-alpine
COPY requirements.txt .
RUN apk update && \
apk add make automake gcc g++ subversion python3-dev && \
apk add gcc musl-dev python3-dev libffi-dev openssl-dev && \
apk add chromium chromium-chromedriver && \
pip install -r /requirements.txt && \
rm -rf /root/.[acpw]* ipaexg00301*
COPY . /app
Then when I run my program I get this error message.
"selenium.common.exceptions.WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 127"
I've tried everything to get this to work to no avail. Help would be appreciated.
I have a docker container that requires a couple Python packages to be installed. I added some commands to install them, but the packages are not available in the container. I think I'm added them wrong, is there a certain way I need to copy over what was installed? Any help appreciated.
Specifically, I should be able to run lottie.py in my container after installing it, but it does not exist, nor does pip3 even though the install is successful.
Below is my Dockerfile:
FROM alpine AS builder
COPY . /go/src/matterbridge
RUN apk --no-cache add go git
WORKDIR /go/src/matterbridge
RUN go build -mod vendor -o /bin/matterbridge
FROM python:3
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir lottie cairosvg
RUN apt-get install libcairo2-dev
FROM alpine
RUN apk --no-cache add ca-certificates mailcap
COPY --from=builder /bin/matterbridge /bin/matterbridge
RUN mkdir /etc/matterbridge \
&& touch /etc/matterbridge/matterbridge.toml \
&& ln -sf /matterbridge.toml /etc/matterbridge/matterbridge.toml
ENTRYPOINT ["/bin/matterbridge", "-conf", "/etc/matterbridge/matterbridge.toml"]
Was given help in the comments that I'm not copying anything over from the Python image. I was able to find python:alpine and use that in my config below
FROM alpine AS builder
COPY . /go/src/matterbridge
RUN apk --no-cache add go git
WORKDIR /go/src/matterbridge
RUN go build -mod vendor -o /bin/matterbridge
FROM python:alpine
RUN apk --no-cache add ca-certificates mailcap
RUN apk --update add libxml2-dev libxslt-dev libffi-dev gcc musl-dev libgcc openssl-dev curl
RUN apk add jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev
RUN apk add --no-cache --virtual .pynacl_deps build-base python3-dev py3-pip
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir lottie cairosvg
COPY --from=builder /bin/matterbridge /bin/matterbridge
RUN mkdir /etc/matterbridge \
&& touch /etc/matterbridge/matterbridge.toml \
&& ln -sf /matterbridge.toml /etc/matterbridge/matterbridge.toml
ENTRYPOINT ["/bin/matterbridge", "-conf", "/etc/matterbridge/matterbridge.toml"]
I have a Python project with Docker that uses gdal package. For pretty long time everything were ok, but now I get an error while building an image:
My Dockerfile:
FROM python:3.6.8-alpine3.10
ENV PYTHONUNBUFFERED 1
RUN mkdir /app
WORKDIR /app
ADD requirements.txt /app/
RUN apk update && \
apk add \
--repository http://dl-cdn.alpinelinux.org/alpine/edge/community \
--repository http://dl-cdn.alpinelinux.org/alpine/edge/main \
geos gdal gdal-dev geos-dev proj-dev && \
apk add gcc linux-headers musl musl-dev postgresql-dev gdal gdal-dev \
geos-dev proj-dev zlib-dev jpeg-dev \
libpng libpng-dev uriparser-dev
RUN pip install -r requirements.txt
ADD . /app/
Do anyone know a solution to the problem?
I don't know the old status, but if you search gdal in official package system now, you will find nothing:
But, with alpine3.11, you could see next:
So, I guess gdal has been deleted from alpine3.10 official repo, you may have to build it from source code by yourself if you insist work on alpine3.10 or directly update your base to python:3.6-alpine3.11.
I have the following dockerfile:
FROM alpine:latest
ADD crontab.txt /crontab.txt
ADD script.sh /script.sh
COPY entry.sh /entry.sh
ADD app /app
RUN chmod 755 /script.sh /entry.sh
RUN /usr/bin/crontab /crontab.txt
# install dependencies
# the lapack package is only in the community repository
RUN echo "http://dl-8.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories \
&& apk update \
&& apk add --no-cache python3 py-pip py3-setuptools python-dev py3-lxml py3-requests py3-numpy py3-cssselect py3-pandas
RUN apk --update add --no-cache \
lapack-dev \
gcc \
freetype-dev
# Install dependencies
RUN apk add --no-cache --virtual .build-deps \
gfortran \
musl-dev \
g++
RUN ln -s /usr/include/locale.h /usr/include/xlocale.h
#RUN pip3 install cython
RUN pip3 install pymongo xlrd
CMD ["/entry.sh"]
when I try to build the dockerfile, I received the error saying that py3-pandas (missing).
I am wondering whether that's the Alpine package management issue.
I can reproduce your issue by doing just:
FROM alpine:latest
RUN echo "http://dl-8.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories \
&& apk update \
&& apk add --no-cache py3-pandas
Some comments here:
If you are going to use the edge/testing packages repository already, then you would be better using the alpine:edge image, although I would not advise this for a production server, of course
Using another package repository than the default one on apk can be done on a one run basis using the option
-X, --repository REPO Use packages from REPO
From apk --help
The package python-dev does not exist in the edge/testing repository, you should use python3-dev
Important disclaimer: this solution does involve using the testing repository and the edge rolling release branch of Alpine, and is thus not recommended for production use.
So, in the end, all your dependencies can be installed doing:
FROM alpine:edge
RUN apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing \
py-pip \
py3-setuptools \
python3-dev \
py3-lxml \
py3-requests \
py3-numpy \
py3-cssselect \
py3-pandas \
lapack-dev \
gcc \
freetype-dev
Note that I didn't add the build dependancies there, as they don't seems to be related to your issue at hand
Don't use alpine images for python; there's too many issues. Use a something like python:slim-buster. You can then just install your requirements with:
FROM python:3.8.4-slim-buster
COPY ./requirements.txt /requirements.txt
RUN pip install -r /requirements.txt