I'm using Docker to build my application. I'm using pip to install the packages from requirements.txt but the package is not including in the build.
FROM python:3.4
WORKDIR /app
ADD . /app
RUN apt-get update && apt-get install -y \
python3-pip python-pip\
cron \
unixodbc \
unixodbc-dev \
python3-dev \
python3-setuptools \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip
RUN pip install sendgrid
RUN pip3 install -r requirements.txt
ENV CONFIG_ENV .env
ADD validator-cron /etc/cron.d/validator-cron-job
RUN chmod 0644 /etc/cron.d/validator-cron-job
RUN touch /var/log/cron.log
CMD cron && tail -f /var/log/cron.log
I'm installing sendgrid using pip but I'm getting ImportError: no module found error.
I have resolved the issue.
It is coming because of the python path.
Related
This question already has answers here:
How to copy files from dockerfile to host?
(6 answers)
Closed 3 months ago.
I have a simply Dockerfile to build python requirements into a zip file to be uploaded to AWS Lambda.
FROM amazonlinux:2.0.20221004.0
RUN yum install -y python37 && \
yum install -y python3-pip && \
yum install -y zip && \
yum clean all
RUN python3.7 -m pip install --upgrade pip && \
python3.7 -m pip install virtualenv
COPY aws_requirements.txt .
RUN pip install -r aws_requirements.txt -t ./python
RUN zip -r python.zip ./py
Is there a way to copy the python.zip out of the image to the host during the dockerfile?
With buildkit, you can output the result to a directory on the host instead of pushing to a registry. So you can put the zip file in a scratch image and then output that:
FROM amazonlinux:2.0.20221004.0 as build
RUN yum install -y python37 && \
yum install -y python3-pip && \
yum install -y zip && \
yum clean all
RUN python3.7 -m pip install --upgrade pip && \
python3.7 -m pip install virtualenv
COPY aws_requirements.txt .
RUN pip install -r aws_requirements.txt -t ./python
RUN zip -r /python.zip ./py
FROM scratch as artifact
COPY --from=build /python.zip /python.zip
Then running:
docker build --output type=local,dest=out .
Will create a out/python.zip file.
I am trying to build in Bamboo and got this error,
Failed to build pyarrow
21-Sep-2022 06:24:14 ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
21-Sep-2022 06:24:15 The command '/bin/sh -c pip install --upgrade pip && pip install pyarrow' returned a non-zero code: 1
21-Sep-2022 06:24:15 =An error occurred when executing task 'DockerBuild'.
This error occurs only when I add pyarrow or fastparquet in requirements.txt.
This is my requirements.txt file:
requests
urllib3
fastapi
uvicorn[standard]
gunicorn
pytest-cov
prometheus-fastapi-instrumentator
prometheus_client
fastapi-health
python-decouple
ecs-logging
fastapi_health
psycopg2
arrow
anyio
asgiref
certifi
charset-normalizer
click
colorama
h11
idna
python-dotenv
pydantic
sniffio
starlette
typing_extensions
datetime
fastapi_resource_server
sendgrid
PyJWT==2.4.0
bcrypt==3.2.
cryptography==37.0.2
passlib
jose
jira
adal==1.2.7
aiohttp==3.8.1
aiosignal==1.2.0
async-timeout==4.0.2
azure-core==1.25.0
azure-identity==1.10.0
azure-storage-blob==12.13.1
pandas==1.4.4
multidict==6.0.2
numpy==1.23.2
ordered-set==4.1.0
oauthlib==3.2.0
packaging==21.3
python-dateutil==2.8.2
pytz==2022.2.1
requests-oauthlib==1.3.1
six==1.16.0
yarl==1.8.1
Below is my dockerfile:
FROM python:3.10.4-alpine3.15
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.12/main' >> /etc/apk/repositories
RUN apk upgrade && apk add make gcc g++
RUN apk update
RUN apk add libffi-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN apk add --no-cache musl-dev linux-headers g++
RUN pip install --upgrade pip && pip install arrow && pip install pyarrow
RUN pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "1000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
Is this error because of the python image?
I am still learning docker so not sure what went wrong here. Can anyone please help me in understanding this?
I have changed the docker file and built it from source since I came to know that pyarrow wheels are not provided for alpine.
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "5000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0","-m 3000m"]
I built a ubuntu image using the following Dockerfile:
FROM ubuntu:20.04
# Disable Prompt During Packages Installation
ARG DEBIAN_FRONTEND=noninteractive
# Add 32bit architecture
RUN dpkg --add-architecture i386 \
&& apt-get update \
&& apt-get install -y libc6:i386 libncurses5:i386 libstdc++6:i386 zlib1g:i386
RUN apt-get update && apt-get install -y locales && rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
RUN apt-get update && apt-get install -y \
iputils-ping \
python3 python3-pip
# Copy app to container
COPY . /app
WORKDIR /app
# Install pip requirements
COPY requirements.txt /app
RUN python3 -m pip install -r requirements.txt
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["bash"]
I've been trying to run a 32bit app (hence the first run command in the Dockerfile) I have inside the my_app directory using:
./app
but I keep getting
bash: ./app: No such file or directory
I build your docker file with no error, do you have more detail ?
I have the following Dockerfile, currently working locally in my device:
FROM python:3.7-slim-buster
WORKDIR /app
COPY . /app
VOLUME /app
RUN chmod +x /app/cat/sitemap_download.py
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
ARG VERSION=3.7.4
RUN apt update && \
apt install -y bash wget && \
wget -O /tmp/nordrepo.deb https://repo.nordvpn.com/deb/nordvpn/debian/pool/main/nordvpn-release_1.0.0_all.deb && \
apt install -y /tmp/nordrepo.deb && \
apt update && \
apt install -y nordvpn=$VERSION && \
apt remove -y wget nordvpn-release
RUN apt-get clean \
&& apt-get -y update
RUN apt-get -y install python3-dev \
python3-psycopg2 \
&& apt-get -y install build-essential
RUN pip install --upgrade pip
RUN pip install -r cat/requirements.txt
RUN pip install awscli
ENTRYPOINT ["sh", "-c", "./entrypoint.sh"]
But when I deploy it to Fargate, the container stops before reaching the steady state with:
sh: 1: ./entrypoint.sh: not found
Edit: Adding entrypoint.sh file for clarification:
#!/bin/env sh
# start process, but it should exit once the file is in S3
/app/cat/sitemap_download.py
# Once the process is done, we are good to scale down the service
aws ecs update-service --cluster cluster_name --region eu-west-1 --service service-name --desired-count 0
I have tried modifying ENTRYPOINT to use it as exec form, or with full path but always get the same issue. Any ideas on what am I doing wrong?
I've managed to fix it now.
Changing the Dockerfile to look as follows solves the issue:
COPY . /app
VOLUME /app
RUN chmod +x /app/cat/sitemap_download.py
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
ARG VERSION=3.7.4
RUN apt update && \
apt install -y bash wget && \
wget -O /tmp/nordrepo.deb https://repo.nordvpn.com/deb/nordvpn/debian/pool/main/nordvpn-release_1.0.0_all.deb && \
apt install -y /tmp/nordrepo.deb && \
apt update && \
apt install -y nordvpn=$VERSION && \
apt remove -y wget nordvpn-release
RUN apt-get clean \
&& apt-get -y update
RUN apt-get -y install python3-dev \
python3-psycopg2 \
&& apt-get -y install build-essential
RUN pip install --upgrade pip
RUN pip install -r cat/requirements.txt
RUN pip install awscli
ENTRYPOINT ["/bin/bash"]
CMD ["./entrypoint.sh"]
I tried this after reading: What is the difference between CMD and ENTRYPOINT in a Dockerfile?
I believe this syntax fixes it because with entrypoint I'm indicating bash to be run at start, and then passing the script as parameter.
I have a Dockerfile that is like:
FROM java:8
LABEL maintainer="CMS"
RUN apt-get install python-pip
RUN pip install awscli
....
.....
[Error: Unable to locate package python-pip]
My end goal is to have java8 and aws-cli installed. Also I don't want to use curl statements in the Dockerfile. Also I don't want to use the plain ubuntu image.
How should I go about doing it?
The error says Pip is not installed. Try installing it properly. If installed try executing same commands to verify.
try to update your docker file to
FROM java:8
LABEL maintainer="CMS"
RUN apt-get update && apt-get install -y \
software-properties-common
RUN add-apt-repository universe
RUN apt-get update && apt-get install -y \
python3.4 \
python3-pip
RUN pip install awscli
....
.....
If you want to base it on top of openjdk:8 image, try the following:
FROM openjdk:8
RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends \
python3-setuptools \
python3-pip \
; \
rm -rf /var/lib/apt/lists/*
RUN pip3 --no-cache-dir install -U awscli
RUN apt-get clean
The other option is to use Alpine distribution:
FROM openjdk:8-alpine
RUN set -eux; \
apk add python3 ; \
pip3 --no-cache-dir install -U awscli
Sources:
https://bitbucket.org/vodkaseledka/openjdk8-awscli
https://bitbucket.org/vodkaseledka/openjdk8-awscli-alpine
Or you can get pre-builds from here:
https://hub.docker.com/repository/docker/savnn/openjdk8-awscli
https://hub.docker.com/repository/docker/savnn/openjdk8-awscli-alpine
this work for me: create dockerfile
FROM openjdk:8-alpine
RUN apk update;
RUN set -eux; \
apk add python3 ; \
pip3 --no-cache-dir install -U awscli; \
pip3 install --upgrade pip;
RUN apk add groff
use docker build . -t aws then run: docker run -it aws /bin/sh