My dockerfile looks as below :
FROM python:2.7 as builder
RUN pip install pika
RUN pip install requests
RUN pip install simplejson
RUN pip install datetime
RUN pip install grequests
RUN pip install urllib
RUN pip install pandas
COPY Action.py ./Action.py
COPY UtilFunctions.py ./UtilFunctions.py
WORKDIR /app
COPY . .
FROM apline
WORKDIR /app
COPY --from=builder /app /app
CMD [ "python","-u","./Action.py" ]
While building -> sudo docker build --rm -t rule1-test .
Gives following error ->
Step 1 : FROM python:2.7 as builder
Error parsing reference: "python:2.7 as builder" is not a valid repository/tag
Docker version is as below :
Docker version 1.12.6, build 88a4867/1.12.6
Is multistage not supported on this version
I have installed docker on centos machine using
yum install docker
Multi-stage builds are a new feature in Docker 17.05, so you have to update your Docker version to 17.05 or a newer version.
Related
I want to host aspnet 6.0 application in docker container on AlmaLinux (or any centos compatible destro). The application also needs to have maprdrill odbc client that is available only in rpm package. Problem is the base images Microsoft provides are based on debian, alpine and ubuntu only (see here) and none of them let me install rpm package.
I also tried to convert the rpm to deb using alien, conversion was success and package was installed but due to some unknown reason the installation created a layer of 400mb that takes only 80mb on centos and the client appears to not work as well.
What other options I've to install rpm package in aspnet 6 app docker image?
Here is my docker file that understandably fails on line 29:
# base
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
# build
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["HelloWorld/HelloWorld.csproj", "HelloWorld/"]
RUN dotnet restore "HelloWorld/HelloWorld.csproj"
COPY . .
WORKDIR "/src/HelloWorld"
RUN dotnet build "HelloWorld.csproj" -c Release -r rhel-x64 --self-contained false -o /app/build
# publish
FROM build AS publish
RUN dotnet publish "HelloWorld.csproj" -c Release -r rhel-x64 --self-contained false -o /app/publish
FROM base AS final
RUN apt update
WORKDIR /app
COPY --from=publish /app/publish .
WORKDIR /mapr
RUN apt-get -yq install wget
RUN wget http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.5.1.1002/maprdrill-1.5.1.1002-1.el7.x86_64.rpm
RUN rpm -i maprdrill-1.5.1.1002-1.el7.x86_64.rpm
ENTRYPOINT ["dotnet", "HelloWorld.dll"]
I am trying to build this public.ecr.aws/lambda/python:3.6 based Dockerfile with a requirements.txt file that contains some libraries that need gcc/g++ to build. I'm getting an error of a missing Python.h file despite the fact that I installed the python development package and /usr/include/python3.6m/Python.h exists in the file system.
Dockerfile
FROM public.ecr.aws/lambda/python:3.6
RUN yum install -y gcc gcc-c++ python36-devel.x86_64
RUN pip install --upgrade pip && \
pip install cyquant
COPY app.py ./
CMD ["app.handler"]
When I build this with
docker build -t redux .
I get the following error
cyquant/dimensions.cpp:4:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
Notice, however, that my Dockerfile yum installs the development package. I have also tried the yum package python36-devel.i686 with no change.
What am I doing wrong?
The pip that you're executing lives in /var/lang/bin/pip whereas the python you're installing lives in the /usr prefix
presumably you could use /usr/bin/pip directly to install, but I'm not sure whether that works correctly with the lambda environment
I was able to duplicate the behavior of the AWS Lambda functionality without their Docker image and it works just fine. This is the Dockerfile I am using.
ARG FUNCTION_DIR="/function/"
FROM python:3.6 AS build
ARG FUNCTION_DIR
ARG NETRC_PATH
RUN echo "${NETRC_PATH}" > /root/.netrc
RUN mkdir -p ${FUNCTION_DIR}
COPY requirements.txt ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
RUN pip install --upgrade pip && \
pip install --target ${FUNCTION_DIR} awslambdaric && \
pip install --target ${FUNCTION_DIR} --no-warn-script-location -r requirements.txt
FROM python:3.6
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build ${FUNCTION_DIR} ${FUNCTION_DIR}
COPY main.py ${FUNCTION_DIR}
ENV MPLCONFIGDIR=/tmp/mplconfig
ENTRYPOINT ["/usr/local/bin/python", "-m", "awslambdaric"]
CMD ["main.handler"]
I am trying to update a dependency in docker using poetry, I have added
RUN poetry update
RUN poetry install -n
in Dockerfile but it doesn't update the package. There is an import error with an older version of tortoise ORM, which requires an upgrade (verified by running the project without docker and with virtualenv and with the newer package) which persists with the changes to the Dockerfile.
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7
RUN pip install poetry
# RUN poetry config virtualenvs.create false
COPY poetry.lock pyproject.toml ./
# for poetry
RUN mkdir -p /app/app/
RUN touch /app/app/__init__.py
RUN poetry update
RUN poetry install -n
COPY ./app /app/app
EXPOSE 8000
my Dockerfile for reference.
I am running my monolith application in a docker container and k8s on GKE.
The application contains python & node dependencies also webpack for front end bundle.
We have implemented CI/CD which is taking around 5-6 min to build & deploy new version to k8s cluster.
Main goal is to reduce the build time as much possible. Written Dockerfile is multi stage.
Webpack is taking more time to generate the bundle.To buid docker image i am using already high config worker.
To reduce time i tried using the Kaniko builder.
Issue :
As docker cache layers for python code it's working perfectly. But when there is any changes in JS or CSS file we have to generate bundle.
When there is any changes in JS & CSS file instead if generate new bundle its use caching layer.
Is there any way to separate out build new bundle or use cache by passing some value to docker file.
Here is my docker file :
FROM python:3.5 AS python-build
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt &&\
pip3 install Flask-JWT-Extended==3.20.0
ADD . /app
FROM node:10-alpine AS node-build
WORKDIR /app
COPY --from=python-build ./app/app/static/package.json app/static/
COPY --from=python-build ./app ./
WORKDIR /app/app/static
RUN npm cache verify && npm install && npm install -g --unsafe-perm node-sass && npm run sass && npm run build
FROM python:3.5-slim
COPY --from=python-build /root/.cache /root/.cache
WORKDIR /app
COPY --from=node-build ./app ./
RUN apt-get update -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt
EXPOSE 9595
CMD python3 run.py
I would suggest to create separate build pipelines for your docker images, where you know that the requirements for npm and pip aren't so frequent.
This will incredibly improve the speed, reducing the time of access to npm and pip registries.
Use a private docker registry (the official one or something like VMWare harbor or SonaType Nexus OSS).
You store those build images on your registry and use them whenever something on the project changes.
Something like this:
First Docker Builder // python-builder:YOUR_TAG [gitrev, date, etc.)
docker build --no-cache -t python-builder:YOUR_TAG -f Dockerfile.python.build .
FROM python:3.5
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt &&\
pip3 install Flask-JWT-Extended==3.20.0
Second Docker Builder // js-builder:YOUR_TAG [gitrev, date, etc.)
docker build --no-cache -t js-builder:YOUR_TAG -f Dockerfile.js.build .
FROM node:10-alpine
WORKDIR /app
COPY app/static/package.json /app/app/static
WORKDIR /app/app/static
RUN npm cache verify && npm install && npm install -g --unsafe-perm node-sass
Your Application Multi-stage build:
docker build --no-cache -t app_delivery:YOUR_TAG -f Dockerfile.app .
FROM python-builder:YOUR_TAG as python-build
# Nothing, already "stoned" in another build process
FROM js-builder:YOUR_TAG AS node-build
ADD ##### YOUR JS/CSS files only here, required from npm! ###
RUN npm run sass && npm run build
FROM python:3.5-slim
COPY . /app # your original clean app
COPY --from=python-build #### only the files installed with the pip command
WORKDIR /app
COPY --from=node-build ##### Only the generated files from npm here! ###
RUN apt-get update -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt
EXPOSE 9595
CMD python3 run.py
A question is: why do you install curl and execute again the pip install -r requirements.txt command in the final docker image?
Triggering every time an apt-get update and install without cleaning the apt cache /var/cache/apt folder produces a bigger image.
As suggestion, use the docker build command with the option --no-cache to avoid caching result:
docker build --no-cache -t your_image:your_tag -f your_dockerfile .
Remarks:
You'll have 3 separate Dockerfiles, as I listed above.
Build the Docker images 1 and 2 only if you change your python-pip and node-npm requirements, otherwise keep them fixed for your project.
If any dependency requirement changes, then update the docker image involved and then the multistage one to point to the latest built image.
You should always build only the source code of your project (CSS, JS, python). In this way, you have also guaranteed reproducible builds.
To optimize your environment and copy files across the multi-stage builders, try to use virtualenv for python build.
I am trying to build a Docker image for my API with the following Dockerfile:
FROM microsoft/dotnet AS build-env
ARG source
RUN echo "source: $source"
WORKDIR /app
RUN apt-get update
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash
RUN apt-get install nodejs
RUN node -v
RUN npm -v
# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore
#Copy everything else & build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM microsoft/dotnet
WORKDIR /app
COPY --from=build-env /app/out .
EXPOSE 80
ENTRYPOINT ["dotnet", "API_App.dll"]
However, when I run the docker build command, I keep getting the following error:
Unable to locate package nodejs
The command '/bin/sh -c apt-get install nodejs returned a non-zero code: 100
Can someone tell me why I am getting this error?
Node Version: 8.11.3
npm Version: 5.6.0
You may occasionally experience some cache issues when the live repositories you’re pulling data from have changed.
To fix this, modify the Dockerfile to do a cleanup and update of the sources before you install any new packages.
...
# clean and update sources
RUN apt-get clean && apt-get update
...
This answer is from digitalocean-issue