When I build just the main image, all the packages instead. But as soon as I turn it into a multi-stage build and it gets to RUN apt-get install -y python3-pip, I get "E: Unable to locate package in multistage Docker build"
FROM gcc:8.2.0 as builder
# FROM ownyourbits/debiandev:latest
RUN apt-get update
# RUN apt-get install -y libxerces-c-dev automake cmake libboost-all-dev build-essential
RUN apt-get install -y libxerces-c-dev automake cmake libboost-all-dev build-essential
RUN git clone https://github.com/mypackage/mypackage-d.git
WORKDIR /mypackage-d/
RUN autoreconf -if
RUN ./configure --enable-silent-rules 'CFLAGS=-g -O0 -w' 'CXXFLAGS=-g -O0 -w' 'LDFLAGS=-g -O0 -w'
RUN make
RUN make install
RUN ls .
# Main Image
FROM library/python:3.7-stretch
COPY --from=builder /mypackage-d/mypackaged.bin /mypackage-d
RUN apt-get update
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y postgresql-client
RUN apt-get install -y libxerces-c-dev
# For VIM
RUN apt-get install -y apt-file
RUN apt-file update
RUN apt-get install -y vim
RUN pip install --upgrade pip
COPY requirements.txt /
RUN pip3 install --trusted-host pypi.org -r /requirements.txt
WORKDIR /code
ENTRYPOINT ["/bin/bash", "start.sh"]
Moving the COPY --from=builder command below the apt-get install and pip install statements worked for me.
Related
I have a python code which runs fine to pull data from an API but I am getting issues to run it via docker. I am using pyodbc to load data into SQLServer in my python code. Here is my dockerfile:
FROM python:3.9.2
RUN apt-get update -y && apt-get install -y --no-install-recommends \
unixodbc-dev \
unixodbc \
libpq-dev
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD ["python3","LoadAPI_data.py"]
After creating the docker image, when I am trying to run the docker image, I get the following error:
Error !!!!: ('01000', "[01000] [unixODBC][Driver Manager]Can't open
lib 'ODBC Driver 17 for SQL Server' : file not found (0)
(SQLDriverConnect)")
Can anyone let me know how do I get rid of this error?
I was able to get my code running by updating my dockerfile to run installation of SQL DB as well as python. Here is what my new dockerfile looks like.
FROM ubuntu:18.04
RUN apt-get update -y && \
apt-get install -y \
libpq-dev \
gcc \
python3-pip \
unixodbc-dev
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential g++-5\
&& rm -rf /var/lib/apt/lists/*
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get install -y --allow-unauthenticated msodbcsql17
RUN pip3 install pyodbc
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD ["python3","LoadAPI_data.py"]
Following is my Dockerfile :-
FROM ubuntu:18.04 AS builder
RUN apt update -y
RUN apt install python3.8 -y && apt install python3-pip -y
RUN apt install build-essential automake pkg-config libtool libffi-dev libgmp-dev -y
RUN apt install libsecp256k1-dev -y
RUN apt install openjdk-8-jre -y
RUN apt install git -y
RUN apt install libkrb5-dev -y
RUN apt install vim -y
RUN mkdir /opt/app
RUN chown -R root:root /opt/app
COPY ["requirements.txt","/opt/app/requirements.txt"]
SHELL ["/bin/bash", "-c"]
WORKDIR /opt/app
RUN pip3 install -r requirements.txt && apt-get -y clean all
RUN mkdir /opt/app/
RUN chown -R root:root /opt/app/
RUN cd /opt/app/
RUN git clone -b master https://bitbucket.org/heroes/test.git
CMD ["bash","/opt/app/bin/connect.sh"]
Docker image is generating with an image file size of 1.7G. I need to have OpenJDK hence cannot use a standard python package as a base package. When I perform docker history , I can see 2 or 3 layers (installing packages above like Python3.8, OpenJDK and libsecp256k1-dev) taking up to 400MB to 500MB in size. Ubuntu as a base image takes only 64 MB however rest of size is taking by my dockerfile layers.
I believe I need to re-write the dockerfile in order to reduce the file size which I did but nothing happened concrete.
Please assist me on reducing the image less than 1 GB at least.
[Update]
Below is my updated Dockerfile:-
FROM ubuntu:18.04 AS builder
WORKDIR /opt/app
COPY requirements.txt /opt/app/aws/requirements.txt
RUN mkdir -p /opt/app/aws \
&& apt-get update -yq \
&& apt-get install -y python3.8 python3-pip openjdk-8-jre -yq && apt-get -y clean all \
&& chown -R root:root /opt/app && cd /opt/app/aws && pip3 install -r requirements.txt
FROM alpine
COPY --from=builder /opt/app /opt/app
SHELL ["/bin/bash", "-c"]
CMD ["bash","/opt/app/aws/bin/connector/connect.sh"]
Screenshot of image size:-
After removing unwanted libraries like git, etc and using the multi-stage build, the image is now approx 1.7 GB which I believe is a lot. Any suggestion to improve this?
You have multiple issues going on.
First, each of your RUN apt install is increasing your image size, you should have them all in the same RUN stage, and at the end of the stage, delete all cached apt files.
Second, you're installing unnecessary stuff. Why would you need vim and git for instance? Why are you installing build-essential and other build-related stuff if you're not building anything?
Third, it seems you tried to do a multi-stage build but ended up adding everything to the same image. Read up on python multi-stage builds.
If we consider best practices instead of multiple RUN use single RUN.
For example
RUN apt-get update -yq \
&& apt-get install -y python3-dev build-essential -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt \
&& apt-get purge -y --auto-remove gcc python3-dev build-essential
you can use multistage builds if you don't require git in your final image you can remove in final stage
Also if possible you can use alpine version also.
Try disabling recommended packages of APT with --no-install-recommends, you can read more about it from here.
Now the image is smaller:
FROM ubuntu:18.04 AS builder
RUN apt update -y
RUN apt install python3-pip -y
RUN apt install build-essential automake pkg-config libtool libffi-dev libgmp-dev -y
RUN apt install libsecp256k1-dev -y
RUN apt install openjdk-8-jre-headless -y
RUN apt install git -y
RUN apt install libkrb5-dev -y
RUN apt install vim -y
RUN mkdir /opt/app
RUN chown -R root:root /opt/app
COPY ["requirements.txt","/opt/app/requirements.txt"]
SHELL ["/bin/bash", "-c"]
WORKDIR /opt/app
RUN pip3 install -r requirements.txt && apt-get -y clean all
RUN mkdir /opt/app/
RUN chown -R root:root /opt/app/
RUN cd /opt/app/
RUN git clone -b master https://bitbucket.org/heroes/test.git
CMD ["bash","/opt/app/bin/connect.sh"]
I have create in docker 2 contains 1 to run the MSSQL server and the other a python container with the code to read data from an .xlsx file and inserting it into SQL server.
My Dockerfile has the below code :
FROM python:3.6-alpine
RUN apk update
RUN apk add gcc libc-dev g++ libffi-dev libxml2 unixodbc-dev mariadb-dev postgresql-dev
FROM continuumio/miniconda3
ADD test.py /
RUN apt-get update -y \
&& apt install python3 -y \
&& apt install python3-pip -y \
&& apt install python3-venv -y \
&& python3 -m venv venv
RUN apt-get -y install curl
**#Install FreeTDS and dependencies for PyODBC**
RUN apt-get update && apt-get install -y tdsodbc unixodbc-dev \
&& apt install unixodbc-bin -y \
&& apt-get clean -y
RUN apt-get update
RUN apt-get install -y tdsodbc unixodbc-dev
RUN apt install unixodbc-bin -y
RUN apt-get clean -y
RUN pip install pandas
RUN pip install pyodbc
RUN pip install DateTime
RUN pip install multiprocess
RUN pip install threaded
CMD [ "python", "./test.py" ]
It compiles successfully but fails every time i run the container with the below error :
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")
I have been trying this for days but found no resolution.
Believe I need to install ODBC driver 17, if so how do I add it to my Dockerfile?
I have to install Python3.7 and pip3 for Python3.7 on my Docker Ubuntu18.04. I can install the 3.7, but I cannot get rid of pip3 for Python3.6:
FROM ubuntu:18.04
# ...
RUN apt-get update && apt-get install -y \
software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y \
python3.7 \
python3-pip
RUN python3.7 -m pip install pip
RUN apt-get update && apt-get install -y \
python3-distutils \
python3-setuptools
and I have
root#ef0c924ba7fa:/tornado_api# python3.7 --version
Python 3.7.3
root#ef0c924ba7fa:/tornado_api# pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
while it should be pip3 under /usr/lib/python3.7/
Currently, I get
root#ef0c924ba7fa:/tornado_api# which pip3
/usr/bin/pip3
root#ef0c924ba7fa:/tornado_api# readlink $(which pip3)
root#ef0c924ba7fa:/tornado_api#
It looks like this has gone stale, nevertheless, I was wondering whether by simply doing a python3.7 -m pip install --upgrade pip
FROM ubuntu:18.04
# ...
RUN apt-get update && apt-get install -y \
software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y \
python3.7 \
python3-pip
RUN python3.7 -m pip install pip
RUN apt-get update && apt-get install -y \
python3-distutils \
python3-setuptools
RUN python3.7 -m pip install pip --upgrade pip
Try 'sudo apt purge pip3' or 'sudo apt-get purge pip3'
If that does not work then try uninstalling pip3 with pip3. (I'm not that sure how)
My next thing to try is to update pip3 with 'pip3 install pip3' (I think)
If those don't work then I don't know.
If you don't need any complex installations, you can simply use a python:3.7 docker image as base. It is anyways a linux based image with all the python requirements installed.
Explore different python images and usage: https://hub.docker.com/_/python
Eg.:
FROM python:3
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD [ "python", "./your-daemon-or-script.py" ]
Or if your requirement needs the same ubuntu image used, you can refer to this answer: Install pip in docker
Just reinstall
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.7 get-pip.py --force-reinstall
This work fine for me...
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y software-properties-common gcc && \
add-apt-repository -y ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.8 python3-distutils python3-pip python3-apt
I have a Dockerfile and I need to include different lines depending on whether I'm running it on my development environment or a raspberry pi.
Can I add in some sort of architecture dependent IF statement around the only lines that vary?
# x64 version (shortened)
FROM node:10
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN apt-get -y update
RUN apt-get -y install build-essential g++
RUN echo 'deb http://deb.debian.org/debian stretch main' > /etc/apt/sources.list
RUN apt-get -y update
RUN apt-get -y install ruby2.3 ruby2.3-dev
The apt source between architectures varies.
# ARM / Raspbian version. (shortened)
FROM node:10
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN apt-get -y update
RUN apt-get -y install build-essential g++
RUN wget https://archive.raspbian.org/raspbian.public.key -O - | apt-key add -
RUN echo 'deb http://archive.raspbian.org/raspbian/ stretch main' > /etc/apt/sources.list
RUN apt-get -y update
RUN apt-get -y install ruby2.3 ruby2.3-dev
In docker file you can use ARG do define parameter for your build process like:
FROM node:10
ARG platform=x64
documentation for it
you can call it like this to change default value
docker build --build-arg platform=arm
and inside your docker file it behaves like any other variable so you can if on it:
RUN if [ "$platform" = "arm" ]; then ... else ... fi