I've got a small Dockerfile where I'm trying to get to a point where I can RUN pip3 bdist_wheel successfully. That is, without getting this error:
unknown command "bdist_wheel" - maybe you meant "wheel"
I've tried installing everything mentioned in this answer, but no luck.
Minimal repro Dockerfile and docker build output:
FROM ubuntu:18.04
RUN apt-get update \
&& apt-get install -qyy -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
file \
gcc \
python3 \
python3-dev \
python3-pip \
python3-setuptools \
python3-venv \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN pip3 install --cache-dir=/tmp/pipcache --upgrade pip && rm -rf /tmp/pipcache
RUN pip install --cache-dir=/tmp/pipcache poetry && rm -rf /tmp/pipcache
WORKDIR /src/app
RUN poetry new .
RUN poetry add gevent
The most relevant part of the output is of course
error: invalid command 'bdist_wheel'
First, some notes to the error: invalid command 'bdist_wheel' output. When running pip install <pkgname>, pip will try to find a prebuilt wheel that matches your target platform. If it doesn't find one, it tries to build a wheel itself -- the source dist is downloaded and pip wheel is run to produce the wheel. On success, the built wheel is installed. On any failure (the wheel package not installed, python setup.py bdist_wheel failed or whatever), pip will fallback to the second option, which is the distutils installation method: running python setup.py install over the unpacked source dist. This is what you can observe in the log you posted:
Failed building wheel for gevent
...
Running setup.py install for gevent: started
Only when the setup.py install also fails, the installation is failed unconditionally. So while pip can't indeed build the wheel b/c the wheel package is not installed, it is not an issue for the failing installation. You can fix this by adding wheel to development packages:
RUN poetry add --dev wheel
RUN poetry add gevent
but this is an optional thing and won't affect the build result.
Now, to the real error:
Running '(cd "/tmp/pip-build-ek9pxyw2/gevent/deps/libev" && sh ./configure -C > configure-output.txt )' in /tmp/pip-build-ek9pxyw2/gevent
config.status: error: in `/tmp/pip-build-ek9pxyw2/gevent/deps/libev':
config.status: error: Something went wrong bootstrapping makefile fragments
for automatic dependency tracking. Try re-running configure with the
'--disable-dependency-tracking' option to at least be able to build
the package (albeit without support for automatic dependency tracking).
See `config.log' for more details
Something went wrong bootstrapping makefile fragments usually means that you're missing make. Install it in addition to the rest:
RUN apt install -y make
After doing that and rerunning the build, I've got the last error
error: src/gevent/libev/corecext.c: No such file or directory
This is because gevent needs Cython for generating the C extension sources. Install it before installing gevent:
RUN poetry add --dev cython
RUN poetry add gevent
The complete Dockerfile for reference, changes in bold:
FROM ubuntu:18.04
RUN apt-get update \
&& apt-get install -qyy -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
file \
gcc \
python3 \
python3-dev \
python3-pip \
python3-setuptools \
python3-venv \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN pip3 install --cache-dir=/tmp/pipcache --upgrade pip && rm -rf /tmp/pipcache
RUN pip install --cache-dir=/tmp/pipcache poetry && rm -rf /tmp/pipcache
WORKDIR /src/app
RUN poetry new .
RUN apt update
RUN apt install -y make
RUN poetry add --dev wheel cython
RUN poetry add gevent
Neither wheel nor cython are required to actually run gevent, so they can be safely uninstalled afterwards to reduce the image size.
I'm trying to install Libressl in a python base docker image. The Python image has openssl by default.
My Dockerfile code:
FROM python:3.7
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install git cpp make dh-autoreconf -y
RUN pip3 install requests
RUN git clone https://github.com/libressl-portable/portable.git /portable
RUN cd /portable \
./autogen.sh \
./configure --prefix=/opt/libressl --enable-nc \
make check \
make install
RUN echo "alias openssl='/opt/libressl/bin/openssl'" >> ~/.bashrc
COPY . /app
WORKDIR /app
CMD ["python3", "./debug.py"]
But, I found that git clone is done well, but the next command failed.
Even autogen.sh doesn't seem to be executed.
And how can I source that .bashrc file?
When I use source ~/.bashrc, source command cannot be found because the command is run with /bin/sh.
What could be a problem in my Dockerfile?
Thanks :)
To combine several command calls use operator &&. Additionally you can use one make call with two targets
cd /portable && \
./autogen.sh && \
./configure --prefix=/opt/libressl --enable-nc && \
make check install
The answer from #alexander is great.
Another approach for it is: executing shell script from Dockerfile and have all the shell command in one place. make the Dockerfile more elegant.
for example :
FROM python:3.7
COPY . /app
RUN ./app/script.sh
WORKDIR /app
CMD ["python3", "./debug.py"]
and the script.sh (you write as simple shell script) (copy\past what you published without testing ) :
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install git cpp make dh-autoreconf -y
pip3 install requests
git clone https://github.com/libressl-portable/portable.git /portable
cd /portable \
./autogen.sh \
./configure --prefix=/opt/libressl --enable-nc \
make check
make install
echo "alias openssl='/opt/libressl/bin/openssl'" >> ~/.bashrc
I am new to docker container and I want to build an image with basic environment. Here is part of my Dockerfile:
FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
ARG CTAGS_DIR=~/tools/ctags
ARG RIPGREP_DIR=~/tools/ripgrep
ARG ANACONDA_DIR=~/tools/anaconda
ARG NVIM_DIR=~/tools/nvim
ARG NVIM_CONFIG_DIR=~/.config/nvim
# Install common dev tools
RUN apt-get update --allow-unauthenticated \
&& apt-get install --allow-unauthenticated -y git curl autoconf pkg-config zsh
# Install anaconda
COPY ./packages/Anaconda3-2019.07-Linux-x86_64.sh /tmp/anaconda.sh
RUN chmod u+x /tmp/anaconda.sh \
&& bash /tmp/anaconda.sh -b -p ${ANACONDA_DIR} \
&& rm /tmp/anaconda.sh
ENV PATH=${ANACONDA_DIR}/bin:$PATH
# RUN echo $PATH && ls -l /root/tools/anaconda/bin|grep pip
RUN echo $PATH && ls -l ~/tools/anaconda/bin|grep pip
# Python packages
RUN pip install pynvim jedi pylint
The build process fails at the pip install step complaining that
/bin/sh: 1: pip: not found
The command '/bin/sh -c pip install pynvim jedi pylint' returned a non-zero code: 127
But the output of command
RUN echo $PATH && ls -l ~/tools/anaconda/bin|grep pip
is the following
~/tools/anaconda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-rwxrwxr-x 1 root root 231 Sep 28 08:19 pip
which suggests that PATH is set and pip is foundable. I am not sure what is the problem here.
The only explanation is that PATH is set but is not correctly set. I do not know why.
Can someone experience explain what happened? What is wrong with my Dockerfile?
I don't see pip in the base image you used in your Dockerfile, you can check the offical Dockerfile, nor in the base image of nvidia/cuda, you can check the base image too 10.0-cudnn7-devel-ubuntu18.04
Installed pip and then try
FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
RUN apt update && apt install python3-pip -y
RUN pip3 --version
I have this file Dockerfile.nlu
FROM chatbot/spacy:latest
WORKDIR /app
COPY nlu ./agent_nlu
RUN python –m rasa_nlu.train --config agent_nlu/config.yml --data agent_nlu/data/ --path agent_nlu/agent --fixed_model_name default
and I get the error below:
]$ sudo docker build -t nlu:latest -f docker/Dockerfile.nlu .
Sending build context to Docker daemon 9.216kB
Step 1/4 : FROM chatbot/spacy:latest
---> 496dc6a38abb
Step 2/4 : WORKDIR /app
---> Using cache
---> 7f02012c8452
Step 3/4 : COPY nlu ./agent_nlu
COPY failed: stat /var/lib/docker/tmp/docker-builder363868051/nlu: no such file or directory
It doesn't look like Docker can find the nlu directory. Are you sure it exists? Are you sure that you are executing the command from the correct directory?
But you also aren't installing Rasa at all or any of it's requirements. Is there a reason you aren't using the pre-built Rasa images? available here with docs here.
Here is a fully functional Docker file pulled from their repo.
FROM python:3.6-slim
ENV RASA_NLU_DOCKER="YES" \
RASA_NLU_HOME=/app \
RASA_NLU_PYTHON_PACKAGES=/usr/local/lib/python3.6/dist-packages
# Run updates, install basics and cleanup
# - build-essential: Compile specific dependencies
# - git-core: Checkout git repos
RUN apt-get update -qq \
&& apt-get install -y --no-install-recommends build-essential git-core openssl libssl-dev libffi6 libffi-dev curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
WORKDIR ${RASA_NLU_HOME}
COPY . ${RASA_NLU_HOME}
# use bash always
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN pip install -r alt_requirements/requirements_spacy_sklearn.txt
RUN pip install -e .
RUN pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.0.0/en_core_web_md-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link en_core_web_md en \
&& pip install https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link de_core_news_sm de
COPY sample_configs/config_spacy.yml ${RASA_NLU_HOME}/config.yml
VOLUME ["/app/projects", "/app/logs", "/app/data"]
EXPOSE 5000
ENTRYPOINT ["./entrypoint.sh"]
CMD ["start", "-c", "config.yml", "--path", "/app/projects"]
I'm trying to connect a pyodbc python script running in a docker container to login to a MSSQL database I have tried all sorts of docker files, but not been able to make the connection (fails when bulding the docker or when python tries to connect), Does anyone have a working dockerfile, using pyodbc:
Dockerfile:
# Use an official Python runtime as a parent image
FROM python:2.7-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt
# Run app.py when the container launches
CMD ["python", "App.py"]
requirements.TXT
pyodbc
App.Py
import pyodbc
connection = pyodbc.connect('Driver={SQL Server};'
'Server=xxxx;'
'Database=xxx;'
'UID=xxxx;'
'PWD=xxxx')
cursor = connection.cursor()
cursor.execute("SELECT [Id],[Name] FROM [DCMM].[config].[Models]")
for row in cursor.fetchall():
print(row.Name)
connection.close()
Bulding the container
docker build -t sqltest .
Output:
Sending build context to Docker daemon 4.096kB
Step 1/5 : FROM python:2.7-slim
---> 426d65ab9a72
Step 2/5 : WORKDIR /app
---> Using cache
---> 725f35122880
Step 3/5 : ADD . /app
---> 3feb8b7744f7
Removing intermediate container 4214091a111a
Step 4/5 : RUN pip install -r requirements.txt
---> Running in 27aa4dcfe738
Collecting pyodbc (from -r requirements.txt (line 1))
Downloading pyodbc-4.0.17.tar.gz (196kB)
Building wheels for collected packages: pyodbc
Running setup.py bdist_wheel for pyodbc: started
Running setup.py bdist_wheel for pyodbc: finished with status 'error'
Failed building wheel for pyodbc
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpa3S13tpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=4.0.17 -DSQL_WCHART_CONVERT=1 -I/usr/local/include/python2.7 -c src/cursor.cpp -o build/temp.linux-x86_64-2.7/src/cursor.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Running setup.py clean for pyodbc
Failed to build pyodbc
Installing collected packages: pyodbc
Running setup.py install for pyodbc: started
Running setup.py install for pyodbc: finished with status 'error'
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-BV4sRM-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=4.0.17 -DSQL_WCHART_CONVERT=1 -I/usr/local/include/python2.7 -c src/cursor.cpp -o build/temp.linux-x86_64-2.7/src/cursor.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-BV4sRM-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-EfWsmy/pyodbc/
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
Need to Run:
sudo apt-get install gcc
need to add a odbcinst.ini file containing:
[FreeTDS]Description=FreeTDS Driver Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
need to add folowing to docker file
ADD odbcinst.ini /etc/odbcinst.ini
RUN apt-get update
RUN apt-get install -y tdsodbc unixodbc-dev
RUN apt install unixodbc-bin -y
RUN apt-get clean -y
need to change connection in .py to
connection = pyodbc.connect('Driver={FreeTDS};'
'Server=xxxxx;'
'Database=DCMM;'
'UID=xxxxx;'
'PWD=xxxxx')
Now the container compiles, and gets data from SQL server
Running through this recently I found it was necessary to additionally include the following line (note that it did not build without this step):
RUN apt-get install --reinstall build-essential -y
The full Dockerfile looks as follows:
# parent image
FROM python:3.7-slim
# install FreeTDS and dependencies
RUN apt-get update \
&& apt-get install unixodbc -y \
&& apt-get install unixodbc-dev -y \
&& apt-get install freetds-dev -y \
&& apt-get install freetds-bin -y \
&& apt-get install tdsodbc -y \
&& apt-get install --reinstall build-essential -y
# populate "ocbcinst.ini"
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini
# install pyodbc (and, optionally, sqlalchemy)
RUN pip install --trusted-host pypi.python.org pyodbc==4.0.26 sqlalchemy==1.3.5
# run app.py upon container launch
CMD ["python", "app.py"]
Here's one way to then actually establish the connection inside app.py, via sqlalchemy (and assuming port 1433):
import sqlalchemy as sa
args = (username, password, server, database)
connstr = "mssql+pyodbc://{}:{}#{}/{}?driver=FreeTDS&port=1433&odbc_options='TDS_Version=8.0'"
engine = sa.create_engine(connstr.format(*args))
Based on Kåre Rasmussen's answer, here's a complete dockerfile for further use.
Make sure to edit the last two lines according to your architecture! They should reflect the actual paths to libtdsodbc.so and libtdsS.so.
If you're not sure about the paths to libtdsodbc.so and libtdsS.so, try dpkg --search libtdsodbc.so and dpkg --search libtdsS.so.
FROM python:3
#Install FreeTDS and dependencies for PyODBC
RUN apt-get update && apt-get install -y tdsodbc unixodbc-dev \
&& apt install unixodbc-bin -y \
&& apt-get clean -y
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/arm-linux-gnueabi/odbc/libtdsodbc.so\n\
Setup = /usr/lib/arm-linux-gnueabi/odbc/libtdsS.so" >> /etc/odbcinst.ini
Afterwards, install PyODBC, COPY your app and run it.
I was unable to use all of the above resolutions, I was keeping al kind of errors relating to the pyodbc package, in particular:
ImportError: libodbc.so.2: cannot open shared object file: No such file or directory.
I ended up with another resolution which defines the ODBC SQL Server Driver specifically for an Ubuntu 18.04 Docker image, in this case ODBC Driver 17 for SQL Server. In my specific use case I needed to make the connection to my MySQL database server on Azure via Flask SQLAlchemy, but the latter is not a necessity for the Docker configuration.
Dockerfile, with most important part adding the Microsoft repository and installing msodbcsql17 and unixodbc-dev:
# Ubuntu 18.04 base with Python runtime and pyodbc to connect to SQL Server
FROM ubuntu:18.04
WORKDIR /app
# apt-get and system utilities
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential g++-5\
&& rm -rf /var/lib/apt/lists/*
# adding custom Microsoft repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
# install SQL Server drivers
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
# install SQL Server tools
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
RUN /bin/bash -c "source ~/.bashrc"
# python libraries
RUN apt-get update -y && \
apt-get install -y python3-pip python3-dev
# install necessary locales, this prevents any locale errors related to Microsoft packages
RUN apt-get update && apt-get install -y locales \
&& echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \
&& locale-gen
# copy requirements and install packages, I added this for general use
COPY ./requirements.txt > ./requirements.txt
RUN pip3 install -r ./requirements.txt
# you can also use regular install of the packages
RUN pip3 install pyodbc SQLAlchemy
# and if you are also planning to use Flask and Flask-SQLAlchemy
Run pip3 install Flask Flask-SQLAlchemy
COPY ..
# run your app via entrypoint or change the CMD command to your regular command
COPY docker-entrypoint.sh wsgi.py ./
CMD ["./docker-entrypoint.sh"]
This should build without any errors in Docker.
My database url looked like this:
import urllib.parse
# name the sepcific ODBC driver by version number, we installed msodbcsql17
params = urllib.parse.quote_plus("DRIVER={ODBC Driver 17 for SQL Server};SERVER=<your.database.windows.net>;DATABASE=<your-db-name>;UID=<username>;PWD=<password>")
db_uri = "mssql+pyodbc:///?odbc_connect={PARAMS}".format(PARAMS=params)
And for the bonus if you are using Flask-SQLAlchemy, your app config should contain something like this:
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri # from above
Happy coding!
How to install the necessary dependencies for pyodbc is related to the linux distribution and its version (in docker case, that is the base image of your docker image). If none of the above work for you, you can figure out the commands by trying in the docker container instance.
First, exec into the docker container
docker exec -it <container id> bash
Try various ways to get the distribution name and version of your linux. Then try different instructions in Install the Microsoft ODBC driver for SQL Server (Linux)
Here is a working example for Debian 9 based images, deriving exactly as the document instructions.
# Install pyodbc dependencies
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql17
RUN apt-get -y install unixodbc-dev
RUN pip install pyodbc
For me to solve this issue I also had to add the following 2 lines in the dockerfile:
RUN echo MinProtocol = TLSv1.0 >> /etc/ssl/openssl.cnf
RUN echo CipherString = DEFAULT#SECLEVEL=1 >> /etc/ssl/openssl.cnf
For those who wanted to do official microsoft approach to install odbc driver and use python:slim docker image, you can use this as DockerFile:
FROM python:3.9-slim
RUN apt-get -y update && apt-get install -y curl gnupg
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
# download appropriate package for the OS version
# Debian 11
RUN curl https://packages.microsoft.com/config/debian/11/prod.list \
> /etc/apt/sources.list.d/mssql-release.list
RUN exit
RUN apt-get -y update
RUN ACCEPT_EULA=Y apt-get install -y msodbcsql18
Then for sqlalchemy's this can be called:
con_str = f"mssql+pyodbc://{username}:{password}#{host}/{db}?" \
"driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes"
engine = create_engine(con_str)
I created a Gist on GitHub on how to do this. I hope it helps. I had to piece things together from what I found on different resources.
https://gist.github.com/joshatxantie/4bcf5d0243fba63845fce7cc40365a3a
Goodluck!
I fixed this problem by using pypyodbc instead of pyodbc.
pip install pypyodbc==1.3.5
https://pypi.org/project/pypyodbc/
Found the hint here:
https://github.com/Azure/azure-functions-python-worker/issues/249
For not more problem use library for python
pymssql this not need install driver
pip install pymssql
import pymssql
conn = pymssql.connect(server, user, password, "tempdb")
cursor = conn.cursor(as_dict=True)
cursor.execute('SELECT * FROM persons WHERE salesrep=%s', 'John Doe')
for row in cursor:
print("ID=%d, Name=%s" % (row['id'], row['name']))
conn.close()
and work in docker