SOLUTION
I needed to add RUN apk add gcc g++ musl-dev libffi-dev openssl-dev to Dockerfile. Now it works. Just building time increased.
Original Question
I'm new to docker.
I've a class where I run google translate API. It runs on local.
Dockerfile:
FROM python:3.6.6-alpine3.8
# Project files
ARG PROJECT_DIR=/srv/api
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
RUN apk add gcc musl-dev libffi-dev openssl-dev
# Install Python dependencies
COPY ./ ./
RUN pip install --upgrade pip
RUN pip install google-cloud-translate
RUN pip install -r requirements.txt
It crashes when I docker-compose build
ERROR: Complete output from command /usr/local/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9k79v_vp/grpcio/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-3w3baynf --python-tag cp36:
ERROR: Found cython-generated files...
running bdist_wheel
running build
running build_py
running build_project_metadata
creating python_build
creating python_build/lib.linux-x86_64-3.6
creating python_build/lib.linux-x86_64-3.6/grpc
copying src/python/grpcio/grpc/_server.py -> python_build/lib.linux-x86_64-3.6/grpc
<..>
ERROR: Command "/usr/local/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9k79v_vp/grpcio/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-vb12kr37/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-9k79v_vp/grpcio/
Any ideas?
gcc: error trying to exec 'cc1plus': execvp: No such file or directory
You need to install g++ as well.
Related
I am quite new to docker. My problem in short is to create a docker file that contains python with sklearn and pandas which can be used on aws sagemaker.
My current docker file looks like the following:
FROM jupyter/scipy-notebook
RUN pip3 install sagemaker-training
COPY train.py /opt/ml/code/train.py
ENV SAGEMAKER_PROGRAM train.py
However when i try to create this image I get an error at line pip3 install sagemaker-training. The error is the following:
error: command 'gcc' failed: No such file or directory
ERROR: Command errored out with exit status 1: /opt/conda/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fj0cb373/sagemaker-training_66ca9935ed134c95ac11a32e118e4568/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fj0cb373/sagemaker-training_66ca9935ed134c95ac11a32e118e4568/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-o5rzjscd/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.9/sagemaker-training Check the logs for full command output.
The command '/bin/bash -o pipefail -c pip3 install sagemaker-training' returned a non-zero code: 1
If there is a more suitable base image can someone point that out to me? I am generally trying to follow this page https://github.com/aws/sagemaker-training-toolkit.
Note: I realise I can use some sagemaker pre-built containers without using my own docker file. However I am trying to do this for my own learning so I know what to do for projects that can't utilise them.
I adjusted your Dockerfile and it builds successfully for me.
FROM jupyter/scipy-notebook
ARG defaultuser=jovyan
USER root
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && \
apt-get -y install gcc mono-mcs && \
rm -rf /var/lib/apt/lists/*
USER $defaultuser
RUN pip3 install sagemaker-training
COPY train.py /opt/ml/code/train.py
ENV SAGEMAKER_PROGRAM train.py
(I had to adjust for the fact that the default user from the base container isn't root, when installing GCC)
Executing 'docker build -t $name .' failed at following issue. It shows 'mariadb_config not found', but i already installed mariadb on this suse15 linux server.
Collecting mariadb==1.0.4
Downloading mariadb-1.0.4.tar.gz (66 kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-lxx0giq5/mariadb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-lxx0giq5/mariadb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-hicsuruq
cwd: /tmp/pip-install-lxx0giq5/mariadb/
Complete output (17 lines):
/bin/sh: 1: mariadb_config: not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-lxx0giq5/mariadb/setup.py", line 26, in <module>
cfg = get_config(options)
File "/tmp/pip-install-lxx0giq5/mariadb/mariadb_posix.py", line 59, in get_config
cc_version = mariadb_config(config_prg, "cc_version")
File "/tmp/pip-install-lxx0giq5/mariadb/mariadb_posix.py", line 29, in mariadb_config
"mariadb_config not found.\n\nPlease make sure, that MariaDB Connector/C is installed on your system.\n"
OSError: mariadb_config not found.
Please make sure, that MariaDB Connector/C is installed on your system.
Either set the environment variable MARIADB_CONFIG or edit the configuration
file 'site.cfg' and set the 'mariadb_config option, which should point
to the mariadb_config utility.
The MariaDB Download website at <https://downloads.mariadb.com/Connectors/c/>
provides latest stable releease of Connector/C.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
The command '/bin/sh -c pip3 install -r requirements.txt' returned a non-zero code: 1
And related Dockerfile content is as follows.
FROM python:3.6.5
WORKDIR /slic-scripts
COPY requirements.txt .
RUN pip install --upgrade pip
RUN pip3 install -r requirements.txt
COPY . .
CMD ["python3", "/slic-scripts/run_cmd.sh"]
As #Jonathan Jacobson wrote, you need to have the MariaDB Connector installed in your Docker image before proceeding with pip install.
Since python:3.6.5 is based on debian stretch, the connector offered there is 2.3.2 which is not supported by the mariadb module (as stated in the prerequisites). To fix this, you might want to switch to a different image (e.g. 3.6-buster).
Here's a working test:
FROM python:3.6-buster
RUN apt-get update \
&& apt-get -yy install libmariadb-dev
WORKDIR /slic-scripts
COPY requirements.txt .
RUN pip install --upgrade pip
RUN pip3 install -r requirements.txt
COPY . .
CMD ["python3", "/slic-scripts/run_cmd.sh"]
You need to install MariaDB Connector/C inside the Dockerfile, before running pip3 install. It doesn't help that you installed it on your host.
Install MariaDB Connector/C using the below command, which is a dependency.
sudo apt-get install libmariadb3 libmariadb-dev
RUN pip3 install -r requirements.txt
My Dockerfile using a pypy base fails with FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config' when installing llvmlite, a dependency of numba which is listed in my requirements.txt
I tried to follow and update the instructions here
Python numba / llvmlite on Debian 8 - i can't build llvmlite
My error in more detail:
Building wheels for collected packages: llvmlite
Building wheel for llvmlite (setup.py): started
Building wheel for llvmlite (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/pypy3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ux49fegr/llvmlite/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ux49fegr/llvmlite/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-8s6wwump --python-tag pp371
cwd: /tmp/pip-install-ux49fegr/llvmlite/
Complete output (26 lines):
running bdist_wheel
/usr/local/bin/pypy3 /tmp/pip-install-ux49fegr/llvmlite/ffi/build.py
LLVM version... Traceback (most recent call last):
File "/tmp/pip-install-ux49fegr/llvmlite/ffi/build.py", line 105, in main_posix
out = subprocess.check_output([llvm_config, '--version'])
File "/usr/local/lib-python/3/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/local/lib-python/3/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib-python/3/subprocess.py", line 722, in __init__
restore_signals, start_new_session)
File "/usr/local/lib-python/3/subprocess.py", line 1354, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'
I don't want the heavy weight of a conda installation. Is there a way to achieve a numba install for pypy without it?
FROM pypy:3.6-slim-stretch
RUN apt-get update && apt-get install -y gnupg wget software-properties-common
RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - 15CF4D18AF4F7421
RUN apt-add-repository "deb http://apt.llvm.org/stretch/ llvm-toolchain-stretch-6.0 main"
RUN apt-get update
RUN apt-get -y install cython python-llvm build-essential libedit-dev
RUN apt-get -y install libllvm6.0 llvm-6.0-dev llvm-dev
RUN pip install enum34
ENV LLVM_CONFIG=/usr/lib/llvm-6.0-dev/bin/llvm-config pip install llvmlite
ENV LLVM_CONFIG=/usr/lib/llvm-6.0-dev/bin/llvm-config pip install numba
# ENV LD_LIBRARY_PATH /usr/lib/llvm-6.0-dev/lib/
#... other stuff
RUN apt-get clean && apt-get -y update
RUN apt-get -y install python3-dev \
&& apt-get -y install build-essential
# Add requirements
COPY requirements.txt /tmp/
# Install sphinx first as it does not work inside requirements
RUN pip install -U pip && pip install -U sphinx && pip install numpy && pip install cython && \
pip install -r /tmp/requirements.txt
I expect a clean build of the docker image with numba on pypy
WORKAROUND If the error appeared after April the 17th (they released a new version, see history).
As a workaround try installing a previous version
pip3 install llvmlite==0.31.0
At the moment numba won't install on python 3.9.
I changed my base docker image from python:3.9-slim to python:3.8-slim and then pip install numba succeeded
This fixed it for me -
RUN echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-8 main" >> /etc/apt/sources.list
RUN echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-8 main" >> /etc/apt/sources.list
RUN apt-get install -y --no-install-recommends libedit-dev build-essential
RUN apt-get install -y --no-install-recommends llvm-8 llvm-8-dev
RUN LLVM_CONFIG=/usr/bin/llvm-config-8 pip3 install enum34 llvmlite numba
Used version 8, because got an error with the latest llvm version -
RuntimeError: Building llvmlite requires LLVM 7.0.x, 7.1.x or 8.0.x, got '3.7.1'
Reference - https://apt.llvm.org/
The "real" solution would be to encourage llvmlite and/or numba to release a wheel for pypy. But for the immediate problem...
Somehow you are missing an llvm installation, at least according to apt-file:
$ apt-file find llvm-config |grep llvm-6
llvm-6.0: /usr/bin/llvm-config-6.0
llvm-6.0: /usr/lib/llvm-6.0/bin/llvm-config
llvm-6.0: /usr/share/man/man1/llvm-config-6.0.1.gz
For installing numba on Python 3.9 on macOS, the following works:
# install llvm using MacPorts
sudo port install llvm-10
sudo port install llvm_select
sudo port select --set llvm mp-llvm-10
# set environment variable used when building `llvmlite`
export LLVM_CONFIG=/opt/local/bin/llvm-config
# clone, build, and install `llvmlite`
# http://llvmlite.pydata.org/en/latest/admin-guide/install.html#compiling-llvmlite
git clone git#github.com:numba/llvmlite.git
cd llvmlite
python setup.py build
python runtests.py
python setup.py install
cd ..
# clone, build, and install `numba`
# https://numba.readthedocs.io/en/stable/user/installing.html#installing-from-source
git clone git#github.com:numba/numba.git
cd numba
export PATH=/usr/bin/:$PATH # use clang, same as used for building Python 3.9
pip install -v .
cd ..
The above are with:
llvmlite at commit 334411cf638c8e06ba19bfda16b07dd64e9dac3c, and
numba at commit ca8132ba5e43fc3f79767046ed55217aeeabb35e.
according to https://askubuntu.com/questions/1286131/how-do-i-install-llvm-10-on-ubuntu-18-04
I added to Dockerfile the lines
RUN apt-get -y install llvm-10*
RUN rm -f /usr/bin/llvm-config
RUN ln -s /usr/bin/llvm-config-10 /usr/bin/llvm-config
and It worked (llvmlite installed)
This is the solution, obtained from this github issues thread, that worked for me! Essentially, pip just needed an upgrade in the Dockerfile before installing libraries. All I needed to do was add this line in my Dockerfile:
--upgrade pip \
Here is the sample of my Dockerfile:
# Docker commands here ...
RUN APT_INSTALL="apt-get install -y " && \
PIP_INSTALL="python3 -m pip install " && \
apt-get update && \
# more docker commands here ...
DEBIAN_FRONTEND=noninteractive $APT_INSTALL \
python3.6 \
python3.6-dev \
python3-distutils-extra \
python3-pip && \
ln -s /usr/bin/python3.6 /usr/local/bin/python3 && \
$PIP_INSTALL \
--upgrade pip \ # adding this line was my solution
setuptools \
&& \
$PIP_INSTALL \
numpy==1.18.5 \
# more docker commands here...
This is for python 3.6
I'm new to Docker, so there could be something I'm doing wrong. I'm trying to install Adafruit-GPIO in a container, but I keep getting this error and I'm not sure how to solve it. I'm building the file locally on a Windows 10 pc, using Powershell.
I'v made sure that the GCC is installed correctly and it is. I've been searching online for an answer and there isn't much to go on.
Dockerfile:
FROM balenalib/raspberry-pi-debian-python:3.7.2
RUN pip3 install --upgrade pip
RUN sudo pip3 install --upgrade setuptools
RUN sudo apt-get update && apt-get -y install gcc
RUN pip3 install adafruit-gpio
I'm expecting the file to compile and work successfully. Instead I'm getting this,
ERROR: Complete output from command /usr/local/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-qfol0jyh/spidev/setup.py'"'"';f=getattr(tokenize, '"'"'open'"
'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-i37a5wvt/install-record.txt -
-single-version-externally-managed --compile:
ERROR: running install
running build
running build_ext
building 'spidev' extension
creating build
creating build/temp.linux-armv6l-3.7
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.7m -c spidev_module.c -o build/temp.linux-armv6l-3.7/spidev_module.o
In file included from /usr/lib/gcc/arm-linux-gnueabihf/6/include-fixed/syslimits.h:7:0,
from /usr/lib/gcc/arm-linux-gnueabihf/6/include-fixed/limits.h:34,
from /usr/local/include/python3.7m/Python.h:11,
from spidev_module.c:28:
/usr/lib/gcc/arm-linux-gnueabihf/6/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command "/usr/local/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-qfol0jyh/spidev/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);cod
e=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-i37a5wvt/install-record.txt --single-version-external
ly-managed --compile" failed with error code 1 in /tmp/pip-install-qfol0jyh/spidev/
Install build-essential to replace gcc could make you work, see official git.
In fact, when build some source code, always better to install build-essential because it will not only install gcc, but also some dev package. Your error fatal error: limits.h: No such file or directory just because miss these dev package.
In a word, next will works for you:
RUN sudo apt-get update && apt-get -y install build-essential
I'm trying to connect a pyodbc python script running in a docker container to login to a MSSQL database I have tried all sorts of docker files, but not been able to make the connection (fails when bulding the docker or when python tries to connect), Does anyone have a working dockerfile, using pyodbc:
Dockerfile:
# Use an official Python runtime as a parent image
FROM python:2.7-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt
# Run app.py when the container launches
CMD ["python", "App.py"]
requirements.TXT
pyodbc
App.Py
import pyodbc
connection = pyodbc.connect('Driver={SQL Server};'
'Server=xxxx;'
'Database=xxx;'
'UID=xxxx;'
'PWD=xxxx')
cursor = connection.cursor()
cursor.execute("SELECT [Id],[Name] FROM [DCMM].[config].[Models]")
for row in cursor.fetchall():
print(row.Name)
connection.close()
Bulding the container
docker build -t sqltest .
Output:
Sending build context to Docker daemon 4.096kB
Step 1/5 : FROM python:2.7-slim
---> 426d65ab9a72
Step 2/5 : WORKDIR /app
---> Using cache
---> 725f35122880
Step 3/5 : ADD . /app
---> 3feb8b7744f7
Removing intermediate container 4214091a111a
Step 4/5 : RUN pip install -r requirements.txt
---> Running in 27aa4dcfe738
Collecting pyodbc (from -r requirements.txt (line 1))
Downloading pyodbc-4.0.17.tar.gz (196kB)
Building wheels for collected packages: pyodbc
Running setup.py bdist_wheel for pyodbc: started
Running setup.py bdist_wheel for pyodbc: finished with status 'error'
Failed building wheel for pyodbc
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpa3S13tpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=4.0.17 -DSQL_WCHART_CONVERT=1 -I/usr/local/include/python2.7 -c src/cursor.cpp -o build/temp.linux-x86_64-2.7/src/cursor.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Running setup.py clean for pyodbc
Failed to build pyodbc
Installing collected packages: pyodbc
Running setup.py install for pyodbc: started
Running setup.py install for pyodbc: finished with status 'error'
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-BV4sRM-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=4.0.17 -DSQL_WCHART_CONVERT=1 -I/usr/local/include/python2.7 -c src/cursor.cpp -o build/temp.linux-x86_64-2.7/src/cursor.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-BV4sRM-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-EfWsmy/pyodbc/
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
Need to Run:
sudo apt-get install gcc
need to add a odbcinst.ini file containing:
[FreeTDS]Description=FreeTDS Driver Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
need to add folowing to docker file
ADD odbcinst.ini /etc/odbcinst.ini
RUN apt-get update
RUN apt-get install -y tdsodbc unixodbc-dev
RUN apt install unixodbc-bin -y
RUN apt-get clean -y
need to change connection in .py to
connection = pyodbc.connect('Driver={FreeTDS};'
'Server=xxxxx;'
'Database=DCMM;'
'UID=xxxxx;'
'PWD=xxxxx')
Now the container compiles, and gets data from SQL server
Running through this recently I found it was necessary to additionally include the following line (note that it did not build without this step):
RUN apt-get install --reinstall build-essential -y
The full Dockerfile looks as follows:
# parent image
FROM python:3.7-slim
# install FreeTDS and dependencies
RUN apt-get update \
&& apt-get install unixodbc -y \
&& apt-get install unixodbc-dev -y \
&& apt-get install freetds-dev -y \
&& apt-get install freetds-bin -y \
&& apt-get install tdsodbc -y \
&& apt-get install --reinstall build-essential -y
# populate "ocbcinst.ini"
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini
# install pyodbc (and, optionally, sqlalchemy)
RUN pip install --trusted-host pypi.python.org pyodbc==4.0.26 sqlalchemy==1.3.5
# run app.py upon container launch
CMD ["python", "app.py"]
Here's one way to then actually establish the connection inside app.py, via sqlalchemy (and assuming port 1433):
import sqlalchemy as sa
args = (username, password, server, database)
connstr = "mssql+pyodbc://{}:{}#{}/{}?driver=FreeTDS&port=1433&odbc_options='TDS_Version=8.0'"
engine = sa.create_engine(connstr.format(*args))
Based on Kåre Rasmussen's answer, here's a complete dockerfile for further use.
Make sure to edit the last two lines according to your architecture! They should reflect the actual paths to libtdsodbc.so and libtdsS.so.
If you're not sure about the paths to libtdsodbc.so and libtdsS.so, try dpkg --search libtdsodbc.so and dpkg --search libtdsS.so.
FROM python:3
#Install FreeTDS and dependencies for PyODBC
RUN apt-get update && apt-get install -y tdsodbc unixodbc-dev \
&& apt install unixodbc-bin -y \
&& apt-get clean -y
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/arm-linux-gnueabi/odbc/libtdsodbc.so\n\
Setup = /usr/lib/arm-linux-gnueabi/odbc/libtdsS.so" >> /etc/odbcinst.ini
Afterwards, install PyODBC, COPY your app and run it.
I was unable to use all of the above resolutions, I was keeping al kind of errors relating to the pyodbc package, in particular:
ImportError: libodbc.so.2: cannot open shared object file: No such file or directory.
I ended up with another resolution which defines the ODBC SQL Server Driver specifically for an Ubuntu 18.04 Docker image, in this case ODBC Driver 17 for SQL Server. In my specific use case I needed to make the connection to my MySQL database server on Azure via Flask SQLAlchemy, but the latter is not a necessity for the Docker configuration.
Dockerfile, with most important part adding the Microsoft repository and installing msodbcsql17 and unixodbc-dev:
# Ubuntu 18.04 base with Python runtime and pyodbc to connect to SQL Server
FROM ubuntu:18.04
WORKDIR /app
# apt-get and system utilities
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential g++-5\
&& rm -rf /var/lib/apt/lists/*
# adding custom Microsoft repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
# install SQL Server drivers
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
# install SQL Server tools
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
RUN /bin/bash -c "source ~/.bashrc"
# python libraries
RUN apt-get update -y && \
apt-get install -y python3-pip python3-dev
# install necessary locales, this prevents any locale errors related to Microsoft packages
RUN apt-get update && apt-get install -y locales \
&& echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \
&& locale-gen
# copy requirements and install packages, I added this for general use
COPY ./requirements.txt > ./requirements.txt
RUN pip3 install -r ./requirements.txt
# you can also use regular install of the packages
RUN pip3 install pyodbc SQLAlchemy
# and if you are also planning to use Flask and Flask-SQLAlchemy
Run pip3 install Flask Flask-SQLAlchemy
COPY ..
# run your app via entrypoint or change the CMD command to your regular command
COPY docker-entrypoint.sh wsgi.py ./
CMD ["./docker-entrypoint.sh"]
This should build without any errors in Docker.
My database url looked like this:
import urllib.parse
# name the sepcific ODBC driver by version number, we installed msodbcsql17
params = urllib.parse.quote_plus("DRIVER={ODBC Driver 17 for SQL Server};SERVER=<your.database.windows.net>;DATABASE=<your-db-name>;UID=<username>;PWD=<password>")
db_uri = "mssql+pyodbc:///?odbc_connect={PARAMS}".format(PARAMS=params)
And for the bonus if you are using Flask-SQLAlchemy, your app config should contain something like this:
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri # from above
Happy coding!
How to install the necessary dependencies for pyodbc is related to the linux distribution and its version (in docker case, that is the base image of your docker image). If none of the above work for you, you can figure out the commands by trying in the docker container instance.
First, exec into the docker container
docker exec -it <container id> bash
Try various ways to get the distribution name and version of your linux. Then try different instructions in Install the Microsoft ODBC driver for SQL Server (Linux)
Here is a working example for Debian 9 based images, deriving exactly as the document instructions.
# Install pyodbc dependencies
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql17
RUN apt-get -y install unixodbc-dev
RUN pip install pyodbc
For me to solve this issue I also had to add the following 2 lines in the dockerfile:
RUN echo MinProtocol = TLSv1.0 >> /etc/ssl/openssl.cnf
RUN echo CipherString = DEFAULT#SECLEVEL=1 >> /etc/ssl/openssl.cnf
For those who wanted to do official microsoft approach to install odbc driver and use python:slim docker image, you can use this as DockerFile:
FROM python:3.9-slim
RUN apt-get -y update && apt-get install -y curl gnupg
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
# download appropriate package for the OS version
# Debian 11
RUN curl https://packages.microsoft.com/config/debian/11/prod.list \
> /etc/apt/sources.list.d/mssql-release.list
RUN exit
RUN apt-get -y update
RUN ACCEPT_EULA=Y apt-get install -y msodbcsql18
Then for sqlalchemy's this can be called:
con_str = f"mssql+pyodbc://{username}:{password}#{host}/{db}?" \
"driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes"
engine = create_engine(con_str)
I created a Gist on GitHub on how to do this. I hope it helps. I had to piece things together from what I found on different resources.
https://gist.github.com/joshatxantie/4bcf5d0243fba63845fce7cc40365a3a
Goodluck!
I fixed this problem by using pypyodbc instead of pyodbc.
pip install pypyodbc==1.3.5
https://pypi.org/project/pypyodbc/
Found the hint here:
https://github.com/Azure/azure-functions-python-worker/issues/249
For not more problem use library for python
pymssql this not need install driver
pip install pymssql
import pymssql
conn = pymssql.connect(server, user, password, "tempdb")
cursor = conn.cursor(as_dict=True)
cursor.execute('SELECT * FROM persons WHERE salesrep=%s', 'John Doe')
for row in cursor:
print("ID=%d, Name=%s" % (row['id'], row['name']))
conn.close()
and work in docker