I just made a very simple Docker file in my terminal, basically I did the following:
mkdir pgrouted
cd pgrouted
touch Dockerfile
Now I open the Docker file in the nano editor, and I add the following commands to the Docker file:
FROM ubuntu
MAINTAINER Gautam <gautamx07#yahoo.com>
LABEL Description="pgrouting excercise" Vendor="skanatek" Version="1.0"
ENV BBOX="-122.8,45.4,-122.5,45.6"
# Add pgRouting launchpad repository
RUN sudo apt-add-repository -y ppa:ubuntugis/ppa
RUN sudo apt-add-repository -y ppa:georepublic/pgrouting
RUN sudo apt-get update
# Install pgRouting package (for Ubuntu 14.04)
RUN sudo apt-get install postgresql-9.3-pgrouting
# Install osm2pgrouting package
RUN sudo apt-get install osm2pgrouting
# Install workshop material (optional, but maybe slightly outdated)
RUN sudo apt-get install pgrouting-workshop
# For workshops at conferences and events:
# Download and install from http://trac.osgeo.org/osgeo/wiki/Live_GIS_Workshop_Install
RUN wget --no-check-certificate https://launchpad.net/~georepublic/+archive/pgrouting/+files/pgrouting-workshop_2.0.6-ppa1_all.deb
RUN sudo dpkg -i pgrouting-workshop_2.0.6-ppa1_all.deb
# Review: Not sure weather this should be in the dockerfile
RUN cp -R /usr/share/pgrouting/workshop ~/Desktop/pgrouting-workshop
# Log in as user "user"
RUN psql -U postgres
# Create routing database
RUN CREATE DATABASE routing;
# Add PostGIS functions
RUN CREATE EXTENSION postgis;
# Add pgRouting core functions
CREATE EXTENSION pgrouting;
# Download using Overpass XAPI (larger extracts possible than with default OSM API)
wget --progress=dot:mega -O "sampledata.osm" "http://www.overpass-api.de/api/xapi?*[bbox=${BBOX}][#meta]"
The entire Dockerfile can be see HERE at a glance.
Now when I try to build the Dockerfile, like so:
docker build -t gautam/pgrouted:v1 .
The Dockerfile runs and then I get the below error:
Step 4 : RUN sudo apt-add-repository -y ppa:ubuntugis/ppa
---> Running in c93c3c5fd5e8
sudo: apt-add-repository: command not found
The command '/bin/sh -c sudo apt-add-repository -y ppa:ubuntugis/ppa' returned a non-zero code: 1
Why am I getting this error?
apt-add-repository is just not in the base Ubuntu image. You'll first need to install it. try apt-get install software-properties-common
By the way, you don't need to use sudo in the Dockerfile because the commands run as root by default unless you change to another user with the USER command.
Add these lines before running apt-add-repository command
RUN apt-get update && \
apt-get install -y software-properties-common && \
rm -rf /var/lib/apt/lists/*
thats worked for me:
RUN apt-get update --fix-missing && \
apt-get install -y software-properties-common && \
rm -rf /var/lib/apt/lists/* && \
add-apt-repository ppa:ondrej/php && \
apt install -y nginx php7.4-fpm php7.4-mysql php7.4-curl net-tools telnet php7.4-gd php-mail php7.4 php7.4-common php7.4-sqlite3 php7.4-curl php7.4-intl php7.4-mbstring php7.4-xmlrpc php7.4-mysql php7.4-gd php7.4-xml php7.4-cli php7.4-zip php7.4-soap unzip && \
rm -rf /var/lib/apt/lists/* && \
apt clean
Trying to run Dockerfile and it fails at installing npm.
ERROR:
Unable to correct problems, you have held broken packages.
The command '/bin/sh -c apt-get install -y npm' returned a non-zero code: 100
Dockerfile:
FROM ubuntu:14.04
MAINTAINER Giacomo Vacca "giacomo.vacca#gmail.com"
ENV REFRESHED_AT 2015-01-19
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get autoremove
RUN npm -v
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install --yes curl
RUN curl --silent --location http://deb.nodesource.com/setup_0.10 | sudo bash -
RUN apt-get install -y nodejs
RUN apt-get install --yes build-essential
RUN rm /usr/bin/node
# needs this to find the nodejs exec
RUN ln -s /usr/bin/nodejs /usr/bin/node
RUN apt-get install -y npm <--- FAIL
RUN /usr/bin/npm install socket.io#0.9.14
EXPOSE 8080
ENTRYPOINT ["/usr/bin/node", "/root/server.js"]
You don't need to install npm from the ubuntu distribution with:
RUN apt-get install -y npm
because it's already installed by the nodejs package from nodesource. You can check it with:
dpkg -L nodejs | grep "/usr/bin/npm"
Getting error while installing SDKMAN! in Ubuntu 16.04 docker image.
FROM ubuntu:16.04
RUN apt-get update
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -qq -y install curl
RUN curl -s https://get.sdkman.io | bash
RUN chmod a+x "$HOME/.sdkman/bin/sdkman-init.sh"
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
make sure you have curl, wget, unzip & zip. With them I am able to install Sdkman successfully. Following is my Docker content
FROM ubuntu:18.04
RUN apt-get update
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -qq -y install curl wget unzip zip
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
TL;DR
Install unzip & zip, which means change
RUN apt-get -qq -y install curl
to
RUN apt-get -qq -y install curl unzip zip
or better
RUN apt-get -qq -y install \
curl \
unzip \
zip
Explanation
When you try to build the Dockerfile, you will get
.....
Step 5/6 : RUN curl -s https://get.sdkman.io | bash
---> Running in 1ce678a59561
--- SDKMAN LOGO ---
Now attempting installation...
Looking for a previous installation of SDKMAN...
Looking for unzip...
Not found.
======================================================================================================
Please install unzip on your system using your favourite package manager.
Restart after installing unzip.
======================================================================================================
Removing intermediate container 1ce678a59561
---> 22211eafd50c
Step 6/6 : RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
---> Running in 1c5cb7d79ef0
/bin/sh: /root/.sdkman/bin/sdkman-init.sh: No such file or directory
The command '/bin/sh -c source "$HOME/.sdkman/bin/sdkman-init.sh"' returned a non-zero code: 1
What you need to do is written just there. This part:
======================================================================================================
Please install unzip on your system using your favourite package manager.
Restart after installing unzip.
======================================================================================================
When you install unzip, you get the same error with zip. After installing it, everything works fine.
So, read your logs/command output. :-)
*P.S. It would be better if curl -s https://get.sdkman.io | bash exited with non-zero code. This way it fails on the next command. But that is not a thing you can fix ;) *
It looks like the sdkman install failed.
When I ran your code above it complained about missing the unzip and zip packages.
After satisfying the dependencies, you'll also need to mark the init script as executable with:
chmod a+x "$HOME/.sdkman/bin/sdkman-init.sh"
So your Dockerfile should look something like:
FROM ubuntu:16.04
RUN apt-get update
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -q -y install curl zip unzip
RUN curl -s https://get.sdkman.io | bash
RUN chmod a+x "$HOME/.sdkman/bin/sdkman-init.sh"
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
P.S: Beaten to the punch!
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get -qq -y install \
curl \
unzip \
zip
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -qq -y install curl
RUN curl -s https://get.sdkman.io | bash
RUN chmod a+x "$HOME/.sdkman/bin/sdkman-init.sh"
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
docker pull kubile/ubuntu-sdkman:23.04
You can try this image.
This Dockerfile seems to work with versions current as of February 2023:
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl zip unzip
RUN curl -s "https://get.sdkman.io" | bash
# this SHELL command is needed to allow using source
SHELL ["/bin/bash", "-c"]
# seems you need to put 'sdk install...'' lines in same RUN command as 'source...'.
RUN source "/root/.sdkman/bin/sdkman-init.sh" \
&& sdk install java 19.0.2-tem \
&& sdk install sbt 1.8.2 \
&& sdk install scala 2.13.10
I'm trying to connect a pyodbc python script running in a docker container to login to a MSSQL database I have tried all sorts of docker files, but not been able to make the connection (fails when bulding the docker or when python tries to connect), Does anyone have a working dockerfile, using pyodbc:
Dockerfile:
# Use an official Python runtime as a parent image
FROM python:2.7-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt
# Run app.py when the container launches
CMD ["python", "App.py"]
requirements.TXT
pyodbc
App.Py
import pyodbc
connection = pyodbc.connect('Driver={SQL Server};'
'Server=xxxx;'
'Database=xxx;'
'UID=xxxx;'
'PWD=xxxx')
cursor = connection.cursor()
cursor.execute("SELECT [Id],[Name] FROM [DCMM].[config].[Models]")
for row in cursor.fetchall():
print(row.Name)
connection.close()
Bulding the container
docker build -t sqltest .
Output:
Sending build context to Docker daemon 4.096kB
Step 1/5 : FROM python:2.7-slim
---> 426d65ab9a72
Step 2/5 : WORKDIR /app
---> Using cache
---> 725f35122880
Step 3/5 : ADD . /app
---> 3feb8b7744f7
Removing intermediate container 4214091a111a
Step 4/5 : RUN pip install -r requirements.txt
---> Running in 27aa4dcfe738
Collecting pyodbc (from -r requirements.txt (line 1))
Downloading pyodbc-4.0.17.tar.gz (196kB)
Building wheels for collected packages: pyodbc
Running setup.py bdist_wheel for pyodbc: started
Running setup.py bdist_wheel for pyodbc: finished with status 'error'
Failed building wheel for pyodbc
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpa3S13tpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=4.0.17 -DSQL_WCHART_CONVERT=1 -I/usr/local/include/python2.7 -c src/cursor.cpp -o build/temp.linux-x86_64-2.7/src/cursor.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Running setup.py clean for pyodbc
Failed to build pyodbc
Installing collected packages: pyodbc
Running setup.py install for pyodbc: started
Running setup.py install for pyodbc: finished with status 'error'
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-BV4sRM-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=4.0.17 -DSQL_WCHART_CONVERT=1 -I/usr/local/include/python2.7 -c src/cursor.cpp -o build/temp.linux-x86_64-2.7/src/cursor.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EfWsmy/pyodbc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-BV4sRM-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-EfWsmy/pyodbc/
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
Need to Run:
sudo apt-get install gcc
need to add a odbcinst.ini file containing:
[FreeTDS]Description=FreeTDS Driver Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
need to add folowing to docker file
ADD odbcinst.ini /etc/odbcinst.ini
RUN apt-get update
RUN apt-get install -y tdsodbc unixodbc-dev
RUN apt install unixodbc-bin -y
RUN apt-get clean -y
need to change connection in .py to
connection = pyodbc.connect('Driver={FreeTDS};'
'Server=xxxxx;'
'Database=DCMM;'
'UID=xxxxx;'
'PWD=xxxxx')
Now the container compiles, and gets data from SQL server
Running through this recently I found it was necessary to additionally include the following line (note that it did not build without this step):
RUN apt-get install --reinstall build-essential -y
The full Dockerfile looks as follows:
# parent image
FROM python:3.7-slim
# install FreeTDS and dependencies
RUN apt-get update \
&& apt-get install unixodbc -y \
&& apt-get install unixodbc-dev -y \
&& apt-get install freetds-dev -y \
&& apt-get install freetds-bin -y \
&& apt-get install tdsodbc -y \
&& apt-get install --reinstall build-essential -y
# populate "ocbcinst.ini"
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini
# install pyodbc (and, optionally, sqlalchemy)
RUN pip install --trusted-host pypi.python.org pyodbc==4.0.26 sqlalchemy==1.3.5
# run app.py upon container launch
CMD ["python", "app.py"]
Here's one way to then actually establish the connection inside app.py, via sqlalchemy (and assuming port 1433):
import sqlalchemy as sa
args = (username, password, server, database)
connstr = "mssql+pyodbc://{}:{}#{}/{}?driver=FreeTDS&port=1433&odbc_options='TDS_Version=8.0'"
engine = sa.create_engine(connstr.format(*args))
Based on Kåre Rasmussen's answer, here's a complete dockerfile for further use.
Make sure to edit the last two lines according to your architecture! They should reflect the actual paths to libtdsodbc.so and libtdsS.so.
If you're not sure about the paths to libtdsodbc.so and libtdsS.so, try dpkg --search libtdsodbc.so and dpkg --search libtdsS.so.
FROM python:3
#Install FreeTDS and dependencies for PyODBC
RUN apt-get update && apt-get install -y tdsodbc unixodbc-dev \
&& apt install unixodbc-bin -y \
&& apt-get clean -y
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/arm-linux-gnueabi/odbc/libtdsodbc.so\n\
Setup = /usr/lib/arm-linux-gnueabi/odbc/libtdsS.so" >> /etc/odbcinst.ini
Afterwards, install PyODBC, COPY your app and run it.
I was unable to use all of the above resolutions, I was keeping al kind of errors relating to the pyodbc package, in particular:
ImportError: libodbc.so.2: cannot open shared object file: No such file or directory.
I ended up with another resolution which defines the ODBC SQL Server Driver specifically for an Ubuntu 18.04 Docker image, in this case ODBC Driver 17 for SQL Server. In my specific use case I needed to make the connection to my MySQL database server on Azure via Flask SQLAlchemy, but the latter is not a necessity for the Docker configuration.
Dockerfile, with most important part adding the Microsoft repository and installing msodbcsql17 and unixodbc-dev:
# Ubuntu 18.04 base with Python runtime and pyodbc to connect to SQL Server
FROM ubuntu:18.04
WORKDIR /app
# apt-get and system utilities
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential g++-5\
&& rm -rf /var/lib/apt/lists/*
# adding custom Microsoft repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
# install SQL Server drivers
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
# install SQL Server tools
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
RUN /bin/bash -c "source ~/.bashrc"
# python libraries
RUN apt-get update -y && \
apt-get install -y python3-pip python3-dev
# install necessary locales, this prevents any locale errors related to Microsoft packages
RUN apt-get update && apt-get install -y locales \
&& echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \
&& locale-gen
# copy requirements and install packages, I added this for general use
COPY ./requirements.txt > ./requirements.txt
RUN pip3 install -r ./requirements.txt
# you can also use regular install of the packages
RUN pip3 install pyodbc SQLAlchemy
# and if you are also planning to use Flask and Flask-SQLAlchemy
Run pip3 install Flask Flask-SQLAlchemy
COPY ..
# run your app via entrypoint or change the CMD command to your regular command
COPY docker-entrypoint.sh wsgi.py ./
CMD ["./docker-entrypoint.sh"]
This should build without any errors in Docker.
My database url looked like this:
import urllib.parse
# name the sepcific ODBC driver by version number, we installed msodbcsql17
params = urllib.parse.quote_plus("DRIVER={ODBC Driver 17 for SQL Server};SERVER=<your.database.windows.net>;DATABASE=<your-db-name>;UID=<username>;PWD=<password>")
db_uri = "mssql+pyodbc:///?odbc_connect={PARAMS}".format(PARAMS=params)
And for the bonus if you are using Flask-SQLAlchemy, your app config should contain something like this:
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri # from above
Happy coding!
How to install the necessary dependencies for pyodbc is related to the linux distribution and its version (in docker case, that is the base image of your docker image). If none of the above work for you, you can figure out the commands by trying in the docker container instance.
First, exec into the docker container
docker exec -it <container id> bash
Try various ways to get the distribution name and version of your linux. Then try different instructions in Install the Microsoft ODBC driver for SQL Server (Linux)
Here is a working example for Debian 9 based images, deriving exactly as the document instructions.
# Install pyodbc dependencies
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql17
RUN apt-get -y install unixodbc-dev
RUN pip install pyodbc
For me to solve this issue I also had to add the following 2 lines in the dockerfile:
RUN echo MinProtocol = TLSv1.0 >> /etc/ssl/openssl.cnf
RUN echo CipherString = DEFAULT#SECLEVEL=1 >> /etc/ssl/openssl.cnf
For those who wanted to do official microsoft approach to install odbc driver and use python:slim docker image, you can use this as DockerFile:
FROM python:3.9-slim
RUN apt-get -y update && apt-get install -y curl gnupg
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
# download appropriate package for the OS version
# Debian 11
RUN curl https://packages.microsoft.com/config/debian/11/prod.list \
> /etc/apt/sources.list.d/mssql-release.list
RUN exit
RUN apt-get -y update
RUN ACCEPT_EULA=Y apt-get install -y msodbcsql18
Then for sqlalchemy's this can be called:
con_str = f"mssql+pyodbc://{username}:{password}#{host}/{db}?" \
"driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes"
engine = create_engine(con_str)
I created a Gist on GitHub on how to do this. I hope it helps. I had to piece things together from what I found on different resources.
https://gist.github.com/joshatxantie/4bcf5d0243fba63845fce7cc40365a3a
Goodluck!
I fixed this problem by using pypyodbc instead of pyodbc.
pip install pypyodbc==1.3.5
https://pypi.org/project/pypyodbc/
Found the hint here:
https://github.com/Azure/azure-functions-python-worker/issues/249
For not more problem use library for python
pymssql this not need install driver
pip install pymssql
import pymssql
conn = pymssql.connect(server, user, password, "tempdb")
cursor = conn.cursor(as_dict=True)
cursor.execute('SELECT * FROM persons WHERE salesrep=%s', 'John Doe')
for row in cursor:
print("ID=%d, Name=%s" % (row['id'], row['name']))
conn.close()
and work in docker
I am following cloudera cdh4 installation guide.
My base file
FROM ubuntu:precise
RUN apt-get update -y
#RUN apt-get install -y curl
RUN apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:webupd8team/java
RUN apt-get update -y
RUN echo debconf shared/accepted-oracle-license-v1-1 select true | \
debconf-set-selections
RUN apt-get install -y oracle-java7-installer
#Checking java version
RUN java -version
My hadoop installation file
java_ubuntu is the image build from my base file.
FROM java_ubuntu:latest
RUN apt-get update -y
RUN apt-get install -y curl
RUN curl http://archive.cloudera.com/cdh4/one-click-install/precise/amd64/cdh4-repository_1.0_all.deb > cdh4-repository_1.0_all.deb
RUN dpkg -i cdh4-repository_1.0_all.deb
RUN curl -s http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/archive.key | apt-key add -
RUN apt-get update -y
RUN apt-get install -y hadoop-0.20-conf-pseudo
#Check for /etc/hadoop/conf.pseudo.mrl to verfiy hadoop packages
RUN echo "dhis"
RUN dpkg -L hadoop-0.20-conf-pseudo
Supervisor part
hadoop_ubuntu is the image build from my hadoop installation docker file
FROM hadoop_ubuntu:latest
USER hdfs
RUN hdfs namenode -format
USER root
RUN apt-get install -y supervisor
RUN echo "[supervisord] nodameon=true [program=namenode] command=/etc/init.d/hadoop-hdfs-namenode -D" > /etc/supervisorconf.d
CMD ["/usr/bin/supervisord"]
Program is successfully build. But namenode is not starting up? How to use supervisor?
You have your config in /etc/supervisorconf.d and I don't believe that's the right location.
It should be /etc/supervisor/conf.d/supervisord.conf instead.
Also it's easier to maintain if you make a file locally and then use the COPY instruction to put it in the image.
Then as someone mentioned you can connect to the container after it's running (docker exec -it <container id> /bin/bash) and then run supervisorctl to see what's running and what might be wrong.
Perhaps you need line breaks in your supervisor.conf. Try hand crafting one and COPY it into your dockerfile for testing.
Docker and supervisord