Source To Image not finding C++ libs - docker

I am creating a custom Builder Image using S2i dotnet core. This will run in OpenShift linux container
I have modified the custom builder image and included few lines to copy few dlls and ".so" files
When running the container in OpenShift I am facing the below error
error says
"unable to load shared library 'CustomCppWrapper' or one of its dependencies. In order to help diagnose loading problems,
consider setting the LD_DEBUG environment variable: libWrapperName: cannot open shared object file: No such file or directory"
I have set the LD_DEBUG environment variable and found below few errors
/lib64/libstdc++.so.6: error: version lookup error: version `CXXABI_1.3.8' not found (required by /opt/app-root/app/libCWrappeNamer.so) (fatal)
/lib64/libstdc++.so.6: error: version lookup error: version `CXXABI_1.3.8' not found (required by ./libCWrappeNamer.so) (fatal)
I did below command and found below
ldd libCWrappeNamer.so
./libCWrappeNamer.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./libCWrappeNamer.so)
./libCWrappeNamer.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /ab/sdk/customlib/gcc540/lib/libabc.so)
./libCWrappeNamer.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /ab/sdk/customlib/gcc540/lib/libxmlc.so)
Below is my Custom Docker file builder image
FROM dotnet/dotnet-31-runtime-rhel7
# This image provides a .NET Core 3.1 environment you can use to run your .NET
# applications.
ENV PATH=/opt/app-root/src/.local/bin:/opt/app-root/src/bin:/opt/app-root/node_modules/.bin:${PATH} \
STI_SCRIPTS_PATH=/usr/libexec/s2i
LABEL io.k8s.description="Platform for building and running .NET Core 3.1 applications" \
io.openshift.tags="builder,.net,dotnet,dotnetcore,rh-dotnet31"
# Labels consumed by Red Hat build service
LABEL name="dotnet/dotnet-31-rhel7" \
com.redhat.component="rh-dotnet31-container" \
version="3.1" \
release="1" \
architecture="x86_64"
#-------------------------- COPY CPP LIBS
COPY CustomCppWrapper.lib /opt/app-root/app
COPY libCWrappeNamer.so /opt/app-root/app
#----------------------------------
# Labels consumed by Eclipse JBoss OpenShift plugin
LABEL com.redhat.dev-mode="DEV_MODE:false" \
com.redhat.deployments-dir="/opt/app-root/src"
# Switch to root for package installs
USER 0
# Copy the S2I scripts from the specific language image to $STI_SCRIPTS_PATH.
COPY ./s2i/bin/ /usr/libexec/s2i
RUN INSTALL_PKGS="rh-nodejs10-npm rh-nodejs10-nodejs-nodemon rh-dotnet31-dotnet-sdk-3.1 rsync" && \
yum install -y --setopt=tsflags=nodocs --disablerepo=\* \
--enablerepo=rhel-7-server-rpms,rhel-server-rhscl-7-rpms,rhel-7-server-dotnet-rpms \
$INSTALL_PKGS && \
rpm -V $INSTALL_PKGS && \
yum clean all -y && \
# yum cache files may still exist (and quite large in size)
rm -rf /var/cache/yum/*
# Directory with the sources is set as the working directory.
RUN mkdir /opt/app-root/src
WORKDIR /opt/app-root/src
# Trigger first time actions.
RUN scl enable rh-dotnet31 'dotnet help'
# Build the container tool.
RUN /usr/libexec/s2i/container-tool build-tool
# Since $HOME is set to /opt/app-root, the yum install may have created config
# directories (such as ~/.pki/nssdb) there. These will be owned by root and can
# cause actions that work on all of /opt/app-root to fail. So we need to fix
# the permissions on those too.
RUN chown -R 1001:0 /opt/app-root && fix-permissions /opt/app-root
ENV ENABLED_COLLECTIONS="$ENABLED_COLLECTIONS rh-nodejs10" \
# Needed for the `dotnet watch` to detect changes in a container.
DOTNET_USE_POLLING_FILE_WATCHER=true
# Run container by default as user with id 1001 (default)
USER 1001
# Set the default CMD to print the usage of the language image.
CMD /usr/libexec/s2i/usage

Your code depends on libstdc++.so.6 but it would seem that version isn't installed
In your Dockerfile, add the yum install command that should do it. It would depend on what operating system you're using, but for RHEL 7, for example, you could do:
RUN yum install -y libstdc++
With more details of the operating system I can give a more specific command
In this specific examples the Dockerfile could look something like this:
FROM centos:7
RUN yum install -y libstdc++
CMD ["/bin/bash"]

Related

Containerized Azure Function does not see databricks odbc driver

I have below Dockerfile:
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS installer-env
COPY . /src/dotnet-function-app
RUN cd /src/dotnet-function-app && \
mkdir -p /home/site/wwwroot && \
dotnet publish *.csproj --output /home/site/wwwroot
FROM mcr.microsoft.com/azure-functions/dotnet:4
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
#ODBCINI=/etc/odbc.in \
#ODBCSYSINI=/etc/odbcinst.ini \
#SIMBASPARKINI=/opt/simba/spark/lib/64/simba.sparkodbc.ini
WORKDIR ./home/site/wwwroot
COPY --from=installer-env /home/site/wwwroot /home/site/wwwroot
RUN apt update && apt install -y apt-utils odbcinst1debian2 libodbc1 odbcinst vim unixodbc unixodbc-dev freetds-dev curl tdsodbc unzip libsasl2-modules-gssapi-mit
RUN curl -sL https://databricks.com/wp-content/uploads/drivers-2020/SimbaSparkODBC-2.6.16.1019-Debian-64bit.zip -o databricksOdbc.zip && unzip databricksOdbc.zip
RUN dpkg -i SimbaSparkODBC-2.6.16.1019-Debian-64bit/simbaspark_2.6.16.1019-2_amd64.deb
RUN export ODBCINI=/etc/odbc.ini ODBCSYSINI=/etc/odbcinst.ini SIMBASPARKINI=/opt/simba/spark/lib/64/simba.sparkodbc.ini
Purpose why this azure function app is containerized is to enable using databricks odbc driver to connect to azure databricks instance and delta lake. I have read on stackoverflow, in other thread, that there is no other way for installing custom drivers if app service is not containerized. I thought it should work the same with function app if is containerized.
Unfortunatelly I get exception that:
ERROR [01000] [unixODBC][Driver Manager]Can't open lib 'Simba Spark ODBC Driver' : file not found
or
Dependency unixODBC with minimum version 2.3.1 is required. Unable to load shared library 'libodbc.so.2' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibodbc.so.2: cannot open shared object file: No such file or directory
even if I point drivers in this line:
RUN export ODBCINI=/etc/odbc.ini ODBCSYSINI=/etc/odbcinst.ini SIMBASPARKINI=/opt/simba/spark/lib/64/simba.sparkodbc.ini
Looks like home/site/wwwroot can not access above folders. What interessting I also tried to copy content of /etc to /home/site/wwwroot/bin to set enrironment variable pointing from that folder, but it is not possible to copy:
WORKDIR /etc
COPY . /home/site/wwwroot/bin
Generally, I pass connection details to databricks instance in connection string, but I also tried to point /etc files by below command:
RUN gawk -i inplace '{ print } ENDFILE { print "[ODBC Drivers]" }' /etc/odbcinst.ini
but I get exception during building that:
gawk: inplace:59: warning: inplace::begin: Cannot stat '/etc/odbcinst.ini' (No such file or directory)

Building a multi-architecture Docker image, but Dockerfile requires different packages depending on architecture

I am experimenting with docker's buildx and noticed that everything seems to be straight forward except for one thing. My Dockerfile needs to pull certain packages depending on the architecture.
For example, here's a piece of the Dockerfile:
FROM XYZ
# Set environment variable for non-interactive install
ARG DEBIAN_FRONTEND=noninteractive
# Run basic commands to update the image and install basic stuff.
RUN apt update && \
apt dist-upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" && \
apt autoremove -y && \
apt clean -y && \
...
# Install amazon-ssm-agent
mkdir /tmp/ssm && \
curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb -o /tmp/ssm/amazon-ssm-agent.deb && \
As you can see from above, the command is set to pull down the Amazon SSM agent using a hard-coded link.
What's the best way to approach this? Should I just modify this Dockerfile to create a bunch of if conditions?
Docker automatically defines a set of ARGs for you when you're using the BuildKit backend (which is now the default). You need to declare that ARG, and then (within the RUN command) you can use an environment variable $TARGETOS to refer to the target operating system (the documentation suggests linux or windows).
FROM ...
# Must be explicitly declared, and after FROM
ARG TARGETOS
# Then it can be used like a normal environment variable
RUN curl https://s3.amazonaws.com/ec2-downloads-$TARGETOS/...
There is a similar $TARGETPLATFORM if you need to build either x86 or ARM images, but its syntax doesn't necessarily match what's in this URL. If $TARGETPLATFORM is either amd64 or arm, you may need to reconstruct the Debian architecture string. You can set a shell variable within a single RUN command and it will last until the end of that command, but no longer.
ARG TARGETPLATFORM
RUN DEBARCH="$TARGETPLATFORM"; \
if [ "$DEBARCH" = "arm" ]; then DEBARCH=arm64; fi; \
curl .../debian-$DEBARCH/...

Run vscode in docker

I want to run vscode in docker for internal test, i've created the following
FROM debian:stable
RUN apt-get update && apt-get install -y apt-transport-https curl gpg
RUN curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg \
&& install -o root -g root -m 644 microsoft.gpg /etc/apt/trusted.gpg.d/ \
&& echo "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" > /etc/apt/sources.list.d/vscode.list
RUN apt-get update && apt-get install -y code libx11-xcb-dev libasound2
RUN code --user-data-dir="~/.vscode-root"
I use to build
docker build -t vscode .
I use to run
docker run vscode code -v
when I run it like this I got error
You are trying to start vscode as a super user which is not recommended. If you really want to, you must specify an alternate user data directory using the --user-data-dir argument.
I just want to verify it by running something like RUN code -v how can I do it ?
should I change the user ? I just want to run vscode in docker and use some vscode apis
Have you tried using VSCode's built in functionality for developing in a container?
Checkout this page which describes how to do this:
Developing inside a Container
You can try out some of the sample container configurations provided by VSCode and use any of those devcontainer.json files as an example to configure a custom development container to your liking. According to the page above:
Workspace files are mounted from the local file system or copied or cloned into the container. Extensions are installed and run inside the container, where they have full access to the tools, platform, and file system. This means that you can seamlessly switch your entire development environment just by connecting to a different container.
This is a very handy way to have different environments for development that are isolated within the container.

Succesfully created a virtualenv (using "mkproject") in Dockerfile, but can't run "workon" properly

Edit: Solved- typo
I have a Dockerfile that successfully creates a virtualenv using virtualenvwrapper (along with setting up a heap of "standard" settings/packages in our normal environment). I am using the resulting image as a "base image" for further use. All good so far. However, the following Dockerfile (based of the first image, "base_image_14.04") falls down at the last line:
FROM base_image_14.04
USER root
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update && apt-get install -y \
libproj0 libproj-dev \
libgeos-c1v5 libgeos-dev \
libjpeg62 libjpeg-dev \
zlib1g zlib1g-dev \
libfreetype6 libfreetype6-dev \
libgdal20 libgdal-dev \
&& rm -rf /var/lib/apt/lists
USER webdev
RUN ["/bin/bash", "-ic", "mkproject maproxy"]
EXPOSE 80
WORKDIR $PROJECT_HOME/mapproxy
ADD ./requirements.txt .
RUN ["/bin/bash", "-ic", "workon mapproxy && pip install -r requirements.txt"]
The "mkproject mapproxy" works fine. If I comment out the last line it builds successfully and I can spin up the container and run "workon mapproxy" manually, not a problem. But when I try and build with the last line, it gives a workon error:
ERROR: Environment 'mapproxy' does not exist. Create it with 'mkvirtualenv mapproxy'.
workon is being called, but for some reason it can't find the mapproxy virtualenv.
WORKON_HOME & PROJECT_HOME both exist (defined in the parent image) and point to the correct locations (and are used successfully by "mkproject mapproxy").
So why is workon returning an error when the mapproxy virtualenv exists? The same error happens when I isolate that last line into a third Dockerfile building on the second.
Solved: It was a simple typo. mkproject maproxy instead of mapproxy. :sigh:
I am trying to build a docker image and am running into similar problems.
First question was why use a virtual env in docker? The main reason in a nutshell is to minimize effort to migrate an existing and working approach into a docker container. I will eventually use docker-compose, but I wanted to start by getting my feet wet with it all in a single docker container.
In my first attempt I installed almost everything with apt-get, including uwsgi. I installed my app "globally" with pip3. The app has command line functionality and a separate flask web app, hence the need for uwsgi. The command line functionality works, but when I make a request of the flask app uwsgi / python has a problem with locale: Fatal Python error: Py_Initialize: Unable to get the locale encoding and ImportError: No module named 'encodings
I have stripped away all my app specific additions to narrow down the problem. This is the Dockerfile I'm using:
# Docker image definition for testing
FROM ubuntu:xenial
# Create a user
RUN useradd -G sudo -ms /bin/bash tester
RUN echo 'tester:password' | chpasswd
WORKDIR /home/tester
# Skipping apt-get update to save some build time. Some are kept
# to insure they are the same as on host setup.
RUN apt-get install -y python3 python3-dev python3-pip \
virtualenv virtualenvwrapper sudo nano && \
apt-get clean -qy
# After above, can we use those installed in rest of Dockerfile?
# Yes, but not always, such as with virtualenvwrapper. What about
# virtualenv? How do you "source" the script? Doesn't appear to be
# installed, as bash complains "source needs a single parameter"
ENV VIRTUALENVWRAPPER_PYTHON /usr/bin/python3
ENV VIRTUALENVWRAPPER_VIRTUALENV /usr/bin/virtualenv
RUN ["/bin/bash", "-c", "source", "/usr/share/virtualenvwrapper/virtualenvwrapper.sh"]
# Create a virtualenv so uwsgi can find locale
# RUN mkdir /home/tester/.virtualenv && virtualenv -p`which python3` /home/bts_tools/.virtualenv/bts_tools
RUN mkvirtualenv -p`which python3` bts_tools && \
workon bts_tools && \
pip3 --disable-pip-version-check install --upgrade bts_tools
USER tester
ENTRYPOINT ["/bin/bash"]
CMD ["--login"]
The build fails on the line I try to source the virtualenvwrapper script. Bash complains source needs an argument - the file to be sourced. So I comment out the RUN lines and it builds without error. When I run the resulting container I see all the additions to the ENV that virtualenvwrapper makes (you can see all of them by executing the "set" command without any args), and the script to be sourced is there too.
So my question is why doesn't docker find them? How does the docker build process work if the results of any previous RUNs or ENVs aren't applied for subsequent use in the Dockerfile? I know some things are applied and work, for example if you apt-get nginx you can refer to /etc/nginx or alter things under that folder. You can create a user and set it's password or cd into its home folder for example. If I move the WORKDIR before the RUN useradd -G I see a warning from useradd the home folder already exists. I tried to use the "time" program to time how long it takes to do various things in the Dockerfile and docker complains it can't find 'time'.
So what exactly is going on? I have spent the last 3 days trying to figure this out. It just shouldn't be this difficult. What am I missing?
Parts of the bts_tools flask app worked when I wasn't using virtual envs. Most of the app didn't work, and the issue was this locale problem. Since everything works on the host outside of docker, and after trying to alter the PATH, PYTHONHOME, PYTHONPATH in my uwsgi start script to overcome the dreaded "locale encoding" fatal error, I decided to try to replicate the host setup as closely as possible since that didn't have the locale issue. When I have had that problem before I could run dpkg-reconfigure python3 or fix with changes to PATH or ENV settings. If you google the problem you'll see many people have difficulties with python & locale. It's almost enough reason to avoid using python!
I posted this elsewhere about locale issue, if it helps.

building in docker using buildpack-deps, but dependencies don't seem to be installed?

I'm trying to write a Dockerfile to build Kaldi (an open source speech recognition system) based on the "buildpack-deps:jessie-scm" image. This is my Dockerfile:
FROM buildpack-deps:jessie-scm
RUN apt-get update
RUN apt-get install -y python2.7 libtool python libtool-bin make
RUN mkdir /opt/kaldi
RUN git clone https://github.com/kaldi-asr/kaldi.git /opt/kaldi --depth=1
RUN ln -s -f bash /bin/sh
WORKDIR /opt/kaldi
RUN cd tools/extras && ./check_dependencies.sh
RUN cd tools && ./install_portaudio.sh
RUN cd tools && make -j 4 && make clean
RUN cd src && ./configure --shared --use-cuda=no && make depend && make -j 4 && make -j 4 online onlinebin online2 && make clean
This fails at the "check_dependencies.sh" script, which is complaining that various base dependencies aren't installed (g++, zlib, automake, autoconf, patch, bzip2) ... but the description of the image that I'm basing this on (https://github.com/docker-library/buildpack-deps/blob/587934fb063d770d0611e94b57c9dd7a38edf928/jessie/Dockerfile) suggests that all of these dependencies should be available in the base image. Why is my build failing here?
I should note that I've attempted these build steps on a bare Debian Jessie system with the required dependencies installed and they were successful there, so I don't think it's a problem with the build scripts provided with Kaldi, but definitely a Docker-related issue.
Looks like I've misunderstood the different tags for the buildpack-deps image. The tags *-scm don't add source control tools to the bundled build tools and libraries, they only apply the source control tools, and the build tools are then added on top of those tools. So I should just be using buildpack-deps:jessie not buildpack-deps:jessie-scm (the latter of which is basically a bare Debian system with git etc installed but nothing else).

Resources