Cupy config shows no cuda device - docker

I built a docker contaiener to be deployed on HPC+GPU via singularity. when i run
cp.show_config()
OS: Linux-5.4.0-135-generic-x86_64-with-glibc2.31
Python Version: 3.9.12
CuPy Version: 9.6.0
CuPy Platform: NVIDIA CUDA
NumPy Version: 1.21.5
SciPy Version: 1.6.0
Cython Build Version: 0.29.24
Cython Runtime Version: 0.29.28
CUDA Root: /opt/conda/envs/rapids
nvcc PATH: None
CUDA Build Version: 11020
CUDA Driver Version: 11060
CUDA Runtime Version: CUDARuntimeError('cudaErrorNoDevice: no
CUDA-capable device is detected')
cuBLAS Version: (available)
cuFFT Version: 10400
cuRAND Version: 10203
cuSOLVER Version: (11, 3, 4)
cuSPARSE Version: (available)
NVRTC Version: (11, 2)
Thrust Version: 101000
CUB Build Version: 101000
Jitify Build Version: 65946d2
cuDNN Build Version: None
cuDNN Version: None
NCCL Build Version: 21104
NCCL Runtime Version: 21210
cuTENSOR Version: None
cuSPARSELt Build Version: None
this is my dockerfile
# Pulls the basic Image from NVIDIA repository
FROM rapidsai/rapidsai:22.04-cuda11.2-runtime-ubuntu20.04-py3.9
# OS install cuda toolkit
RUN apt-get update
RUN apt-get install -y cuda-toolkit-11.2
# Pulls the basic Image from NVIDIA repository
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
--fix-missing git python3-setuptools python3-pip build-essential libcurl4-gnutls-dev \
zlib1g-dev rsync vim cmake tabix && \
apt-get clean
# Conda install on base env cudnn
RUN conda install --yes -c conda-forge cudnn=8.0.5.39
# Adding env directory to path and activate rapids env
ENV PATH /opt/conda/envs/rapids/bin:$PATH
RUN /bin/bash -c "source activate rapids"
# Install libraries needed in the examples
RUN pip install \
scanpy==1.9.1 wget pytabix dash-daq \
dash-html-components dash-bootstrap-components dash-core-components \
pytest utils tensorflow
RUN pip install --upgrade tensorflow-gpu
WORKDIR /workspace
ENV HOME /workspace
RUN mkdir -p /.singularity.d/env
RUN echo "#!/usr/bin/env bash" > /.singularity.d/env/99-custom_prompt.sh
RUN echo 'PS1="[${SINGULARITY_NAME%.*}]\u#\h:\w\$ "' >> /.singularity.d/env/99-custom_prompt.sh
RUN conda install batchspawner

I hope this could help other members
# Pulls the basic Image from NVIDIA repository
FROM rapidsai/rapidsai:22.04-cuda11.2-runtime-ubuntu20.04-py3.9
# OS install cuda toolkit
RUN apt-get update
RUN apt-get install -y software-properties-common \
cuda-toolkit-11.2 \
python3-setuptools
# Add Nvidia cudnn repository
ENV OS=ubuntu2004
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/cuda-${OS}.pin
RUN mv cuda-${OS}.pin /etc/apt/preferences.d/cuda-repository-pin-600
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/7fa2af80.pub
RUN add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/ /"
RUN apt-get update
# Fix Cuda and Cudnn version
ENV cudnn_version=8.1.1.33
ENV cuda_version=cuda11.2
# Install CUDNN
RUN apt-get install -y libcudnn8=${cudnn_version}-1+${cuda_version}
RUN apt-get install -y libcudnn8-dev=${cudnn_version}-1+${cuda_version}
# Adding env directory to path and activate rapids env
ENV PATH /opt/conda/envs/rapids/bin:$PATH
RUN /bin/bash -c "source activate rapids"
# Install libraries needed in the examples
RUN pip install \
scanpy==1.9.1 \
pytabix \
dash-daq \
dash-html-components \
dash-bootstrap-components \
dash-core-components \
pytest \
utils tensorflow
#
WORKDIR /workspace
ENV HOME /workspace
#

Related

ERR: "Biopython requires Python 3.6 or later. Python 2.7 detected". Even though I'm running python version 3.10.6

I'm trying to build a docker image from the docker file:
FROM ubuntu:18.04
# Install tzdata
RUN apt-get update &&\
DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
# Install Python, Bowtie2 and Java
RUN apt-get install -y python3.10 python3-pip \
openjdk-8-jdk \
bowtie2 \
wget
RUN apt-get install --yes --no-install-recommends \
zlib1g-dev \
libbz2-dev \
liblzma-dev
# Install RSeQC
RUN apt-get install -y python-pip &&\
pip install RSeQC
# Install biopython=1.80
RUN pip install biopython
# Install Atria
RUN wget https://github.com/cihga39871/Atria/releases/download/v3.1.2/atria-3.1.2-linux.tar.gz && \
tar -zxf atria-3.1.2-linux.tar.gz && \
mv atria-3.1.2/bin/atria /usr/local/bin/atria && \
chmod +x /usr/local/bin/atria
#Atria dependencies
RUN apt-get install pigz pbzip2
# Install findtail
RUN wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/findtail/findtail_v1.01 && \
mv findtail_v1.01 /usr/local/bin/findtail_v1.01 && \
chmod +x /usr/local/bin/findtail_v1.01
# Cleanup
RUN apt clean
But while on Step 6/10 : RUN pip install biopython it gives the error :
---> Running in 062f9c27cc15
Collecting biopython
Downloading https://files.pythonhosted.org/packages/3d/2f/d9df24de05d651c5e686ee8fea3afe3985c03ef9ca02f4cc1e7ea10aa31e/biopython-1.77.tar.gz (16.8MB)
Complete output from command python setup.py egg_info:
Biopython requires Python 3.6 or later. Python 2.7 detected.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-mgUybW/biopython/
The command '/bin/sh -c pip install biopython' returned a non-zero code: 1
I've check the python version on my pc. I'm running python version 3.10.6 on my OS and in the dockerfile I'm trying to incorporate python 3.10 version. Where is this error coming from?
The default Python version in Ubuntu 18.04 is 3.6. However, on line 19, you are installing the python-pip package, i.e. the Python 2 version of the pip package manager, which in turn depends on the Python 2.7 package and thus will install Python 2.7.
Which means from this point on, you have two Python versions installed on the system, with two Python VM executables, python and python3, and two pip executables, pip and pip3.
It is not clear why you install Python 2.7 and python-pip on line 19 when you have already installed Python 3.10 and python3-pip on line 8. Nor is it clear why you install BioPython using pip, i.e. the Python 2 version instead of pip3, i.e. the Python 3 version.
I would not install Python 2 at all, since it has not been maintained or supported for several years. I would also not use Ubuntu 18.04, which will be out of standard support in 4 months, unless you are paying for the Extended Security Maintenance. The latest Long-Term Support version of Ubuntu is 22.04.1 which has standard support until April 2027 and extended support until April 2032.

Docker exec container pytest fail

I am using the dev image so i can have the cuda compiler, now the issue is that when running the CI, as below I get that error, but if I build the standard container (commented line in dockerfile).
CONTAINER=$(docker run -d gpu-test)
docker exec $CONTAINER pytest
OCI runtime exec failed: exec failed: unable to start container process: exec: "pytest": executable file not found in $PATH: unknown
Dockerfile:
# Pulls the basic Image from NVIDIA repository
FROM rapidsai/rapidsai-dev:22.04-cuda11.5-devel-ubuntu20.04-py3.9
# FROM rapidsai/rapidsai:22.04-cuda11.5-runtime-ubuntu20.04-py3.9
# Updates OS libraries
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
--fix-missing git python3-setuptools python3-pip build-essential libcurl4-gnutls-dev \
zlib1g-dev rsync vim nano cmake tabix
# Install libraries needed in the examples
RUN /opt/conda/envs/rapids/bin/pip install \
scanpy==1.9.1 wget pytabix dash-daq \
dash-html-components dash-bootstrap-components dash-core-components \
utils pytest
RUN /opt/conda/envs/rapids/bin/pip install \
torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
WORKDIR /workspace
ENV HOME /workspace

Errors that occur when installing anaconda using Docker on M1 chip MacBooks

I tried to build an existing dockerfile on my MacBookPro with M1 chip, but I got the following error when installing anaconda3.
Why does it output such an error?
Also, how can I fix this?
In docker-compose, the platform is set to linux/arm64 and the cpus is 2, but the same error is output.
PREFIX=/opt/anaconda3
Unpacking payload ...
/lib64/ld-linux-x86-64.so.2: No such file or directory
/lib64/ld-linux-x86-64.so.2: No such file or directory
My Dockerfile is below.
FROM ubuntu:latest
#update
RUN apt-get -y update && apt-get install -y \
sudo \
wget \
gcc \
g++ \
vim
#install anaconda3
WORKDIR /opt
#download anaconda package and install anaconda
# archive => https://repo.continuum.io/archive/
RUN wget https://repo.continuum.io/archive/Anaconda3-2020.07-Linux-x86_64.sh && \
sh /opt/Anaconda3-2020.07-Linux-x86_64.sh -b -p /opt/anaconda3 && \
rm -f Anaconda3-2020.07-Linux-x86_64.sh
ENV PATH /opt/anaconda3/bin:$PATH
#update pip and conda
RUN pip install -U pip
WORKDIR /code
ADD requirements.txt /code
RUN pip install -r requirements.txt
WORKDIR /
#execute jupyter lab as a default command
CMD ["jupyter","lab","--ip=0.0.0.0","--allow-root","--LabApp.token=''"]
My docker-compose.yml file is below
version: "3"
services:
jupyter:
image: investment-project:1.0.0
container_name: investment-jupyter
build: .
platform: linux/arm64/v8
cpus: "2"
volumes:
- $PWD:/tmp/working
working_dir: /tmp/working
ports:
- 8888:8888
command: jupyter notebook --ip=0.0.0.0 --allow-root --LabApp.token=''

Speed up image building docker with docker compose

i am a newby in docker and I need an installation based on ubuntu of logstash.
I try the official logstash image and without success I can't run it so I decided to build my own installation based on my needs.
It works well but takes a lot of time to build it.
I wonder how can I improve (speed up) my building
This is my Dockerfile
FROM ubuntu:18.04
# RUN adduser --disabled-password --gecos "" ubuntu
# RUN usermod -aG sudo ubuntu
# RUN echo "ubuntu ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
RUN apt-get update && \
apt-get autoclean && \
apt-get autoremove
RUN echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
RUN apt-get install curl ca-certificates apt-utils wget apt-transport-https default-jre gnupg apt-transport-https software-properties-common -y
# RUN update-alternatives --config java
ENV LANG C.UTF-8
ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
ENV PATH $JAVA_HOME/bin:$PATH
RUN wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
RUN apt update
RUN apt install ruby-full -y
RUN ruby --version
RUN apt install jruby -y
#install jruby (last version)
RUN jruby --version
RUN apt install logstash
#install plugins and bundles.
RUN cd /usr/share/logstash && gem install bundler
COPY logstash-pcap-test.conf /usr/share/logstash/
RUN mkdir /home/logstash/ && mkdir /home/logstash/traffic
COPY /traffic-example/* /home/logstash/traffic/
WORKDIR /usr/share/logstash
CMD ["/bin/bash","-c","bin/logstash -f logstash-pcap-test.conf --config.reload.automatic"]
And this is my docker-compose
version: "3"
services:
logstash_test:
build:
context: .
dockerfile: container/Dockerfile
image: logstash_test:latest
container_name: logstash_test
hostname: logstash_test
ports:
- 9600:9600
- 8089:8089
networks:
- elknetwork
networks:
elknetwork:
driver: bridge
Any thoughts?

Dockerfile fails with llvm-config error for numba install

My Dockerfile using a pypy base fails with FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config' when installing llvmlite, a dependency of numba which is listed in my requirements.txt
I tried to follow and update the instructions here
Python numba / llvmlite on Debian 8 - i can't build llvmlite
My error in more detail:
Building wheels for collected packages: llvmlite
Building wheel for llvmlite (setup.py): started
Building wheel for llvmlite (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/pypy3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ux49fegr/llvmlite/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ux49fegr/llvmlite/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-8s6wwump --python-tag pp371
cwd: /tmp/pip-install-ux49fegr/llvmlite/
Complete output (26 lines):
running bdist_wheel
/usr/local/bin/pypy3 /tmp/pip-install-ux49fegr/llvmlite/ffi/build.py
LLVM version... Traceback (most recent call last):
File "/tmp/pip-install-ux49fegr/llvmlite/ffi/build.py", line 105, in main_posix
out = subprocess.check_output([llvm_config, '--version'])
File "/usr/local/lib-python/3/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/local/lib-python/3/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib-python/3/subprocess.py", line 722, in __init__
restore_signals, start_new_session)
File "/usr/local/lib-python/3/subprocess.py", line 1354, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'
I don't want the heavy weight of a conda installation. Is there a way to achieve a numba install for pypy without it?
FROM pypy:3.6-slim-stretch
RUN apt-get update && apt-get install -y gnupg wget software-properties-common
RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - 15CF4D18AF4F7421
RUN apt-add-repository "deb http://apt.llvm.org/stretch/ llvm-toolchain-stretch-6.0 main"
RUN apt-get update
RUN apt-get -y install cython python-llvm build-essential libedit-dev
RUN apt-get -y install libllvm6.0 llvm-6.0-dev llvm-dev
RUN pip install enum34
ENV LLVM_CONFIG=/usr/lib/llvm-6.0-dev/bin/llvm-config pip install llvmlite
ENV LLVM_CONFIG=/usr/lib/llvm-6.0-dev/bin/llvm-config pip install numba
# ENV LD_LIBRARY_PATH /usr/lib/llvm-6.0-dev/lib/
#... other stuff
RUN apt-get clean && apt-get -y update
RUN apt-get -y install python3-dev \
&& apt-get -y install build-essential
# Add requirements
COPY requirements.txt /tmp/
# Install sphinx first as it does not work inside requirements
RUN pip install -U pip && pip install -U sphinx && pip install numpy && pip install cython && \
pip install -r /tmp/requirements.txt
I expect a clean build of the docker image with numba on pypy
WORKAROUND If the error appeared after April the 17th (they released a new version, see history).
As a workaround try installing a previous version
pip3 install llvmlite==0.31.0
At the moment numba won't install on python 3.9.
I changed my base docker image from python:3.9-slim to python:3.8-slim and then pip install numba succeeded
This fixed it for me -
RUN echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-8 main" >> /etc/apt/sources.list
RUN echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-8 main" >> /etc/apt/sources.list
RUN apt-get install -y --no-install-recommends libedit-dev build-essential
RUN apt-get install -y --no-install-recommends llvm-8 llvm-8-dev
RUN LLVM_CONFIG=/usr/bin/llvm-config-8 pip3 install enum34 llvmlite numba
Used version 8, because got an error with the latest llvm version -
RuntimeError: Building llvmlite requires LLVM 7.0.x, 7.1.x or 8.0.x, got '3.7.1'
Reference - https://apt.llvm.org/
The "real" solution would be to encourage llvmlite and/or numba to release a wheel for pypy. But for the immediate problem...
Somehow you are missing an llvm installation, at least according to apt-file:
$ apt-file find llvm-config |grep llvm-6
llvm-6.0: /usr/bin/llvm-config-6.0
llvm-6.0: /usr/lib/llvm-6.0/bin/llvm-config
llvm-6.0: /usr/share/man/man1/llvm-config-6.0.1.gz
For installing numba on Python 3.9 on macOS, the following works:
# install llvm using MacPorts
sudo port install llvm-10
sudo port install llvm_select
sudo port select --set llvm mp-llvm-10
# set environment variable used when building `llvmlite`
export LLVM_CONFIG=/opt/local/bin/llvm-config
# clone, build, and install `llvmlite`
# http://llvmlite.pydata.org/en/latest/admin-guide/install.html#compiling-llvmlite
git clone git#github.com:numba/llvmlite.git
cd llvmlite
python setup.py build
python runtests.py
python setup.py install
cd ..
# clone, build, and install `numba`
# https://numba.readthedocs.io/en/stable/user/installing.html#installing-from-source
git clone git#github.com:numba/numba.git
cd numba
export PATH=/usr/bin/:$PATH # use clang, same as used for building Python 3.9
pip install -v .
cd ..
The above are with:
llvmlite at commit 334411cf638c8e06ba19bfda16b07dd64e9dac3c, and
numba at commit ca8132ba5e43fc3f79767046ed55217aeeabb35e.
according to https://askubuntu.com/questions/1286131/how-do-i-install-llvm-10-on-ubuntu-18-04
I added to Dockerfile the lines
RUN apt-get -y install llvm-10*
RUN rm -f /usr/bin/llvm-config
RUN ln -s /usr/bin/llvm-config-10 /usr/bin/llvm-config
and It worked (llvmlite installed)
This is the solution, obtained from this github issues thread, that worked for me! Essentially, pip just needed an upgrade in the Dockerfile before installing libraries. All I needed to do was add this line in my Dockerfile:
--upgrade pip \
Here is the sample of my Dockerfile:
# Docker commands here ...
RUN APT_INSTALL="apt-get install -y " && \
PIP_INSTALL="python3 -m pip install " && \
apt-get update && \
# more docker commands here ...
DEBIAN_FRONTEND=noninteractive $APT_INSTALL \
python3.6 \
python3.6-dev \
python3-distutils-extra \
python3-pip && \
ln -s /usr/bin/python3.6 /usr/local/bin/python3 && \
$PIP_INSTALL \
--upgrade pip \ # adding this line was my solution
setuptools \
&& \
$PIP_INSTALL \
numpy==1.18.5 \
# more docker commands here...
This is for python 3.6

Resources