I have the following Dockerfile:
# We will use Ubuntu for our image
FROM ubuntu:latest
# Updating Ubuntu packages
ARG CLOJURE_TOOLS_VERSION=1.10.1.507
RUN apt-get -qq update && apt-get -qq -y install curl wget bzip2 openjdk-8-jdk-headless \
&& curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh \
# && curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o /tmp/miniconda.sh \
&& bash /tmp/miniconda.sh -bfp /usr/local \
&& rm -rf /tmp/miniconda.sh \
&& conda install -y python=3 \
&& conda update conda \
&& curl -o install-clojure https://download.clojure.org/install/linux-install-${CLOJURE_TOOLS_VERSION}.sh \
&& chmod +x install-clojure \
&& ./install-clojure && rm install-clojure \
# no need to install lein
&& wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein \
&& chmod a+x lein \
&& mv lein /usr/bin \
&& apt-get -qq -y autoremove \
&& apt-get autoclean \
&& rm -rf /var/lib/apt/lists/* /var/log/dpkg.log \
&& conda clean --all --yes
ENV PATH /opt/conda/bin:$PATH
RUN conda create -n pyclj python=3.7 && conda install -n pyclj numpy mxnet \
&& conda install -c conda-forge opencv
## To install pip packages into the pyclj environment do
RUN conda run -n pyclj python3 -mpip install numpy opencv-python
FROM openjdk:8-alpine
RUN lein uberjar
COPY target/uberjar/vendo.jar /vendo/app.jar
EXPOSE 3000
CMD ["java", "-jar", "/vendo/app.jar", "--server.port=$PORT"]
And I'm pushing my project using git push heroku master, and I get the error:
remote: Step 8/11 : RUN lein uberjar
remote: ---> Running in 07533c6b0e9c
remote: /bin/sh: lein: not found
remote: The command '/bin/sh -c lein uberjar' returned a non-zero code: 127
Suggesting that lein wasn't installed. The wget in that first RUN command is supposed to install lein. How do I fix this issue?
You need to add a RUN lein to run the self-install.
Related
I'm not a Docker expert and I'm trying to create a container for machine learning projects. This is mostly for academic purpose, as I'm studying machine learning.
I wrote a dockerfile (and a devcontainer.json file to open the container in vscode) that runs fine until I add the lines to build tensorflow. I found three problems but I don't know what I'm missing:
I got a warning about the fact that Bazel installation isn't a release version
When the building phase arrives at ./configure I can't configure through prompt question
I got an error at the very building phase with
ERROR: The project you're trying to build requires Bazel 5.3.0 (specified in /docklearning/tensorflow/.bazelversion), but it wasn't found in /usr/bin.
This is the Dockerfile:
FROM nvidia/cuda:12.0.0-runtime-ubuntu22.04 as base
ARG USER_UID=1000
#switch to non-interactive frontend
ENV DEBIAN_FRONTEND=noninteractive
WORKDIR /docklearning
ADD . /docklearning
# Install packages
RUN apt-get update -q && apt-get install -q -y --no-install-recommends \
apt-transport-https curl gnupg apt-utils wget gcc g++ npm unzip build-essential ca-certificates curl git gh \
make nano iproute2 nano openssh-client openssl procps \
software-properties-common bzip2 subversion neofetch \
fontconfig && \
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg && \
mv bazel-archive-keyring.gpg /usr/share/keyrings && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/bazel-archive-keyring.gpg] https://storage.googleapis.com/bazel-apt stable jdk1.8" \
| tee /etc/apt/sources.list.d/bazel.list && \
apt update && apt install -q -y bazel && \
apt-get full-upgrade -q -y && \
cd ~ && \
wget https://github.com/ryanoasis/nerd-fonts/releases/download/v2.1.0/Meslo.zip && \
mkdir -p .local/share/fonts && \
unzip Meslo.zip -d .local/share/fonts && \
cd .local/share/fonts && rm *Windows* && \
cd ~ && \
rm Meslo.zip && \
fc-cache -fv && \
apt-get install -y zsh zsh-doc chroma
# Install anaconda
RUN wget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh -O Anaconda.sh && \
/bin/bash Anaconda.sh -b -p /opt/conda && \
rm Anaconda.sh
# Install LSD for ls substitute and clean up
RUN wget https://github.com/Peltoche/lsd/releases/download/0.23.1/lsd_0.23.1_amd64.deb -P /tmp && \
dpkg -i /tmp/lsd_0.23.1_amd64.deb && \
rm /tmp/lsd_0.23.1_amd64.deb && \
apt-get autoremove -y && \
apt-get autoclean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*/apt/lists/* && \
useradd -r -m -s /bin/bash -u ${USER_UID} docklearning
# Add conda to PATH
ENV PATH=/opt/conda/bin:$PATH
ENV HOME=/home/docklearning
# Make zsh default shell
RUN chsh -s /usr/bin/zsh docklearning
# link conda to /etc/profile.d
RUN ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
# Update Conda Base env
RUN conda update -n base --all -y && \
conda install -c conda-forge bandit opt_einsum keras-preprocessing && \
conda clean -a -q -y
# Download Tensorflow from github and build from source
RUN git clone https://github.com/tensorflow/tensorflow.git && \
cd tensorflow && \
./configure && \
bazel build --config=opt --config=cuda --cxxopt="-mavx2" //tensorflow/tools/pip_package:build_pip_package && \
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg && \
pip install /tmp/tensorflow_pkg/tensorflow-*.whl && \
cd .. && \
rm -rf tensorflow
# Create shell history file
RUN mkdir ${HOME}/zsh_history && \
chown docklearning ${HOME}/zsh_history && \
mkdir ${HOME}/.ssh
# Switch to internal user
USER docklearning
WORKDIR ${HOME}
# Copy user configuration files
COPY --chown=docklearning ./config/.aliases.sh ./
COPY --chown=docklearning ./config/.bashrc ./
COPY --chown=docklearning ./config/.nanorc ./
# Configure Zsh for internal user
ENV ZSH=${HOME}/.oh-my-zsh
ENV ZSH_CUSTOM=${ZSH}/custom
ENV ZSH_PLUGINS=${ZSH_CUSTOM}/plugins
ENV ZSH_THEMES=${ZSH_CUSTOM}/themes
RUN wget -qO- https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh | zsh || true
RUN git clone --single-branch --branch 'master' --depth 1 https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_PLUGINS}/zsh-syntax-highlighting \
&& git clone --single-branch --branch 'master' --depth 1 https://github.com/zsh-users/zsh-autosuggestions ${ZSH_PLUGINS}/zsh-autosuggestions \
&& git clone --single-branch --depth 1 https://github.com/romkatv/powerlevel10k.git ${ZSH_THEMES}/powerlevel10k
COPY --chown=docklearning ./config/.p10k.zsh ./
COPY --chown=docklearning ./config/.zshrc ./
CMD [ "/bin/zsh" ]
I wouldn't say this question about docker or TensorFlow. It is about package installation and OS configuration.
In this case, you need to fix the bazel version first by specifying it explicitly:
apt install -q -y bazel-5.3.0
But that package is quite wired and doesn't create a symlink, so you need to create it:
ln -s /bin/bazel-5.3.0 /bin/bazel
You can validate if bazel installed by running it:
`bazel --version
bazel 5.3.0`
I am trying to create a GPU microservice using Nvidia cuda Base image, but during the docker build, I am facing Driver not found issue, can someone point out what is missing here?
DockerFile:
FROM nvidia/cuda:10.1-devel
# Install some basic utilities
RUN apt-get update && apt-get install -y \
curl \
ca-certificates \
sudo \
git \
bzip2 \
libx11-6 \
&& rm -rf /var/lib/apt/lists/*
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PATH=/home/user/miniconda/bin:$PATH
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda install -y python==3.7 \
&& conda clean -ya
ENV PATH="/usr/local/cuda-10.1/bin:$PATH"
ENV LD_LIBRARY_PATH="/usr/local/cuda-10.1/lib64:$LD_LIBRARY_PATH"
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
ENV NVIDIA_VISIBLE_DEVICES=all
ENV FORCE_CUDA="1"
RUN conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
RUN pip install -v -e .
Error:
"/home/user/miniconda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1013, in _get_cuda_arch_flags
capability = torch.cuda.get_device_capability()
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 320, in get_device_capability
prop = get_device_properties(device)
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 325, in get_device_properties
_lazy_init() # will define _get_device_properties and _CudaDeviceProperties
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 196, in _lazy_init
_check_driver()
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 101, in _check_driver
http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx
The issues happens during execution of last step in docker file.
I tried using multiple Nvidia base docker images, but didn't help much. (cuda:10.1-base-ubuntu18.04, cuda:10.1-runtime-ubuntu18.04)
Any pointers appreciated.
After lot of trial and errors and going through a lot of documentation, this is what worked fine.
ARG PYTORCH=1.3
ARG CUDA=10.1
ARG CUDNN=7
FROM pytorch/pytorch:1.3-cuda10.1-cudnn7-devel
RUN mkdir /app
WORKDIR /app
ENV TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0+PTX"
ENV TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
ENV CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"
RUN apt-get update && apt-get install -y libglib2.0-0 libsm6 libxrender-dev libxext6 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install some basic utilities
RUN apt-get update && apt-get install -y \
curl \
ca-certificates \
sudo \
git \
bzip2 \
libx11-6 \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update && \
apt-get install -y --no-install-recommends \
build-essential g++ \
libglib2.0-0 libsm6 libxrender-dev libxext6 wget
# Create a non-root user and switch to it
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
&& chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user
# All users can use /home/user as their home directory
ENV HOME=/home/user
RUN chmod 777 /home/user
# Install Miniconda and Python 3.7
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PATH=/home/user/miniconda/bin:$PATH
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda install -y python==3.7 \
&& conda clean -ya
RUN conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
RUN pip install -v -e .
Hope this helps!
Good luck!
I have the following Dockerfile:
FROM ubuntu:latest
RUN apt-get -qq update && apt-get -qq -y install wget\
&& wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein \
&& chmod a+x lein \
&& cp lein /usr/bin
RUN "lein -v"
to download lein from the url, and put it in /usr/bin. But still the
RUN lein -v
command doesn't work.
I get the error:
remote: Step 18/22 : RUN lein -v
remote: ---> Running in e5f404275fe2
remote: /bin/sh: 1: lein -v: not found
remote: The command '/bin/sh -c lein -v' returned a non-zero code: 127
remote:
remote: Verifying deploy...
remote:
remote: ! Push rejected to appname
On Heroku one-off dyno itself, the command works.
$ /bin/sh -c "lein -v"
the problem is that docker RUN expects the command unquoted, as is, or as an array of command and its args:
one of RUN lein -v or RUN ["lein", "-v"] should do the trick.
the other problem, that your image doesn't have java installed, so the command would fail anyway. So you need to install it somehow. Your final Dockerfile could look like this:
FROM ubuntu:latest
RUN apt-get -qq update && apt-get -qq -y install wget\
&& wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein \
&& chmod a+x lein \
&& cp lein /usr/bin
RUN DEBIAN_FRONTEND=noninteractive \
apt-get -y install default-jre-headless && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN ["lein", "-v"]
-- EDIT --
In fact, adding the java dependency and changing to RUN ["lein", "-v"] doesn't work either. These are the first fifteen steps of the Dockerfile:
ARG CLOJURE_TOOLS_VERSION=1.10.1.507
RUN apt-get -qq update && apt-get -qq -y install curl wget bzip2 openjdk-8-jdk-headless\
&& curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh \
# && curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o /tmp/miniconda.sh \
&& bash /tmp/miniconda.sh -bfp /usr/local \
&& rm -rf /tmp/miniconda.sh \
&& conda install -y python=3 \
&& conda update conda \
&& curl -o install-clojure https://download.clojure.org/install/linux-install-${CLOJURE_TOOLS_VERSION}.sh \
&& chmod +x install-clojure \
&& ./install-clojure && rm install-clojure \
# no need to install lein
&& wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein > /usr/bin/lein \
&& chmod 777 /usr/bin/lein \
&& apt-get -qq -y autoremove \
&& apt-get autoclean \
&& rm -rf /var/lib/apt/lists/* /var/log/dpkg.log \
&& conda clean --all --yes
ENV PATH /usr/bin:$PATH
ENV NODE_VERSION=12.18.1
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash
ENV NVM_DIR=/root/.nvm
RUN . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION}
ENV PATH="/root/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"
RUN node --version
RUN npm --version
# ENV PATH /opt/conda/bin:$PATH
# RUN conda create -n pyclj python=3.7 && conda install -n pyclj numpy mxnet \
# && conda install -c conda-forge opencv
# ## To install pip packages into the pyclj environment do
# RUN conda run -n pyclj python3 -mpip install numpy opencv-python
FROM openjdk:8-alpine
RUN ["lein", "-v"]
and it gives the error
remote: Step 15/19 : RUN ["lein", "-v"]
remote: ---> Running in b817213d45b5
remote: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"lein\": executable file not found in $PATH": unknown
remote:
remote: Verifying deploy...
remote:
remote: ! Push rejected to
I have the following heroku.yml:
build:
docker:
web: Dockerfile
run:
web: java $JVM_OPTS -cp target/myapp.jar clojure.main -m myapp.application
And I'm pushing to heroku, and the Docker container loads correctly and the app is deployed too without any warnings or errors, but the app crashes, with these logs:
2020-04-07T22:58:22.151943+00:00 heroku[web.1]: State changed from starting to crashed
2020-04-07T22:58:22.157295+00:00 heroku[web.1]: State changed from crashed to starting
2020-04-07T22:58:22.098498+00:00 app[web.1]: /bin/sh: 1: java: not found
2020-04-07T22:58:25.714638+00:00 heroku[web.1]: State changed from starting to crashed
2020-04-07T22:58:25.654321+00:00 app[web.1]: /bin/sh: 1: java: not found
Why is java not found and what am I doing wrong?
---EDIT---
This is the docker file
# We will use Ubuntu for our image
FROM ubuntu:latest
# Updating Ubuntu packages
ARG CLOJURE_TOOLS_VERSION=1.10.1.507
RUN apt-get -qq update && apt-get -qq -y install curl wget bzip2 openjdk-8-jdk-headless \
&& curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh \
&& bash /tmp/miniconda.sh -bfp /usr/local \
&& rm -rf /tmp/miniconda.sh \
&& conda install -y python=3 \
&& conda update conda \
&& curl -o install-clojure https://download.clojure.org/install/linux-install-${CLOJURE_TOOLS_VERSION}.sh \
&& chmod +x install-clojure \
&& ./install-clojure && rm install-clojure \
&& wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein \
&& chmod a+x lein \
&& mv lein /usr/bin \
&& apt-get -qq -y autoremove \
&& apt-get autoclean \
&& rm -rf /var/lib/apt/lists/* /var/log/dpkg.log \
&& conda clean --all --yes
ENV PATH /opt/conda/bin:$PATH
RUN conda create -n pyclj python=3.6 && conda install -n pyclj numpy mxnet
## To install pip packages into the pyclj environment do
RUN conda run -n pyclj python3 -mpip install numpy
I’m on a Mac and trying to build a new container (new to Docker), I can get Anaconda installed fine and updating Anaconda from within the container works, however when I try to run conda update conda from the Dockerfile I get the below error:
What am I doing wrong?
Thanks!
The command '/bin/sh -c conda update conda' returned a non-zero code: 127
FROM ubuntu:18.04
RUN apt-get update && \
apt-get -y install curl && \
apt-get -y install python3 && \
apt-get -y install python3-pip && \
python3 -m pip install --upgrade pip && \
apt-get -y install wget && \
apt-get -y install vim && \
pip3 install tensorflow && \
pip3 install keras
RUN wget --quiet https://repo.anaconda.com/archive/Anaconda3-5.3.0-Linux-x86_64.sh -O ~/anaconda.sh && \
/bin/bash ~/anaconda.sh -b -p /opt/conda && \
rm ~/anaconda.sh && \
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
echo "conda activate base" >> ~/.bashrc
RUN conda update conda
RUN conda install opencv
RUN conda install matplotlib
RUN conda install pandas
RUN conda install seaborn