Why am I getting permission issues while writing a file even after using --user with chmod777? - docker

I have built an image and run that container as --user gauravs. I used --user gauravs to build my image, provided all sudo functionalities to the user and modify the permissions to 777, but still, I am getting a permission error whenever I try to write any file (give commands in my script). I need to manually do touch \<file\> then chmod 777 \<file\>. May I request you to tell me what I am doing wrong?
What I have found when I run any experiment, ps aux|grep <pid> shows some random name associated with it, and that random name is later associated with mine.
awoodard 2805117 8778 0.6 27876664 5175500 ? Rsl 10:31 8783:54 python3 main.py --gpu_num 1
gauravs 2808325 0.0 0.0 9032 2708 pts/1 S+ 12:11 0:00 grep --color=auto 2805117
The Dockerfile I used:
FROM nvidia/cuda:11.1-devel-ubuntu20.04
RUN apt-get update
RUN apt-get -y --no-install-recommends install apt-utils
RUN apt-get -y --no-install-recommends install curl ca-certificates
RUN apt-get -y --no-install-recommends install sudo \
git \
bzip2 \
libx11-6 \
wget \
build-essential \
pkg-config \
vim
RUN mkdir /app
WORKDIR /app
VOLUME /app
CMD bash
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
#RUN apt-get -y --no-install-recommends install python3-dev python3-pip
#ENV user gauravs
#RUN adduser --disabled-password --gecos '' --shell /bin/bash ${user} \
# && chown -R ${user}:${user} /app \
# && adduser ${user} sudo
#RUN echo "%sudo ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers
#USER ${user}
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
&& chown -R user:user /app \
&& adduser user sudo
RUN echo "%sudo ALL=(ALL) NOPASSWD:ALL">/etc/sudoers
USER user
#ENV PATH=/home/${user}/.local/bin:$PATH
ENV PATH=/home/user/.local/bin:$PATH
RUN sudo chmod 777 -R /app
#RUN sudo chmod 777 /home/${user}
RUN sudo chmod 777 /home/user
RUN sudo apt-get -y --no-install-recommends install python3-dev
RUN sudo apt-get -y --no-install-recommends install python3-pip
# Install Dependencies
RUN pip3 install --user --upgrade setuptools
RUN pip3 install --user wheel
RUN sudo apt-get -y --no-install-recommends install nvidia-cuda-toolkit
COPY requirements.txt .
RUN pip3 install --user -r requirements.txt
RUN pip3 uninstall -y torch
#RUN pip3 install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip3 install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
RUN pip3 install --user torchtext==0.9.0

Related

How to grant access to a non privileged user to run conda in node docker container?

I am running a container from this dockerfile:
FROM node:14
RUN apt update && apt upgrade
RUN apt install -y wget build-essential libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev
RUN wget https://www.python.org/ftp/python/3.9.4/Python-3.9.4.tgz
RUN tar xzf Python-3.9.4.tgz
RUN cd Python-3.9.4 \
&& ./configure --enable-optimizations \
&& make install
RUN alias python=python3
RUN wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh
RUN bash Mambaforge-$(uname)-$(uname -m).sh -b
RUN ln -s /root/mambaforge/bin/conda /usr/bin/conda
RUN conda init
RUN conda create -n myenv nodejs=14 r=4 r-import r-dplyr r-tidyr
# USER node
With the above code, I have access to conda in the shell. But if I uncomment the last line, I get:
conda: Permission denied
I have tried:
chmod -R a+rwx /usr/bin/conda
and
RUN chown -R node:node /usr/bin/conda
RUN chmod 755 /usr/bin/conda
I found on the internet, none of which worked. What should I do to give the node user the required permission?

NVIDIA Driver Not found during Nvidia + Cuda - Docker Image build

I am trying to create a GPU microservice using Nvidia cuda Base image, but during the docker build, I am facing Driver not found issue, can someone point out what is missing here?
DockerFile:
FROM nvidia/cuda:10.1-devel
# Install some basic utilities
RUN apt-get update && apt-get install -y \
curl \
ca-certificates \
sudo \
git \
bzip2 \
libx11-6 \
&& rm -rf /var/lib/apt/lists/*
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PATH=/home/user/miniconda/bin:$PATH
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda install -y python==3.7 \
&& conda clean -ya
ENV PATH="/usr/local/cuda-10.1/bin:$PATH"
ENV LD_LIBRARY_PATH="/usr/local/cuda-10.1/lib64:$LD_LIBRARY_PATH"
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
ENV NVIDIA_VISIBLE_DEVICES=all
ENV FORCE_CUDA="1"
RUN conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
RUN pip install -v -e .
Error:
"/home/user/miniconda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1013, in _get_cuda_arch_flags
capability = torch.cuda.get_device_capability()
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 320, in get_device_capability
prop = get_device_properties(device)
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 325, in get_device_properties
_lazy_init() # will define _get_device_properties and _CudaDeviceProperties
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 196, in _lazy_init
_check_driver()
File "/home/user/miniconda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 101, in _check_driver
http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx
The issues happens during execution of last step in docker file.
I tried using multiple Nvidia base docker images, but didn't help much. (cuda:10.1-base-ubuntu18.04, cuda:10.1-runtime-ubuntu18.04)
Any pointers appreciated.
After lot of trial and errors and going through a lot of documentation, this is what worked fine.
ARG PYTORCH=1.3
ARG CUDA=10.1
ARG CUDNN=7
FROM pytorch/pytorch:1.3-cuda10.1-cudnn7-devel
RUN mkdir /app
WORKDIR /app
ENV TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0+PTX"
ENV TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
ENV CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"
RUN apt-get update && apt-get install -y libglib2.0-0 libsm6 libxrender-dev libxext6 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install some basic utilities
RUN apt-get update && apt-get install -y \
curl \
ca-certificates \
sudo \
git \
bzip2 \
libx11-6 \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update && \
apt-get install -y --no-install-recommends \
build-essential g++ \
libglib2.0-0 libsm6 libxrender-dev libxext6 wget
# Create a non-root user and switch to it
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
&& chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user
# All users can use /home/user as their home directory
ENV HOME=/home/user
RUN chmod 777 /home/user
# Install Miniconda and Python 3.7
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PATH=/home/user/miniconda/bin:$PATH
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda install -y python==3.7 \
&& conda clean -ya
RUN conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
RUN pip install -v -e .
Hope this helps!
Good luck!

Entrypoint not found when deployed to Fargate. Locally works

I have the following Dockerfile, currently working locally in my device:
FROM python:3.7-slim-buster
WORKDIR /app
COPY . /app
VOLUME /app
RUN chmod +x /app/cat/sitemap_download.py
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
ARG VERSION=3.7.4
RUN apt update && \
apt install -y bash wget && \
wget -O /tmp/nordrepo.deb https://repo.nordvpn.com/deb/nordvpn/debian/pool/main/nordvpn-release_1.0.0_all.deb && \
apt install -y /tmp/nordrepo.deb && \
apt update && \
apt install -y nordvpn=$VERSION && \
apt remove -y wget nordvpn-release
RUN apt-get clean \
&& apt-get -y update
RUN apt-get -y install python3-dev \
python3-psycopg2 \
&& apt-get -y install build-essential
RUN pip install --upgrade pip
RUN pip install -r cat/requirements.txt
RUN pip install awscli
ENTRYPOINT ["sh", "-c", "./entrypoint.sh"]
But when I deploy it to Fargate, the container stops before reaching the steady state with:
sh: 1: ./entrypoint.sh: not found
Edit: Adding entrypoint.sh file for clarification:
#!/bin/env sh
# start process, but it should exit once the file is in S3
/app/cat/sitemap_download.py
# Once the process is done, we are good to scale down the service
aws ecs update-service --cluster cluster_name --region eu-west-1 --service service-name --desired-count 0
I have tried modifying ENTRYPOINT to use it as exec form, or with full path but always get the same issue. Any ideas on what am I doing wrong?
I've managed to fix it now.
Changing the Dockerfile to look as follows solves the issue:
COPY . /app
VOLUME /app
RUN chmod +x /app/cat/sitemap_download.py
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
ARG VERSION=3.7.4
RUN apt update && \
apt install -y bash wget && \
wget -O /tmp/nordrepo.deb https://repo.nordvpn.com/deb/nordvpn/debian/pool/main/nordvpn-release_1.0.0_all.deb && \
apt install -y /tmp/nordrepo.deb && \
apt update && \
apt install -y nordvpn=$VERSION && \
apt remove -y wget nordvpn-release
RUN apt-get clean \
&& apt-get -y update
RUN apt-get -y install python3-dev \
python3-psycopg2 \
&& apt-get -y install build-essential
RUN pip install --upgrade pip
RUN pip install -r cat/requirements.txt
RUN pip install awscli
ENTRYPOINT ["/bin/bash"]
CMD ["./entrypoint.sh"]
I tried this after reading: What is the difference between CMD and ENTRYPOINT in a Dockerfile?
I believe this syntax fixes it because with entrypoint I'm indicating bash to be run at start, and then passing the script as parameter.

How to install homebrew on Ubuntu inside Docker container

When I try to install homebrew on Ubuntu 18.04
# Dockerfile
FROM ubuntu:18.04
RUN apt-get update && apt-get install build-essential curl file git -y
RUN sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
getting errors:
==> Add Ruby to your PATH by running: PATH=/root/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/current/bin:$PATH
Don't run this as root!
Is there a reason you can't use the official image (docker pull linuxbrew/linuxbrew)? It is based on Ubuntu 16.04 / Xenial.
If you must use Bionic (18.04), the correct way to install homebrew will be to follow the steps in the official Dockerfile.
But to get your Dockerfile working, you need to install ruby, create a non-root user and execute the installation script as that user. Like so,
FROM ubuntu:18.04
RUN apt-get update && \
apt-get install build-essential curl file git ruby-full locales --no-install-recommends -y && \
rm -rf /var/lib/apt/lists/*
RUN localedef -i en_US -f UTF-8 en_US.UTF-8
RUN useradd -m -s /bin/bash linuxbrew && \
echo 'linuxbrew ALL=(ALL) NOPASSWD:ALL' >>/etc/sudoers
USER linuxbrew
RUN sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
USER root
ENV PATH="/home/linuxbrew/.linuxbrew/bin:${PATH}"
PS: I have added --no-install-recommends to ignore optional dependencies and rm -rf /var/lib/apt/lists/* to remove apt-get leftovers thus reducing the image size. Also, locales is added to install UTF-8 or brew will throw a warning when you run the command.
The new correct way is:
RUN apt-get update && \
apt-get install -y -q --allow-unauthenticated \
git \
sudo
RUN useradd -m -s /bin/zsh linuxbrew && \
usermod -aG sudo linuxbrew && \
mkdir -p /home/linuxbrew/.linuxbrew && \
chown -R linuxbrew: /home/linuxbrew/.linuxbrew
USER linuxbrew
RUN /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
Gabriel's answer mostly worked for me, but was missing one step. I needed to chown the /home/linuxbrew/.linuxbrew folder to the user that would be running Homebrew:
RUN apt-get update && \
apt-get install -y -q --allow-unauthenticated \
git \
sudo
RUN useradd -m -s /bin/zsh linuxbrew && \
usermod -aG sudo linuxbrew && \
mkdir -p /home/linuxbrew/.linuxbrew && \
chown -R linuxbrew: /home/linuxbrew/.linuxbrew
USER linuxbrew
RUN /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
USER root
RUN chown -R $CONTAINER_USER: /home/linuxbrew/.linuxbrew

docker run error: Unable to access jarfile

Docker image is built but when I want to run it, it shows this error:
Error: Unable to access jarfile rest-service-1.0.jar
My OS is Ubuntu 18.04.1 LTS and I use docker build -t doc-service & docker run doc-service.
This is my Dockerfile:
FROM ubuntu:16.04
MAINTAINER Frederico Apostolo <frederico.apostolo#blockfactory.com> (#fapostolo)
RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y software-properties-common python-software-properties language-pack-en-base
RUN add-apt-repository ppa:webupd8team/java
RUN apt-get update && apt-get update --fix-missing && apt-get -y --allow-downgrades --allow-remove-essential --allow-change-held-packages upgrade \
&& echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections \
&& apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages curl vim unzip wget oracle-java8-installer \
&& apt-get clean && rm -rf /var/cache/* /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle/
run java -version
run echo $JAVA_HOME
#use locate for debug
RUN apt-get update && apt-get install -y locate mlocate && updatedb
#LIBREOFFICE START
RUN apt-get update && apt-get update --fix-missing && apt-get install -y -q libreoffice \
libreoffice-writer ure libreoffice-java-common libreoffice-core libreoffice-common \
fonts-opensymbol hyphen-fr hyphen-de hyphen-en-us hyphen-it hyphen-ru fonts-dejavu \
fonts-dejavu-core fonts-dejavu-extra fonts-noto fonts-dustin fonts-f500 fonts-fanwood \
fonts-freefont-ttf fonts-liberation fonts-lmodern fonts-lyx fonts-sil-gentium \
fonts-texgyre fonts-tlwg-purisa
#LIBREOFFICE END
#font configuration
COPY 00-odt-template-renderer-fontconfig.conf /etc/fonts/conf.d
RUN mkdir /document-service /document-service/fonts /document-service/module /document-service/logs
# local settings
RUN echo "127.0.0.1 http://www.arbs.local http://arbs.local www.arbs.local arbs.local" >> /etc/hosts
# && mkdir /logs/ && echo "dummy" >> /logs/errors.log
#EXPOSE 2115
COPY document-service-java_with_user_arg.sh /
RUN chmod +x /document-service-java_with_user_arg.sh
RUN apt-get update && apt-get -y --no-install-recommends install \
ca-certificates \
curl
RUN gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.4/gosu-$(dpkg --print-architecture)" \
&& curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.4/gosu-$(dpkg --print-architecture).asc" \
&& gpg --verify /usr/local/bin/gosu.asc \
&& rm /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu
ENV LANG="en_US.UTF-8"
# In case someone loses the Dockerfile
# Needs to be in the end so it doesn't invalidate unaltered cache whenever the file is updated.
RUN rm -rf /etc/Dockerfile
ADD Dockerfile /etc/Dockerfile
ENTRYPOINT ["/document-service-java_with_user_arg.sh"]
this is document-service-java_with_user_arg.sh:
#!/bin/bash
USER_ID=${LOCAL_USER_ID:-9001}
USER_NAME=${LOCAL_USER_NAME:-jetty}
echo "Starting user: $USER_NAME with UID : $USER_ID"
useradd --shell /bin/bash --home-dir /document-service/dockerhome --non-unique --uid $USER_ID $USER_NAME
cd /document-service
/usr/local/bin/gosu $USER_NAME "$#" java -jar rest-service-1.0.jar
Can anyone help me on this?
Based on the comments, you must add the JAR when building the image by defining in your Dockerfile :
COPY rest-service-1.0.jar /document-service/rest-service-1.0.jar
You could also just use :
COPY rest-service-1.0.jar /rest-service-1.0.jar
, and remove cd /document-service in your entrypoint script, as on ubuntu:16.04 images, default working directory is /. My opinion is that setting the working directory in the script is safer, so you should just go for the first solution.
Note that you could also use ADD instead of COPY (as you already did in your Dockerfile), but here only COPY is necessary (read this post if you want more info : What is the difference between the `COPY` and `ADD` commands in a Dockerfile?).
Finally, I suggest you to add the COPY line at the end of your Dockerfile, so that if a new JAR is built, image won't be rebuilt from scratch but from an existing layer, speeding up build time.
it looking error about workdir
you must select workdir for this copy format
try WORKDIR /yourpath/

Resources