Permission denied trying to mkdir in build with jenkins and kubernetes - docker

I need to create a folder in one of the stages of a build in jenkins, but I'm getting a permission error when it tries to run mkdir.
dockerfile:
FROM debian:latest
USER root
WORKDIR /root
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y gnupg2
RUN DEBIAN_FRONTEND="noninteractive" apt install -y apt-transport-https ca-certificates software-properties-common curl git jq wget unzip
RUN curl -s https://storage.googleapis.com/golang/go1.15.6.linux-amd64.tar.gz| tar -v -C /usr/local -xz
RUN export PATH=$PATH:/usr/local/go/bin
#
# Docker
#
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-key fingerprint 0EBFCD88
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io
ENV PATH=$PATH:/usr/local/go/bin
RUN export PATH=$PATH:/usr/local/go/bin
WORKDIR /app/src/
Stage of Jenkinsfile:
stage('Restore') {
try {
timeout(time: 10, unit: 'MINUTES') {
dir('.') {
sh "mv * /root/go/src/Proj/"
}
dir('/root/go/src/Proj/') {
sh "mkdir ./${repoName}"
}
}
} catch (Exception err) {
cleanWS()
error("[FAILED]: " + err.getMessage())
}
}
The mv command works fine, but when it comes to mkdir, it gives me this error:
java.nio.file.AccessDeniedException: /root/go/src
Does anyone know how could I set this permission?

Related

Docker build for ros melodic and nvidia/cudagl is taking alot of memory

I'm trying to build an image in my docker image. It will contain nvidia/gadugl and ROS melodic.
Here:
FROM nvidia/cudagl:11.4.2-base-ubuntu18.04
# Minimal setup
RUN apt-get update \
&& apt-get install -y locales lsb-release
ARG DEBIAN_FRONTEND=noninteractive
RUN dpkg-reconfigure locales
# Install ROS melodic
RUN sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
RUN apt-get update \
&& apt-get install -y --no-install-recommends ros-melodic-desktop-full
RUN apt-get install -y --no-install-recommends python3-rosdep
RUN rosdep init \
&& rosdep fix-permissions \
&& rosdep update
RUN echo "source /opt/ros/melodic/setup.bash" >> ~/.bashrc
I don't know how long it's suppsoed to take and how much GB, but this doesn't seem correct to me:
Sending build context to Docker daemon 94.48GB
Maybe I'm wrong?

Install google-chrome-stable with docker

I get an error when I do docker-compose build.
I am using apple siilcon and I was able to build it with intel without any problems.
Does anyone know if something is wrong?
Contents of docker
RUN apt-get update && apt-get install -y unzip && \
CHROME_DRIVER_VERSION=`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE` && \
wget -N http://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip -P ~/ && \
unzip ~/chromedriver_linux64.zip -d ~/ && \
rm ~/chromedriver_linux64.zip && \
chown root:root ~/chromedriver && \
chmod 755 ~/chromedriver && \
mv ~/chromedriver /usr/bin/chromedriver && \
sh -c 'wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -' && \
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get update
RUN apt-get install -y google-chrome-stable
Error
=> ERROR [ 5/15] RUN apt-get install -y google-chrome-stable 3.2s
------
> [ 5/15] RUN apt-get install -y google-chrome-stable:
#8 1.090 Reading package lists...
#8 2.038 Building dependency tree...
#8 2.154 Reading state information...
#8 2.214 E: Unable to locate package google-chrome-stable
You can add the platform flag to the FROM statement in your Dockerfile. This will ensure the docker container builds the correct architecture every time.
I used this for Debian Linux and Chrome headless:
FROM --platform=linux/amd64 python:3.9

how to start docker container from gradle?

I am trying to switch from Cmake to gradle. I want to configure gradle to work as follow
$ cd myapp && ls myapp
$ Dockerfile build.gradle src
$ gradle build
Build the docker image from docker file
start container
build the application
The docker image contains complete environment for my app.
FROM debian:stretch
RUN apt-get update -y && apt install -y git \
python3-dev libncurses5-dev libxml2-dev \
libedit-dev swig doxygen graphviz xz-utils ninja-build
RUN echo "deb http://ftp.de.debian.org/debian stretch main" >> /etc/apt/source.list
RUN apt-get update && apt-get install -y openjdk-8-jre openjdk-8-jdk
# Clang 8 as a compiler
RUN apt-get update && apt-get install -y \
xz-utils \
build-essential \
curl \
&& rm -rf /var/lib/apt/lists/* \
&& curl -SL http://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz \
| tar -xJC . && \
mv clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04 clang_8.0.0 && \
echo 'export PATH=/clang_8.0.0/bin:$PATH' >> ~/.bashrc && \
echo 'export LD_LIBRARY_PATH=/clang_8.0.0/lib:LD_LIBRARY_PATH' >> ~/.bashrc
#
RUN apt-get update
#install sdkman
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -qq -y install curl wget unzip zip
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
#install gradle
RUN yes | /bin/bash -l -c 'sdk install gradle 6.1'
PS: This is cpp project
You can build a docker image from Gradle tasks by using **com.bmuschko:gradle-docker-plugin:3.1.0 plugin**
buildscript {
repositories {
jcenter()
mavenCentral()
}
dependencies {
classpath 'com.bmuschko:gradle-docker-plugin:3.1.0'
}
}
apply plugin: 'com.bmuschko.docker-remote-api'
import com.bmuschko.gradle.docker.tasks.image.Dockerfile
import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
import com.bmuschko.gradle.docker.tasks.image.*
task buildImage(type: DockerBuildImage) {
group = ''
inputDir = file('.')
tag = 'image name:'+tag
}
read the documentation for more details https://bmuschko.github.io/gradle-docker-plugin/
Build an image from Gradle task - ./gradlew taskname
To start container and run the command inside it you can use CMD or ENTRYPOINT and specify the command in Dockerfile
CMD ["start.sh"]
in start.sh you can specify your command to be executed after running the container
Let me slightly clean up that Dockerfile first:
FROM debian:stretch
RUN echo "deb http://ftp.de.debian.org/debian stretch main" >> /etc/apt/source.list
RUN apt-get update -y && apt install -qq -y \
python3-dev libncurses5-dev libxml2-dev \
libedit-dev swig doxygen graphviz xz-utils ninja-build \
openjdk-8-jre openjdk-8-jdk \
xz-utils curl git build-essential wget unzip zip
# Clang 8 as a compiler
RUN curl -SL http://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz \
| tar -xJC . && \
mv clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04 clang_8.0.0 && \
echo 'export PATH=/clang_8.0.0/bin:$PATH' >> ~/.bashrc && \
echo 'export LD_LIBRARY_PATH=/clang_8.0.0/lib:LD_LIBRARY_PATH' >> ~/.bashrc
#install sdkman
RUN ln -fs /bin/bash /bin/sh
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
RUN yes | /bin/bash -l -c 'sdk install gradle 6.1'
RUN mkdir /src /work
WORKDIR /src
ENTRYPOINT gradle build -p /src
The important bits are at the bottom: it creates a /src directory and executes gradle build there. All that remains for you is to make that directory available when you build.
Assuming you built the container once with docker build -t my-build-container ., you can run it as follows:
docker run -v $(pwd):/src my-build-container
Depending on your build system, this might pollute your source tree with various build artifacts owned by root. If so, consider switching to out-of-tree builds by changing the default working directory to /work instead. All build results will go to /work, and you can extract them from the container afterwards.
Add docker plugin first
buildscript {
dependencies {
classpath("se.transmode.gradle:gradle-docker:1.2")
}
}
create simple task like this in build.gradle file
task buildDocker(type: Docker, dependsOn: build) {
push = false
project.group = 'testProject'
project.archivesBaseName = jar.baseName
applicationName = jar.baseName
dockerfile = file('src/main/docker/Dockerfile')
doFirst {
copy {
from jar
into stageDir
}
}
}

apt-get error: Version '5:19.03.4~3-0~ubuntu-bionic' for 'docker-ce' was not found

Documentation
provides syntax to install specific version of docker-ce:
$ sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io
On similar line, below dockerfile uses the above syntax:
FROM jenkins/jenkins:lts
ENV DEBIAN_FRONTEND=noninteractive
USER root
ARG DOCKER_GID=497
# Create Docker Group with GID
# Set default value of 497 if DOCKER_GID set to blank string by Docker compose
RUN groupadd -g ${DOCKER_GID:-497} docker
# Install base packages for docker, docker-compose & ansible
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys AA8E81B4331F7F50 && \
RUN apt-get update -y && \
apt-get -y install bc \
gawk \
libffi-dev \
musl-dev \
apt-transport-https \
curl \
python3 \
python3-dev \
python3-setuptools \
gcc \
make \
libssl-dev \
python3-pip
# Used at build time but not runtime
ARG DOCKER_VERSION=5:19.03.4~3-0~ubuntu-bionic
# Install the latest Docker CE binaries and add user `jenkins` to the docker group
RUN apt-get update && \
apt-get -y install apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common && \
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(lsb_release -cs) \
stable" && \
apt-get update && \
apt-get -y install docker-ce=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} \
docker-ce-cli=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} \
containerd.io && \
usermod -aG docker jenkins
ARG DOCKER_COMPOSE=1.24.1
# Install docker compose
RUN curl -L "https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE:-1.24.1}/docker-compose-$(uname -s)-$(uname -m)" \
-o /usr/local/bin/docker-compose && \
chmod +x /usr/local/bin/docker-compose
RUN pip3 install ansible boto3
# Change to jenkins user
USER jenkins
# Add jenkins plugin
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/plugins.sh /usr/share/jenkins/plugins.txt
fails at line below(on build):
apt-get -y install docker-ce=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} \
docker-ce-cli=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} \
containerd.io && \
where default values are retrieved from command: apt-cache madison docker-ce | awk 'NR==1{print $3}' in my local docker host
where docker-compose build gives below error:
Reading state information...
E: Version '5:19.03.4~3-0~ubuntu-bionic' for 'docker-ce' was not found
E: Version '5:19.03.4~3-0~ubuntu-bionic' for 'docker-ce-cli' was not found
ERROR: Service 'jenkins' failed to build: The command '/bin/sh -c apt-get update && apt-get -y install apt-transport-https ca-certificates curl gnupg-agent software-properties-common && curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable" && apt-get update && apt-get -y install docker-ce=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} docker-ce-cli=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} containerd.io && usermod -aG docker jenkins' returned a non-zero code: 100
apt-get -y install docker-ce docker-ce-cli containerd.io is able to download and install the latest version of ubuntu packages, but why download and install of specific version of ubuntu package fails?
apt-get -y install docker-ce=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} \
docker-ce-cli=${DOCKER_VERSION:-5:19.03.4~3-0~ubuntu-bionic} \
containerd.io && \
You've selected Docker versions based on what's available on your build host, not what's available inside the container image you're building. The jenkins:lts image is based on Debian Stretch, not Ubuntu Bionic.
Dockerfiles are actually just running fairly ordinary Docker operations. So, for example, you can run docker run -ti -u root jenkins/jenkins:lts /bin/bash, run your RUN scripts by hand, and check the apt-cache output inside the container:
# apt-cache madison docker-ce
docker-ce | 5:19.03.4~3-0~debian-stretch | https://download.docker.com/linux/debian stretch/stable amd64 Packages
docker-ce | 5:19.03.3~3-0~debian-stretch | https://download.docker.com/linux/debian stretch/stable amd64 Packages
docker-ce | 5:19.03.2~3-0~debian-stretch | https://download.docker.com/linux/debian stretch/stable amd64 Packages
docker-ce | 5:19.03.1~3-0~debian-stretch | https://download.docker.com/linux/debian stretch/stable amd64 Packages
docker-ce | 5:19.03.0~3-0~debian-stretch | https://download.docker.com/linux/debian stretch/stable amd64 Packages
Also, a failed docker build should leave the partially-complete image around; so you can use that directly to investigate a failure. As an example with a trivially failing step RUN false:
⋮
Removing intermediate container baaeab34bb8c
---> 6d34bab07796
Step 3/3 : RUN false
---> Running in 8347f442dfaa
The command '/bin/sh -c false' returned a non-zero code: 1
The 6d34bab07796 image is left around. You can pass that to docker run and investigate why the command failed. The 8347f442dfaa container is also left around, though exited; you can use the various docker container subcommands to investigate it as well.

Dockerfile and including other repositories for use with apt-get

Whilst the particular base linux docker image we use currently escapes me, the problem I have is I need to include the Postgresql Client.
apt-get update allowed me to install it, but I discovered it was only version 9, and I needed 10 to match my Postgres DB.
If the docker is running and I used exec bash to access it I can run the following commands to install version 10...
echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" >> /etc/apt/sources.list.d/pgdg.list
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
apt-get update
apt-get -y install postgresql-client-10
So I then set about adding these to my Dockerfile so it would be included automatically, the result being...
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" >> /etc/apt/sources.list.d/pgdg.list
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN apt-get update
RUN apt-get -y install postgresql-client-10
When Visual Studio (Mac 2019) builds the docker, it completely ignores this. When I built it from the command line, the output suggested it had done as it should, what Docker Cloud builds it I get the error...
Step 17/24 : COPY --from=publish /app .
---> 091e71bd17c0
Step 18/24 : RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" >> /etc/apt/sources.list.d/pgdg.list
---> Running in 8582e2dba910
/bin/sh: can't create /etc/apt/sources.list.d/pgdg.list: nonexistent directory
Removing intermediate container 8582e2dba910
The command '/bin/sh -c echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" >> /etc/apt/sources.list.d/pgdg.list' returned a non-zero code: 1
Note the 3rd line up about can't create pgdg.list
So how should I go about doing this?
Your approach with the additional Dockerfile statements is sound. However, it's unclear how you're extending your previous container image to include these.
Usually you'd have something of the form:
FROM my-original-image:its-version
RUN ...
RUN ...
RUN apt-get update
RUN apt-get -y install postgresql-client-10
This approach would add postgresql-client-10 to whatever's installed in my-original-image:its-version.
An alternative approach is to start from the base from which my-original-image:its-version uses and install postgresql-client-10 instead of the previous version. Your choice.
If my-original-image were created using:
FROM some-base
RUN apt update && apt install -y postgresql-client-9
You could:
FROM some-base
RUN apt update && apt install -y postgresql-client-10
NB You'd have to duplicate the installation of other dependencies too.
Update
See instructions for Dockerizing Postgres here. The Postgres PGP key has possibly been dropped from your image and is causing errors.
FROM debian:stretch-slim
RUN apt-get update && apt-get install -y \
gnupg2 \
wget \
&& apt-key adv \
--keyserver hkp://p80.pool.sks-keyservers.net:80 \
--recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8 \
&& echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" \
>> /etc/apt/sources.list.d/pgdg.list \
&& wget \
--quiet \
--output-document - \
https://www.postgresql.org/media/keys/ACCC4CF8.asc \
| apt-key add - \
&& apt-get update \
&& apt-get install -y \
postgresql-client-10 \
&& rm -rf /var/lib/apt/lists/*

Resources