We were able to successfully add the deployment to Azuredevops Agent pool and could execute the pipeline on them by following the [Microsoft docs][1].
I used below docker file to install the software inside the container.
FROM ubuntu:18.04
# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl4 \
libicu60 \
libunwind8 \
netcat \
libssl1.0 \
maven \
python \
python3 \
docker \
&& rm -rf /var/lib/apt/lists/*
RUN curl -LsS https://aka.ms/InstallAzureCLIDeb | bash \
&& rm -rf /var/lib/apt/lists/*
# Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'.
ENV TARGETARCH=linux-x64
WORKDIR /azp
COPY ./vstsagent/ .
COPY ./start.sh .
RUN chmod +x start.sh
ENTRYPOINT ["./start.sh"]
But Now I am confused with below points
How to set Maven and java home directories along with Mavens custom setting.xml and node and gradle custom properties files in side this AKS based agents?
Even though I put Docker software to install within the conatiner, it seems docker is not getting installed. So how we can run docker related tasks in our pipelines like "build image" nad push Image tasks within this aks based build agents?
Related
I have been trying for a while to copy files via ssh from a remote server (not gihub) inside the docker image I want to build, but I can't connect to host. Here is the Dockerfile up until the critical point:
FROM r-base:latest
### Install libs
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
openssh-server \
openssh-client \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libssl-dev \
libxml2 \
libxml2-dev \
libv8-dev \
curl \
gnupg \
git
COPY ./setup setup
RUN mv setup/.ssh ~/.ssh
RUN touch ~/.ssh/known_hosts
RUN chmod -R 400 ~/.ssh
RUN ssh-agent sh -c 'ssh-add /root/.ssh/id_rsa'
#RUN eval "$(ssh-agent -s)"
#RUN ssh-add -K ~/.ssh/id_rsa #This is commented out as it causes an error
RUN ssh-keyscan hostname > ~/.ssh/known_host
RUN ssh-keygen -R hostname
## THIS IS THE COMMAND WE NEED TO RUN...
RUN scp -r user#hostname:/path/to/folder ./
The owner of the folder is user. The id_rsa.pub was added to the authorized_keys file of the user user on the host, and ssh was restarted there. However I get a Failed authentication error. I tried to use my personal id_rsa which works from the command line, but it also fails inside docker. Is this a docker issuor is it solvable?
I finally managed to do it by generating a key with the command suggested in this post
So to reproduce my case, locally:
cd setup/.ssh/
ssh-keygen -q -t rsa -N '' -f id_rsa
Then on the server add the id_rsa.pub contents to the known hosts for the user user. Can copy the contents to clipboard using xclip: xclip -sel clip < setup/.ssh/id_rsa.pub)
Dockerfile:
I have been trying for a while to copy files via ssh from a remote server (not gihub) inside the docker image I want to build, but I can't connect to host. Here is the Dockerfile up until the critical point:
FROM r-base:latest
### Install libs
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
openssh-server \
openssh-client \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libssl-dev \
libxml2 \
libxml2-dev \
libv8-dev \
curl \
gnupg \
git
COPY ./setup setup
RUN chmod -R 600 ~/.ssh
RUN echo "IdentityFile /root/.ssh/id_rsa" >> /etc/ssh/ssh_config
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
## THIS IS THE COMMAND WE NEED TO RUN...
RUN scp -r user#hostname:/path/to/folder ./
There’s no specific requirement that you must do everything inside your Dockerfile. Especially things that require remote ssh access are better done outside Docker: consider that anyone who gets your image later on can docker cp a valid ssh key out of it and potentially get access to your internal systems.
For Docker caching reasons, it’s also not a good idea to git clone or otherwise try to remotely retrieve your application from inside the Dockerfile. If you re-run docker build, and nothing else in your Dockerfile has changed, then Docker will skip over the scp step too, even if the remote content has changed.
My general recommendation would be to copy this content from outside the Dockerfile, then build it
# Using whatever credentials are in your local ssh-agent
scp -r user#hostname:/path/to/stuff dist/
# Then your Dockerfile doesn’t need scp or credentials
docker build .
Your Dockerfile then doesn’t need a bunch of extra packages that are only relevant to this path: you should be able to remove sudo openssh-server openssh-client xtail curl gnupg git without actually affecting the single main process you’re trying to run inside your container.
I need to build the dockerfile that downloads jenkins.war and through it jenkins-cli.jar need to be downloaded.
I have conf.xml also to configure it.**
Then I need that image to run in the bash, which needs to run that jar file commands.
Here is the code:
FROM ubuntu:14.04
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:webupd8team/java -y && \
apt-get update && \
echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \
apt-get install -f -y oracle-java8-installer && \
apt install -y default-jre curl wget git nano; \
apt-get clean
# Install dependencies
RUN apt-get -y update && \
apt-get -yqq --no-install-recommends install git bzip2 curl unzip && \
apt-get update
ENV JAVA_HOME /usr
ENV PATH $JAVA_HOME/bin:$PATH
# copy jenkins war file to the container
ADD http://mirrors.jenkins.io/war-stable/2.107.1/jenkins.war /opt/jenkins.war
RUN chmod 644 /opt/jenkins.war
ENV JENKINS_HOME /jenkins
# configure the container to run jenkins, mapping container port 8080 to that host port
RUN mkdir /jenkins/
RUN echo 2.107.1 > /jenkins/jenkins.install.UpgradeWizard.state
RUN echo 2.107.1 > /jenkins/jenkins.install.InstallUtil.lastExecVersion
CMD ["nohup","java", "-jar", "/opt/jenkins.war"]
EXPOSE 8080
VOLUME /jenkins
#COPY jenkins-cli.jar /jenkins/jenkins-cli.jar
#jenkins-cli installation
ENV JENKINS_URL "http://192.168.99.100:8080"
RUN curl --insecure http://192.168.99.100:8080/jnlpJars/jenkins-cli.jar \
--output /jenkins/jenkins-cli.jar
CMD ["java","-jar","/jenkins/jenkins-cli.jar","-noCertificateCheck","-noKeyAuth"]
Here is what im getting.
MY ASSUMPTION
Do I need to run along congf.xml?If yes , then HOW?
Do I need to be running jenkins.war instance in background??? HOW?
Thank you in advance
If you see the reference , I can find those comments.
There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect.
in your dockerfile, there are multiple CMD commands. only the last one will be executed.
If you want to run multiple commands at once. try bash scripts. here is the example
#!/bin/bash
echo "Starting sshd"
exec /usr/sbin/sshd -D &
if [ -z "$1" ];
then
tail -f $HADOOP_INSTALL/logs/*
fi
I have created a bunch of Local deployment pipeline jobs, these jobs do things like remove an existing container, build a service locally, build a docker image, run the container - etc. These are not CI/CD jobs, just small pipelines for deploying locally during dev.
What I want to do now is make this available to all our devs, so they can just simply spin up a local instance of jenkins that already contains the jobs.
My docker file is reasonably straight forward...
FROM jenkins:latest
USER root
RUN apt-get update
RUN apt-get install -y sudo
RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
# Docker
RUN apt-get update
RUN apt-get dist-upgrade -y
RUN apt-get install apt-transport-https ca-certificates -y
RUN sh -c "echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list"
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
RUN apt-get update
RUN apt-cache policy docker-engine
RUN apt-get install docker-engine -y
# .NET Core CLI dependencies
RUN echo "deb [arch=amd64] http://llvm.org/apt/jessie/ llvm-toolchain-jessie-3.6 main" > /etc/apt/sources.list.d/llvm.list \
&& wget -q -O - http://llvm.org/apt/llvm-snapshot.gpg.key|apt-key add - \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
clang-3.5 \
libc6 \
libcurl3 \
libgcc1 \
libicu52 \
liblldb-3.6 \
liblttng-ust0 \
libssl1.0.0 \
libstdc++6 \
libtinfo5 \
libunwind8 \
libuuid1 \
zlib1g \
&& rm -rf /var/lib/apt/lists/*
#DotNetCore
RUN curl -sSL -o dotnet.tar.gz https://go.microsoft.com/fwlink/?linkid=847105
RUN mkdir -p /opt/dotnet && tar zxf dotnet.tar.gz -C /opt/dotnet
RUN ln -s /opt/dotnet/dotnet /usr/local/bin
# Minimal Jenkins Plugins
RUN /usr/local/bin/install-plugins.sh git matrix-auth workflow-aggregator docker-workflow blueocean credentials-binding
# Skip initial setup
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
COPY LocallyDeployIdentityConfig.xml /var/jenkins_home/jobs/identity/config.xml
USER jenkins
What I thought I could do is simply copy a job config file into the /jobs/jobname folder and the job would appear, but not only does this not appear, but now I cannot manually create jobs either. I now get a java.io.IOException "No such file or directory" - Note when I exec into the running container, the job and jobname directories exist and my config file is in there.
Any ideas?
For anyone who is interested - I found a better solution. I simply map the jobs folder to a folder on my host, that way I can put the created jobs into source control and edit then add them without having to build a new docker image.
Sorted.
Jobs need to bootstrapped while the Jenkins starts can be copied to /usr/share/jenkins/ref/jobs/ folder.
But keep in mind that if the jobs(or any) already exist in Jenkins home folder, updates from /usr/share/jenkins/ref/jobs/ folder won't have any effect unless you end the files with *.override name.
https://github.com/jenkinsci/docker/blob/master/jenkins-support#L110
Dockerfile
# First time building of jenkins with the preconfigured job
COPY job_name/config.xml /usr/share/jenkins/ref/jobs/job_name/config.xml
# But if jobs need to be updated, suffix the file names with '.override'.
COPY job_name/config.xml.override /usr/share/jenkins/ref/jobs/job_name/config.xml.override
I maintain the jobs in a bootstrap folder together with configs etc.
To add a job (i.e. seedjob) I need to add the following to the Dockerfile:
# copy seedjob
COPY bootstrap/seedjob.xml /usr/share/jenkins/ref/jobs/seedjob/config.xml
I am working on continuous integration of a .net project using jenkins. By far, i am able to set a jenkins job in window. But now i need to replicate this all in jenkins running as docker container. I am able to start jenkins in docker, using github as source repository but when i try to build this project, it fails. My project is using asp.net core so i am assuming it should run on linux as well (which is the OS of docker virtual machine).
What i am missing here? Any help is highly appreciated
I'm working on a project with .Net core and we started to use Jenkins in Docker container, so far the only way I found was to create a custom Jenkins image. This is my docker file:
FROM jenkins
USER root
# Work around https://github.com/dotnet/cli/issues/1582 until Docker releases a
# fix (https://github.com/docker/docker/issues/20818). This workaround allows
# the container to be run with the default seccomp Docker settings by avoiding
# the restart_syscall made by LTTng which causes a failed assertion.
ENV LTTNG_UST_REGISTER_TIMEOUT 0
# Install .NET CLI dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libc6 \
libcurl3 \
libgcc1 \
libgssapi-krb5-2 \
libicu52 \
liblttng-ust0 \
libssl1.0.0 \
libstdc++6 \
libunwind8 \
libuuid1 \
zlib1g \
&& rm -rf /var/lib/apt/lists/*
# Install .NET Core SDK
ENV DOTNET_SDK_VERSION 1.0.0-preview2-003131
ENV DOTNET_SDK_DOWNLOAD_URL https://dotnetcli.blob.core.windows.net/dotnet/preview/Binaries/$DOTNET_SDK_VERSION/dotnet-dev-debian-x64.$DOTNET_SDK_VERSION.tar.gz
RUN curl -SL $DOTNET_SDK_DOWNLOAD_URL --output dotnet.tar.gz \
&& mkdir -p /usr/share/dotnet \
&& tar -zxf dotnet.tar.gz -C /usr/share/dotnet \
&& rm dotnet.tar.gz \
&& ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
# Trigger the population of the local package cache
ENV NUGET_XMLDOC_MODE skip
RUN mkdir warmup \
&& cd warmup \
&& dotnet new \
&& cd .. \
&& rm -rf warmup \
&& rm -rf /tmp/NuGetScratch
USER jenkins
I'm still looking for a better way to do it.
The Jenikins task should invoke dotnet commands to build. MSBuild is not yet supported for dotnet.
Basically, it has to do something similar to what we do in KoreBuild:
dotnet restore
dotnet build / dotnet publish
dotnet test
etc
I'm not sure if I'm using docker wrong but I have a base image called repo/base and it looks like
# Pull base image.
FROM centos:centos7
# add yum repos
ADD yum-repos/* /etc/yum.repos.d/
ADD certs/RPM-GPG-KEY-EPEL-7 /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
# Upgrading system
RUN rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch && \
yum -y install epel-release \
wget \
git \
tar \
nodejs \
npm \
libicu-devel \
logstash-forwarder \
rhnpush \
monit \
java-1.8.0-openjdk-devel
ADD runner/* /
RUN chmod +x /runner.sh && chmod +x /service-wrapper.sh
ENTRYPOINT ["/runner.sh"]
Really not that big of a deal. I push this to an artifactory. Then I create a test image from it
FROM repo/base
RUN echo "foo"
Build it and push it to the repo. Here it looks like the same hashes are being pushed out AGAIN, so its as if docker isn't registering that the images already exist remote.
Is this normal or is it somehow related to my remote artifactory?