Clone git repo from github with dockerfile - docker

I'm trying to clone GitHub repo with below docker file after installing java,maven
FROM openjdk:8-jdk-alpine
LABEL WebAutomation Test <waaanjula#gmail.com>
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
# 1- Define a constant with the version of maven you want to install
ARG MAVEN_VERSION=3.8.1
# 3- Define the SHA key to validate the maven download
ARG SHA=0ec48eb515d93f8515d4abe465570dfded6fa13a3ceb9aab8031428442d9912ec20f066b2afbf56964ffe1ceb56f80321b50db73cf77a0e2445ad0211fb8e38d
# 4- Define the URL where maven can be downloaded from
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
#5- Create the directories, download maven, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
# 6- Define environmental variables required by Maven, like Maven_Home directory and where the maven repo is located
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# # Install dependencies required to git clone.
RUN apk update && \
apk add --update git && \
apk add --update openssh
# 1. Create the SSH directory.
# 2. Populate the private key file.
# 3. Set the required permissions.
# 4. Add github to our list of known hosts for ssh.
RUN mkdir -p /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
ADD id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod -R 700 /root/.ssh/ && \
touch ~/.ssh/known_hosts && \
ssh-keyscan -t rsa github.com >> /root/.ssh/known_hosts
RUN mkdir /usr/AutomationProject/ && \
chmod -R 600 /usr/AutomationProject/
# # Clone a repository (MatificWebAutomation project)
RUN git clone git#github.com:anjulaw/Selenium_Keyword_driven_Example.git /usr/AutomationProject
CMD [""]
and I'm getting below error
"git#github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists."
This is my first time using dockerfiles, but from what I have read (and taken from working configs) I cannot see why this doesn't work.
My id_rsa is in the same folder as my dockerfile and is a copy of my local key which can clone this repo no problem.

You need to create a folder with keys in the local project folder, transfer there the generated keys that are already installed in the remote repository.
# SSH Keys
ADD .ssh/id_rsa /root/.ssh/id_rsa
ADD .ssh/id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod 600 /root/.ssh/id_rsa &&\
chmod 600 /root/.ssh/id_rsa.pub
It turns out that after each build, the key will be generated, which means that after each build of this Dockerfile, you will have to reconfigure the keys on the remote repository 💁

Related

Unable to Copy s3 files inside Docker container

Am new to docker and aws. I am trying to create a Jmeter Image and pass on the JMX script during runtime. For that, i thought copying files from S3 inside a container will be a best fit. So initially i tried to copy the files from s3 to my local host using the below command
aws s3 cp s3://bucketname/sample.jmx .
I was able to download the file successfully into my local system.
After then i have created a docker images with latest AWS CLI installed and tried the same, the message shows "download: s3://bucketname/sample.jmx to current folder " but am not able to see the file.
But on the other hand, i was able to copy the file from docker to S3 using the command
aws s3 cp /tmp/sample.jmx s3://bucketname/
Further details :
Image on - alpine:3.12.4
Credentials - Passed inline with the docker run command like below
docker run -it --rm -e AWS_DEFAULT_REGION='us-east-2' -e AWS_ACCESS_KEY_ID='aaaaaa' -e AWS_SECRET_ACCESS_KEY='dsfssdfds' dockerimage aws s3 cp s3://bucketname/sample.jmx /tmp
Complete Docker file :
FROM alpine:3.12.4
ARG JMETER_VERSION="5.3"
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV JMETER_DOWNLOAD_URL https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz
# Install extra packages
# Set TimeZone, See: https://github.com/gliderlabs/docker-alpine/issues/136#issuecomment-612751142
ARG TZ="Europe/Amsterdam"
ENV TZ ${TZ}
RUN apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update openjdk8-jre tzdata curl unzip bash \
&& apk add --no-cache nss \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /tmp/dependencies \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} > /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz \
&& mkdir -p /opt \
&& tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt \
&& rm -rf /tmp/dependencies
# TODO: plugins (later)
# && unzip -oq "/tmp/dependencies/JMeterPlugins-*.zip" -d $JMETER_HOME
# Set global PATH such that "jmeter" command is found
ENV PATH $PATH:$JMETER_BIN
RUN apk update && \
apk add --no-cache python3 py3-pip\
&& pip3 install --upgrade pip
RUN pip3 --no-cache-dir install --upgrade awscli
ENV PATH $PATH:/usr/bin/aws
CMD ["/bin/bash"]
I would really need some help here.

Docker build for Go project with GitLab private repositories

I am having some issues getting my build to work with dependencies in private GitLab repositories. All-in-all it is a multistage build, but the stage where I am attempting to build my Go project is listed below. This works locally for me, so there is an issue somewhere getting this working in Docker:
FROM golang:1.16.8-alpine3.14 as BuildStage
RUN apk update && apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client bash
RUN mkdir /root/.ssh
RUN ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
COPY localRsa /root/.ssh/id_rsa
RUN chmod 0400 /root/.ssh/id_rsa
RUN eval $(ssh-agent -s) && ssh-add /root/.ssh/id_rsa
WORKDIR $GOPATH/src/myproject
COPY . .
ENV GOPRIVATE="gitlab.com/MyGitLabUser"
RUN git config --global url."git#gitlab.com".insteadOf "https://gitlab.com"
RUN go mod download
RUN go mod verify
RUN GOOS=linux GOARCH=amd64 \
go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
The error message I get:
go: gitlab.com/MyProject/Sub1/Sub2/some-library#v0.0.6: reading gitlab.com/MyProject/Sub1/Sub2/some-library.git/go.mod at revision v0.0.6: unknown revision v0.0.6
The release definitely exists and is working locally. I am missing a step somewhere.
Update:
Cloning the project works from that stage if I add:
git clone git#gitlab.com:MyProject/Sub1/Sub2/some-library.git
Makes me think I'm missing something in my Go configuration or linking Go with Git.
Issues
After putting together everything that everyone provided here (thanks, by the way). I was able to rework my Dockerfile to do exactly what I needed and get things to work! So first, let me go through and enumerate all of the problems that were in my original submission:
Passing in the SSH key is completely unnecessary (see here - hat tip #RakeshGupta)
Using go mod download -x helped a lot to search out there for more specific information (hat tip #mh-cbon)
I was able to simplify more (see here - hat tip #sytech)
One of the big things was a typo that I fixed, so now it's: git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
Updated Dockerfile
FROM golang:1.16.8-alpine3.14 as BuildStage
# setup Git & SSL (for getting dependencies)
RUN apk update && \
apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client && \
update-ca-certificates
ENV GOPRIVATE="gitlab.com/MyProject"
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
RUN git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
# setup an application user
ENV USER=appuser
ENV UID=10001
RUN adduser --disabled-password \
--gecos "" \
--home "/nonexistent" \
--shell "/sbin/nologin" \
--no-create-home \
--uid "${UID}" "${USER}"
# build the project
WORKDIR $GOPATH/src/myproject
COPY . .
# make sure Go knows the packages are private
RUN go env -w GOPRIVATE="gitlab.com/MyProject/*"
# build the binary
RUN --mount=type=ssh go mod download -x && go mod verify
RUN --mount=type=ssh GOOS=linux GOARCH=amd64 go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
Updated Build Command
So that's cute and all, but I need BuildKit with SSH:
eval "$(minikube docker-env)"
DOCKER_BUILDKIT=1 docker build --ssh default -t myservice:latest .

Building Go apps with private modules in Docker

I'm trying to build a go project in a docker container that relies on private submodules.
I was hoping that --mount=type=ssh would pass my ssh credentials to the container and it'd work. Currently I can build locally with just make the GOPRIVATE variable set and the git config update.
Here is my relevant Dockerfile currently
# syntax = docker/dockerfile:experimental
FROM golang:1.14.3-alpine AS build
RUN apk add --no-cache git \
openssh-client \
ca-certificates
WORKDIR /src
ENV GIT_TERMINAL_PROMPT=1
ENV GOPRIVATE="gitlab.com/company_foo"
RUN git config --global url."ssh://git#gitlab.com".insteadOf "https://gitlab.com"
# Authorize SSH Host
# Skip Host verification for git
RUN mkdir -p /root/.ssh && \
chmod 0700 /root/.ssh && \
ssh-keyscan gitlab.com > /root/.ssh/known_hosts &&\
chmod 644 /root/.ssh/known_hosts && touch /root/.ssh/config \
&& echo "StrictHostKeyChecking no" > /root/.ssh/config
COPY go.mod go.sum .
RUN --mount=type=ssh mkdir -p /var/ssh && \
GIT_SSH_COMMAND="ssh -o \"ControlMaster auto\" -o \"ControlPersist 300\" -o \"ControlPath /var/ssh/%r#%h:%p\"" \
go mod download
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build go build -o api-server ./cmd/api-server
RUN --mount=type=cache,target=/root/.cache/go-build go build -o migrations ./cmd/migrations
I've also tried adding a CI_JOB_TOKEN with
RUN echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc
but this also didn't work. Perhaps I did it wrong.
All of this results in the failure:
revision v0.0.3: unknown revision v0.0.3
relating to one of our private repos.
Any advice would be appreciate.
I'm absolutely at a lost.
This workes for me.
FROM golang:1.14
ARG USERNAME=user1
ARG PASSWORD=secret
WORKDIR /app
ADD . .
ENV GOPRIVATE=private.git.local/*
RUN echo "machine private.git.local login $USERNAME password $PASSWORD" > ~/.netrc
RUN go build -o testGo main.go
CMD ["/app/testGo"]
pass your gitlab_token to docker file from gitlab_ci.yaml and do the following steps
RUN git config --global url."https://oauth2:$GITLAB_TOKEN#gitlab.com/".insteadOf "https://git#gitlab.com/"
add your repo as GO_PRIVATE
ENV GOPRIVATE=gitlab.com/*
copy .netrc file to docker root
COPY confidential/.netrc /root/.netrc
.netrc file will have the following structure
machine gitlab.com
login gitlab_user
password p#$$word

Can you install command-line packages in jib docker image?

I need to install command line tools like jq, curl etc in the docker image created by maven jib plugin. How can I achieve this? Any help would be greatly appreciated. Thanks.
As explained in the other answer, using a base image customized with pre-installed tools that rarely change is a good solution.
Alternatively, you may put curl using Jib's <extraDirectories> feature, which enables adding arbitrary files to the target image. Check the Maven and Gradle docs for more details. As explained in the docs, you will also need to configure <permissions> to set executable bits to curl.
If you prefer, you could even set up your Maven or Gradle builds to download curl and unpack it. Here's an example Jib setup (showing both Maven and Gradle) from the Jib repository.
Adding a reference Dockerfile and you can build your own base image by creating your Dockerfile and then build it.
FROM openjdk:8-jdk-alpine
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
ARG MAVEN_VERSION=3.6.1
ARG USER_HOME_DIR="/root"
ARG SHA=b4880fb7a3d81edd190a029440cdf17f308621af68475a4fe976296e71ff4a4b546dd6d8a58aaafba334d309cc11e638c52808a4b0e818fc0fd544226d952544
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# Downloading and installing Gradle
# 1- Define a constant with the version of gradle you want to install
ARG GRADLE_VERSION=4.0.1
# 2- Define the URL where gradle can be downloaded from
ARG GRADLE_BASE_URL=https://services.gradle.org/distributions
# 3- Define the SHA key to validate the gradle download
# obtained from here https://gradle.org/release-checksums/
ARG GRADLE_SHA=d717e46200d1359893f891dab047fdab98784143ac76861b53c50dbd03b44fd4
# 4- Create the directories, download gradle, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/gradle /usr/share/gradle/ref \
&& echo "Downlaoding gradle hash" \
&& curl -fsSL -o /tmp/gradle.zip ${GRADLE_BASE_URL}/gradle-${GRADLE_VERSION}-bin.zip \
\
&& echo "Checking download hash" \
&& echo "${GRADLE_SHA} /tmp/gradle.zip" | sha256sum -c - \
\
&& echo "Unziping gradle" \
&& unzip -d /usr/share/gradle /tmp/gradle.zip \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/gradle.zip \
&& ln -s /usr/share/gradle/gradle-${GRADLE_VERSION} /usr/bin/gradle
# 5- Define environmental variables required by gradle
ENV GRADLE_VERSION 4.0.1
ENV GRADLE_HOME /usr/bin/gradle
ENV GRADLE_USER_HOME /cache
ENV PATH $PATH:$GRADLE_HOME/bin
VOLUME $GRADLE_USER_HOME
CMD [""]
Ref:- https://docs.docker.com/engine/reference/builder/
Once your custom image is ready, push it to Registry and then reference it in jib in following manner.
mvn compile jib:build \
-Djib.from.image=customImage

Running dep ensure -vendor-only inside Docker Hangs not able to pull private Repos

My Dockerfile:
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
 RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only
When i am building this docker file it hangs at RUN dep ensure -vendor-only
It fails to pull the dependencies which are private repos
Is there any possiblities to store git credentials inside Docker or any way to build Docker with one or more private repos of GOlang
Use some thing like this
# ensure that the private Github repo is
# accessed using SSH instead of HTTPS
RUN ssh-keyscan github.com > /root/.ssh/known_hosts
RUN echo "$SSH_KEY" > /root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa
RUN echo '[url "ssh://git#github.com/*your_repo*/"]' >> /root/.gitconfig && echo 'insteadOf = https://github.com/*your_repo*/' >> /root/.gitconfig
Refer this to add ssh key to your git repo
Adding .netrc file will pass credentials inside the docker containers and helps to pull more than one private repositories to build dependencies
#vim .netrc
machine github.com
login < your github token >
add those 2 lines and pass your github token
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
COPY .netrc /root/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only

Resources