Building Go apps with private modules in Docker - docker

I'm trying to build a go project in a docker container that relies on private submodules.
I was hoping that --mount=type=ssh would pass my ssh credentials to the container and it'd work. Currently I can build locally with just make the GOPRIVATE variable set and the git config update.
Here is my relevant Dockerfile currently
# syntax = docker/dockerfile:experimental
FROM golang:1.14.3-alpine AS build
RUN apk add --no-cache git \
openssh-client \
ca-certificates
WORKDIR /src
ENV GIT_TERMINAL_PROMPT=1
ENV GOPRIVATE="gitlab.com/company_foo"
RUN git config --global url."ssh://git#gitlab.com".insteadOf "https://gitlab.com"
# Authorize SSH Host
# Skip Host verification for git
RUN mkdir -p /root/.ssh && \
chmod 0700 /root/.ssh && \
ssh-keyscan gitlab.com > /root/.ssh/known_hosts &&\
chmod 644 /root/.ssh/known_hosts && touch /root/.ssh/config \
&& echo "StrictHostKeyChecking no" > /root/.ssh/config
COPY go.mod go.sum .
RUN --mount=type=ssh mkdir -p /var/ssh && \
GIT_SSH_COMMAND="ssh -o \"ControlMaster auto\" -o \"ControlPersist 300\" -o \"ControlPath /var/ssh/%r#%h:%p\"" \
go mod download
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build go build -o api-server ./cmd/api-server
RUN --mount=type=cache,target=/root/.cache/go-build go build -o migrations ./cmd/migrations
I've also tried adding a CI_JOB_TOKEN with
RUN echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc
but this also didn't work. Perhaps I did it wrong.
All of this results in the failure:
revision v0.0.3: unknown revision v0.0.3
relating to one of our private repos.
Any advice would be appreciate.
I'm absolutely at a lost.

This workes for me.
FROM golang:1.14
ARG USERNAME=user1
ARG PASSWORD=secret
WORKDIR /app
ADD . .
ENV GOPRIVATE=private.git.local/*
RUN echo "machine private.git.local login $USERNAME password $PASSWORD" > ~/.netrc
RUN go build -o testGo main.go
CMD ["/app/testGo"]

pass your gitlab_token to docker file from gitlab_ci.yaml and do the following steps
RUN git config --global url."https://oauth2:$GITLAB_TOKEN#gitlab.com/".insteadOf "https://git#gitlab.com/"
add your repo as GO_PRIVATE
ENV GOPRIVATE=gitlab.com/*
copy .netrc file to docker root
COPY confidential/.netrc /root/.netrc
.netrc file will have the following structure
machine gitlab.com
login gitlab_user
password p#$$word

Related

Docker build for Go project with GitLab private repositories

I am having some issues getting my build to work with dependencies in private GitLab repositories. All-in-all it is a multistage build, but the stage where I am attempting to build my Go project is listed below. This works locally for me, so there is an issue somewhere getting this working in Docker:
FROM golang:1.16.8-alpine3.14 as BuildStage
RUN apk update && apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client bash
RUN mkdir /root/.ssh
RUN ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
COPY localRsa /root/.ssh/id_rsa
RUN chmod 0400 /root/.ssh/id_rsa
RUN eval $(ssh-agent -s) && ssh-add /root/.ssh/id_rsa
WORKDIR $GOPATH/src/myproject
COPY . .
ENV GOPRIVATE="gitlab.com/MyGitLabUser"
RUN git config --global url."git#gitlab.com".insteadOf "https://gitlab.com"
RUN go mod download
RUN go mod verify
RUN GOOS=linux GOARCH=amd64 \
go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
The error message I get:
go: gitlab.com/MyProject/Sub1/Sub2/some-library#v0.0.6: reading gitlab.com/MyProject/Sub1/Sub2/some-library.git/go.mod at revision v0.0.6: unknown revision v0.0.6
The release definitely exists and is working locally. I am missing a step somewhere.
Update:
Cloning the project works from that stage if I add:
git clone git#gitlab.com:MyProject/Sub1/Sub2/some-library.git
Makes me think I'm missing something in my Go configuration or linking Go with Git.
Issues
After putting together everything that everyone provided here (thanks, by the way). I was able to rework my Dockerfile to do exactly what I needed and get things to work! So first, let me go through and enumerate all of the problems that were in my original submission:
Passing in the SSH key is completely unnecessary (see here - hat tip #RakeshGupta)
Using go mod download -x helped a lot to search out there for more specific information (hat tip #mh-cbon)
I was able to simplify more (see here - hat tip #sytech)
One of the big things was a typo that I fixed, so now it's: git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
Updated Dockerfile
FROM golang:1.16.8-alpine3.14 as BuildStage
# setup Git & SSL (for getting dependencies)
RUN apk update && \
apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client && \
update-ca-certificates
ENV GOPRIVATE="gitlab.com/MyProject"
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
RUN git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
# setup an application user
ENV USER=appuser
ENV UID=10001
RUN adduser --disabled-password \
--gecos "" \
--home "/nonexistent" \
--shell "/sbin/nologin" \
--no-create-home \
--uid "${UID}" "${USER}"
# build the project
WORKDIR $GOPATH/src/myproject
COPY . .
# make sure Go knows the packages are private
RUN go env -w GOPRIVATE="gitlab.com/MyProject/*"
# build the binary
RUN --mount=type=ssh go mod download -x && go mod verify
RUN --mount=type=ssh GOOS=linux GOARCH=amd64 go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
Updated Build Command
So that's cute and all, but I need BuildKit with SSH:
eval "$(minikube docker-env)"
DOCKER_BUILDKIT=1 docker build --ssh default -t myservice:latest .

Clone git repo from github with dockerfile

I'm trying to clone GitHub repo with below docker file after installing java,maven
FROM openjdk:8-jdk-alpine
LABEL WebAutomation Test <waaanjula#gmail.com>
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
# 1- Define a constant with the version of maven you want to install
ARG MAVEN_VERSION=3.8.1
# 3- Define the SHA key to validate the maven download
ARG SHA=0ec48eb515d93f8515d4abe465570dfded6fa13a3ceb9aab8031428442d9912ec20f066b2afbf56964ffe1ceb56f80321b50db73cf77a0e2445ad0211fb8e38d
# 4- Define the URL where maven can be downloaded from
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
#5- Create the directories, download maven, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
# 6- Define environmental variables required by Maven, like Maven_Home directory and where the maven repo is located
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# # Install dependencies required to git clone.
RUN apk update && \
apk add --update git && \
apk add --update openssh
# 1. Create the SSH directory.
# 2. Populate the private key file.
# 3. Set the required permissions.
# 4. Add github to our list of known hosts for ssh.
RUN mkdir -p /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
ADD id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod -R 700 /root/.ssh/ && \
touch ~/.ssh/known_hosts && \
ssh-keyscan -t rsa github.com >> /root/.ssh/known_hosts
RUN mkdir /usr/AutomationProject/ && \
chmod -R 600 /usr/AutomationProject/
# # Clone a repository (MatificWebAutomation project)
RUN git clone git#github.com:anjulaw/Selenium_Keyword_driven_Example.git /usr/AutomationProject
CMD [""]
and I'm getting below error
"git#github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists."
This is my first time using dockerfiles, but from what I have read (and taken from working configs) I cannot see why this doesn't work.
My id_rsa is in the same folder as my dockerfile and is a copy of my local key which can clone this repo no problem.
You need to create a folder with keys in the local project folder, transfer there the generated keys that are already installed in the remote repository.
# SSH Keys
ADD .ssh/id_rsa /root/.ssh/id_rsa
ADD .ssh/id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod 600 /root/.ssh/id_rsa &&\
chmod 600 /root/.ssh/id_rsa.pub
It turns out that after each build, the key will be generated, which means that after each build of this Dockerfile, you will have to reconfigure the keys on the remote repository 💁

Pass ssh-agent to dockerfile to install private repository modules

I am trying to automate a docker build in Jenkins pipeline. In my dockerfile, I basically build a node application. In my npm install, I have some private git repositories which need os bindings and so have to be installed in the container. When I run this manually, I transfer my ssh keys (id_rsa) to dockerfile which is used for doing npm install. Now, my problem is when running this task in jenkins pipeline, I will be configuring a ssh-agent(Jenkins plugin). It will not be possible to extract private key from ssh-agent. How should I pass my ssh-agent to my dockerfile.
EDIT 1:
I got it partially working by this:
Docker Build Command:
DOCKER_BUILDKIT=1 docker build --no-cache -t $DOCKER_REGISTRY_URL/$IMAGE_NAME:v$BUILD_NUMBER --ssh default . &&
Then in Docker file:
This works fine:
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT -o StrictHostKeyChecking=no"
git clone git#github.com:****
Weird thing is this doesn't work:
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT -o StrictHostKeyChecking=no" npm install git+ssh//git#github.com:****
I feel this is something to do with StrictHostKeyChecking=no
I finally got it working by using ROOT user in Dockerfile and setting the npm cache to root.
The problem was that git was using the /root/.ssh folder while npm was using a different path - /home/.ssh as it's npm cache was set on /home/.ssh
For anyone still struggling, this is the config I used
Docker Build Command:
DOCKER_BUILDKIT=1 docker build --no-cache -t test --ssh default .
Dockerfile:
USER root
RUN apt-get update && \
apt-get install -y \
git \
openssh-server \
openssh-client
RUN mkdir -p -m 600 /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts && echo "Host *\n StrictHostKeyChecking no" > /root/.ssh/config
RUN echo "Check ssh_config" && cat /root/.ssh/config
RUN rm -rf node_modules
RUN npm config set cache /root
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT" npm install

MultiStage Docker Image

I have created a Multistage dockerfile.
FROM dot-portal.de.pri.o2.com:8079/centos:centos7 as base
# Install a basic SSH server GIT, UNZIP, LSOF and JDK 8
ADD myrepo.repo /etc/yum.repos.d/myrepo.repo
RUN rm -rf /etc/yum.repos.d/CentOS*
COPY docker /opt/docker
RUN yum repolist refresh && yum clean all && yum repolist && yum install -y openssh-server git unzip lsof epel-release bzip2 fontconfig && yum clean all && rm -rf /var/cache/yum && cd /opt/docker && rpm -ivh *.rpm && rm -rf /opt/docker
ADD docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
RUN ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
# update sshd settings, create jenkins user, set jenkins user pw, generate ssh keys
RUN sed -i 's|session required pam_loginuid.so|session optional pam_loginuid.so|g' /etc/pam.d/sshd \
&& mkdir -p /var/run/sshd \
&& useradd -u 1000 -m -s /bin/bash jenkins \
&& echo "jenkins:jenkins" | chpasswd \
&& usermod -aG wheel jenkins \
&& /usr/bin/ssh-keygen -A \
&& sed -i "s/#UsePrivilegeSeparation.*/UsePrivilegeSeparation no/g" /etc/ssh/sshd_config \
&& sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config \
&& echo export JAVA_HOME="/`alternatives --display java | grep best | cut -d "/" -f 2-6`" >> /etc/environment \
&& usermod -a -G docker jenkins \
&& sed 's#session\s*required\s*pam_loginuid.so#session optional pam_loginuid.so#g' -i /etc/pam.d/sshd
RUN git config --global http.sslVerify false && mkdir -p /opt/app/jenkins
COPY --chown=jenkins:jenkins jenkins_tools /opt/app/jenkins/jenkins_tools/
ADD node-v10.13.0-linux-x64.tar.gz /opt/app/jenkins/jenkins_tools
ENV CI_TOOLS_HOME /opt/app/jenkins/jenkins_tools
ENV ARCH Linux-x86_64
ENV CI_TOOLS_HOME /opt/app/jenkins/jenkins_tools
ENV JAVA_HOME /usr/java/jdk1.8.0_241-amd64/
COPY dot-portal.de.pri.o2.com.crt $CI_TOOLS_HOME/dot-portal.de.pri.o2.com.crt
RUN cd $CI_TOOLS_HOME && echo yes | keytool -importcert -file dot-portal.de.pri.o2.com.crt -alias "dot-portal.de.pri.o2.com" -keystore "$JAVA_HOME/jre/lib/security/cacerts" -storepass changeit
COPY --chown=jenkins:jenkins authorized_keys /home/jenkins/.ssh/
RUN mkdir /root/.ssh && mkdir -p /opt/app/home/jenkins && chown -R jenkins:jenkins /opt/app/home/jenkins
COPY dockerslave /etc/default/docker
RUN ln -s $CI_TOOLS_HOME/phantomjs-2.1.1-linux-x86_64/bin/phantomjs /usr/bin/ && chown -R jenkins:jenkins /opt/app/jenkins/jenkins_tools
FROM base
COPY --from=base /opt/app/jenkins/jenkins_tools /opt/app/jenkins/jenkins_tools
COPY --from=base /etc/default/docker /etc/default/docker
COPY --from=base /usr/bin/phantomjs /usr/bin/phantomjs
ENV CI_TOOLS_HOME /opt/app/jenkins/jenkins_tools
ENV ARCH Linux-x86_64
ENV CI_TOOLS_HOME /opt/app/jenkins/jenkins_tools
ENV JAVA_HOME /usr/java/jdk1.8.0_241-amd64/
ENV MAVEN_HOME /opt/app/jenkins/jenkins_tools/nonarch/maven-3.3.9
RUN echo "export PATH=$CI_TOOLS_HOME/node-v10.13.0-linux-x64/bin/:$CI_TOOLS_HOME/phantomjs-2.1.1-linux-x86_64/bin/:$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH" >> ~/.bashrc
# Standard SSH port
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D", "-e"]
My Question is, the package and user which I installed/created in the base Image wont be reflect in the Final Image. From Base only I'm creating final Image, but when I connect to the container, I'm unable to see the user "jenkins" created in base image and the packages also missing.
What are you trying to achieve with a multistage docker build here? If your second stage starts from the result of first stage in a build with two stages, you might as well have only one stage...
Usually multi stage builds are used because one uses the first stage (e.g. "builder") to build some artefact which requires a lot of development tools. In the second stage only that artefact will be copied into a small base image to get a final image which is very small as it does not contain the tooling needed for building the artefact.
Example from the docker docs:
FROM golang:1.7.3 as builder
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]
In Your case I am pretty sure that your final image does contain the jenkins user, because your second stage is started from the base stage. However these copy commands which copy from the base image will reset the permissions to root in your second stage:
COPY --from=base /opt/app/jenkins/jenkins_tools /opt/app/jenkins/jenkins_tools
COPY --from=base /etc/default/docker /etc/default/docker
COPY --from=base /usr/bin/phantomjs /usr/bin/phantomjs
This copy commands are redundant as you start FROM base anyways. If you'd remove these lines the jenkins_tools directory will be correctly owned by the jenkins user.
Note that you can change the user which is used to execute commands during the docker build with USER <username>

Running dep ensure -vendor-only inside Docker Hangs not able to pull private Repos

My Dockerfile:
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
 RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only
When i am building this docker file it hangs at RUN dep ensure -vendor-only
It fails to pull the dependencies which are private repos
Is there any possiblities to store git credentials inside Docker or any way to build Docker with one or more private repos of GOlang
Use some thing like this
# ensure that the private Github repo is
# accessed using SSH instead of HTTPS
RUN ssh-keyscan github.com > /root/.ssh/known_hosts
RUN echo "$SSH_KEY" > /root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa
RUN echo '[url "ssh://git#github.com/*your_repo*/"]' >> /root/.gitconfig && echo 'insteadOf = https://github.com/*your_repo*/' >> /root/.gitconfig
Refer this to add ssh key to your git repo
Adding .netrc file will pass credentials inside the docker containers and helps to pull more than one private repositories to build dependencies
#vim .netrc
machine github.com
login < your github token >
add those 2 lines and pass your github token
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
COPY .netrc /root/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only

Resources