Docker build for Go project with GitLab private repositories - docker

I am having some issues getting my build to work with dependencies in private GitLab repositories. All-in-all it is a multistage build, but the stage where I am attempting to build my Go project is listed below. This works locally for me, so there is an issue somewhere getting this working in Docker:
FROM golang:1.16.8-alpine3.14 as BuildStage
RUN apk update && apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client bash
RUN mkdir /root/.ssh
RUN ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
COPY localRsa /root/.ssh/id_rsa
RUN chmod 0400 /root/.ssh/id_rsa
RUN eval $(ssh-agent -s) && ssh-add /root/.ssh/id_rsa
WORKDIR $GOPATH/src/myproject
COPY . .
ENV GOPRIVATE="gitlab.com/MyGitLabUser"
RUN git config --global url."git#gitlab.com".insteadOf "https://gitlab.com"
RUN go mod download
RUN go mod verify
RUN GOOS=linux GOARCH=amd64 \
go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
The error message I get:
go: gitlab.com/MyProject/Sub1/Sub2/some-library#v0.0.6: reading gitlab.com/MyProject/Sub1/Sub2/some-library.git/go.mod at revision v0.0.6: unknown revision v0.0.6
The release definitely exists and is working locally. I am missing a step somewhere.
Update:
Cloning the project works from that stage if I add:
git clone git#gitlab.com:MyProject/Sub1/Sub2/some-library.git
Makes me think I'm missing something in my Go configuration or linking Go with Git.

Issues
After putting together everything that everyone provided here (thanks, by the way). I was able to rework my Dockerfile to do exactly what I needed and get things to work! So first, let me go through and enumerate all of the problems that were in my original submission:
Passing in the SSH key is completely unnecessary (see here - hat tip #RakeshGupta)
Using go mod download -x helped a lot to search out there for more specific information (hat tip #mh-cbon)
I was able to simplify more (see here - hat tip #sytech)
One of the big things was a typo that I fixed, so now it's: git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
Updated Dockerfile
FROM golang:1.16.8-alpine3.14 as BuildStage
# setup Git & SSL (for getting dependencies)
RUN apk update && \
apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client && \
update-ca-certificates
ENV GOPRIVATE="gitlab.com/MyProject"
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
RUN git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
# setup an application user
ENV USER=appuser
ENV UID=10001
RUN adduser --disabled-password \
--gecos "" \
--home "/nonexistent" \
--shell "/sbin/nologin" \
--no-create-home \
--uid "${UID}" "${USER}"
# build the project
WORKDIR $GOPATH/src/myproject
COPY . .
# make sure Go knows the packages are private
RUN go env -w GOPRIVATE="gitlab.com/MyProject/*"
# build the binary
RUN --mount=type=ssh go mod download -x && go mod verify
RUN --mount=type=ssh GOOS=linux GOARCH=amd64 go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
Updated Build Command
So that's cute and all, but I need BuildKit with SSH:
eval "$(minikube docker-env)"
DOCKER_BUILDKIT=1 docker build --ssh default -t myservice:latest .

Related

How does this Dockerfile actually run logstash without an entrypoint or cmd?

Just doing a container start on this official logstash docker container does make logstash properly run, given the right config.
It does not have an entrypoint or cmd, or anything of the sort though. I am also not issuing one on the start command. So, how is logstash actually getting executed in this case?
I need to know because I need to edit the command for other reasons. We're working on running it in kubernetes but are just testing with local docker for now.
https://github.com/elastic/logstash/blob/7.15/Dockerfile
Copied for easy reference:
FROM ubuntu:bionic
RUN apt-get update && \
apt-get install -y zlib1g-dev build-essential vim rake git curl libssl-dev libreadline-dev libyaml-dev \
libxml2-dev libxslt-dev openjdk-11-jdk-headless curl iputils-ping netcat && \
apt-get clean
WORKDIR /root
RUN adduser --disabled-password --gecos "" --home /home/logstash logstash && \
mkdir -p /usr/local/share/ruby-build && \
mkdir -p /opt/logstash && \
mkdir -p /opt/logstash/data && \
mkdir -p /mnt/host && \
chown logstash:logstash /opt/logstash
USER logstash
WORKDIR /home/logstash
# used by the purge policy
LABEL retention="keep"
# Setup gradle wrapper. When running any `gradle` command, a `settings.gradle` is expected (and will soon be required).
# This section adds the gradle wrapper, `settings.gradle` and sets the permissions (setting the user to root for `chown`
# and working directory to allow this and then reverts back to the previous working directory and user.
COPY --chown=logstash:logstash gradlew /opt/logstash/gradlew
COPY --chown=logstash:logstash gradle/wrapper /opt/logstash/gradle/wrapper
COPY --chown=logstash:logstash settings.gradle /opt/logstash/settings.gradle
WORKDIR /opt/logstash
RUN for iter in `seq 1 10`; do ./gradlew wrapper --warning-mode all && exit_code=0 && break || exit_code=$? && echo "gradlew error: retry $iter in 10s" && sleep 10; done; exit $exit_code
WORKDIR /home/logstash
ADD versions.yml /opt/logstash/versions.yml
ADD LICENSE.txt /opt/logstash/LICENSE.txt
ADD NOTICE.TXT /opt/logstash/NOTICE.TXT
ADD licenses /opt/logstash/licenses
ADD CONTRIBUTORS /opt/logstash/CONTRIBUTORS
ADD Gemfile.template Gemfile.jruby-2.5.lock.* /opt/logstash/
ADD Rakefile /opt/logstash/Rakefile
ADD build.gradle /opt/logstash/build.gradle
ADD rubyUtils.gradle /opt/logstash/rubyUtils.gradle
ADD rakelib /opt/logstash/rakelib
ADD config /opt/logstash/config
ADD spec /opt/logstash/spec
ADD qa /opt/logstash/qa
ADD lib /opt/logstash/lib
ADD pkg /opt/logstash/pkg
ADD tools /opt/logstash/tools
ADD logstash-core /opt/logstash/logstash-core
ADD logstash-core-plugin-api /opt/logstash/logstash-core-plugin-api
ADD bin /opt/logstash/bin
ADD modules /opt/logstash/modules
ADD x-pack /opt/logstash/x-pack
ADD ci /opt/logstash/ci
USER root
RUN rm -rf build && \
mkdir -p build && \
chown -R logstash:logstash /opt/logstash
USER logstash
WORKDIR /opt/logstash
LABEL retention="prune"
If you look at the final layer on the image here, it looks like there is an ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]. The Dockerfile you've linked might not be the one used to build the image.

Clone git repo from github with dockerfile

I'm trying to clone GitHub repo with below docker file after installing java,maven
FROM openjdk:8-jdk-alpine
LABEL WebAutomation Test <waaanjula#gmail.com>
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
# 1- Define a constant with the version of maven you want to install
ARG MAVEN_VERSION=3.8.1
# 3- Define the SHA key to validate the maven download
ARG SHA=0ec48eb515d93f8515d4abe465570dfded6fa13a3ceb9aab8031428442d9912ec20f066b2afbf56964ffe1ceb56f80321b50db73cf77a0e2445ad0211fb8e38d
# 4- Define the URL where maven can be downloaded from
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
#5- Create the directories, download maven, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
# 6- Define environmental variables required by Maven, like Maven_Home directory and where the maven repo is located
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# # Install dependencies required to git clone.
RUN apk update && \
apk add --update git && \
apk add --update openssh
# 1. Create the SSH directory.
# 2. Populate the private key file.
# 3. Set the required permissions.
# 4. Add github to our list of known hosts for ssh.
RUN mkdir -p /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
ADD id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod -R 700 /root/.ssh/ && \
touch ~/.ssh/known_hosts && \
ssh-keyscan -t rsa github.com >> /root/.ssh/known_hosts
RUN mkdir /usr/AutomationProject/ && \
chmod -R 600 /usr/AutomationProject/
# # Clone a repository (MatificWebAutomation project)
RUN git clone git#github.com:anjulaw/Selenium_Keyword_driven_Example.git /usr/AutomationProject
CMD [""]
and I'm getting below error
"git#github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists."
This is my first time using dockerfiles, but from what I have read (and taken from working configs) I cannot see why this doesn't work.
My id_rsa is in the same folder as my dockerfile and is a copy of my local key which can clone this repo no problem.
You need to create a folder with keys in the local project folder, transfer there the generated keys that are already installed in the remote repository.
# SSH Keys
ADD .ssh/id_rsa /root/.ssh/id_rsa
ADD .ssh/id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod 600 /root/.ssh/id_rsa &&\
chmod 600 /root/.ssh/id_rsa.pub
It turns out that after each build, the key will be generated, which means that after each build of this Dockerfile, you will have to reconfigure the keys on the remote repository 💁

Pass ssh-agent to dockerfile to install private repository modules

I am trying to automate a docker build in Jenkins pipeline. In my dockerfile, I basically build a node application. In my npm install, I have some private git repositories which need os bindings and so have to be installed in the container. When I run this manually, I transfer my ssh keys (id_rsa) to dockerfile which is used for doing npm install. Now, my problem is when running this task in jenkins pipeline, I will be configuring a ssh-agent(Jenkins plugin). It will not be possible to extract private key from ssh-agent. How should I pass my ssh-agent to my dockerfile.
EDIT 1:
I got it partially working by this:
Docker Build Command:
DOCKER_BUILDKIT=1 docker build --no-cache -t $DOCKER_REGISTRY_URL/$IMAGE_NAME:v$BUILD_NUMBER --ssh default . &&
Then in Docker file:
This works fine:
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT -o StrictHostKeyChecking=no"
git clone git#github.com:****
Weird thing is this doesn't work:
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT -o StrictHostKeyChecking=no" npm install git+ssh//git#github.com:****
I feel this is something to do with StrictHostKeyChecking=no
I finally got it working by using ROOT user in Dockerfile and setting the npm cache to root.
The problem was that git was using the /root/.ssh folder while npm was using a different path - /home/.ssh as it's npm cache was set on /home/.ssh
For anyone still struggling, this is the config I used
Docker Build Command:
DOCKER_BUILDKIT=1 docker build --no-cache -t test --ssh default .
Dockerfile:
USER root
RUN apt-get update && \
apt-get install -y \
git \
openssh-server \
openssh-client
RUN mkdir -p -m 600 /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts && echo "Host *\n StrictHostKeyChecking no" > /root/.ssh/config
RUN echo "Check ssh_config" && cat /root/.ssh/config
RUN rm -rf node_modules
RUN npm config set cache /root
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT" npm install

Building Go apps with private modules in Docker

I'm trying to build a go project in a docker container that relies on private submodules.
I was hoping that --mount=type=ssh would pass my ssh credentials to the container and it'd work. Currently I can build locally with just make the GOPRIVATE variable set and the git config update.
Here is my relevant Dockerfile currently
# syntax = docker/dockerfile:experimental
FROM golang:1.14.3-alpine AS build
RUN apk add --no-cache git \
openssh-client \
ca-certificates
WORKDIR /src
ENV GIT_TERMINAL_PROMPT=1
ENV GOPRIVATE="gitlab.com/company_foo"
RUN git config --global url."ssh://git#gitlab.com".insteadOf "https://gitlab.com"
# Authorize SSH Host
# Skip Host verification for git
RUN mkdir -p /root/.ssh && \
chmod 0700 /root/.ssh && \
ssh-keyscan gitlab.com > /root/.ssh/known_hosts &&\
chmod 644 /root/.ssh/known_hosts && touch /root/.ssh/config \
&& echo "StrictHostKeyChecking no" > /root/.ssh/config
COPY go.mod go.sum .
RUN --mount=type=ssh mkdir -p /var/ssh && \
GIT_SSH_COMMAND="ssh -o \"ControlMaster auto\" -o \"ControlPersist 300\" -o \"ControlPath /var/ssh/%r#%h:%p\"" \
go mod download
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build go build -o api-server ./cmd/api-server
RUN --mount=type=cache,target=/root/.cache/go-build go build -o migrations ./cmd/migrations
I've also tried adding a CI_JOB_TOKEN with
RUN echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc
but this also didn't work. Perhaps I did it wrong.
All of this results in the failure:
revision v0.0.3: unknown revision v0.0.3
relating to one of our private repos.
Any advice would be appreciate.
I'm absolutely at a lost.
This workes for me.
FROM golang:1.14
ARG USERNAME=user1
ARG PASSWORD=secret
WORKDIR /app
ADD . .
ENV GOPRIVATE=private.git.local/*
RUN echo "machine private.git.local login $USERNAME password $PASSWORD" > ~/.netrc
RUN go build -o testGo main.go
CMD ["/app/testGo"]
pass your gitlab_token to docker file from gitlab_ci.yaml and do the following steps
RUN git config --global url."https://oauth2:$GITLAB_TOKEN#gitlab.com/".insteadOf "https://git#gitlab.com/"
add your repo as GO_PRIVATE
ENV GOPRIVATE=gitlab.com/*
copy .netrc file to docker root
COPY confidential/.netrc /root/.netrc
.netrc file will have the following structure
machine gitlab.com
login gitlab_user
password p#$$word

Running dep ensure -vendor-only inside Docker Hangs not able to pull private Repos

My Dockerfile:
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
 RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only
When i am building this docker file it hangs at RUN dep ensure -vendor-only
It fails to pull the dependencies which are private repos
Is there any possiblities to store git credentials inside Docker or any way to build Docker with one or more private repos of GOlang
Use some thing like this
# ensure that the private Github repo is
# accessed using SSH instead of HTTPS
RUN ssh-keyscan github.com > /root/.ssh/known_hosts
RUN echo "$SSH_KEY" > /root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa
RUN echo '[url "ssh://git#github.com/*your_repo*/"]' >> /root/.gitconfig && echo 'insteadOf = https://github.com/*your_repo*/' >> /root/.gitconfig
Refer this to add ssh key to your git repo
Adding .netrc file will pass credentials inside the docker containers and helps to pull more than one private repositories to build dependencies
#vim .netrc
machine github.com
login < your github token >
add those 2 lines and pass your github token
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
COPY .netrc /root/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only

Resources