Pass ssh-agent to dockerfile to install private repository modules - docker

I am trying to automate a docker build in Jenkins pipeline. In my dockerfile, I basically build a node application. In my npm install, I have some private git repositories which need os bindings and so have to be installed in the container. When I run this manually, I transfer my ssh keys (id_rsa) to dockerfile which is used for doing npm install. Now, my problem is when running this task in jenkins pipeline, I will be configuring a ssh-agent(Jenkins plugin). It will not be possible to extract private key from ssh-agent. How should I pass my ssh-agent to my dockerfile.
EDIT 1:
I got it partially working by this:
Docker Build Command:
DOCKER_BUILDKIT=1 docker build --no-cache -t $DOCKER_REGISTRY_URL/$IMAGE_NAME:v$BUILD_NUMBER --ssh default . &&
Then in Docker file:
This works fine:
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT -o StrictHostKeyChecking=no"
git clone git#github.com:****
Weird thing is this doesn't work:
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT -o StrictHostKeyChecking=no" npm install git+ssh//git#github.com:****
I feel this is something to do with StrictHostKeyChecking=no

I finally got it working by using ROOT user in Dockerfile and setting the npm cache to root.
The problem was that git was using the /root/.ssh folder while npm was using a different path - /home/.ssh as it's npm cache was set on /home/.ssh
For anyone still struggling, this is the config I used
Docker Build Command:
DOCKER_BUILDKIT=1 docker build --no-cache -t test --ssh default .
Dockerfile:
USER root
RUN apt-get update && \
apt-get install -y \
git \
openssh-server \
openssh-client
RUN mkdir -p -m 600 /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts && echo "Host *\n StrictHostKeyChecking no" > /root/.ssh/config
RUN echo "Check ssh_config" && cat /root/.ssh/config
RUN rm -rf node_modules
RUN npm config set cache /root
RUN --mount=type=ssh GIT_SSH_COMMAND="ssh -vvvT" npm install

Related

Docker Buildkit SSH/Github woes

I'm trying to write a Dockerfile that pulls a private repository from github. The problem is that I can't get Docker buildkit to use my SSH key properly. Even using the precise instructions and example code from their website does not work. Here is what I did:
Created a passphraseless SSH key using ssh-keygen -t ed25519 -C my_email#my_company.com
Copied the public key and added it as a Github deploy key to my repository
ssh-added the key
Ran the Dockerfile
# syntax=docker/dockerfile:1
FROM alpine
# Install ssh client and git
RUN apk add --no-cache openssh-client git
# Download public key for github.com
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# Clone private repository
RUN --mount=type=ssh git clone git#github.com:myorg/myproject.git myproject
replacing myproject and myorg appropriately. Copy-pasting the git clone command from the Dockerfile to the terminal works. Running DOCKER_BUILDKIT=1 docker build --ssh default and DOCKER_BUILDKIT=1 docker build --ssh default=/path/to/key both fail with the error
> [4/4] RUN --mount=type=ssh git clone git#github.com/myorg/myrepository.git myrepository
#9 0.262 fatal: repository 'git#github.com/myorg/myrepository.git' does not exist
What could be going on here? I'm using Docker 20.10.12 build e91ed57 on MacOS 10.14.6.
This is a workaround rather than a solution to the problem. Instead of the --ssh option, use secrets.
# syntax=docker/dockerfile:experimental
FROM alpine
RUN apk add openssh-client git
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
ENV GIT_SSH_COMMAND="ssh -i /run/secrets/deploy_key"
RUN --mount=type=secret,id=deploy_key git clone git#github.com:myorg/myrepository.git myrepository
building with the command
DOCKER_BUILDKIT=1 docker build --no-cache --secret id=deploy_key,src=/Users/Holmes5/.ssh/deploy_key .

Docker build for Go project with GitLab private repositories

I am having some issues getting my build to work with dependencies in private GitLab repositories. All-in-all it is a multistage build, but the stage where I am attempting to build my Go project is listed below. This works locally for me, so there is an issue somewhere getting this working in Docker:
FROM golang:1.16.8-alpine3.14 as BuildStage
RUN apk update && apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client bash
RUN mkdir /root/.ssh
RUN ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
COPY localRsa /root/.ssh/id_rsa
RUN chmod 0400 /root/.ssh/id_rsa
RUN eval $(ssh-agent -s) && ssh-add /root/.ssh/id_rsa
WORKDIR $GOPATH/src/myproject
COPY . .
ENV GOPRIVATE="gitlab.com/MyGitLabUser"
RUN git config --global url."git#gitlab.com".insteadOf "https://gitlab.com"
RUN go mod download
RUN go mod verify
RUN GOOS=linux GOARCH=amd64 \
go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
The error message I get:
go: gitlab.com/MyProject/Sub1/Sub2/some-library#v0.0.6: reading gitlab.com/MyProject/Sub1/Sub2/some-library.git/go.mod at revision v0.0.6: unknown revision v0.0.6
The release definitely exists and is working locally. I am missing a step somewhere.
Update:
Cloning the project works from that stage if I add:
git clone git#gitlab.com:MyProject/Sub1/Sub2/some-library.git
Makes me think I'm missing something in my Go configuration or linking Go with Git.
Issues
After putting together everything that everyone provided here (thanks, by the way). I was able to rework my Dockerfile to do exactly what I needed and get things to work! So first, let me go through and enumerate all of the problems that were in my original submission:
Passing in the SSH key is completely unnecessary (see here - hat tip #RakeshGupta)
Using go mod download -x helped a lot to search out there for more specific information (hat tip #mh-cbon)
I was able to simplify more (see here - hat tip #sytech)
One of the big things was a typo that I fixed, so now it's: git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
Updated Dockerfile
FROM golang:1.16.8-alpine3.14 as BuildStage
# setup Git & SSL (for getting dependencies)
RUN apk update && \
apk add --no-cache git ca-certificates tzdata gcc libc-dev openssh-client && \
update-ca-certificates
ENV GOPRIVATE="gitlab.com/MyProject"
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
RUN git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
# setup an application user
ENV USER=appuser
ENV UID=10001
RUN adduser --disabled-password \
--gecos "" \
--home "/nonexistent" \
--shell "/sbin/nologin" \
--no-create-home \
--uid "${UID}" "${USER}"
# build the project
WORKDIR $GOPATH/src/myproject
COPY . .
# make sure Go knows the packages are private
RUN go env -w GOPRIVATE="gitlab.com/MyProject/*"
# build the binary
RUN --mount=type=ssh go mod download -x && go mod verify
RUN --mount=type=ssh GOOS=linux GOARCH=amd64 go build -ldflags='-w -s -extldflags "-static"' -tags musl -a -o /go/bin/mybinary
Updated Build Command
So that's cute and all, but I need BuildKit with SSH:
eval "$(minikube docker-env)"
DOCKER_BUILDKIT=1 docker build --ssh default -t myservice:latest .

Building Go apps with private modules in Docker

I'm trying to build a go project in a docker container that relies on private submodules.
I was hoping that --mount=type=ssh would pass my ssh credentials to the container and it'd work. Currently I can build locally with just make the GOPRIVATE variable set and the git config update.
Here is my relevant Dockerfile currently
# syntax = docker/dockerfile:experimental
FROM golang:1.14.3-alpine AS build
RUN apk add --no-cache git \
openssh-client \
ca-certificates
WORKDIR /src
ENV GIT_TERMINAL_PROMPT=1
ENV GOPRIVATE="gitlab.com/company_foo"
RUN git config --global url."ssh://git#gitlab.com".insteadOf "https://gitlab.com"
# Authorize SSH Host
# Skip Host verification for git
RUN mkdir -p /root/.ssh && \
chmod 0700 /root/.ssh && \
ssh-keyscan gitlab.com > /root/.ssh/known_hosts &&\
chmod 644 /root/.ssh/known_hosts && touch /root/.ssh/config \
&& echo "StrictHostKeyChecking no" > /root/.ssh/config
COPY go.mod go.sum .
RUN --mount=type=ssh mkdir -p /var/ssh && \
GIT_SSH_COMMAND="ssh -o \"ControlMaster auto\" -o \"ControlPersist 300\" -o \"ControlPath /var/ssh/%r#%h:%p\"" \
go mod download
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build go build -o api-server ./cmd/api-server
RUN --mount=type=cache,target=/root/.cache/go-build go build -o migrations ./cmd/migrations
I've also tried adding a CI_JOB_TOKEN with
RUN echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc
but this also didn't work. Perhaps I did it wrong.
All of this results in the failure:
revision v0.0.3: unknown revision v0.0.3
relating to one of our private repos.
Any advice would be appreciate.
I'm absolutely at a lost.
This workes for me.
FROM golang:1.14
ARG USERNAME=user1
ARG PASSWORD=secret
WORKDIR /app
ADD . .
ENV GOPRIVATE=private.git.local/*
RUN echo "machine private.git.local login $USERNAME password $PASSWORD" > ~/.netrc
RUN go build -o testGo main.go
CMD ["/app/testGo"]
pass your gitlab_token to docker file from gitlab_ci.yaml and do the following steps
RUN git config --global url."https://oauth2:$GITLAB_TOKEN#gitlab.com/".insteadOf "https://git#gitlab.com/"
add your repo as GO_PRIVATE
ENV GOPRIVATE=gitlab.com/*
copy .netrc file to docker root
COPY confidential/.netrc /root/.netrc
.netrc file will have the following structure
machine gitlab.com
login gitlab_user
password p#$$word

Running dep ensure -vendor-only inside Docker Hangs not able to pull private Repos

My Dockerfile:
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
 RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only
When i am building this docker file it hangs at RUN dep ensure -vendor-only
It fails to pull the dependencies which are private repos
Is there any possiblities to store git credentials inside Docker or any way to build Docker with one or more private repos of GOlang
Use some thing like this
# ensure that the private Github repo is
# accessed using SSH instead of HTTPS
RUN ssh-keyscan github.com > /root/.ssh/known_hosts
RUN echo "$SSH_KEY" > /root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa
RUN echo '[url "ssh://git#github.com/*your_repo*/"]' >> /root/.gitconfig && echo 'insteadOf = https://github.com/*your_repo*/' >> /root/.gitconfig
Refer this to add ssh key to your git repo
Adding .netrc file will pass credentials inside the docker containers and helps to pull more than one private repositories to build dependencies
#vim .netrc
machine github.com
login < your github token >
add those 2 lines and pass your github token
FROM golang:1.11.4
RUN apt-get update && apt-get install git bash curl -yqq
ENV ENV test
ENV GIT_TERMINAL_PROMPT=1
ENV GITHUB_TOKEN XXXXXXXXXXXXXXXXXX 
RUN curl -Ls https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz | tar xz -C /tmp \
&& mv /tmp/linux-amd64/glide /usr/bin/
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN mkdir -p $GOPATH/src/github.com/<Myrepo>/
COPY . $GOPATH/src/github.com/<Myrepo>/
COPY .netrc /root/
WORKDIR $GOPATH/src/github.com/<Myrepo>/
RUN dep ensure -vendor-only

Dockerfile: Permission denied during build when running ssh-agent on /tmp

So I'm trying to create an image, which adds a SSH private key to /tmp, runs ssh-agent on it, does a git clone and then deletes the key again.
This is the idea I'm trying to accomplish
Dockerfile:
FROM node:4.2.4
MAINTAINER Me
CMD ["/bin/bash"]
ENV GIT_SSL_NO_VERIFY=1
ENV https_proxy="httpsproxy"
ENV http_proxy="httpproxy"
ENV no_proxy="exceptions"
ADD projectfolder/key /tmp/
RUN ssh-agent /tmp
WORKDIR /usr/src/app
RUN git clone git#gitlab.private.address:something/target.git
RUN rm /tmp/key
WORKDIR /usr/src/app/target
RUN npm install
EXPOSE 3001
Now the problem lies within the build-process. I use the following command to build:
docker build -t samprog/targetimage:4.2.4 -f projectfolder/dockerfile .
The layers up to "ADD projectfolder/key /tmp/" work just fine, though the "RUN ssh-agent /tmp" layer doesn't want to cooperate.
Error code:
Step 9 : RUN ssh-agent /tmp/temp
---> Running in d2ed7c8870ae
/tmp: Permission denied
The command '/bin/sh -c ssh-agent /tmp' returned a non-zero code: 1
Any ideas? Since I thought it was a permission issue, where the directory was already created by the parent image, I created a /tmp/temp and put the key in there. Doesn't work either, same error.
I'm using Docker version 1.10.3 on SLES12 SP1
I did it. What I did is, I got rid of ssh-agent. I simply copied the ~/.ssh- directory of my docker-host into the /root/.ssh of the image and it worked.
Do not use the ~ though, copy the ~/.ssh-directory inside the projectfolder first and then with the dockerfile inside the container.
Final dockerfile looked as follows:
FROM node:4.2.4
MAINTAINER me
CMD["/bin/bash"]
ENV GIT_SSL_NO_VERIFY=1
ENV https_proxy="httpsproxy"
ENV http_proxy="httpproxy"
ENV no_proxy="exceptions"
ADD projectfolder/.ssh /root/.ssh
WORKDIR /usr/src/app
RUN git clone git#gitlab.private.address:something/target.git
RUN rm -r /root/.ssh
WORKDIR /urs/src/app/target
RUN npm set registry http://local-npm-registry
RUN npm install
EXPOSE 3001
The dockerfile still has to be improved on efficiency and stuff, but it works! Eureka!
The image now has to be squashed and it should be safe to use, though we only use it in our local registry.
I have faced with the same problem with maven:3-alpine. It was solved when I properly installed openssh-client:
RUN apk --update add openssh-client
Then copied keys with known hosts to the image:
ADD id_rsa /root/.ssh/
ADD id_rsa.pub /root/.ssh/
ADD known_hosts /root/.ssh/
And ran git clone command inline (with ssh-agent and ssh-add):
RUN eval $(ssh-agent -s) \
&& ssh-add \
&& git clone ssh://git#private.address:port/project/project.git
Complete docker file:
FROM maven:3-alpine
RUN apk update
RUN apk add python
RUN apk add ansible
RUN apk add git
RUN apk --update add openssh-client
ADD id_rsa /root/.ssh/
ADD id_rsa.pub /root/.ssh/
ADD known_hosts /root/.ssh/
RUN eval $(ssh-agent -s) \
&& ssh-add \
&& git clone ssh://git#private.address:port/project/project.git
ADD hosts /etc/ansible/hosts
RUN ansible all -m ping --ask-pass
I had the same issue while executing any bash command when building my Dockerfile.
I solved by adding RUN chmod -R 777 ./ like suggested in the answer of this question. I think this is a workaround, I'm not sure why docker in ubuntu has permission issues when building a container.

Resources