I have specified a git clone command in the docker file as below.
RUN git clone https://github.com/zhaoyi0113/test.git
but I got this error when build the docker image:
Cloning into 'test'...
fatal: could not read Username for 'https://github.com': No such device or address
I wonder why it doesn't work. I am able to run this command on my host. Is there anything different if I list it on docker file?
You can pass credentials as arguments to container. This should work
FROM alpine:3.8
RUN apk update && apk upgrade && \
apk add --no-cache bash git openssh
ARG username
ARG password
RUN git clone https://${username}:${password}#github.com/username/repository.git
ENTRYPOINT ["sleep 10"]
but it might be unsafe if you want to distribute that image
then build
docker build \
--no-cache \
-t git-app:latest \
--build-arg username=user \
--build-arg password=qwerty \
.
I know what wrong with this problem. The issue is because the repo I am cloning is a private repo which means it requires credential to connect to github. Fixing it by passing credentials to the container.
Related
My Git repo isn't changing in my Docker image even when rebuilding with a new tag.
I run git clone in a build and it works:
# dockerfile
FROM python:3.8
ENV PYTHONUNBUFFERED 1
# To fix GPG key error when running apt-get update
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub
RUN apt-get update && apt-get install -y git
RUN git clone https://github.com/my_git_username/repo_name.git
WORKDIR /repo_name
I build:
docker build -t username/exam:0.1
Then, I push a file named tmp.txt to that repo, and run a new build:
docker build -t username/exam:0.2
I expect that this command fails, because I changed the repo to private, but it succeeds. So I went inside the container and checked the contents:
docker run -it username/exam:0.2
import os
print(os.listdir(os.getcwd()))
And I confirm that tmp.txt does not exist (which is the previous state of the repo).
This is what I would expect:
=> ERROR [6/7] RUN git clone https://github.com/my_git_username/repo_name.git 1.1s
------
> > [6/7] RUN git clone https://github.com/my_git_username/repo_name.git:
> #10 0.632 Cloning into 'repo_name'...
> #10 1.007 fatal: could not read Username for 'https://github.com': No such device or address
> ------
> executor failed running [/bin/sh -c git clone https://github.com/my_git_username/repo_name.git]: exit code: 128
What I need help with:
Why is the git repo at the previous commit in the second image?
How can I automatically bring the latest modifications without modifying the Dockerfile?
Use docker build --no-cache. Docker caches layers, and is likely caching the git clone layer (step), because the command hasn't changed and the checksums of the previous layers haven't changed.
Example: docker build --no-cache -t username/exam:0.3 ..
I am trying to build a docker image on an Ubuntu machine, where one of the dependencies is
an internally hosted Gitlab project (IT IS NOT HOSTED ON GITLAB.COM).
The docker file is:
FROM rayproject/ray:1.12.0-py39-cpu
RUN git config --global user.name <MY USER NAME>
RUN git config --global http.sslVerify false
COPY .ssh/id_rsa /home/ray/.ssh/id_rsa
RUN sudo chmod -R 777 ~/.ssh/*
RUN eval "$(ssh-agent -s)"
RUN eval `ssh-agent -s` && ssh-add $HOME/.ssh/id_rsa
RUN pip install git+https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git
I am attempting to build the docker file with:
docker build .
I receive this error message when building the docker image:
Step 8/29 : RUN pip install git+https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git
---> Running in b9e11319c84d
Collecting git+https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git
Cloning https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git to /tmp/pip-req-build-l1qklujg
Running command git clone -q https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git /tmp/pip-req-build-l1qklujg
fatal: could not read Username for 'https://gitlab.<EMPLOYERS DOMAIN>.com': No such device or address
WARNING: Discarding git+https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git. Command errored out with exit status 128: git clone -q https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git /tmp/pip-req-build-l1qklujg Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git /tmp/pip-req-build-l1qklujg Check the logs for full command output.
However, from my machine, I can run
pip install https://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git
without being asked to enter my username/password.
I'm unsure whether or not my machine was configured to use the RSA key to login into the internal Gitlab.
How can I install the package when building the docker image?
Use ssh instead of https. i.e.
RUN pip install git+ssh://gitlab.<EMPLOYERS DOMAIN>.com/xyz/my_project.git
Just to recall a security issue that wouldn't fit in a mere comment:
Every datum that we COPY within the Dockerfile is kept forever
(even if we do RUN rm -fr something afterwards!),
so this means here that anyone that can pull the Docker image can retrieve the contents of the /home/ray/.ssh/id_rsa file… to avoid this, two ways:
either do the cloning of the repo outside of the Dockerfile from GitLab-CI (then do a mere COPY my_project my_project then RUN pip install ./my_project)
or use the docker build --ssh feature.
When attempting to build a GoLang-based Docker Image, the Docker executor runs into the following error:
. . .go: $GIT_REPO#v1.9.11: reading $GIT_REPO/go.mod at revision v0.0.07: unknown revision v0.0.07
at the following RUN instruction from the Dockerfile used:
RUN go build . . .
where GIT_REPO represents the private repo. full path, including owner and name.
The Docker executor encounters this error with go1.13.x and higher; the Docker executor does not encounter this error with go1.12.x.
The vendor dir. contains all required packages. Tags are confirmed to be present.
Proper SSH keys were even added to the private Go common repo. with successful
git clone . . .
commands outside of building Docker images, but still encountering the same error above.
EDIT:
Verify your remote repo in bitbucket.org actually has the v0.0.7 tag you're trying to build against.
While a local build may work if the git tag exists locally - a docker build will pull from the remote source and fail with an error like go.mod at revision v0.0.7: unknown revision v0.0.7 - if the tag does not exist remotely.
To push your local tags to the remote repo:
git push --tags
For more granular tag operations see.
Docker builds by default can only access public repos. Since you need access to a private repo, you need to include a read-ssh key to the Docker build process (keys should never be checked into the repo!).
It is critically important, however, you do this in a multi-stage build, so you do not include your SSH keys in the final image.
This blog post walks through all the steps. But to include a working example:
To build the docker image:
SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)" \
docker build -t "myapp:v0.0.1" --build-arg SSH_PRIVATE_KEY .
And the Dockerfile using a bitbucket.org private repo site:
FROM golang:1.14.6 AS build
WORKDIR /bld
COPY *.go go.mod go.sum ./
ARG SSH_PRIVATE_KEY
# ***NEVER*** DO THIS IN A SINGLE-STAGE DOCKER BUILD (see below)
RUN \
mkdir -p ~/.ssh && \
umask 0077 && \
echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa && \
git config --global url."git#bitbucket.org:".insteadOf https://bitbucket.org/ && \
ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
RUN \
go get && \
CGO_ENABLED=0 go build -o app
# final stage of multi-stage: will appropriately *NOT* include SSH keys
FROM scratch
COPY --from=build \
/etc/ssl /etc/ssl
COPY --from=build \
/bld/app /app/myapp
CMD ["/app/myapp"]
There are two problems to solve here:
1. How to allow Docker to access local SSH keys, safely?
2. How to tell Go not to use public registry to fetch private packages?
Short Answers
Since Docker v18.09, there's a built-in solution to handle SSH authentication during the build phase (more). It's also much easier and safer, compared to passing build arguments, and eliminates the need for a multi-stage Docker build.
Go has a GOPRIVATE environment variable to identify private packages. (more)
Long Answer
Step-by-step:
1. Make sure ssh-agent is setup and knows the SSH key
Github has a quick guide on this subject, explaining the process for different operating systems. See Generating a new SSH key and adding it to SSH agent.
2. Enable BuildKit for Docker
Without BuildKit, docker build won't recognize --ssh option.
From the Docker reference:
Easiest way from a fresh install of docker is to set the
DOCKER_BUILDKIT=1 environment variable when invoking the docker build
command, such as:
$ DOCKER_BUILDKIT=1 docker build .
To enable docker BuildKit by
default, set daemon configuration in /etc/docker/daemon.json feature
to true and restart the daemon:
{ "features": { "buildkit": true } }
Docker Desktop users can manage daemon configurations via Preferences > Docker Engine.
4. Update Dockerfile
4.1. Make sure Git uses SSH instead of HTTPs
Go tends to fetch public packages via HTTPs. You can adjust this behavior by updating git configurations:
RUN git config --global url.git#github.com:.insteadOf https://github.com/
You should probably do this on your local machine as well.
4.2. Request SSH access where it's required
Each RUN command the needs SSH access should be mounted with type=ssh. For
Example:
RUN --mount=type=ssh git clone ...
4.3. Make sure Go knows your private packages
Update GOPRIVATE variable:
RUN go env -w GOPRIVATE="github.com/your-org/private-repo"
Putting all of it together, in following sample of a Dockerfile:
FROM golang:1.16.3-alpine3.13
RUN apk update
RUN apk add git openssh
RUN mkdir /app
ADD . /app
WORKDIR /app
# You can replace github.com with any other Git host
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# Make sure git uses SSH to fetch packages, not HTTPs
RUN git config --global url.git#github.com:.insteadOf https://github.com/
# Make Go knows which packages are private.
RUN go env -w GOPRIVATE="github.com/your-org/private-repo"
# GOPRIVATE is a comma separated list glob-patterns.
# You can use a wildcard to match every repo in an organization:
# e.g.: GOPRIVATE="github.com/your-org/*"
# Mount the build command with type `ssh`.
RUN --mount=type=ssh go get && go build -o main .
CMD ["/app/main"]
6. Build the image with --ssh option:
Having BuildKit enabled by default:
$ docker build --ssh default -t my-app:latest .
I want to create an image , with git pre-installed that can log in Github during docker-run.
This image suppose to get Github username and token in docker-run command through -e option.
For example :
sudo docker run -it --rm -e "username=parsalotfy" -e "token=secretsecretsecret" -e "email=parsalotfy#gmail.com" gitloggedin
I wrote Dockerfile like this :
FROM alpine
RUN apk update
RUN apk upgrade
RUN apk add git
RUN git clone https://$username:$token#github.com/$username/$repo.git
RUN cd $repo/
RUN git config user.email $email
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
But when I want to build this Dockerfile, this error occurs :
What am I doing wrong here?
And How can I have a Github user logged in passing username and password through -e option after docker-run ?
Thank you.
As someone mentioned the reference link in comments you'll need to declare your environment variables before use like ENV EMAIL me#example.com or just ENV EMAIL then using it
I am creating my own Dockerfile based on the Jenkins Docker image only to add some pre-installed packages in the image (build-essential, etc...).
In the Jenkins Dockerfile they used the ARG command to create the jenkins user and group: See these lines.
Here is my Dockerfile:
FROM jenkins
USER root
RUN apt-get update && apt-get install -y build-essential
USER jenkins
But when I build it to fit my jenkins host user by using this line:
docker build --tag my-jenkins \
--build-arg user=jenkins \
--build-arg group=jenkins \
--build-arg uid=$(id -u jenkins) \
--build-arg gid=$(id -g jenkins) \
.
I have got this error:
One or more build-args [gid group uid user] were not consumed, failing build.
Is there a way to do this or is it impossible?
It's impossible. The image is already built (using the defaults). You could write a quick script to build your own jenkins image (passing the args) then build your sub-image from that.