I want to create an image , with git pre-installed that can log in Github during docker-run.
This image suppose to get Github username and token in docker-run command through -e option.
For example :
sudo docker run -it --rm -e "username=parsalotfy" -e "token=secretsecretsecret" -e "email=parsalotfy#gmail.com" gitloggedin
I wrote Dockerfile like this :
FROM alpine
RUN apk update
RUN apk upgrade
RUN apk add git
RUN git clone https://$username:$token#github.com/$username/$repo.git
RUN cd $repo/
RUN git config user.email $email
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
But when I want to build this Dockerfile, this error occurs :
What am I doing wrong here?
And How can I have a Github user logged in passing username and password through -e option after docker-run ?
Thank you.
As someone mentioned the reference link in comments you'll need to declare your environment variables before use like ENV EMAIL me#example.com or just ENV EMAIL then using it
Related
I just started learning docker. To teach myself, I managed to containerize bandit (a python code scanner) but I'm not able to see the output of the scan before the container destroys itself. How can I copy the output file from inside the container to the host, or otherwise save it?
Right now i'm just using bandit to scan itself basically :)
Dockerfile
FROM python:3-alpine
WORKDIR /
RUN pip install bandit
RUN apk update && apk upgrade
RUN apk add git
RUN git clone https://github.com/PyCQA/bandit.git ./code-to-scan
CMD [ "python -m bandit -r ./code-to-scan -o bandit.txt" ]
You can mount a volume on you host where you can share the output of bandit.
For example, you can run your container with:
docker run -v $(pwd)/output:/tmp/output -t your_awesome_container:latest
And you in your dockerfile:
...
CMD [ "python -m bandit -r ./code-to-scan -o /tmp/bandit.txt" ]
This way the bandit.txt file will be found in the output folder.
Better place the code in your image not in the root directory.
I did some adjustments to your Dockerfile.
FROM python:3-alpine
WORKDIR /usr/myapp
RUN pip install bandit
RUN apk update && apk upgrade
RUN apk add git
RUN git clone https://github.com/PyCQA/bandit.git .
CMD [ "bandit","-r",".","-o","bandit.txt" ]`
This clones git in your WORKDIR.
Note the CMD, it is an array, so just devide all commands and args as in the Dockerfile about.
I put the the Dockerfile in my D:\test directory (Windows).
docker build -t test .
docker run -v D:/test/:/usr/myapp test
It will generate you bandit.txt in the test folder.
After the code is execute the container exits, as there are nothing else to do.
you can also put --rm to remove the container once it finishs.
docker run --rm -v D:/test/:/usr/myapp test
I have a Dockerfile like below:
# syntax=docker/dockerfile:1
FROM continuumio/miniconda3
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
# Create directory to store our application
WORKDIR /app
## The following three commands adapted from Dockerfile snippet at
## https://docs.docker.com/develop/develop-images/build_enhancements/#using-ssh-to-access-private-data-in-builds
# Install ssh client and git
RUN apt-get upgrade && apt-get update && apt-get install openssh-client git -y
# Download public key for gitlab.com
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
# clone my-repo, authenticating using client's ssh-agent.
RUN --mount=type=ssh git clone git#gitlab.com:mycompany/data-products/my-repo.git /app/
# set up python (conda) environment to run application
RUN conda create --name recenv --file conda-linux-64.lock
# run my-package with the conda environment we just created.
CMD ["conda", "run", "-n", "recenv", "python", "-m", "my_package.train" "path/to/gcp/service/account.json"]
This dockerfile builds successfully with docker build . --no-cache --tag my-package --ssh default but fails (as expected) on docker run my-package:latest with:
FileNotFoundError: [Errno 2] No such file or directory: path/to/gcp/service/account.json
So I've gotten the ssh secrets management working so the RUN ...git clone step uses my ssh/rsa creds successfully. But I'm having trouble using my other secret - my gcp service account json file. The difference is I only need the ssh secret in a RUN step but I need my gcp service account secret in my CMD step.
While everything I've read, such as docker docs page on using the --secret flag, tutorials and SO answers I've found, all reference how to pass in a secret to be used in a RUN step but not the CMD step. But I need to pass my GCP service account json file to my CMD step.
I could just COPY the file into my container, but from my reading that's supposedly not a great solution from a security standpoint.
What is the recommended, secure way of passing a secret json file to the CMD step of a docker container?
When attempting to build a GoLang-based Docker Image, the Docker executor runs into the following error:
. . .go: $GIT_REPO#v1.9.11: reading $GIT_REPO/go.mod at revision v0.0.07: unknown revision v0.0.07
at the following RUN instruction from the Dockerfile used:
RUN go build . . .
where GIT_REPO represents the private repo. full path, including owner and name.
The Docker executor encounters this error with go1.13.x and higher; the Docker executor does not encounter this error with go1.12.x.
The vendor dir. contains all required packages. Tags are confirmed to be present.
Proper SSH keys were even added to the private Go common repo. with successful
git clone . . .
commands outside of building Docker images, but still encountering the same error above.
EDIT:
Verify your remote repo in bitbucket.org actually has the v0.0.7 tag you're trying to build against.
While a local build may work if the git tag exists locally - a docker build will pull from the remote source and fail with an error like go.mod at revision v0.0.7: unknown revision v0.0.7 - if the tag does not exist remotely.
To push your local tags to the remote repo:
git push --tags
For more granular tag operations see.
Docker builds by default can only access public repos. Since you need access to a private repo, you need to include a read-ssh key to the Docker build process (keys should never be checked into the repo!).
It is critically important, however, you do this in a multi-stage build, so you do not include your SSH keys in the final image.
This blog post walks through all the steps. But to include a working example:
To build the docker image:
SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)" \
docker build -t "myapp:v0.0.1" --build-arg SSH_PRIVATE_KEY .
And the Dockerfile using a bitbucket.org private repo site:
FROM golang:1.14.6 AS build
WORKDIR /bld
COPY *.go go.mod go.sum ./
ARG SSH_PRIVATE_KEY
# ***NEVER*** DO THIS IN A SINGLE-STAGE DOCKER BUILD (see below)
RUN \
mkdir -p ~/.ssh && \
umask 0077 && \
echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa && \
git config --global url."git#bitbucket.org:".insteadOf https://bitbucket.org/ && \
ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
RUN \
go get && \
CGO_ENABLED=0 go build -o app
# final stage of multi-stage: will appropriately *NOT* include SSH keys
FROM scratch
COPY --from=build \
/etc/ssl /etc/ssl
COPY --from=build \
/bld/app /app/myapp
CMD ["/app/myapp"]
There are two problems to solve here:
1. How to allow Docker to access local SSH keys, safely?
2. How to tell Go not to use public registry to fetch private packages?
Short Answers
Since Docker v18.09, there's a built-in solution to handle SSH authentication during the build phase (more). It's also much easier and safer, compared to passing build arguments, and eliminates the need for a multi-stage Docker build.
Go has a GOPRIVATE environment variable to identify private packages. (more)
Long Answer
Step-by-step:
1. Make sure ssh-agent is setup and knows the SSH key
Github has a quick guide on this subject, explaining the process for different operating systems. See Generating a new SSH key and adding it to SSH agent.
2. Enable BuildKit for Docker
Without BuildKit, docker build won't recognize --ssh option.
From the Docker reference:
Easiest way from a fresh install of docker is to set the
DOCKER_BUILDKIT=1 environment variable when invoking the docker build
command, such as:
$ DOCKER_BUILDKIT=1 docker build .
To enable docker BuildKit by
default, set daemon configuration in /etc/docker/daemon.json feature
to true and restart the daemon:
{ "features": { "buildkit": true } }
Docker Desktop users can manage daemon configurations via Preferences > Docker Engine.
4. Update Dockerfile
4.1. Make sure Git uses SSH instead of HTTPs
Go tends to fetch public packages via HTTPs. You can adjust this behavior by updating git configurations:
RUN git config --global url.git#github.com:.insteadOf https://github.com/
You should probably do this on your local machine as well.
4.2. Request SSH access where it's required
Each RUN command the needs SSH access should be mounted with type=ssh. For
Example:
RUN --mount=type=ssh git clone ...
4.3. Make sure Go knows your private packages
Update GOPRIVATE variable:
RUN go env -w GOPRIVATE="github.com/your-org/private-repo"
Putting all of it together, in following sample of a Dockerfile:
FROM golang:1.16.3-alpine3.13
RUN apk update
RUN apk add git openssh
RUN mkdir /app
ADD . /app
WORKDIR /app
# You can replace github.com with any other Git host
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# Make sure git uses SSH to fetch packages, not HTTPs
RUN git config --global url.git#github.com:.insteadOf https://github.com/
# Make Go knows which packages are private.
RUN go env -w GOPRIVATE="github.com/your-org/private-repo"
# GOPRIVATE is a comma separated list glob-patterns.
# You can use a wildcard to match every repo in an organization:
# e.g.: GOPRIVATE="github.com/your-org/*"
# Mount the build command with type `ssh`.
RUN --mount=type=ssh go get && go build -o main .
CMD ["/app/main"]
6. Build the image with --ssh option:
Having BuildKit enabled by default:
$ docker build --ssh default -t my-app:latest .
I have specified a git clone command in the docker file as below.
RUN git clone https://github.com/zhaoyi0113/test.git
but I got this error when build the docker image:
Cloning into 'test'...
fatal: could not read Username for 'https://github.com': No such device or address
I wonder why it doesn't work. I am able to run this command on my host. Is there anything different if I list it on docker file?
You can pass credentials as arguments to container. This should work
FROM alpine:3.8
RUN apk update && apk upgrade && \
apk add --no-cache bash git openssh
ARG username
ARG password
RUN git clone https://${username}:${password}#github.com/username/repository.git
ENTRYPOINT ["sleep 10"]
but it might be unsafe if you want to distribute that image
then build
docker build \
--no-cache \
-t git-app:latest \
--build-arg username=user \
--build-arg password=qwerty \
.
I know what wrong with this problem. The issue is because the repo I am cloning is a private repo which means it requires credential to connect to github. Fixing it by passing credentials to the container.
In docker I want to do this:
git clone XYZ
cd XYZ
make XYZ
However because there is no cd command, I have to pass in the full path everytime (make XYZ /fullpath). Any good solutions for this?
To change into another directory use WORKDIR. All the RUN, CMD and ENTRYPOINT commands after WORKDIR will be executed from that directory.
RUN git clone XYZ
WORKDIR "/XYZ"
RUN make
You can run a script, or a more complex parameter to the RUN. Here is an example from a Dockerfile I've downloaded to look at previously:
RUN cd /opt && unzip treeio.zip && mv treeio-master treeio && \
rm -f treeio.zip && cd treeio && pip install -r requirements.pip
Because of the use of '&&', it will only get to the final 'pip install' command if all the previous commands have succeeded.
In fact, since every RUN creates a new commit & (currently) an AUFS layer, if you have too many commands in the Dockerfile, you will use up the limits, so merging the RUNs (when the file is stable) can be a very useful thing to do.
I was wondering if two times WORKDIR will work or not, but it worked :)
FROM ubuntu:18.04
RUN apt-get update && \
apt-get install -y python3.6
WORKDIR /usr/src
COPY ./ ./
WORKDIR /usr/src/src
CMD ["python3", "app.py"]
You can use single RUN command for all of them
RUN git clone XYZ && \
cd XYZ && \
make XYZ
In case you want to change the working directory for the container when you run a docker image, you can use the -w (short for --workdir) option:
docker run -it -w /some/valid/directory/inside/docker {image-name}
Ref:
docker run options: https://docs.docker.com/engine/reference/commandline/run/#options
Mind that if you must run in bash shell, you need not just the RUN make, but you need to call the bash shell before, since in Docker, you are in the sh shell by default.
Taken from /bin/sh: 1: gvm: not found, which would say more or less:
Your shell is /bin/sh, but source expects /bin/bash, perhaps because it
puts its initialization in ~/.bashrc.
In other words, this problem can occur in any setting where the "sh" shell is used instead of the "bash", causing "/bin/sh: 1: MY_COMMAND: not found".
In the Dockerfile case, use the recommended
RUN /bin/bash -c 'source /opt/ros/melodic/setup.bash'
or with the "[]" (which I would rather not use):
RUN ["/bin/bash", "-c", "source /opt/ros/melodic/setup.bash"]
Every new RUN of a bash is isolated, "starting at 0". For example, mind that setting WORKDIR /MY_PROJECT before the bash commands in the Dockerfile does not affect the bash commands since the starting folder would have to be set in the ".bashrc" again. It needs cd /MY_PROJECT even if you have set WORKDIR.
Side-note: do not forget the first "/" before "opt/../...". Else, it will throw the error:
/bin/bash: opt/ros/melodic/setup.bash: No such file or directory
Works:
=> [stage-2 18/21] RUN ["/bin/bash", "-c", "source /opt/ros/melodic/setup.bash"] 0.5s
=> [stage-2 19/21] [...]
See “/bin/sh: 1: MY_COMMAND: not found” at SuperUser for some more details on how this looks with many lines, or how you would fill the ".bashrc" instead. But that goes a bit beyond the actual question here.
PS: You might also put the commands you want to execute in a single bash script and run that bash script in the Dockerfile (though I would rather put the bash commands in the Dockerfile as well, just my opinion):
#!/bin/bash
set -e
source /opt/ros/melodic/setup.bash