We are trying to set up an OpenShift flow as close to the defaults as possible, without having to use Jenkins.
So far I have managed to adapt a s2i-generated Dockerfile to our needs (to get the benefits of the s2i images) and set up a simple "oc new-app" pipeline which creates a build following a git repository pushing the generated docker image to an image stream and a deployment triggered by the image stream update pushing to a default pod. This works well.
Now we want to additionally have this docker image pushed to an external docker repository (the RED HAT CONTAINER REGISTRY known by OpenShift) and I would essentially like to keep the existing flow instead of adapting it to use the external registry instead of the internal.
So, can I set up a "build" or something like it which does nothing but take that image and push it to the docker registry? How do I set it up using the oc command?
Would you like to push the image builded by buildConfig to external registry, right ?
I think the work flow is like this:Using Docker Credentials for Private Registries.
Create credential secret
$ oc create secret generic <secret_name> \
--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
--type=kubernetes.io/dockerconfigjson
Link the secret with builder ServiceAccount.
Configure the build image push target in the buildConfig as follows.
spec:
output:
to:
kind: DockerImage
name: external.registry.io/subpath/imagename:latest
pushSecret:
name: <secret_name>
I hope it help you.
You can use the skopeo command-line tool to copy container images from one docker registry to another without needing to build the image a second time.
This is an officially supported tool by Red Hat that is part of the new suite of tools (buildah, podman, skopeo, etc) to support running OpenShift and Kubernetes on top of the daemonless cri-o container runtime instead of docker.
Either log in to the source and destination registries via docker login or use the --src-creds and --dest-creds flags.
skopeo copy \
--src-creds=testuser:testpassword \
--dest-creds=otheruser:otherpassword \
docker://myregistrydomain.com:5000/dockerimage:tag \
docker://mysecondregistrydomain.com:5000/dockerimage:tag
Related
I was wondering is it possible to use cached docker images in gitlab registry for gitlab-ci?
for example, I want to use node:16.3.0-alpine docker image, can I cache it in my gitlab registry and pull it from that and speed up my gitlab ci instead of pulling it from docker hub?
Yes, GitLab's dependency proxy features allow you to configure GitLab as a "pull through cache". This is also beneficial for working around rate limits of upstream sources like dockerhub.
It should be faster in most cases to use the dependency proxy, but not necessarily so. It's possible that dockerhub can be more performant than a small self-hosted server, for example. GitLab runners are also remote with respect to the registry and not necessarily any "closer" to the GitLab registry than any other registry over the internet. So, keep that in mind.
As a side note, the absolute fastest way to retrieve cached images is to self-host your GitLab runners and hold images directly on the host. That way, when jobs start, if the image already exists on the host, the job will start immediately because it does not need to pull the image (depending on your pull configuration). (that is, assuming you're using images in the image: declaration for your job)
I'm using a corporate Gitlab instance where for some reason the Dependency Proxy feature has been disabled. The other option you have is to create a new Docker image on your local machine, then push it into the Container Registry of your personal Gitlab project.
# First create a one-line Dockerfile containing "FROM node:16.3.0-alpine"
docker pull node:16.3.0-alpine
docker build . -t registry.example.com/group/project/image
docker login registry.example.com -u <username> -p <token>
docker push registry.example.com/group/project/image
where the image tag should be constructed based on the example given on your project's private Container Registry page.
Now in your CI job, you just change image: node:16.3.0-alpine to image: registry.example.com/group/project/image. You may have to run the docker login command (using a deploy token for credentials, see Settings -> Repository) in the before_script section -- I think maybe newer versions of Gitlab will have the runner authenticate to the private Container Registry using system credentials, but that could vary depending on how it's configured.
TL;DR Must you put your private Docker URI in your Dockerfile FROM command if the parent image is in a private registry?
I thought this should be an easily answerable question, but cannot find a good set of Google keywords...
Detail:
I have three repos, all built with separate CI calls; the order of these CI executions is correct for this DAG (i.e. parent image will be available when a child needs it):
Repo 1 holds a Dockerfile that constructs a base image with dependencies. It is slow moving
Repo 2 & 3 hold applications that are built in a Dockerfile that pulls FROM the image built in Repo 1. These change frequently.
As I understand it, if you don't specify a repo URI in your FROM command, docker assumes you are pulling from DockerHub. Is this correct?
If these images are stored in a private registry, is it true that the private registry must be explicitly included in the child Dockerfiles? Is there another way?
Docker looks for the image name, on your local repository, if it does not find the image there, it would pull from the docker hub.
At first thought, it feels intuitive that we should be able to configure a private repository as the default, and then use it just as we would use docker hub.
However, this seems to be a topic of lengthy discussion, you can follow the discussion here.
Unfortunately, at the time of writing this, to build from a private repo, you will need to specify the complete URI in your Dockerfile:
FROM <aws_account_id>.dkr.ecr.<aws region>.amazonaws.com/<private image name>:latest
You will need to configure your Docker daemon to authenticate and pull from your private ECR repository:
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
and then go for
docker build .
Alternatively, you can use arguments, to construct your ECR URI. This will keep things clean and parameterised in your Dockerfile while being explicit that you are using a private repo.
Eg:
In your Dockerfile
ARG PRIVATE_REPO
FROM ${PRIVATE_REPO}your_image_name
And build the docker image with:
docker build . --build-arg PRIVATE_REPO=aws_account_id.dkr.ecr.region.amazonaws.com/
Your basic assumptions are correct: if your base image is in a private registry, you generally must include the registry name in downstream Dockerfiles' FROM lines, docker run commands, Compose and Kubernetes image: lines, and so on. If the registry part of the image name is absent, it defaults to docker.io.
# Expands to docker.io/library/alpine:latest
FROM alpine
This only actually matters if the image isn't already present on the local machine, in which case Docker will automatically pull it. If it's already present, Docker doesn't worry about where it came from and just uses it as-is. If you have a manual multi-stage build, but don't actually push any of the intermediate stages anywhere, it doesn't actually matter if you tag it for any particular repository.
# "step 2" Dockerfile
# Build with: `docker build -t step2 -f step2.Dockerfile .`
# Starts from image in internal repository
FROM registry.example.com/img/step1
# Application Dockerfile
# Depends on "step 2" image being built first
# Since we expect step2:latest image to be present locally, won't pull
FROM step2
I am using ubuntu 20.04
while following book -> https://www.oreilly.com/library/view/kubeflow-for-machine/9781492050117/
on page 17, it says the following (only relevant parts) which I don't understand....
You will want to store container images called a
container registry. The container registry will be accessed by your Kubeflow cluster.
I am going to use docker hub as container registry. Next
we'll assume that you've set your container registry via an environment variable
$CONTAINER_REGISTRY, in your shell" NOTE: If you use registry that isn't on Google Cloud
Platform, you will need to configure Kubeflow pipelines container builder to have access to
your registry by following the Kaniko configuration guide -> https://oreil.ly/88Ep-
First, I do not understand how to set container registry through environment variable, am I supposed to give it a link??
Second, I've gone into Kaniko config guide and did everything as told -> creating config.json with "auth":"mypassword for dockerhub". After that In the book it says:
To make sure your docker installation is properly configured, you can write one line Dc and
push it to your registry."
Example 2.7 Specify the new container is built on top of Kubeflow's
container
FROM gcr.io/kubeflow-images-public/tensorflow-2.1.0-notebook-cpu:1.0.0
Example 2.8 Build new container and push to registry for use
IMAGE="${CONTAINER_REGISTRY}/kubeflow/test:v1"
docker build -t "${IMAGE}" -f Dockerfile . docker push "${IMAGE}"
I've created Dockerfile with code from Example2.7 inside it, then ran code from Example 2.8 however not working.
Make sure that:
You set the environment variable using export CONTAINER_REGISTRY=docker.io/your_username in your terminal (or in your ~/.bash_profile and run source ~/.bash_profile).
Your .docker/config.json does not have your password in plain text but in base64, for example the output of echo -n 'username:password' | base64
The docker build and the docker push are two separate commands, in your example they're seen as one command, unlike the book.
I have a situation where I need to wait for an image to show up on a given docker registry (this is an OpenShift external registry) before continuing my script with additional oc-commands.
Before that happens, the external docker registry has no knowledge what so ever of this image. Afterwards it is available as :latest.
Can this be done programatically? (Preferably without trying to download the image)
My order of preference:
oc command
docker command
A REST api (using oc or docker credentials)
assuming the repository from openshift works similarly to dockerhub, you can do this:
curl --silent -f -lSL https://index.docker.io/v1/repositories/$1/tags/$2 > /dev/null
source
I need to pull all images from an openshift template file, in my case it's openwhisk.
I'm trying to deploy this project on a private network so I don't have access to docker's official repository from there and thus need to push the images myself.
I was hoping there is a script/tool to automate this process.
There is no such available tool/script but you can write small shell script to do it.
If public dockerhub registry not allowed then either use private separate registry
or
Pull the image in your local laptop then tag it and push to openshift registry.
After pushing all the image to openshift, import your openshift template to deploy your application.
Below is the steps for single image. you can define list of image and loop it over the list.
docker pull imagename
oc login https://127.0.0.1:8443 --token=<hidden_token> #copy from https://your_openshift_server:port/console/command-line
oc project test
oc create imagestream imagename
docker login -u `oc whoami` -p `oc whoami --show-token` your_openshift_server:port
docker tag imagename your_openshift_server:port/openshift_projectname/imagename:tag
docker push your_openshift_server:port/openshift_projectname/imagename:tag
you can get more details on page suggested by graham-dumpleton
.
Graham Dumpleton's book talks about this. You create a list (JSON) of all the images used and import that into the openshift namespace. Since your OpenShift is offline/disconnected, you'll also change any remote registry to the URL of the internal, hosted registry.
Example that imports all JBoss images: https://github.com/projectatomic/adb-utils/blob/master/services/openshift/templates/common/jboss-image-streams.json