How to supply multiple ImagePullSecrets to an s2i build in OpenShift - docker

I'm trying to build an OCI image using the s2i mechanism of OpenShift.
The Dockerfile is multi-stage, both base images are hosted inside two different, only privately accessible docker repositories.
My BuildConfig looks like this:
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
name: ${NAME}
spec:
failedBuildsHistoryLimit: 5
nodeSelector: { }
output:
pushSecret:
name: ${PUSH_SECRET}
to:
kind: DockerImage
name: >-
${IMAGE_NAME}:${IMAGE_VERSION}
postCommit: { }
resources: { }
runPolicy: Serial
source:
binary: { }
type: Binary
strategy:
dockerStrategy:
pullSecret:
name: ${PULL_SECRET}
type: Docker
successfulBuildsHistoryLimit: 5
triggers: [ ]
I know I can supply the ImagePullSecret to use with the .spec.strategy.dockerStrategy.pullSecret.name field. But how can I use multiple pull secrets, i.e. enable the build to pull from two separate private docker registries?
Creating the ImagePullSecrets in OpenShift and then assigning them to the builder ServiceAccount did not work, as s2i seems to ignore this configuration and only use the secrets listed in the BuildConfig.

The field : spec.strategy.dockerStrategy.pullSecret must contains only one pullSecret.
pullSecret: LocalObjectReference: The name of a Secret that would be used for setting up the authentication for pulling the container images from the private Docker registries.
Make sure you have define all registry server name/auth inside your pull secret, for example:
// dockerconfig.json
{
"auths":{
"registry.yourdomain.com":{"auth":"<hash>","email":"<email_address>"},
"cloud.openshift.com":{"auth":"<hash>","email":"<email_address>"},
"quay.io/repository-main":{"auth":"<hash>","email":"<email_address>"}
}
}
// Create pull secret from dockerconfig file
$ oc create secret generic <pull_secret_name> \
--from-file=.dockerconfigjson=<path/to/dockerconfig.json> \
--type=kubernetes.io/dockerconfigjson
// Add the secret to your service account
$ oc secrets link builder <pull_secret_name> --for=pull
When configuring the BuildConfig, the PullSecret can be overridden to manually choose the ImagePullSecret used by the OpenShift Build when there are multiple to choose from in the Builder ServiceAccount.

You can specify multiple auths inside a single pull secret.

Related

Is there a way to configure docker hub pro user in kubernetes?

We've just bought a docker hub pro user so that we don't have to worry about pull rate limits.
Now, I'm currently having a problem trying to to set the docker hub pro user. Is there a way to set the credentials for hub.docker.com globally?
In the kubernetes docs I found following article: Kubernetes | Configure nodes for private registry
On every node I executed a docker login with the credentials, copied the config.json to /var/lib/kubelet and restarted kubelet. But I'm still getting an ErrImagePull because of those rate limits.
I've copied the config.json to the following places:
/var/lib/kubelet/config.json
/var/lib/kubelet/.dockercfg
/root/.docker/config.json
/.docker/config.json
There is an option to use a secret for authentification. The problem is, that we would need to edit hundreds of statefulsets, deployments and deamonsets. So it would be great to set the docker user globally.
Here's the config.json:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "[redacted]"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.13 (linux)"
}
}
To check if it actually logs in with the user I've created an access token in my account. There I can see the last login with said token. The last login was when I executed the docker login command. So the images that I try to pull aren't using those credentials.
Any ideas?
Thank you!
Kubernetes implements this using image pull secrets. This doc does a better job at walking through the process.
Using the Docker config.json:
kubectl create secret generic regcred \
--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
--type=kubernetes.io/dockerconfigjson
Or you can pass the settings directly:
kubectl create secret docker-registry <name> --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
Then use those secrets in your pod definitions:
apiVersion: v1
kind: Pod
metadata:
name: foo
namespace: awesomeapps
spec:
containers:
- name: foo
image: janedoe/awesomeapp:v1
imagePullSecrets:
- name: myregistrykey
Or to use the secret at a user level (Add image pull secret to service account)
kubectl get serviceaccounts default -o yaml > ./sa.yaml
open the sa.yaml file, delete line with key resourceVersion, add lines with imagePullSecrets: and save.
kind: ServiceAccount
metadata:
creationTimestamp: "2020-11-22T21:41:53Z"
name: default
namespace: default
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: afad07eb-f58e-4012-9ccf-0ac9762981d5
secrets:
- name: default-token-gkmp7
imagePullSecrets:
- name: regcred
Finally replace the serviceaccount with the new updated sa.yaml file
kubectl replace serviceaccount default -f ./sa.yaml
We use docker-registry as a proxy cache in our Kubernetes clusters, Docker Hub credentials may be set in the configuration. Docker daemons on Kubernetes nodes are configured to use the proxy by setting registry-mirror in /etc/docker/daemon.json.
This way, you do not need to modify any Kubernetes manifest to include pull secrets. Our complete setup is described in a blog post.
I ran into the same problem as OP. It turns out, putting docker credential files for kubelet works for kubernetes version 1.18 or higher. I have tested here and can confirm that kubelet 1.18 picks up the config.json placed in /var/lib/kubelet correctly and authenticates the docker registry.

How to deploy private organization image from docker hub to kubernetes

Current Situation
I have a Kubernetes cluster created on DigitalOcean. I want to deploy a Docker image that is hosted in a private that in turn belongs to an organization in Docker Hub.
Docker Hub organization name (sample): myorg
Docker Hub repository name (sample): myorg/mo-server
So in order to push a new image I use docker push myorg/mo-server
(Note: The example above contains a dash (-) in the name of the image which I have in the real name as well)
Problem
When I try to deploy that docker image to kubernetes using kubectl the deployment always ends up in status ErrImagePull. Error message:
ailed to pull image "index.docker.io/myorg/mo-server": rpc error: code = Unknown desc = Error response from daemon: pull access denied for myorg/mo-server, repository does not exist or may require 'docker login'
What I tried so far
Because it is a private repository I'm creating a secret beforehand. For this, I'm using the username and E-Mail of myself.
set DOCKER_REGISTRY_SERVER=https://index.docker.io/v1/
set DOCKER_USER=sarensw
set DOCKER_EMAIL=stephan#myorg.com
set DOCKER_PASSWORD=...
The credentials are the same as when I use docker login. Then I create a secret using:
kubectl create secret docker-registry regcred
--docker-server=%DOCKER_REGISTRY_SERVER%
--docker-username=%DOCKER_USER%
--docker-password=%DOCKER_PASSWORD%
--docker-email=%DOCKER_EMAIL%
Then, I use kubectl create to create a new deployment.
kubectl create -f ci\helper\kub-deploy-staging.yaml
kub-deploy-staging.yaml looks as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mo-server
spec:
replicas: 1
selector:
matchLabels:
app: mo-server
template:
metadata:
labels:
app: mo-server
spec:
containers:
- name: mo-server
image: index.docker.io/myorg/mo-server
imagePullPolicy: Always
command: [ "echo", "SUCCESS" ]
imagePullSecrets:
- name: regcred
The result is ErrImagePull as described above.
I'm pretty sure that the image: index.docker.io/myorg/mo-server is the culprit because it is an organization image that I try to use with a normal account. And all the tutorials for accessing a private image do not take organizations into account.
So what am I doing wrong?
(one of many similar) references: https://gist.github.com/rkuzsma/b9a0e342c56479f5e58d654b1341f01e
I suspect this happens because of the docker registry variable with which you have created your secret, please try substituting index.docker.io with registry.hub.docker.com as this is the official dockerhub registry URL. If you are using Google cloud you can also try docker.io
As I see you are trying to set your variable with "set" command, and please try now with "export" as in mentioned in gist file that you follow
export DOCKER_REGISTRY_SERVER=https://index.docker.io/v1/
export DOCKER_USER=Type your dockerhub username, same as when you `docker login`
export DOCKER_EMAIL=Type your dockerhub email, same as when you `docker login`
export DOCKER_PASSWORD=Type your dockerhub pw, same as when you `docker login`
kubectl create secret docker-registry myregistrykey \
--docker-server=$DOCKER_REGISTRY_SERVER \
--docker-username=$DOCKER_USER \
--docker-password=$DOCKER_PASSWORD \
--docker-email=$DOCKER_EMAIL
Then try again and let us know the result please.

Usage of Kubectl command and deployment of pods using Kubernetes and Jenkins

I am trying to implement CI/CD pipeline for my spring boot microservice deployment. Here I have some sample microservices. When I am exploring about Kubernetes , I found that pods, services, replica sets/ controller, statefulsets etc. I understood those Kubernetes terminologies properly. And I am planning to use Docker hub for my image registry.
My Requirement
When there is a commit made to my SVN code repository, then the Jenkins need to pull code from Subversion repository and need to build the project , create docker image, push into Docker hub - as mentioned earlier. And after that need to deploy into my test environment from Dockerhub by pulling by Jenkins.
My Confusion
When am I creating services and pods, how I can define the docker image path within pod/services / statefulsets? Since it pulling from Docker hub for deployment.
Can I directly add kubectl command within Jenkins pipeline schedule job? How can I use kubectl command for Kubernetes deployment?
Jenkins can do anything you can do given that the tools are installed and accessible. So an easy solution is to install docker and kubectl on Jenkins and provide him with the correct kube config so he can access the cluster. So if your host can use kubectl you can have a look at the $HOME/.kube/config file.
So in your job you can just use kubectl like you do from your host.
Regarding the images from Docker Hub:
Docker Hub is the default Docker Registry for Docker anyway so normally there is no need to change anything in your cluster only if you want to use your own Private Hosted Registry. If you are running your cluster at any cloud provider I would use there Docker registries because they are better integrated.
So this part of a deployment will pull nginx from Docker Hub no need to specify anything special for it:
spec:
containers:
- name: nginx
Image: nginx:1.7.9
So ensure Jenkins can do the following things from command line:
build Docker images
Push Docker Images (make sure you called docker login on Jenkins)
Access your cluster via kubectl get pods
So an easy pipeline needs to simply do this steps:
trigger on SVN change
checkout code
create a unique version which could be Build number, SVN Revision, Date)
Build / Test
Build Docker Image
tag Docker Image with unique version
push Docker Image
change image line in Kubernetes deployment.yaml to newly build version (if your are using Jenkins Pipeline you can use readYaml, writeYaml to achive this)
call kubectl apply -f deployment.yaml
Depending on your build system and languages used there are some useful tools which can help building and pushing the Docker Image and ensuring a unique tag. For example for Java and Maven you can use Maven CI Friendly Versions with any maven docker plugin or jib.
To create deployment you need to create a yaml file.
In the yaml file you the row:
image: oronboni/serviceb
Leads you to the container that in this case in DockerHub:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: serviceb
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: serviceb
template:
metadata:
labels:
app: serviceb
spec:
containers:
- name: serviceb
image: oronboni/serviceb
ports:
- containerPort: 5002
I strongly suggest that you will see the kubernetes deployment webinar in the link below:
https://m.youtube.com/watch?v=_vHTaIJm9uY
Good luck.

Does Kubernetes kubelet support Docker Credential Stores for private registries?

Docker has a mechanism for retrieving Docker registry passwords from a remote store, instead of just storing them in a config file - this mechanism is called a Credentials Store. It has a similar mechanism that are used to retrieve a password for a specific registry called Credential Helpers.
Basically, it involves defining a value in ~/.docker/config.json that is interpreted as the name of an executable.
{
"credsStore": "osxkeychain"
}
The value of the credsStore key has a prefix docker-credential- pre-pended to it and if that executable (e.g. docker-credential-osxkeychain) exists on the path then it will be executed and is expected to echo the username and password to stdout, which Docker will use to log in to a private registry. The idea is that the executable reaches out to a store and retrieves your password for you, so you don't have to have lots of files laying around in your cluster with your username/password encoded in them.
I can't get a Kubernetes kubelet to make use of this credential store. It seems to just ignore it and when Kubernetes attempts to download from a private registry I get a "no basic auth credentials" error. If I just have a config.json with the username / password in it then kubelet works ok.
Does Kubernetes support Docker credential stores/credential helpers and if so, how do I get them to work?
For reference, kubelet is running through systemd, the credential store executable is on the path and the config.json file is being read.
As of the moment of writing Kubernetes v1.14 does not support credential helpers as per official docs Configuring Nodes to Authenticate to a Private Registry
Note: Kubernetes as of now only supports the auths and HttpHeaders section of docker config. This means credential helpers (credHelpers or credsStore) are not supported.
Yes, Kubernetes has the same mechanism called secrets but with extended functionality, and it includes specific secret type called docker-registry. You can create your specific secret with credentials for docker registry:
$ kubectl create secret docker-registry myregistrykey \
--docker-server=DOCKER_REGISTRY_SERVER \
--docker-username=DOCKER_USER \
--docker-password=DOCKER_PASSWORD \
--docker-email=DOCKER_EMAIL
secret "myregistrykey" created.
and use it:
apiVersion: v1
kind: Pod
metadata:
name: foo
namespace: awesomeapps
spec:
containers:
- name: foo
image: janedoe/awesomeapp:v1
imagePullSecrets:
- name: myregistrykey

Openshift imagestream "Import failed (Unauthorized)" for private external secure registry

May be I'm not getting something right, but my ImageStream returnes "! error: Import failed (Unauthorized): you may not have access to the Docker image "my_registry:5000/project/my_image:latest"".
I have set up all needed steps to connect to external registry (created secret and added it to current projects's serviceaccount/default and serviceaccount/builder accounts). All deploymentconfigs with specified image: my_registry:5000/project/my_image:latest are working great, node can successfully pull the image and create a pod.
But when I am making image stream with:
from:
kind: DockerImage
name: my_registry:5000/project/my_image:latest
I get error that I am not authorised.
So what am i doing wrong? Is there any additional account I should give rights for pull?
oc describe sa/builder
Name: builder
Namespace: nginx
Labels: <none>
Image pull secrets: builder-dockercfg-8ogvt
my_registry
Mountable secrets: builder-token-v6w8q
builder-dockercfg-8ogvt
my_registry
Tokens: builder-token-0j8p5
builder-token-v6w8q
and
oc describe sa/default
Name: default
Namespace: nginx
Labels: <none>
Image pull secrets: default-dockercfg-wmm1h
my_registry
Mountable secrets: default-token-st7k9
default-dockercfg-wmm1h
Tokens: default-token-m2aoq
default-token-st7k9
The solution depends upon your particular infrastructure configuration, but here are some pointers which worked for me -
Assuming your private external registry has Certificates, please check if those certificates are properly imported, if thats not the case, then please add the registry as insecure.
Docker pull, build config, imagestream pull - all work in different manner.
Also it is recommended that pull secret name should be same as hostname of registry authentication endpoint. (If not using insecure registry).
For ex. Registry FQDN Name:5000/yourapp:latest (Certificates need this to work properly).
Please take a look here
oc secrets link default <pull_secret_name> --for=pull
I ran into the same problem when I was trying to import an image from a docker registry hosted in another Openshift cluster. After some debugging I found the problem: Unable to find a secret to match https://docker-dev.xxxx.com:443/openshift/token (docker-dev.xxxx.com:443/openshift/token)
The Openshift Docker registry is using the OAuth of Openshift. So you have to create a secret where the --docker-server is pointing to the /openshift/token endpoint. eg:
oc secrets new-dockercfg registry.example.com \
--docker-server=https://registry.example.com:443/openshift/token \
--docker-username=default/puller-sa \
--docker-password=<token> \
--docker-email=someone#example.com

Resources