I am failing to pull from my private Docker Hub repository into my local Kubernetes setup running on Vagrant:
Container "hellonode" in pod "hellonode-n1hox" is waiting to start: image can't be
pulled
Failed to pull image "username/hellonode": Error: image username/hellonode:latest not found
I have set up Kubernetes locally via Vagrant as described here and created a secret named "dockerhub" with kubectl create secret docker-registry dockerhub --docker-server=https://registry.hub.docker.com/ --docker-username=username --docker-password=... --docker-email=... which I supplied as the image pull secret.
I am running Kubernetes 1.2.0.
To pull a private DockerHub hosted image from a Kubernetes YAML:
Run these commands:
DOCKER_REGISTRY_SERVER=docker.io
DOCKER_USER=Type your dockerhub username, same as when you `docker login`
DOCKER_EMAIL=Type your dockerhub email, same as when you `docker login`
DOCKER_PASSWORD=Type your dockerhub pw, same as when you `docker login`
kubectl create secret docker-registry myregistrykey \
--docker-server=$DOCKER_REGISTRY_SERVER \
--docker-username=$DOCKER_USER \
--docker-password=$DOCKER_PASSWORD \
--docker-email=$DOCKER_EMAIL
If your username on DockerHub is DOCKER_USER, and your private repo is called PRIVATE_REPO_NAME, and the image you want to pull is tagged as latest, create this example.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: whatever
spec:
containers:
- name: whatever
image: DOCKER_USER/PRIVATE_REPO_NAME:latest
imagePullPolicy: Always
command: [ "echo", "SUCCESS" ]
imagePullSecrets:
- name: myregistrykey
Then run:
kubectl create -f example.yaml
Create k8 Secret:
apiVersion: v1
kind: Secret
metadata:
name: repositorySecretKey
data:
.dockerconfigjson: <base64 encoded docker auth config>
type: kubernetes.io/dockerconfigjson
Then in pod or rc config mention the secret. Example :
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: test-pod
image: quay.io/example/hello:1.1
imagePullSecrets:
- name: repositorySecretKey
Docker auth config
{
"https://quay.io": {
"email": ".",
"auth": "<base64 encoded auth token>"
}
}
Or
kubectl create secret docker-registry myregistrykey \
--docker-server=DOCKER_REGISTRY_SERVER \
--docker-username=DOCKER_USER \
--docker-password=DOCKER_PASSWORD \
--docker-email=DOCKER_EMAIL
I solved using the following Kubectl command :
kubectl create secret docker-registry your-key-name\
--docker-server=docker.io \
--docker-username=DOCKER_USER \
--docker-password=DOCKER_PASSWORD \
--docker-email=DOCKER_EMAIL
You can follow these instructions on how to configure nodes to authenticate to a private repository in order to configure the kubelets to make Docker use your credentials, or follow +Phagun Baya's solution with imagePullSecrets that applies to pods.
Just in case anyone else is stuck using kubectl from Windows -
set secretname="secret1"
set username="dockerhubUsername"
set pw="dockerhubPassword"
set email="dockerhubEmail#domain.com"
kubectl create secret docker-registry %secretname% --docker-username=%username% --docker-password=%pw% --docker-email=%email%
Related
Is it possible to replace the usage of Google Cloud Storage buckets with an alternative on-premises solution so that it is possible to run e.g. Kubeflow Pipelines completely independent from the Google Cloud Platform?
Yes it is possible. You can use minio, it's like s3/gs but it runs on a persistent volume of your on-premises storage.
Here are the instructions on how to use it as a kfserving inference storage:
Validate that minio is running in your kubeflow installation:
$ kubectl get svc -n kubeflow |grep minio
minio-service ClusterIP 10.101.143.255 <none> 9000/TCP 81d
Enable a tunnel for your minio:
$ kubectl port-forward svc/minio-service -n kubeflow 9000:9000
Forwarding from 127.0.0.1:9000 -> 9000
Forwarding from [::1]:9000 -> 9000
Browse http://localhost:9000 to get to the minio UI and create a bucket/upload your model. Credentials minio/minio123. Alternatively you can use the mc command to do it from your terminal:
$ mc ls minio/models/flowers/0001/
[2020-03-26 13:16:57 CET] 1.7MiB saved_model.pb
[2020-04-25 13:37:09 CEST] 0B variables/
Create a secret&serviceaccount for the minio access, note that the s3-endpoint defines the path to the minio, keyid&acceskey are the credentials encoded in base64:
$ kubectl get secret mysecret -n homelab -o yaml
apiVersion: v1
data:
awsAccessKeyID: bWluaW8=
awsSecretAccessKey: bWluaW8xMjM=
kind: Secret
metadata:
annotations:
serving.kubeflow.org/s3-endpoint: minio-service.kubeflow:9000
serving.kubeflow.org/s3-usehttps: "0"
name: mysecret
namespace: homelab
$ kubectl get serviceAccount -n homelab sa -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa
namespace: homelab
secrets:
- name: mysecret
Finally, create your inferenceservice as follows:
$ kubectl get inferenceservice tensorflow-flowers -n homelab -o yaml
apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
name: tensorflow-flowers
namespace: homelab
spec:
default:
predictor:
serviceAccountName: sa
tensorflow:
storageUri: s3://models/flowers
I've deployed my first app on my Kubernetes prod cluster a month ago.
I could deploy my 2 services (front / back) from gitlab registry.
Now, I pushed a new docker image to gitlab registry and would like to redeploy it in prod:
Here is my deployment file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
labels:
app: espace-client-client
name: espace-client-client
namespace: espace-client
spec:
replicas: 1
strategy: {}
template:
metadata:
labels:
app: espace-client-client
spec:
containers:
- envFrom:
- secretRef:
name: espace-client-client-env
image: registry.gitlab.com/xxx/espace_client/client:latest
name: espace-client-client
ports:
- containerPort: 3000
resources: {}
restartPolicy: Always
imagePullSecrets:
- name: gitlab-registry
I have no clue what is inside gitlab-registry. I didn't do it myself, and the people who did it left the crew :( Nevertheless, I have all the permissions, so, I only need to know what to put in the secret, and maybe delete it and recreate it.
It seems that secret is based on my .docker/config.json
➜ espace-client git:(k8s) ✗ kubectl describe secrets gitlab-registry
Name: gitlab-registry
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/dockerconfigjson
Data
====
.dockerconfigjson: 174 bytes
I tried to delete existing secret, logout with
docker logout registry.gitlab.com
kubectl delete secret gitlab-registry
Then login again:
docker login registry.gitlab.com -u myGitlabUser
Password:
Login Succeeded
and pull image with:
docker pull registry.gitlab.com/xxx/espace_client/client:latest
which worked.
file: ~/.docker/config.json is looking weird:
{
"auths": {
"registry.gitlab.com": {}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.6 (linux)"
},
"credsStore": "secretservice"
}
It doesn't seem to contain any credential...
Then I recreate my secret
kubectl create secret generic gitlab-registry \
--from-file=.dockerconfigjson=/home/julien/.docker/config.json \
--type=kubernetes.io/dockerconfigjson
I also tried to do :
kubectl create secret docker-registry gitlab-registry --docker-server=registry.gitlab.com --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
and deploy again:
kubectl rollout restart deployment/espace-client-client -n espace-client
but I still have the same error:
Error from server (BadRequest): container "espace-client-client" in pod "espace-client-client-6c8b88f795-wcrlh" is waiting to start: trying and failing to pull image
You have to update the gitlab-registry secret because this item is used to let Kubelet to pull the protected image using credentials.
Please, delete the old secret with kubectl -n yournamespace delete secret gitlab-registry and recreate it typing credentials:
kubectl -n yournamespace create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD[ --docker-email=DOCKER_EMAIL]
where:
- DOCKER_REGISTRY_SERVER is the GitLab Docker registry instance
- DOCKER_USER is the username of the robot account to pull images
- DOCKER_PASSWORD is the password attached to the robot account
You could ignore docker-email since it's not mandatory (note the square brackets).
After following the link below, I can successfully pull my private images in Docker Hub from my Pods: Pull from Private repo
However, attempting to pull a Docker Store image doesn't seem to work.
I am able to pull this store image locally on my deskop using docker pull store/oracle/database-instantclient:12.2.0.1 and the same credentials that have been stored in Kubernetes as a secret.
What is the correct way to pull a Docker Store image from Kubernetes Pods?
Working pod config for my private repo/image:
image: index.docker.io/<privaterepo>/<privateimage>
I have tried the following in my pod config, none work:
image: store/oracle/database-instantclient:12.2.0.1
image: oracle/database-instantclient:12.2.0.1
image: index.docker.io/oracle/database-instantclient:12.2.0.1
image: index.docker.io/store/oracle/database-instantclient:12.2.0.1
All of the above attempts return the same error (with different image paths):
Failed to pull image "store/oracle/database-instantclient:12.2.0.1": rpc error: code = Unknown desc = Error response from daemon: repository store/oracle/database-instantclient not found: does not exist or no pull access
I managed to run this in minikube by setting up a secret with my docker login:
kubectl create secret docker-registry dockerstore \
--docker-server=index.docker.io/v1/ \
--docker-username={docker store username} \
--docker-password={docker store password} \
--docker-email={your email}
Then kubectl create -f testreplicaset.yaml
on
#testreplicaset.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: oracle-instantclient
labels:
app: oracle-instantclient
spec:
replicas: 1
selector:
matchLabels:
app: oracle-instantclient
template:
metadata:
labels:
app: oracle-instantclient
spec:
containers:
- name: oracle-instantclient-container
image: store/oracle/database-instantclient:12.2.0.1
env:
ports:
imagePullSecrets:
- name: dockerstore
I can't tell exactly why it doesn't work for you, but it might give more clues if you ssh into your kubernetes node and try docker pull in there.
I am planning to deploy an application in my kubernetes-clustering infra.
I pushed image to dockerhub repo. How can I pull image from dockerhub?
One line command to create a Docker registry secret
kubectl create secret docker-registry regcred --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email> -n <your-namespace>
Then you can use it in your deployment file under spec
spec:
containers:
- name: private-reg-container-name
image: <your-private-image>
imagePullSecrets:
- name: regcred
More details:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-in-the-cluster-that-holds-your-authorization-token
Kubernetes run docker pull pseudo/your-image:latest under the hood. image field in Kubernetes resources is simply the docker image to run.
spec:
containers:
- name: app
image: pseudo/your-image:latest
[...]
As the docker image name contains no specific docker registry url, the default is docker.io. Your image is in fact docker.io/pseudo/your-image:latest
If your image is hosted in a private docker hub repo, you need to specify an image pull secret in the spec field.
spec:
containers:
- name: app
image: pseudo/your-image:latest
imagePullSecrets:
- name: dockerhub-credential
Here is the documentation to create the secret containing your docker hub login: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
using docker pull or kubectl set image
example yaml deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
start container and show status deployment with kubectl get deployments
result
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 18s
and now update image in kubernetes using set image
kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
and show status update image with rollout
kubectl rollout status deployment/nginx-deployment
Note: ngnix is name of container ->name
containers:
- name: nginx
image: nginx:1.14.2
nginx:1.16.1 is image version in docker hub, is recommendable change version for update
if you decided remove update and rollback to the previous revision, use rollout undo
kubectl rollout undo deployment/nginx-deployment
for more information, use the documentation
Create a docker registry secret:
#!/bin/bash
for ns in $(kubectl get namespaces |grep -v NAME|awk '{print $1}')
do
kubectl create secret docker-registry docker.registry \
--docker-username=<MyAccountName> \
--docker-password='MyDockerHubPassword' -n $ns
done
Patch all the dynamic service accounts in all the namesapces with the secret you created in step 1
for ns in $(kubectl get namespaces|grep -v NAME|awk '{print $1}')
do
for sa in $(kubectl -n $ns get sa|grep -v SECRETS|awk '{print $1}')
do
kubectl patch serviceaccount $sa -p '{"imagePullSecrets": [{"name": "docker.registry"}]}' -n $ns
if [ $? -eq 0 ]; then
echo $ns $sa patched
else
echo Error patching $ns $sa
fi
done
done
You can patch only specific namespaces, if you wish.
Let me know how it goes.
I have an ubuntu installed on my laptop.
I started a private docker registry (ssl enabled + htpasswd secured) and added it on overlay network (so it can be accessed from other hosts/vms)
here is the code (docker-compose.yaml):
version: "3"
services:
registry:
restart: always
image: registry:2
ports:
- 5000:5000
environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
REGISTRY_HTTP_TLS_KEY: /certs/domain.key
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
volumes:
- /certs:/certs
- ~/caas_rd/workspace/ci_cd_pipeline/registry_setup:/auth
networks:
- overlaynetwork
networks:
overlaynetwork:
so my registry is running in the following link (with dns, i can verify it in browser) : https://home-thinkpad-t420s:5000/v2/_catalog
Now I installed "Minikube" on my laptop. && ssh it by "minikube ssh".
I created a folder "/etc/docker/certs.d" on minikube vm & added certificates as per instructions :
https://docs.docker.com/engine/security/certificates/#understanding-the-configuration
I also modified /etc/hosts && appended ca.cert on /etc/ssl/certs/ca-certificates.crt.
and restarted docker service on minikube vm by : sudo systemctl restart docker.service
after this I am able to pull the images on minikube vm by "docker login & docker pull" & also by "curl with (cacert + username/password)"
above is working perfectly fine, means I can successfully access/pull private registry images inside minikube vm.
Then I tried to create a secret (on my laptop with kubectl create -f ) defined as below:
apiVersion: "v1"
kind: "Secret"
metadata:
name: "ssl-proxy-secret"
namespace: "default"
data:
proxycert: "LS0..."
proxykey: "LS0t..."
htpasswd: "YWRt..."
and created a pod (on my laptop with kubectl create -f ) defined as below:
apiVersion: v1
kind: Pod
metadata:
name: private-jenkins
spec:
containers:
- name: private-jenkins-container
image: home-thinkpad-t420s:5000/my-jenkins
volumeMounts:
- name: secrets
mountPath: /etc/secrets
volumes:
- name: secrets
secret:
secretName: ssl-proxy-secret
but when I try to run this pod, it throws error :
Failed to pull image "home-thinkpad-t420s:5000/my-jenkins": rpc error: code = 2 desc = Error: image my-jenkins not found
Error syncing pod, skipping: failed to "StartContainer" for "private-jenkins-container" with ErrImagePull: "rpc error: code = 2 desc = Error: image my-jenkins not found"
If I am able to pull images inside the minikube vm successfully by curl & docker login/pull......then when why pod creation is failing with above error ?
You need to create a separated kubernetes registry secret instead. Could use this command:
kubectl create secret docker-registry <secret-name> \
--docker-email=<your-email> --docker-username=<registry-user> \
--docker-password=<registry-password> --docker-server=<registry-server-domain>
After that you could update the pod configuration as follow:
apiVersion: v1
kind: Pod
metadata:
name: private-jenkins
spec:
containers:
- name: private-jenkins-container
image: home-thinkpad-t420s:5000/my-jenkins
imagePullSecrets:
- name: <secret-name>
Reference: link
Hope it helps!
Can you remove those double quotes escpecially from the credentials in the secret.yml file and try spinning the pod again ?