I set up a Kubernetes cluster on AWS using kube-up script with one master and two minions. I want to create a pod that uses a private docker image. So I need to add my credential to docker daemons of each minion of the cluster. But I don't know how to log into the minions created by AWS script. What is the recommended way to pass credentials to the docker demons of each minion?
Probably the best method for you is ImagePullSecrets - you will create secret (docker config), which be will be used for image pull. Read more about different concepts of using private registry http://kubernetes.io/docs/user-guide/images/#using-a-private-registry
Explained here: https://kubernetes.io/docs/concepts/containers/images/
There are 3 options for ImagePullPolicy: Always, IfNotPresent and Never
1) example of yaml:
...
spec:
containers:
- name: uses-private-image
image: $PRIVATE_IMAGE_NAME
imagePullPolicy: Always
command: [ "echo", "SUCCESS" ]
2) By default, the kubelet will try to pull each image from the specified registry. However, if the imagePullPolicy property of the container is set to IfNotPresent or Never, then a local image is used (preferentially or exclusively, respectively).
If you want to rely on pre-pulled images as a substitute for registry authentication, you must ensure all nodes in the cluster have the same pre-pulled images.
This can be used to preload certain images for speed or as an alternative to authenticating to a private registry.
All pods will have read access to any pre-pulled images.
Related
I would like to run a K8s Cronjob (or controller) to mirror copies of the actual container images used to a external (ECR) Docker repo. How can I do the equivalent of:
docker pull localimage
docker tag localimage newlocation
docker push newlocation
Kubernetes doesn't have any way to push or rename images, or to manually pull images beyond declaring them in a Pod spec.
The Docker registry system has its own HTTP API. One possibility is, when you discover a new image, manually make the API calls to pull and push it. In this context you wouldn't specifically need to "tag" the image since the image repository, name, and tag only appear as URL components. I'm not specifically aware of a prebuilt tool that can do this, though I'd be very surprised if nobody's built it.
If you can't do this, then the only reliable way to get access to some Docker daemon in Kubernetes is to run one yourself. In this scenario you don't need access to "the real" container system, just somewhere you can target the specific docker commands you list, so you're not limited by the container runtime Kubernetes uses. The one big "but" here is that the Docker daemon must run in a privileged container, which your local environment may not allow.
It's a little unusual to run two containers in one Pod but this is a case where it makes sense. The Docker daemon can run as a prepackaged separate container, tightly bound to its client, as a single unit. Here you don't need persistent storage or anything else that might want the Docker daemon to have a different lifecycle than the thing that's using it; it's just an implementation detail of the copy process. Carefully Googling "docker in docker" kubernetes finds write-ups like this or this that similarly describe this pattern.
By way of illustration, here's a way you might do this in a Kubernetes Job:
apiVersion: batch/v1
kind: Job
metadata: { ... }
spec:
template:
spec:
containers:
- name: cloner
image: docker:latest # just the client and not a daemon
environment:
- name: IMAGE
value: some/image:tag
- name: REGISTRY
value: registry.example.com
- name: DOCKER_HOST
value: tcp://localhost:2375 # pointing at the other container
command:
- /bin/sh
- -c
- |-
docker pull "$IMAGE"
docker tag "$IMAGE" "$REGISTRY/$IMAGE"
docker push "$REGISTRY/$IMAGE"
docker rmi "$IMAGE" "$REGISTRY/$IMAGE"
- name: docker
image: docker:dind
securityContext:
privileged: true # <-- could be a problem with your security team
volumes:
- name: dind-storage
mountPath: /var/lib/docker
volumes:
- name: dind-storage
emptyDir: {} # won't outlive this Pod and that's okay
In practice I suspect you'd want a single long-running process to manage this, maybe running the DinD daemon as a second container in your controller's Deployment.
I'm running microservices on GKE and using skaffold for management.
Everything works fine for a week and suddenly all services are killed (not sure why).
Logging shows this same info for all services:
There is no indication that something is going wrong in the logs before all services fail. It looks like the pods are all killed at the same time by GKE for whatever reason.
What confuses me is why the services do not restart.
kubectl describe pod auth shows a "imagepullbackoff" error.
When I simulate this situation on the test system (same setup) by deleting a pod manually, all services restart just fine.
To deploy the microservices, I use skaffold.
---deployment.yaml for one of the microservices---
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-depl
namespace: development
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
volumes:
- name: google-cloud-key
secret:
secretName: pubsub-key
containers:
- name: auth
image: us.gcr.io/XXXX/auth
volumeMounts:
- name: google-cloud-key
mountPath: /var/secrets/google
env:
- name: NATS_CLIENT_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NATS_URL
value: 'http://nats-srv:4222'
- name: NATS_CLUSTER_ID
value: XXXX
- name: JWT_KEY
valueFrom:
secretKeyRef:
name: jwt-secret
key: JWT_KEY
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
Any idea why the microservices don't restart? Again, they run fine after deploying them with skaffold and also after simulating pod shutdown on the test system ... what changed here?
---- Update 2021.10.30 -------------
After some further digging in the cloud log explorer, I figured out that the pod is trying to pull the previously build image but fails. If I pull the image on cloud console manually using the image name plus tag as listed in the logs, it works just fine (thus the image is there).
The log gives the following reason for the error:
Failed to pull image "us.gcr.io/scout-productive/client:v0.002-72-gaa98dde#sha256:383af5c5989a3e8a5608495e589c67f444d99e7b705cfff29e57f49b900cba33": rpc error: code = NotFound desc = failed to pull and unpack image "us.gcr.io/scout-productive/client#sha256:383af5c5989a3e8a5608495e589c67f444d99e7b705cfff29e57f49b900cba33": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:4e9f2cdf438714c2c4533e28c6c41a89cc6c1b46cf77e54c488db30ca4f5b6f3 (application/vnd.docker.image.rootfs.diff.tar.gzip) from remote: not found"
Where is that "sha256:4e9f2cdf438714c2c4533e28c6c41a89cc6c1b46cf77e54c488db30ca4f5b6f3" coming from that it cannot find according to the error message?
Any help/pointers are much appreciated!
Thanks
When Skaffold causes an image to be built, the image is pushed to a repository (us.gcr.io/scout-productive/client) using a tag generated with your specified tagging policy (v0.002-72-gaa98dde; this tag looks like the result of using Skaffold's default gitCommit tagging policy). The remote registry returns the image digest of the received image, a SHA256 value computed from the image metadata and image file contents (sha256:383af5c5989a3e8a5608495e589c67f444d99e7b705cfff29e57f49b900cba33). The image digest is unique like a fingerprint, whereas a tag is just name → image digest mapping, and which may be updated to point to a different image digest.
When deploying your application, Skaffold rewrites your manifests on-the-fly to reference the specific built container images by using both the generated tag and the image digest. Container runtimes ignore the image tag when an image digest is specified since the digest identifies a unique image.
So the fact that your specific image cannot be resolved means that the image has been deleted from your repository. Are you, or someone on your team, deleting images?
Some people run image garbage collectors to delete old images. Skaffold users do this to delete the interim images generated by skaffold dev between dev loops. But some of these collectors are indiscriminate, such as only keeping images tagged with latest. Since Skaffold tags images using a configured tagging policy, such collectors can delete your Skaffold-built images. To avoid problems, either tune your collector (e.g., only deleted untagged images), or have Skaffold build your dev and production images to different repositories. For example, GCP's Artifact Registry allows having multiple independent repositories in the same project, and your GKE cluster can access all such repositories.
The imagepullbackoff means that kubernetes couldn't download the images from the registry - that could mean that the image with that name/tag doesn't exist, OR the the credentials to the registry is wrong/expired.
From what I see in your deployment.yml there aren't provided any credentials to the registry at all. You can do it by providing imagePullSecret. I never used Skaffold, but my assumptions is that you login to private registry in Skaffold, and use it to deploy the images, so when Kubernetes tries to redownload the image from the registry by itself, it fails because because of lack of authorization.
I can propose two solutions:
1. ImagePullSecret
You can create secret resource witch will contain credentials to private registry, then define that secret in the deployment.yml, that way Kubernetes will be able to authorize at your private registry and pull the images.
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
2. ImagePullPolicy
You can prevent re-downloading your images, by setting ImagePullPolicy:IfNotPresent in the deployment.yml . It will reuse the one which is available locally. Using this method requires defining image tags properly. For example using this with :latest tag can lead to not pulling the "newest latest" image, because from cluster perspective, it already has image with that tag so it won't download a new one.
https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy
To resolve problem with pods being killed, can you share Kubernetes events from the time when pods are being killed? kubectl -n NAMESPACE get events This would maybe give more information.
I am using a CI and have built a Docker Image. I want to pass the image built in the CI to kubectl to take and place it in the cluster I have specified by my kubeconfig. This is as opposed to having the cluster reach out to a registry like dockerhub to retrieve the image. Is this possible?
So far I cannot get this to work and I am thinking I will be forced to create a secret on my cluster to just use my private docker repo. I would like to exhaust my options to not have to use any registry. Also as an alternative I already login to docker on my CI and would like to ideally only have to use those credentials once.
I thought setting the ImagePullPolicy on my deployment might do it but I think it is referring to the cluster context. Which makes me wonder if there is some other way to add an image to my cluster with something like a kubectl create image.
Maybe I am just doing something obvious wrong?
Here is my deploy script on my CI
docker build -t <DOCKERID>/<PROJECT>:latest -t <DOCKERID>/<PROJECT>:$GIT_SHA -f ./<DIR>/Dockerfile ./<DIR>
docker push <DOCKERID>/<PROJECT>:latest
docker push <DOCKERID>/<PROJECT>:$GIT_SHA
kubectl --kubeconfig=$HOME/.kube/config apply -f k8s
kubectl --kubeconfig=$HOME/.kube/config set image deployment/<DEPLOYMENT NAME> <CONTAINER NAME>=<DOCKERID>/<PROJECT>:$SHA
And this Dockerfile:
apiVersion: apps/v1
kind: Deployment
metadata:
name: <deployment name>
spec:
replicas: 3
selector:
matchLabels:
component: <CONTAINER NAME>
template:
metadata:
labels:
component: <CONTAINER NAME>
spec:
containers:
- name: <CONTAINER NAME>
image: <DOCKERID>/<PROJECT>
imagePullPolicy: IfNotPresent
ports:
- containerPort: <PORT>
I want to pass the image [...] to kubectl to take and place it in the cluster [...] as opposed to having the cluster reach out to a registry like dockerhub to retrieve the image. Is this possible?
No. Generally the only way an image gets into a cluster is by a node pulling an image named in a pod spec. And even there "in the cluster" is a little vague; each node will have a different collection of images, based on which pods have ever run there and what's historically been cleaned up.
There are limited exceptions for developer-oriented single-node environments (you can docker build an image directly in a minikube VM, then set a pod to run it with imagePullPolicy: Never) but this wouldn't apply to a typical CI system.
I would like to exhaust my options to not have to use any registry.
Kubernetes essentially requires a registry. If you're using a managed Kubernetes from a public-cloud provider (EKS/GKE/AKS/...), there is probably a matching image registry offering you can use (ECR/GCR/ACR/...).
Preferred way in k8s setting is to update k8s definitions or helm chart values or kustomize values (whichever you are using) with image and its sha256 digest that needs to be deployed.
Using sha256 digest is preferable to tags in production since docker image tags are mutable.
Next, instead of using kubectl from CI/CD it's preferred to use a GitOps tool - either Argo or Flux on the k8s side to pull the correct images based on definitions in Git.
There you need some sort of system to route and manage images - which one should go to which environment. Here is my article with an example how it can be achieved (this one is using my tool): https://itnext.io/building-kubernetes-cicd-pipeline-with-github-actions-argocd-and-reliza-hub-e7120b9be870
To minimize network problems, I created a local repository, mirroring Docker Hub. When applying a deployment, my local registry goes down. Is it possible to automatically change repository source to Docker Hub instead of my local repository? Applying then, redundancy.
Like:
...
containers:
- name: webserver
image-1: registry.mydomain/nginx:latest
image-2: nginx:latest
imagePullPolicy: Always
...
The kubernetes pod spec doesn't include the functionality to specify multiple container images, but the underlying container runtime can often be configured to do that.
Docker
When a specific prefix domain is not provided for an image, The docker.io registry is the first place Docker looks for an unscoped image. This is why docker pull mysql works. Docker has a registry-mirrors setting to modify this behaviour.
In daemon.json add:
{
"registry-mirrors": [ "something.local" ],
}
This will cause Docker to look for non scoped images on something.local first, then docker.io.
I would like to be able to automatically update my Google Cloud Run Services once my image has been updated on Google Container Registry.
I need to update multiple Cloud Run services based on the same image (which has a tag of :latest ), so I expected this to work.
# build & push the container image
- name: "gcr.io/kaniko-project/executor:latest"
args: ["--cache=true", "--cache-ttl=48h", "--destination=gcr.io/project/titan:latest"]
Currently, my titan image gets updated but no new Revision is deployed to Cloud Run.
Google Cloud Run does not automatically deploy a revision when you push a new image to a tag reference. There are many good reasons it doesn’t.
When a Cloud Run revision is deployed, it computes the sha256 hash of the image reference.
Therefore when you specify a container image with :latest tag, Cloud Run uses its sha256 reference to deploy and scale out that revision of your service. When you update :latest tag to point to the new image, Cloud Run will still use the previous image. It would be a dangerous and slippery slope otherwise.
If you need to auto-deploy new revisions to Cloud Run based on a new image push, I recommend two solutions:
Make “gcloud beta run deploy” command a step in your Google Cloud Build process. (easy) https://cloud.google.com/run/docs/continuous-deployment
Write a GCF/Run service that deploys your app to Cloud Run every time there’s a new image pushed by subscribing to Google Cloud Build (or GCR) notifications through PubSub. (much harder)
The default pull policy is IfNotPresent which causes the Kubelet to skip pulling an image if it already exists. If you would like to always force a pull, you can do one of the following:
set the imagePullPolicy of the container to Always.
omit the imagePullPolicy and use :latest as the tag for the image to use.
omit the imagePullPolicy and the tag for the image to use.
enable the AlwaysPullImages admission controller.
Note that you should avoid using :latest tag, see Best Practices for Configuration for more information.
For example, creating a YAML file dummy.yaml
apiVersion: v1
kind: Pod
metadata:
name: foo
spec:
containers:
- name: whatever
image: index.docker.io/DOCKER_USER/PRIVATE_REPO_NAME:latest
imagePullPolicy: Always
command: [ "echo", "SUCCESS" ]
imagePullSecrets:
- name: myregistrykey
Then run:
kubectl create -f dummy.yaml