is there a way to only pull images from a private registry and blocking pulling images from the public registry, docker hub?
I'm using a Nexus registry that I was able to configure and pull images from. I'm trying to implement a way of only push images from my private registry so I can track what docker images I'm using and after that, apply some security analysis in that. But I can still pull images from docker hub in my computer. Is there a way to block that?
There are different ways, how you can do it.
I assume here you are using Kubernetes:
Use an admission controller on the cluster that rewrite Pod spec to your internal registry
Container Runtimes have nowadays options to restrict registries, cri-o and containerd have this option.
If you are using something like CNCF Harbor [1], [2] you can create proxies for 3rd party registries and then user Kyverno to rewrite the Pod spec.
Replace Image Registry with Kyverno
Rather than blocking Pods which come from outside registries, it is also possible to mutate them, so the pulls are directed to approved registries. In some cases, those registries may function as pull-through proxies and can fetch the image if not cached. This policy mutates all images either in the form 'image:tag' or 'example.container-registry.com/image:tag' to be myregistry.corp.com/. Any path in the image name will be preserved. Note that this mutates Pods directly and not their controllers. It can be changed if desired, but if so, may need not match on Pods.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: replace-image-registries
spec:
background: false
rules:
# We only allow a known set of approved registries to be used in our clusters
- name: validate-registries
match:
resources:
kinds:
- Pod
validate:
message: "Unapproved image registry."
pattern:
spec:
containers:
- image: "example.container-registry.com/* | quay.io/* | gcr.io/* | ghcr.io/* | docker.io/*"
# Rewrite all the references for our approved external registries
- name: replace-registries
match:
resources:
kinds:
- Pod
mutate:
patchStrategicMerge:
spec:
containers:
- (name): "*"
image: |-
{{ regex_replace_all('(quay.io|gcr.io|ghcr.io|docker.io)/(.*)', '{{#}}', 'example.container-registry.com/$1/$2') }}
# At this point we expect everything that has a registry prefix to have been transformed
# example.container-registry.com.*. We are left with references like:
#
# - velero/velero:v1.6.2
# - nginx:latest
# - nginx
#
# Without interfering with our newly rewritten references that start with example.container-registry.com
- name: replace-docker
match:
resources:
kinds:
- Pod
mutate:
patchStrategicMerge:
spec:
containers:
- (name): "*"
image: |-
{{ regex_replace_all('^([^example.container-registry.com].*)', '{{#}}', 'example.container-registry.com/docker.io/$1') }}
This repo here has some examples: https://github.com/jvanzyl/kyverno-registries
Related
I'm running microservices on GKE and using skaffold for management.
Everything works fine for a week and suddenly all services are killed (not sure why).
Logging shows this same info for all services:
There is no indication that something is going wrong in the logs before all services fail. It looks like the pods are all killed at the same time by GKE for whatever reason.
What confuses me is why the services do not restart.
kubectl describe pod auth shows a "imagepullbackoff" error.
When I simulate this situation on the test system (same setup) by deleting a pod manually, all services restart just fine.
To deploy the microservices, I use skaffold.
---deployment.yaml for one of the microservices---
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-depl
namespace: development
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
volumes:
- name: google-cloud-key
secret:
secretName: pubsub-key
containers:
- name: auth
image: us.gcr.io/XXXX/auth
volumeMounts:
- name: google-cloud-key
mountPath: /var/secrets/google
env:
- name: NATS_CLIENT_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NATS_URL
value: 'http://nats-srv:4222'
- name: NATS_CLUSTER_ID
value: XXXX
- name: JWT_KEY
valueFrom:
secretKeyRef:
name: jwt-secret
key: JWT_KEY
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
Any idea why the microservices don't restart? Again, they run fine after deploying them with skaffold and also after simulating pod shutdown on the test system ... what changed here?
---- Update 2021.10.30 -------------
After some further digging in the cloud log explorer, I figured out that the pod is trying to pull the previously build image but fails. If I pull the image on cloud console manually using the image name plus tag as listed in the logs, it works just fine (thus the image is there).
The log gives the following reason for the error:
Failed to pull image "us.gcr.io/scout-productive/client:v0.002-72-gaa98dde#sha256:383af5c5989a3e8a5608495e589c67f444d99e7b705cfff29e57f49b900cba33": rpc error: code = NotFound desc = failed to pull and unpack image "us.gcr.io/scout-productive/client#sha256:383af5c5989a3e8a5608495e589c67f444d99e7b705cfff29e57f49b900cba33": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:4e9f2cdf438714c2c4533e28c6c41a89cc6c1b46cf77e54c488db30ca4f5b6f3 (application/vnd.docker.image.rootfs.diff.tar.gzip) from remote: not found"
Where is that "sha256:4e9f2cdf438714c2c4533e28c6c41a89cc6c1b46cf77e54c488db30ca4f5b6f3" coming from that it cannot find according to the error message?
Any help/pointers are much appreciated!
Thanks
When Skaffold causes an image to be built, the image is pushed to a repository (us.gcr.io/scout-productive/client) using a tag generated with your specified tagging policy (v0.002-72-gaa98dde; this tag looks like the result of using Skaffold's default gitCommit tagging policy). The remote registry returns the image digest of the received image, a SHA256 value computed from the image metadata and image file contents (sha256:383af5c5989a3e8a5608495e589c67f444d99e7b705cfff29e57f49b900cba33). The image digest is unique like a fingerprint, whereas a tag is just name → image digest mapping, and which may be updated to point to a different image digest.
When deploying your application, Skaffold rewrites your manifests on-the-fly to reference the specific built container images by using both the generated tag and the image digest. Container runtimes ignore the image tag when an image digest is specified since the digest identifies a unique image.
So the fact that your specific image cannot be resolved means that the image has been deleted from your repository. Are you, or someone on your team, deleting images?
Some people run image garbage collectors to delete old images. Skaffold users do this to delete the interim images generated by skaffold dev between dev loops. But some of these collectors are indiscriminate, such as only keeping images tagged with latest. Since Skaffold tags images using a configured tagging policy, such collectors can delete your Skaffold-built images. To avoid problems, either tune your collector (e.g., only deleted untagged images), or have Skaffold build your dev and production images to different repositories. For example, GCP's Artifact Registry allows having multiple independent repositories in the same project, and your GKE cluster can access all such repositories.
The imagepullbackoff means that kubernetes couldn't download the images from the registry - that could mean that the image with that name/tag doesn't exist, OR the the credentials to the registry is wrong/expired.
From what I see in your deployment.yml there aren't provided any credentials to the registry at all. You can do it by providing imagePullSecret. I never used Skaffold, but my assumptions is that you login to private registry in Skaffold, and use it to deploy the images, so when Kubernetes tries to redownload the image from the registry by itself, it fails because because of lack of authorization.
I can propose two solutions:
1. ImagePullSecret
You can create secret resource witch will contain credentials to private registry, then define that secret in the deployment.yml, that way Kubernetes will be able to authorize at your private registry and pull the images.
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
2. ImagePullPolicy
You can prevent re-downloading your images, by setting ImagePullPolicy:IfNotPresent in the deployment.yml . It will reuse the one which is available locally. Using this method requires defining image tags properly. For example using this with :latest tag can lead to not pulling the "newest latest" image, because from cluster perspective, it already has image with that tag so it won't download a new one.
https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy
To resolve problem with pods being killed, can you share Kubernetes events from the time when pods are being killed? kubectl -n NAMESPACE get events This would maybe give more information.
The dilemma: Deploy multiple app and database container pairs with identical docker image and code, but different config (different clients using subdomains).
What are some logical ways to approach this, as it doesn't seem kubernetes has an integration that would support this kind of setup?
Possible Approaches
Use a single app service for all app deployments, a single database service for all database deployments. Have a single Nginx static file service and deployment running, that will serve static files from a static volume that is shared between the app deployments (all use the same set of static files). Whenever a new deployment is needed, have a bash script copy the app and database .yaml deployment files and sed text replace with the name of the client, and point to the correct configmap (which is manually written ofcourse) and kubectl apply them. A main nginx ingress will handle the incoming traffic and point to the correct pod through the app deployment service
Similar to the above except use a StatefulSet instead of separate deployments, and an init container to copy different configs to mounted volumes (only drawback is you cannot delete an item in the middle of a statefulset, which would be the case if you no longer need a specific container for a client, as well as this seems like a very hacky approach).
Ideally if a StatefulSet could use the downward api to dynamically choose a configmap name based on the index of the stateful set that would resolve the issue (where you would basically make your config files manually with the index in the name, and it would be selected appropriately). Something like:
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
envFrom:
- configMapRef:
name: $(POD_NAME)-config
However that functionality isn't available in kubernetes.
A templating engine like Helm can help with this. (I believe Kustomize, which ships with current Kubernetes, can do this too, but I'm much more familiar with Helm.) The basic idea is that you have a chart that contains the Kubernetes YAML files but can use a templating language (the Go text/template library) to dynamically fill in content.
In this setup generally you'd have Helm create both the ConfigMap and the matching Deployment; in the setup you describe you'd install it separately (a Helm release) for each tenant. Say the Nginx configurations were different enough that you wanted to store them in external files; the core parts of your chart would include
values.yaml (overridable configuration, helm install --set nginxConfig=bar.conf):
# nginxConfig specifies the name of the Nginx configuration
# file to embed.
nginxConfig: foo.conf
templates/configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-{{ .Chart.Name }}-config
data:
nginx.conf: |-
{{ .Files.Get .Values.nginxConfig | indent 4 }}
deployment.yaml:
apiVersion: v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-{{ .Chart.Name }}-nginx
spec:
...
volumes:
- name: nginx-config
configMap:
name: {{ .Release.Name }}-{{ .Chart.Name }}-config
The {{ .Release.Name }}-{{ .Chart.Name }} is a typical convention that allows installing multiple copies of the chart in the same namespace; the first part is a name you give the helm install command and the second part is the name of the chart itself. You can also directly specify the ConfigMap content, referring to other .Values... settings from the values.yaml file, use the ConfigMap as environment variables instead of files, and so on.
While dynamic structural replacement isn't possible (plus or minus, see below for the whole story), I believe you were in the right ballpark with your initContainer: thought; you can use the serviceAccount to fetch the configMap from the API in an initContainer: and then source that environment on startup by the main container:
initContainers:
- command:
- /bin/bash
- -ec
- |
curl -o /whatever/env.sh \
-H "Authorization: Bearer $(cat /var/run/secret/etc/etc)" \
https://${KUBERNETES_SERVICE_HOST}/api/v1/namespaces/${POD_NS}/configmaps/${POD_NAME}-config
volumeMounts:
- name: cfg # etc etc
containers:
- command:
- /bin/bash
- -ec
- "source /whatever/env.sh; exec /usr/bin/my-program"
volumeMounts:
- name: cfg # etc etc
volumes:
- name: cfg
emptyDir: {}
Here we have the ConfigMap fetching inline with the PodSpec, but if you had a docker container specialized for fetching ConfigMaps and serializing them into a format that your main containers could consume, I wouldn't expect the actual solution to be nearly that verbose
A separate, and a lot more complicated (but perhaps elegant) approach is a Mutating Admission Webhook, and it looks like they have even recently formalized your very use case with Pod Presets but it wasn't super clear from the documentation in which version that functionality first appeared, nor if there are any apiserver flags one must twiddle to take advantage of it.
PodPresets has been removed since v1.20, the more elegant solution, based on Mutating Admission Webhook, to solve this problem is available now https://github.com/spoditor/spoditor
Essentially, it uses a custom annotation on the PodSpec template, like:
annotations:
spoditor.io/mount-volume: |
{
"volumes": [
{
"name": "my-volume",
"secret": {
"secretName": "my-secret"
}
}
],
"containers": [
{
"name": "nginx",
"volumeMounts": [
{
"name": "my-volume",
"mountPath": "/etc/secrets/my-volume"
}
]
}
]
}
Now, nginx container in each Pod of the StatefulSet will try to mount its own dedicated secret in the pattern of my-secret-{pod ordinal}.
You will just need to make sure my-secret-0, my-secret-1, so on and so forth exists in the same namespace of the StatefulSet.
There're more advanced usage of the annotation in the documentation of the project.
I'm working on a Docker-based project. The project code is hosted in a private Gitlab installation, git.example.com. With it, the Docker private registry shipped with Gitlab is deployed, registry.example.com.
The project has a CI setup which ends up building Docker images and pushing to the registry, this part works as expected. As Gitlab+Docker registry does not yet support multiple images related to the same Git repo, I'm using the tags workaround which specifies an image as:
registry.example.com/group/my.project:web
registry.example.com/group/my.project:app
etc.
I've created a user and attached it to the projects, logged in via it locally and tried to pull above images, that works as expected.
I've added the ImageStream block as so:
apiVersion: v1
kind: ImageStream
metadata:
name: web
spec:
tags:
-
from:
kind: DockerImage
name: registry.example.com/group/my.project:web
name: latest
This adds the image in the Images section, but it cannot pull it Openshift doesn't have access to the Docker Registry yet. I add a new Docker secret as described here and am now able to see image metadata in Openshift, everything looks as expected.
But, if I add a deployment config, like so:
apiVersion: v1
kind: DeploymentConfig
metadata:
creationTimestamp: null
labels:
service: web
name: web
spec:
replicas: 1
selector:
service: web
strategy:
resources: { }
template:
metadata:
creationTimestamp: null
labels:
service: web
spec:
containers:
-
name: web
ports:
-
containerPort: 80
resources: { }
restartPolicy: Always
test: false
triggers:
-
type: ConfigChange
-
type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- web
from:
kind: ImageStreamTag
name: 'web:latest'
status: { }
I keep getting error:
Failed to pull image "registry.example.com/group/my.project#sha256:3333022641e571d7e4dcae2953d35be8cdf9416b13967b99537c4e8f150f74e4": manifest unknown: manifest unknown
in the Events tab of the pod created. This basically kills my plan to deploy prebuilt images to Openshift.
I know about Docker 1.9 -> 1.10 incompatibility, but this is Openshift 1.4.1, images were pushed with Docker 1.13 so it shouldn't be a problem.
How do I even start debugging this, is there a way to access any sort of log which would explain what's going on? Why is ImageStream able to find everything it needs (and access my registry), but not the DeploymentConfig?
To answer my own question: it seems Docker's Distribution (registry daemon) has a bug which manifests itself in quite a weird way.
Basically, the problem is:
Registry is behind Apache reverse proxy
the image gets built and pushed from CI runner to Gitlab's Registry, digest SHA:1234 (example, of course)
the image gets imported to Openshift, it queries the metadata and Docker Distribution claims the digest is SHA:ABCD, you can reproduce this by pushing and then pulling right away, the digests are supposed to be identical both times, as explained in the link
when Openshift tries to actually pull the image, it will get the dreaded "Manifest unknown" error above (as it's trying to fetch the image using an invalid digest, not by fault of its own)
all symptoms look exactly like with Docker v1 => Docker v2 API changes, except for totally different reasons
I've since moved my Gitlab instance to another machine (where it's behind Nginx) and it works without a problem.
I am running kubeadm alpha version to set up my kubernates cluster.
From kubernates , I am trying to pull docker images which is hosted in nexus repository.
When ever I am trying to create a pods , It is giving "ImagePullBackOff" every time. Can anybody help me on this ?
Detail for this are present in https://github.com/kubernetes/kubernetes/issues/41536
Pod definition :
apiVersion: v1
kind: Pod
metadata:
name: test-pod
labels:
name: test
spec:
containers:
- image: 123.456.789.0:9595/test
name: test
ports:
- containerPort: 8443
imagePullSecrets:
- name: my-secret
You need to refer to the secret you have just created from the Pod definition.
When you create the secret with kubectl create secret docker-registry my-secret --docker-server=123.456.789.0 ... the server must exactly match what's in your Pod definition - including the port number (and if it's a secure one then it also must match up with the docker command line in systemd).
Also, the secret must be in the same namespace where you are creating your Pod, but that seems to be in order.
I received similar error while launching containers from the amazon ECR registry. The issue was that I didn;t mention the exact "Image URI" location in deployment file.
I set up a Kubernetes cluster on AWS using kube-up script with one master and two minions. I want to create a pod that uses a private docker image. So I need to add my credential to docker daemons of each minion of the cluster. But I don't know how to log into the minions created by AWS script. What is the recommended way to pass credentials to the docker demons of each minion?
Probably the best method for you is ImagePullSecrets - you will create secret (docker config), which be will be used for image pull. Read more about different concepts of using private registry http://kubernetes.io/docs/user-guide/images/#using-a-private-registry
Explained here: https://kubernetes.io/docs/concepts/containers/images/
There are 3 options for ImagePullPolicy: Always, IfNotPresent and Never
1) example of yaml:
...
spec:
containers:
- name: uses-private-image
image: $PRIVATE_IMAGE_NAME
imagePullPolicy: Always
command: [ "echo", "SUCCESS" ]
2) By default, the kubelet will try to pull each image from the specified registry. However, if the imagePullPolicy property of the container is set to IfNotPresent or Never, then a local image is used (preferentially or exclusively, respectively).
If you want to rely on pre-pulled images as a substitute for registry authentication, you must ensure all nodes in the cluster have the same pre-pulled images.
This can be used to preload certain images for speed or as an alternative to authenticating to a private registry.
All pods will have read access to any pre-pulled images.