We use Azure Container Registry to pull a larger image (~6Gb) to launch a cluster of many instances.. and it takes unusually long to pull the image.
We were wondering if Azure Container Registry is a truly multi-region service, or at least has a front-end CDN that has per-region local caches?
Have a look at
https://learn.microsoft.com/en-us/azure/container-registry/container-registry-geo-replication
This will allow you to bring your images closer to your different regions where your clusters are created.
Related
I have one global container registry.
I will have many k8s clusters in different cloud providers. For now I use GKE.
I want to have in each k8s cluster a local docker registry cache. It reduces the pulling latency and I will be safer if the global container registry has short downtime.
It should work like: when I deploy something on k8s cluster, the k8s starts pulling the image and goes via this proxy. If this proxy already has this image it will serve it quickly, if not it will pull it from the global container registry and will serve it.
I tried to setup https://hub.docker.com/r/rpardini/docker-registry-proxy
I run it, but I can't configure k8s cluster to use it as a proxy. In docs I see how to do it, but it is ok when you have your own k8s clusters on servers and you can change dockerd or containerd service files, but I have managed k8s in Google Cloud (GKE), so I can't easily permanent change files on nodes.
Do you have any ideas on how to achieve what I want?
I have my own docker private registry created in my host machine[localhost] and I intend to make use of localhost private registry to pull images in google Kubernetes engine.
How do I make it happen?
You won't be able to use either your locally built docker images (which can be listed by running docker images on your local machine) or your locally set up docker private registry (unless you make it available under some public IP which doesn't make much sense if it's your home computer). Those images can be used by your local kubernetes cluster but not by GKE.
In GKE we generally use GCR (Google Container Registry) for storing images that are used by our Kubernetes Engine. We can build them directly from code (e.g. pulled from our github account) on a cloudshell vm (simply click Cloud Shell icon in your GCP Console). You can build them directly on this machine and you can push them to your GCR directly from there.
Alternatively, if you build your images locally, but by "locally" I mean this time the nodes where kubernetes is installed (so in case of GKE they need to be present on every worker node), you can also use them without a need of pulling them from any external registry. The only requirement is that they are available on all kubernetes worker nodes. You can force kubernetes to always use your local images, present on your nodes, instead of trying to pull them from a registry by specifying:
imagePullPolicy: Never
in your Pod or Deployment specification. More details on that you can find in this answer.
I have setup a 3 node cluster (with no Internet access) with 1 manager and 2 worker-nodes using the standard swarm documentation.
How does the swarm manager in swarm mode know about the images present in worker nodes?
Lets say I have image A in worker-node-1 and image B in worker-node-2 and no images in the manager-node.
Now how do I start container for image A using the manager?
Will it start in manager or node-1?
When I query manager for the list of images will it give the whole list with A and B in it?
Does anyone know how this works?
I couldn’t get the details from the documentation.
Docker Swarm manager node may to be a worker one by the second role but not strictly necessary.
Image deployment policy is mapped via docker-compose.yml which has an information like target nodes, networks, hostnames, volumes, etc. in relation of particular service. So, it will start either in specified node or in emptiest default one.
Swarm manager communicates with the worker nodes via Docker networks:
When you initialize a swarm or join a Docker host to an existing swarm, two new networks are created on that Docker host:
an overlay network called ingress, which handles control and data
traffic related to swarm services. When you create a swarm service and
do not connect it to a user-defined overlay network, it connects to
the ingress network by default
a bridge network called
docker_gwbridge, which connects the individual Docker daemon to the
other daemons participating in the swarm.
Reference
During Swarm deployment, the images of it's services are being propagated to worker nodes according to their deployment policy.
The manager node will contain images once the node is the worker one too (correct me, if it won't).
The default configuration with swarm mode is to pull images from a registry server and use pinning to reference a unique hash for those images. This can be adjusted, but there is no internal mechanism to distribute images within a cluster.
For an offline environment, I'd recommend a stand alone registry server accessible to the cluster. You can even run it on the cluster. Push your image there, and point your server l services to the registry for their images to pull. See this doc for details on running a stand alone registry, or any of the many 3rd party options (e.g. Harbor): https://docs.docker.com/registry/
The other option is to disable the image pinning, and manually copy images to each of your swarm nodes. You need to do this in advance of deploying any service changes. You'll also lose the benefit of reused image layers when you manually copy them. Because of all this issues it creates, overhead to manage, and risk of mistakes, I'd recommend against this option.
Run the docker stack deploy command with --with-registry-auth that will give the Workers access to pull the needed image
By default Docker Swarm will pull the latest image from registry when deploying
Is it possible, to pull private images from Docker Hub to a Google Cloud Kubernetes cluster?
Is this recommended, or do I need to push my private images also to Google Cloud?
I read the documentation, but I found nothing that could explain me this clearly. It seems that it is possible, but I don´t know if it's recommended.
There is no restriction to use any registry you want. If you just use the image name, (e.g., image: nginx) in pod specification, the image will be pulled from public docker hub registry with tag assumed as :latest
As mentioned in the Kubernetes documentation:
The image property of a container supports the same syntax as the
docker command does, including private registries and tags. Private
registries may require keys to read images from them.
Using Google Container Registry
Kubernetes has native support for the Google Container Registry (GCR), when running on Google
Compute Engine (GCE). If you are running your cluster on GCE or Google
Kubernetes Engine, simply use the full image name (e.g.
gcr.io/my_project/image:tag). All pods in a cluster will have read
access to images in this registry.
Using AWS EC2 Container Registry
Kubernetes has native support for the AWS EC2 Container Registry, when nodes are AWS EC2 instances.
Simply use the full image name (e.g.
ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag) in the Pod
definition. All users of the cluster who can create pods will be able
to run pods that use any of the images in the ECR registry.
Using Azure Container Registry (ACR)
When using Azure Container Registry you can authenticate using either an admin user or a
service principal. In either case, authentication is done via standard
Docker authentication. These instructions assume the azure-cli command
line tool.
You first need to create a registry and generate credentials, complete
documentation for this can be found in the Azure container registry
documentation.
Configuring Nodes to Authenticate to a Private Repository
Here are the recommended steps to configuring your nodes to use a private
registry. In this example, run these on your desktop/laptop:
Run docker login [server] for each set of credentials you want to use. This updates $HOME/.docker/config.json.
View $HOME/.docker/config.json in an editor to ensure it contains just the credentials you want to use.
Get a list of your nodes, for example:
if you want the names: nodes=$(kubectl get nodes -o jsonpath='{range.items[*].metadata}{.name} {end}')
if you want to get the IPs: nodes=$(kubectl get nodes -o jsonpath='{range
.items[*].status.addresses[?(#.type=="ExternalIP")]}{.address}
{end}')
Copy your local .docker/config.json to the home directory of root on each node.
for example: for n in $nodes; do scp ~/.docker/config.json root#$n:/root/.docker/config.json; done
Use cases:
There are a number of solutions for configuring private registries.
Here are some common use cases and suggested solutions.
Cluster running only non-proprietary (e.g. open-source) images. No need to hide images.
Use public images on the Docker hub.
No configuration required.
On GCE/Google Kubernetes Engine, a local mirror is automatically used for improved speed and availability.
Cluster running some proprietary images which should be hidden to those outside the company, but visible to all cluster users.
Use a hosted private Docker registry.
It may be hosted on the Docker Hub, or elsewhere.
Manually configure .docker/config.json on each node as described above.
Or, run an internal private registry behind your firewall with open read access.
No Kubernetes configuration is required.
Or, when on GCE/Google Kubernetes Engine, use the project’s Google Container Registry.
It will work better with cluster autoscaling than manual node configuration.
Or, on a cluster where changing the node configuration is inconvenient, use imagePullSecrets.
Cluster with a proprietary images, a few of which require stricter access control.
Ensure AlwaysPullImages admission controller is active. Otherwise, all Pods potentially have access to all images.
Move sensitive data into a “Secret” resource, instead of packaging it in an image.
A multi-tenant cluster where each tenant needs own private registry.
Ensure AlwaysPullImages admission controller is active. Otherwise, all Pods of all tenants potentially have access to all
images.
Run a private registry with authorization required.
Generate registry credential for each tenant, put into secret, and populate secret to each tenant namespace.
The tenant adds that secret to imagePullSecrets of each namespace.
Consider reading the Pull an Image from a Private Registry document if you decide to use a private registry.
There are 3 types of registries:
Public (Docker Hub, Docker Cloud, Quay, etc.)
Private: This would be a registry running on your local network. An example would be to run a docker container with a registry image.
Restricted: That is one registry that needs some credentials to validate. Google Container Registry (GCR) in an example.
As you are well saying, in a public registry, such as Docker Hub, you can have private images.
Private and Restricted registries are more secure obviously, as one of them is not even exposed to internet (ideally), and the other one needs credentials.
I guess you can achieve an acceptable security level with any of them. So, it is matter of choice. If you feel your application is critical, and you don't want to run any risk, you should have it in GCR, or in a private registry.
If you feel like it is important, but not critical, you could have it in any public repository, making it private. This will give a layer of security.
Docker pull can be slow sometimes
How can this best be optimised?
Is it possible to set the mirrors?
Any ideas appreciated. I appreciate sometimes it can just be slow network but would be great to speed this up as much as possible.
Not exactly a mirror, but you can setup a registry as a pull through cache:
By running a local registry mirror, you can keep most of the redundant image fetch traffic on your local network.
In this mode a Registry responds to all normal docker pull requests but stores all content locally.
The first time you request an image from your local registry mirror, it pulls the image from the public Docker registry and stores it locally before handing it back to you.
On subsequent requests, the local registry mirror is able to serve the image from its own storage.
You will need to pass the --registry-mirror option to your Docker daemon on startup:
docker --registry-mirror=https://<my-docker-mirror-host> daemon