I have one global container registry.
I will have many k8s clusters in different cloud providers. For now I use GKE.
I want to have in each k8s cluster a local docker registry cache. It reduces the pulling latency and I will be safer if the global container registry has short downtime.
It should work like: when I deploy something on k8s cluster, the k8s starts pulling the image and goes via this proxy. If this proxy already has this image it will serve it quickly, if not it will pull it from the global container registry and will serve it.
I tried to setup https://hub.docker.com/r/rpardini/docker-registry-proxy
I run it, but I can't configure k8s cluster to use it as a proxy. In docs I see how to do it, but it is ok when you have your own k8s clusters on servers and you can change dockerd or containerd service files, but I have managed k8s in Google Cloud (GKE), so I can't easily permanent change files on nodes.
Do you have any ideas on how to achieve what I want?
Related
I’m trying to deploy a docker image on a kubernetes cluster. This cluster is on a server accessible via a ssh connection.
Here is the part of the .yaml I use to pull the image :
spec :
containers :
- name : my_image_name
image : my_private_registry/my_image
my_image is stored in a private registry in another server accessible via a ssh connection (with a proxyjump).
I know how to pull an image from a private registry that is on the same server that the kubernetes cluster. But when it is on another server, I don’t. I'm sure that it is possible to configure kubernetes to make it use the ssh connection, but I didn't find ressources helping me to do that. The others private registry are accessible just by a "docker login" command ?
I’ve found a beginning of solution sending the image using a package named « docker_over_ssh », but it was not really appropriate. I’ve found a solution by pushing the image on dockerhub, but I’m sure that we can do better …
Thank you for your help. I’m sure that it is not complicated, but I’m quite new in kubernetes and docker.
It is quite important to understand, that the docker daemon does not run inside kubernetes. So whatever you provide through kubernetes objects (pods, services...) is usually not accessible from outside.
You could provide a mirror registry for your cluster, so that it is accessible in the usual way.
Another option would be to use ssh port forwarding, if the ssh proxy allows that and use the forwarding host as registry.
I am currently working with a Hyperledger Fabric v1.4 deployment over k8s. The chaincode containers that are generated are basically create by the container running within the peer pods and k8s as such has no knowledge and control of the chaincode containers. In such a scenario where there is a Docker container running along with k8s and k8s has no knowledge of a particular docker container, is it possible for the Docker container to in some way gain access to the k8s master API and gain access to the whole k8s cluster consequently?
My intention with asking this question is to figure out if there is a way to use an container external to any pods in k8s, to cause any undesirable impact to the k8s cluster by gaining unauthorized access to k8s. The chaincode container that I talked about, is created using a trusted template image and the only possible malicious component in the container is a single golang, java or nodejs script that is provided by the user. So my real question here is, "Is it possible using these user scripts gain unauthorized access to the k8s cluster?" And I am primarily focusing on a manager k8s service like Azure Kubernetes Service.
Your question totally changed the meaning so I'll try to rewrite the answer.
You have to remember that the pod you are running the code on by default is limited to just the namespace it's running on. If you didn't gave it any higher privileges. Also the code is not running as root.
You can read about Pod Security Policies and Configure a Security Context for a Pod or Container.
TLDR.
As long as you don't give it any special privileges or rights it should be fairly save for your cluster.
I have my own docker private registry created in my host machine[localhost] and I intend to make use of localhost private registry to pull images in google Kubernetes engine.
How do I make it happen?
You won't be able to use either your locally built docker images (which can be listed by running docker images on your local machine) or your locally set up docker private registry (unless you make it available under some public IP which doesn't make much sense if it's your home computer). Those images can be used by your local kubernetes cluster but not by GKE.
In GKE we generally use GCR (Google Container Registry) for storing images that are used by our Kubernetes Engine. We can build them directly from code (e.g. pulled from our github account) on a cloudshell vm (simply click Cloud Shell icon in your GCP Console). You can build them directly on this machine and you can push them to your GCR directly from there.
Alternatively, if you build your images locally, but by "locally" I mean this time the nodes where kubernetes is installed (so in case of GKE they need to be present on every worker node), you can also use them without a need of pulling them from any external registry. The only requirement is that they are available on all kubernetes worker nodes. You can force kubernetes to always use your local images, present on your nodes, instead of trying to pull them from a registry by specifying:
imagePullPolicy: Never
in your Pod or Deployment specification. More details on that you can find in this answer.
I have setup a 3 node cluster (with no Internet access) with 1 manager and 2 worker-nodes using the standard swarm documentation.
How does the swarm manager in swarm mode know about the images present in worker nodes?
Lets say I have image A in worker-node-1 and image B in worker-node-2 and no images in the manager-node.
Now how do I start container for image A using the manager?
Will it start in manager or node-1?
When I query manager for the list of images will it give the whole list with A and B in it?
Does anyone know how this works?
I couldn’t get the details from the documentation.
Docker Swarm manager node may to be a worker one by the second role but not strictly necessary.
Image deployment policy is mapped via docker-compose.yml which has an information like target nodes, networks, hostnames, volumes, etc. in relation of particular service. So, it will start either in specified node or in emptiest default one.
Swarm manager communicates with the worker nodes via Docker networks:
When you initialize a swarm or join a Docker host to an existing swarm, two new networks are created on that Docker host:
an overlay network called ingress, which handles control and data
traffic related to swarm services. When you create a swarm service and
do not connect it to a user-defined overlay network, it connects to
the ingress network by default
a bridge network called
docker_gwbridge, which connects the individual Docker daemon to the
other daemons participating in the swarm.
Reference
During Swarm deployment, the images of it's services are being propagated to worker nodes according to their deployment policy.
The manager node will contain images once the node is the worker one too (correct me, if it won't).
The default configuration with swarm mode is to pull images from a registry server and use pinning to reference a unique hash for those images. This can be adjusted, but there is no internal mechanism to distribute images within a cluster.
For an offline environment, I'd recommend a stand alone registry server accessible to the cluster. You can even run it on the cluster. Push your image there, and point your server l services to the registry for their images to pull. See this doc for details on running a stand alone registry, or any of the many 3rd party options (e.g. Harbor): https://docs.docker.com/registry/
The other option is to disable the image pinning, and manually copy images to each of your swarm nodes. You need to do this in advance of deploying any service changes. You'll also lose the benefit of reused image layers when you manually copy them. Because of all this issues it creates, overhead to manage, and risk of mistakes, I'd recommend against this option.
Run the docker stack deploy command with --with-registry-auth that will give the Workers access to pull the needed image
By default Docker Swarm will pull the latest image from registry when deploying
Similarly to the user in this question: How do I run private docker images on Google Container Engine:
I'd like to non-dockerhub docker images in GKE.
I would prefer to not have to explicitly prefix my images with a docker IP address.
This is in a live cluster, so re-installing nodes isnt really a good idea.
Is there a way I can fire up new nodes in the cluster with a configuration that adds a new default search path for a registry to the docker daemons on the kubelets, or similar workaround.