We are using Azure IoT Edge in a nested edge configuration for devices at L3 with parents at our top level L4.
Regarding the top level docker registry module we have deployed at the L4 parents, this appears to be wide open and we’re concerned with anyone with access to the L4 port the registry is running on being able to read, write & delete images from this repository.
We have deployed the top level docker registry container as outlined here.
I know the registry “delete” option is disabled by default, but I didn’t see an obvious way to make the registry module read only without possibly breaking the caching feature used by IoT edge?
I also saw there is a "connected registry" in the Azure Container Registry which looks like a preview feature that may address some of our security concerns but we are using JFrog as our registry currently that our top level access to pull images from so switching to Azure container registry won't work.
How can we secure the top level docker registry module? If we add authentication to it, how do we make sure lower level edge devices can still pull images from $upstream?
Related
We have a Docker Registry running that uses native basic authentication with nginx, so images can only be pushed to the Registry after authentication. Is it possible to get the user who pushed the image to the Registry?
It's not part of the registry API. You would need to check the logs of that registry and auth server. It's possible the user may self report who they are by setting a label on the image (or the legacy maintainer field), but I wouldn't depend on that for any security critical tasks.
For more on the registry API, see: https://github.com/opencontainers/distribution-spec
Docker also has their API (which predates OCI) documented at: https://docs.docker.com/registry/spec/api/
This is a bit of a silly setup, but here's what I'm looking at right now:
I'm learning Kubernetes
I want to push custom code to my Kubernetes cluster, which means the code must be available as a Docker image available from some Docker repository (default is Docker Hub)
While I'm willing to pay for Docker Hub if I have to (though I'd rather avoid it), I have concerns about putting my custom code on a third-party service. Sudden rate limits, security breaches, sudden ToS changes, etc
To this end, I'm running my own Docker registry within my Kubernetes cluster
I do not want to configure the Docker clients running on the Kubernetes nodes to trust insecure (HTTP) Docker registries. If I do choose to pull any images from an external registry (e.g. public images like nginx I may pull from Docker Hub instead of hosting locally) then I don't want to be vulnerable to MITM attacks swapping out the image
Ultimately I will have a build tool within the cluster (Jenkins or otherwise) pull my code from git, build the image, and push it to my internal registry. Then all nodes pulling from the registry live within the cluster. Since the registry never needs to receive images from sources outside of the cluster or delivery them to sources outside of the cluster, the registry does not need a NodePort service but can instead be a ClusterIP service.... ultimately
Until I have that ultimate setup ready, I'm building images on my local machine and wish to push them to the registry (from the internet)
Because I don't plan on making the registry accessible from the outside world (eventually), I can't utilize Let's Encrypt to generate valid certs for it (even if I were making my Docker registry available to the outside world, I can't use Let's Encrypt, anyway without writing some extra code to utilize certbot or something)
My plan is to follow the example in this StackOverflow post: generate a self-signed cert and then launch the Docker registry using that certificate. Then use a DaemonSet to make this cert trusted on all nodes in the cluster.
Now that you have the setup, here's the crux of my issue: within my cluster my Docker registry can be accessed via a simple host name (e.g. "docker-registry"), but outside of my cluster I need to either access it via a node IP address or a domain name pointing at a node or a load balancer.
When generating my self-signed cert I was asked to provide a CN / FQDN for the certificate. I put in "docker-registry" -- the internal host name I plan to utilize. I then tried to access my registry locally to push an image to it:
> docker pull ubuntu
> docker tag ubuntu example.com:5000/my-ubuntu
> docker push example.com:5000/my-ubuntu
The push refers to repository [example.com:5000/my-ubuntu]
Get https://example.com:5000/v2/: x509: certificate is valid for docker-registry, not example.com
I can generate a certificate for example.com instead of for docker-registry, however I worry that I'll have issues configuring the service or connecting to my registry from within my cluster if I provide my external domain like this instead of an internal host name.
This is why I'm wondering if I can just say that my self-signed cert applies to both example.com and docker-registry. If not, two other acceptable solutions would be:
Can I tell the Docker client not to verify the host name and just trust the certificate implicitly?
Can I tell the Docker registry to deliver one of two different certificates based on the host name used to access it?
If none of the three options are possible, then I can always just forego pushing images from my local machine and start the process of building images within the cluster -- but I was hoping to put that off until later. I'm learning a lot right now and trying to avoid getting distracted by tangential things.
Probably the easiest way to solve your problem would be to use Docker's insecure-registry feature. The concern you mention about this in your post (that it would open you up to security risks later) probably won't apply as the feature works by specifying specific IP addresses or host names to trust.
For example you could configure something like
{
"insecure-registries" : [ "10.10.10.10:5000" ]
}
and the only IP address that your Docker daemons will access without TLS is the one at that host and port number.
If you don't want to do that, then you'll need to get a trusted TLS certificate in place. The issue you mentioned about having multiple names per cert is usually handled with the Subject Alternative Name field in a cert. (indeed Kubernetes uses that feature quite a bit).
Maybe a simple-to-answer Question: How can I set up a private docker reposiory and limit the Access to only a subset of the Images there that one can pull ? E.g. I have Image1 and Image2 pushed, but want to allow one Image2 being pullable by account USER1 ?
This tends to get into the commercial offerings of docker (DTR). The spec itself for the registry includes all of the capabilities for auth, and you can configure a simple htpasswd based login on the standalone registry. However for the next step up, you get into a token server which docker doesn't have an open source implementation of themselves. You could work around this limitation by deploying multiple registry servers, each with a different set of users in a htpasswd file.
There are various third party implementations of the docker registry that may include these features. In the open source space, there's a project called cesanta/docker_auth that works with docker's stand alone registry and does exactly what you're looking for. The next step up is the harbor project that should be all most organizations need from a registry, but may be more complicated and have more overhead for a small project.
I just read the entire docs on securing a private Docker Registry. In addition to this, there seems to be a "Docker Trusted Registry", which is described as:
Docker Trusted Registry (DTR) lets you run and manage your own Docker image storage service, securely on your own infrastructure behind your company firewall.
Furthermore, the doc goes on to list a DTRs features:
An image registry to store, manage, and collaborate on Docker images
Pluggable storage drivers
Configuration options to let you run DTR in your particular enterprise environment.
Easy, transparent upgrades
Logging, usage and system health metrics
But doesn't the "normal" Docker Registry give me these as well?!?
Are these two things really the same, or is DTR some sort of commercialized offering of a Docker Registry? Or something else? I'm so confused!
New features in Docker Trusted Registry are:
Control access and permissions by user or organisation
Web UI to search and browse repos, manage users and setting
Integrate to CI and CD systems to automate workflows
LDAP/AD integration
Flexible storage
support User audit logs
Soft Delete image tags
Garbage collection
DTR is the paid support service for the registry - see https://hub.docker.com/enterprise/
Basically, they help you setup and will give you support down the road. It might have some niceties, otherwise you are left to your own to figure out how to run the registry.
What is currently the recommended way to mirror a Private Docker Registry?
Mirroring functionality is provided by official docker-registry image but only for the Public Registry.
See documentation:
"Beware that mirroring only works for the public registry. You can not create a mirror for a private registry."
My use-case:
A bigger development team that is working in an office with a limited network. They only pull docker images from registries. Pushing is occasional and handled by Jenkins box hosted in AWS. Most of the images they use resides in our password protected Private Registry (served over https). So it's only natural to mirror/cache the Registry on a machine in a local network. If not for https I would just go for HTTP_PROXY and local squid install.
I'm sure I'm not the only one solving docker dev bandwidth problem. What do you do?
It is now possible to do this with the "proxy" settings in the configuration for a V2 registry. Just put up another registry (on a different server/port from any other private registry you have) and on every docker engine, set the '--registry-mirror' flag to point to it.
Just watch out for accidental pushes - always retag your images to the private registry or a private repository if you wish to keep them private.
Right now, I would recommend using the (new) golang registry (https://github.com/docker/distribution) instead of the (v1) python one, and go with the proxy solution (using HTTP_PROXY + a reverse proxy cache - squid, or whatever else pleases your tastes - I would probably use varnish).
Native support for "mirroring" built into the registry itself will come eventually, and later more flexible transports.