Can a docker registry populated on one host be 'tree-copied' to another machine and be 'turned on' as a pre-populated docker registry served by the new host?
I am working on a project providing Platform-as-a-Service which includes a docker registry service. These run in disconnected environments (not connected to the Internet). One very time consuming aspect of each deployment is creating an empty registry and loading, tagging, and pushing hundreds of docker images (tens of gigabytes of data) from a compressed tar into the registry for each new deployment.
I am thinking it would be faster to do this differently. Instead of a tarball of docker files, could we at 'build time' create and populate the docker registry then and compress that. At deploy time we just unpack the registry into /var/lib/registry or wherever...
But, I don't know if any of the data in the registry is dependent upon, say, a machine ID, certificate, or other aspect of the host upon which the registry was first running.
It seems to me an equivalent question is, if I populate two docker registries running on different machines with the same set of docker images in the same order, will the file contents of the registry folder be the same (or similar, allowing for timestamps and such?)
Every time I search for "docker registry transfer" or "move docker registry to new machine" or similar terms, I am flooded with answers about moving single docker images from one machine or registry to another, but don't see anything about docker registry migration or portability.
I haven't had the time or resources to test this out; maybe someone already expert in docker registry structures could clue me in that this is practical (or can say it absolutely will not work) I can make a better decision about whether to pursue getting the time and machines to demonstrate this approach.
Thank you.
I don't know if any of the data in the registry is dependent upon,
say, a machine ID, certificate, or other aspect of the host upon which
the registry was first running.
The configuration will be associated to the registry, for example, if you are running a secure registry by adding certs to the registry.
In this case, you will have to configure the registry in the same manner on a new machine as on the previous machine, use a configuration manager(like ansible) for that.
Instead of a tarball of docker files, could we at 'build time' create
and populate the docker registry then and compress that. At deploy
time we just unpack the registry into /var/lib/registry or wherever...
Adding to what #DazWilkin already mentioned in the comments, a storage location can be configured which can be,
filesystem: the rootdirectory default is /var/lib/registry
based on the cloud provider if the registry is deployed on a private cloud
Example, S3 bucket for AWS
You can take backup of that like the rootdirectory in case of filesystem or attach the storage location to the new registry.
**Words of caution**
Try to use the exact configuration and version of the docker registry.
Related
I am trying to build a multi-arch image but would like to avoid pushing it to docker hub. I've had a lot of trouble finding out how to control the export options. is there a way to make "--push" push to a registry of my choosing?
Any help is appreciated
Docker provides a container image for a registry server that you may self run even on localhost, see: Deploying a registry server.
There are other servers|services that implement the registry API (see below) but this is a good place to start.
Conventionally, images pushed|pulled default to Docker registry; unless a registry is explicitly specifed, an image e.g. your-image:your-tag defaults to docker.io/my-image:my-tag. In my opinion, it's a good practice to always include this default to be more transparent about this.
If you run Docker's registry image on localhost on the default port 5000, you'll need to take your images with localhost:5000/your-image:your-tag to ensure that when you docker push localhost:5000/your-image:your-tag, the CLI is able to determine your local registry is the intended destination.
Similarly, if you use e.g. Quay registry, images must be prefixed quay.io, Google Artifact Registry, images are prefixed ${REGION}-docker.pkg.dev/${PROJECT}/${REPOSITORY} etc.
IIRC it's not possible to push to Docker's registry (aka dockerhub) without an account so, as long as you ensure you're not logged in, you should not accidentally push images to Docker's registry.
NOTE You only need to use a registry to ease distribution of container images between machines. If you're only interested in local(host) development, you can docker run ... immediately after a successful docker build without any pushing|pulling (beyond interim images, e.g. FROM).
I've set up a Docker Swarm consisting of two VMs on my local network, (1 manager, 1 worker). In the manager node, I created a private registry service and I want to deploy a number of locally built images in my local dev machine (which is not in the swarm) to that registry. The Swarm docs and the dozens of examples I've read in the Internet seem not to go beyond the basics, running commands inside the manager node, building, tagging and pushing images from the manager's local cache to the registry in that same node, and I have that uneasy feeling that I'm missing something right on my face.
I see that my machine could simply join the swarm as a manager, owning the registry. The other nodes would automagically receive the updates and my problem would go away. But does this make sense for a production swarm setting, a cluster of nodes serving production code, depending on my dev's home machine - even as non-worker, manager-only?
Things I've tried:
Retagging my local image to <my.node.manager.ip>/my_app:1.0.0, followed by docker-compose push. I can see this does push the image to the manager's registry, but the service fails to start with the message "No such image: <my.node.manager.ip>/my_app:1.0.0"
Creating a context and, from my machine, run docker-compose --context my_context up --no-start. This (re)creates the image in the manager node's local cache, which I can then push to the registry, but it feels very unwieldy as a deploy process.
Should I run a remote script in the manager node to git pull my code and then do the build/push/docker stack deploy?
TL;DR What's the expected steps to deploy an image/app to a Docker Swarm from a local dev machine outside the swarm? Is this possible? Is this supported by Docker Swarm?
After reading a bit more on private registries and tags, I could finally wrap my head around the necessary tagging for my use case to work. My first approach was halfway right, but I had to change my deploy script so as to:
extract the image field from my docker-compose.yml (in the form localhost:5000/my_app:${MY_APP_VERSION-latest}, to circumvent the "No such image" error)
create a second tag for pushing to the remote registry, replacing "localhost" by my manager node address (where the registry is at)
Tag my locally built image with that tag and docker-compose push it
Deploy the app with docker --context <staging|production> stack deploy my_app
I'm answering myself since I did solve my original problem, but would love to see other DevOps implementations for similar scenarios.
I am looking to a solution to fasten development with docker images and their testing.
Does it exist a solution that would expose all tagged images from a host through a docker registry API ?
This would allow to do:
docker build
List item
docker tags
NOT populating a registry
and have the docker image already available, and usable in kubernetes for example.
I am surprise this doesn't already exists.
How do people iterate on docker images (fixing config, checking everything deploy correctly, etc ...) ?
I was used to do it locally with docker-compose, but using Kubernetes makes a registry mandatory and seems to complicate things a bit more.
I have few docker images composed together in the stack using docker-compose.yml.
Now I want to transfer whole docker compose stack to the other host machine without uploading to the dockerhub,
And deploy it on the docker swarm.
I saw there is a thing called docker compose bundle, would that help?
If you’re deploying on a multi-host swarm (or something similar like Kubernetes or Nomad) you all but need a Docker registry. It doesn’t specifically have to be Docker Hub — quay.io, Amazon’s ECR, Google’s GCR, and self-hosted registries all work fine — but you do need to have pushed the built images somewhere where the orchestrator can retrieve them by name.
I’ve never used docker-compose bundle myself, but its documentation also notes that its operation “requires interaction with a Docker registry”.
The only real alternative is using docker save and docker load to manually move images between machines, but as a manual process it will get tedious very quickly, and you need to make sure an identical set of images are on every machine for consistency. Using a registry will be vastly easier.
The easyest way to do it is to use a Docker registry. The problem with Docker Hub is that you can only have one private registry, the rest must be public or paid.
Thankfully, there are other (free) alternatives:
Deploy your own private registry. Here is a nice tutorial where you can try it in the browser.
Use a free private registry. I personnaly use Codefresh. It can automatically build your image from a private repo (like bitbucket who has free plan too), but you can also just use it like a "simple" docker registry and push and pull your Docker images there.
I'm wondering where Docker's images are exactly stored to in my local host machine.
Can I share my Docker-Image without using the Docker-Hub or a Dockerfile but the 'real' Docker-Image? And what is exactly happening when I 'push' my Docker-Image to Docker-Hub?
Docker images are stored as filesystem layers. Every command in the Dockerfile creates a layer. You can also create layers by using docker commit from the command line after making some changes (via docker run probably).
These layers are stored by default under /var/lib/docker. While you could (theoretically) cherry pick files from there and install it in a different docker server, is probably a bad idea to play with the internal representation used by Docker.
When you push your image, these layers are sent to the registry (the docker hub registry, by default… unless you tag your image with another registry prefix) and stored there. When pulling, the layer id is used to check if you already have the layer locally or it needs to be downloaded. You can use docker history to peek at which layers (other images) are used (and, to some extent, which command created the layer).
As for options to share an image without pushing to the docker hub registry, your best options are:
docker save an image or docker export a container. This will output a tar file to standard output, so you will like to do something like docker save 'dockerizeit/agent' > dk.agent.latest.tar. Then you can use docker load or docker import in a different host.
Host your own private registry. - Outdated, see comments See the docker registry image. We have built an s3 backed registry which you can start and stop as needed (all state is kept on the s3 bucket of your choice) which is trivial to setup. This is also an interesting way of watching what happens when pushing to a registry
Use another registry like quay.io (I haven't personally tried it), although whatever concerns you have with the docker hub will probably apply here too.
Based on this blog, one could share a docker image without a docker registry by executing:
docker save --output latestversion-1.0.0.tar dockerregistry/latestversion:1.0.0
Once this command has been completed, one could copy the image to a server and import it as follows:
docker load --input latestversion-1.0.0.tar
Sending a docker image to a remote server can be done in 3 simple steps:
Locally, save docker image as a .tar:
docker save -o <path for created tar file> <image name>
Locally, use scp to transfer .tar to remote
On remote server, load image into docker:
docker load -i <path to docker image tar file>
[Update]
More recently, there is Amazon AWS ECR (Elastic Container Registry), which provides a Docker image registry to which you can control access by means of the AWS IAM access management service. ECR can also run a CVE (vulnerabilities) check on your image when you push it.
Once you create your ECR, and obtain the "URL" you can push and pull as required, subject to the permissions you create: hence making it private or public as you wish.
Pricing is by amount of data stored, and data transfer costs.
https://aws.amazon.com/ecr/
[Original answer]
If you do not want to use the Docker Hub itself, you can host your own Docker repository under Artifactory by JFrog:
https://www.jfrog.com/confluence/display/RTF/Docker+Repositories
which will then run on your own server(s).
Other hosting suppliers are available, eg CoreOS:
http://www.theregister.co.uk/2014/10/30/coreos_enterprise_registry/
which bought quay.io