I have a docker container with Trivy installed.
I have a remote registry with docker images.
and
I would like to download the docker images to the container for scanning
Challenges
It is hard to run docker within a docker container for pulling the images.
Trivy requires that you have the images locally before it can scan the images, either in a local registry or as a file.
I found two solutions:
Download the images with Skopeo
Download the images with the HTTP API V2
For the API I had a hard time making the authentication work, as it is repository specific, and Scaleways' authentication had unexpected behaviour.
Related
Problem
I would like to do the following:
Create an image with Trivy and Docker installed
Run the container on Kubernetes
Schedule a CRON job to pull all images from a container registry and to scan them, and output the results to stdout
Question
Is there a smart way to do this? Can you even install docker within a docker container?
I ended up doing the following:
Creating an image with Trivy and Skopeo installed
Downloading the docker images with Skopeo
skopeo copy --src-creds=user:password --dest-compress --src-tls-verify=false docker://myrepo.com/mynamespace/ubuntu:latest oci:ubuntu
Scanning the image
trivy image --input ubuntu
An alternative to Skopeo would be to use the Registry HTTP API to download images as suggested by David Maze.
I have something on my mind that is bugging me. When running docker images I see a list of my local images I have in my docker environment. When pulling Images I pull it from a registry and more specific pull the specified tag managed by the repository.
so there is the registry as the big hub to store all image
repositories
and the repository is storing commits/tagged versions of a specific image
But what is docker images then? It's a registry as well isn't it? It holds all images that I've built locally or pulled.
If my claim is valid:
How does it comply with running a private registry (mentioned here https://docs.docker.com/registry/deploying/)
Running this docker run -d -p 5000:5000 --restart=always --name registry registry:2
Would deploy this new registry into my docker images...
So now I have a registry within my registry... registception?
What is the difference besides the custom registry is deployable?
Its not a local image registry as other questions have pointed. It is an image cache. The purpose of the image cache is to avoid having every time to download the same image whenever you do a docker run.
docker images simply lists all the cached images on the machine. Whenever there is newer image on the registry, the image(some layers) are downloaded and cached when doing docker pull .... Also, when a layer exists in the local cache, docker tells you that, example:
Step 2/2 : CMD /bin/bash
---> Using cache
On the other hand, a docker registry is a central repository to store images. It provide a remote api to pull and push images. The local image cache does not have this feature. Images in the local cache are read and stored used local docker commands that simply read files under /var/lib/docker/...
To make things clear, think of Docker remote registries (such as Docker Hub) as the remote Git repositories. You pull Docker images (like git repositories) that you need and you play with it.
Like remote Git repositories such as GitHub\BitBucket, Docker registries are also public and private. Public registries are for public usage and open-source projects. Examples include in like Docker Hub. Where as private registries are for organizational use or for your own. Examples for private registries include Azure Container Registry, EC2 Container Registry etc.
The official Docker Registry image is just a Docker registry image for your own system, you can't share them with others unless you have a server or a public Internet IP address. Think of it as Bonobo Private Git Server for Windows.
Your local image registry as you mentioned are all those images that you have build locally or pulled from a registry public or private you can see it like a local cache of images that you can re use without download or rebuild each time.
Running the registry what actually does is to spin up a server that implements the Docker Registry API which allows users to push, pull, delete and handles the storage of this images and their layers. See it like a central repository like npm, nexus
For example if you run the registry in your.registry.com:5000
You can do things like
docker build -t your.registry.com:5000/my-image:tag .
docker push your.registry.com:5000/my-image:tag
So others that have access to your server can pull it
docker pull your.registry.com:5000/my-image:tag
I have successfully built some Docker images:
Now I would like to start my microservices by docker-compose, unfortunatelly I am unable to pull those images i.e. repository callista/discovery-server not found: does not exist or no pull access I solved this error by logging into my DockerHub account and pushining those images to remote server. But it seems to me like a little overkill to send such larges images (which are likely to change pretty soon) over the Internet over and over again twice (push&pull).
Is it possible to configure Docker to install those images locally and not to pull from remote server?
I use Docker 1.8 and work on Windows 10.
Do you need to run this images in a server different from the one you build then?
If you need you have some alternatives:
As #engineer-dollery said, you can run a registry into your network, than you would not need to send it over the internet, only in your network. Docs: https://docs.docker.com/registry/deploying/
You could use the docker save and docker import to move then around too. Docs: https://docs.docker.com/engine/reference/commandline/save/
But if the server you run the images is the same you build then...
...than you could just add the tag image to your docker-compose services, and do a docker-compose build, as #lauri said, but with the image docker-compose will create a image with that name after the build, and then you could do docker run using than. Or do a docker-compose up --build so it will always build than again if something changes into the Dockerfile
If you define build option in docker-compose.yml, you should be able to build images locally with Docker Compose and then it uses those images without pulling. By default Docker Compose builds images if they are not found locally. If you want to rebuild images just add --build option docker-compose up command docker-compose up --build
Docker Compose build reference:
https://docs.docker.com/compose/compose-file/#build
I'm wondering where Docker's images are exactly stored to in my local host machine.
Can I share my Docker-Image without using the Docker-Hub or a Dockerfile but the 'real' Docker-Image? And what is exactly happening when I 'push' my Docker-Image to Docker-Hub?
Docker images are stored as filesystem layers. Every command in the Dockerfile creates a layer. You can also create layers by using docker commit from the command line after making some changes (via docker run probably).
These layers are stored by default under /var/lib/docker. While you could (theoretically) cherry pick files from there and install it in a different docker server, is probably a bad idea to play with the internal representation used by Docker.
When you push your image, these layers are sent to the registry (the docker hub registry, by default… unless you tag your image with another registry prefix) and stored there. When pulling, the layer id is used to check if you already have the layer locally or it needs to be downloaded. You can use docker history to peek at which layers (other images) are used (and, to some extent, which command created the layer).
As for options to share an image without pushing to the docker hub registry, your best options are:
docker save an image or docker export a container. This will output a tar file to standard output, so you will like to do something like docker save 'dockerizeit/agent' > dk.agent.latest.tar. Then you can use docker load or docker import in a different host.
Host your own private registry. - Outdated, see comments See the docker registry image. We have built an s3 backed registry which you can start and stop as needed (all state is kept on the s3 bucket of your choice) which is trivial to setup. This is also an interesting way of watching what happens when pushing to a registry
Use another registry like quay.io (I haven't personally tried it), although whatever concerns you have with the docker hub will probably apply here too.
Based on this blog, one could share a docker image without a docker registry by executing:
docker save --output latestversion-1.0.0.tar dockerregistry/latestversion:1.0.0
Once this command has been completed, one could copy the image to a server and import it as follows:
docker load --input latestversion-1.0.0.tar
Sending a docker image to a remote server can be done in 3 simple steps:
Locally, save docker image as a .tar:
docker save -o <path for created tar file> <image name>
Locally, use scp to transfer .tar to remote
On remote server, load image into docker:
docker load -i <path to docker image tar file>
[Update]
More recently, there is Amazon AWS ECR (Elastic Container Registry), which provides a Docker image registry to which you can control access by means of the AWS IAM access management service. ECR can also run a CVE (vulnerabilities) check on your image when you push it.
Once you create your ECR, and obtain the "URL" you can push and pull as required, subject to the permissions you create: hence making it private or public as you wish.
Pricing is by amount of data stored, and data transfer costs.
https://aws.amazon.com/ecr/
[Original answer]
If you do not want to use the Docker Hub itself, you can host your own Docker repository under Artifactory by JFrog:
https://www.jfrog.com/confluence/display/RTF/Docker+Repositories
which will then run on your own server(s).
Other hosting suppliers are available, eg CoreOS:
http://www.theregister.co.uk/2014/10/30/coreos_enterprise_registry/
which bought quay.io
Docker remote api v1.6.:
I am using it to POST and build dockerfile. I can see my newly built images in the json image list output by same remote api, http://docs.docker.io/en/latest/api/docker_remote_api_v1.6/#id20 . During build i'm also tagging it with repository name, http://docs.docker.io/en/latest/api/docker_remote_api_v1.6/#id30.
To push this so generated image on the registry am using docker cli. But the reply from docker cli says
"No such id...".
Since am using docker cli to pull images as well i can't pull it as well.
Please help solving this sync issue in reflecting changes make through docker remote api's into docker cli result?