I am trying to run a Nix-built Docker image in tarball form. With docker, docker load -i <path> followed by a docker run works fine. Now I've uploaded the tarball to Artifactory and am trying to run the image on K8s with something like:
$ kubectl run foo-service --image=<internal Artifactory>/foo-service/foo-service-latest.tar.gz
However all I see is:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
foo-service 1 1 1 0 2h
Is it possible to load an image from a (remote) tarball in K8s? If yes, what is the command to do so?
There is no way to do that directly in Kubernetes.
You can do docker load and then docker push to a registry (you can host a private registry in Kubernetes or use a public one) and after that kubectl run.
Minikube also has a registry addon for local development.
Related
A word of warning, this is my first posting, and I am new to docker and Kubernetes with enough knowledge to get me into trouble.
I am confused about where docker container images are being stored and listing images.
To illustrate my confusion I start with the confirmation that "docker images" indicates no image for nginx is present.
Next I create a pod running nginx.
kubectl run nginx --image=nginx is succesful in pulling image "nginx" from github (or that's my assumption):
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8s default-scheduler Successfully assigned default/nginx to minikube
Normal Pulling 8s kubelet Pulling image "nginx"
Normal Pulled 7s kubelet Successfully pulled image "nginx" in 833.30993ms
Normal Created 7s kubelet Created container nginx
Normal Started 7s kubelet Started container nginx
Even though the above output indicates the image is pulled, issuing "docker images" does not include nginx the output.
If I understand correctly, when an image is pulled, it is being stored on my local disk. In my case (Linux) in /var/lib/docker.
So my first question is, why doesn't docker images list it in the output, or is the better question where does docker images look for images?
Next if I issue a docker pull for nginx it is pulled from what I assume to be Github. docker images now includes it in it's output.
Just for my clarification, nothing up to this point involves a private local registry, correct?
I purposefully create a basic local Docker Registry using the docker registry container thinking it would be clearer since that will allow me to explicitly specify a registry but this only results in another issue:
docker run -d \
-p 5000:5000 \
--restart=always \
--name registry \
-v /registry:/var/lib/registry \
registry
I tag and push the nginx image to my newly created local registry:
docker tag nginx localhost:5000/nginx:latest
docker push localhost:5000/nginx:latest
The push refers to repository [localhost:5000/nginx]
2bed47a66c07: Pushed
82caad489ad7: Pushed
d3e1dca44e82: Pushed
c9fcd9c6ced8: Pushed
0664b7821b60: Pushed
9321ff862abb: Pushed
latest: digest: sha256:4424e31f2c366108433ecca7890ad527b243361577180dfd9a5bb36e828abf47 size: 1570
I now delete the original nginx image:
docker rmi nginx
Untagged: nginx:latest
Untagged: nginx#sha256:9522864dd661dcadfd9958f9e0de192a1fdda2c162a35668ab6ac42b465f0603
... and the newely tagged one:
docker rmi localhost:5000/nginx
Untagged: localhost:5000/nginx:latest
Untagged: localhost:5000/nginx#sha256:4424e31f2c366108433ecca7890ad527b243361577180dfd9a5bb36e828abf47
Deleted: sha256:f652ca386ed135a4cbe356333e08ef0816f81b2ac8d0619af01e2b256837ed3e
... but from where are they being deleted?
Now the image nginx should only be present in localhost:5000/? But docker images doesn't show it in it's output.
Moving on, I try to create the nginx pod once more using the image pushed to localhost:5000/nginx:latest.
kubectl run nginx --image=localhost:5000/nginx:latest --image-pull-policy=IfNotPresent
This is the new issue. The connection to localhost:5000 is refused.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 1s kubelet Pulling image "localhost:5000/nginx:latest"
Warning Failed 1s kubelet Failed to pull image "localhost:5000/nginx:latest": rpc error: code = Unknown desc = Error response from daemon: Get "http://localhost:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused
Warning Failed 1s kubelet Error: ErrImagePull
Normal BackOff 0s kubelet Back-off pulling image "localhost:5000/nginx:latest"
Why is it I can pull and push to localhost:5000, but pod creation fails with what appears to be an authorization issue? I try logging into the registry but no matter what I use for the username and user password, login is successful. This confuses me more.
I would try creating/specifying imagePullSecret, but based on docker login outcome, it doesn't make sense.
Clearly I not getting it.
Someone please have pity on me and show where I have lost my way.
I will try to bring some clarity to you despite the fact your question already contains about 1000 questions (and you'll probably have 1000 more after my answer :D)
Before you can begin to understand any of this, you need to learn a few basic things:
Docker produces images which are used by containers - it similar to Virtual Machine, but more lightweight (I'm oversimplifying, but the TL;DR is pretty much that).
Kubernetes is an orchestration tool - it is responsible for starting containers (by using already built images) and tracking their state (i.e. if this container has crashed it should be restarted, or if it's not started it should be started, etc)
Docker can run on any machine. To be able to start a container you need to build an image first. The image is essentially a lightweight mini OS (i.e. alpine, ubuntu, windows, etc) which is configured with only those dependencies you need to run your application. This image is then pushed to a public repository/registry (hub.docker.com) or to a private one. And afterwards it's used for starting containers.
Kubernetes builds on top of this and adds the "automation" layer which is responsible for scheduling and monitoring the containers. For example, you have a group of 10 servers all running nginx. One of those servers restarts - the nginx container will be automatically started by k8s.
A kubernetes cluster is the group of physical machines that are dedicated to the mentioned logical cluster. These machines have labels or tags which define the purpose of physical node and work as a constraint for where a container will be scheduled.
Now that I have explained the minimum basics in an oversimplified way I can move with answering your questions.
When you do docker run nginx - you are instructing docker to pull the nginx image from https://hub.docker.com/_/nginx and then start it on the machine you executed the command on (usually your local machine).
When you do kubectl run nginx --image=nginx - you are instructing Kubernetes to do something similar to 1. but in a cluster. The container will be deployed to a random machine somewhere in the cluster unless you put a nodeSelector or configure affinity. If you put a nodeSelector this container (called Pod in K8S) will be placed on that specific node.
You have started a private registry server on your local machine. It is crucial to know that localhost inside a container will point to the container itself.
It is worth mentioning that some of the kubernetes commands will create their own container for the execution phase of the command. (remember this!)
When you run kubectl run nginx --image=nginx everything works fine, because it is downloading the image from https://hub.docker.com/_/nginx.
When you run kubectl run nginx --image=localhost:5000/nginx you are telling kubernetes to instruct docker to look for the image at localhost which is ambiguous because you have multiple layers of containers running (check 4.). This means the command that will do docker pull localhost:5000/nginx also runs in a docker container -- so there is no service running at port :5000 (the registry is running in a completely different isolated container!) :D
And this is why you are getting Error: ErrImagePull - it can't resolve localhost as it points to itslef.
As for the docker rmi nginx and docker rmi localhost:5000/nginx commands - by running them you removed your local copy of the nginx images.
If you run docker run localhost:5000/nginx on the machine where you started docker run registry you should get a running nginx container.
You should definitely read the Docker Guide BEFORE you try to dig into Kubernetes or nothing will ever make sense.
Your head will stop hurting after that I promise... :D
TL;DR
docker images lists images stored in the docker daemon's data root, by default /var/lib/docker.
You're deploying images to Kubernetes, the images are pulled onto the node on which the pod is scheduled. For example, using Kubernetes in Docker:
kind create cluster
kubectl run nginx --image=nginx
docker exec -it $(kubectl get pod nginx -o jsonpath={.spec.nodeName}) crictl images
crictl is a command-line interface for CRI-compatible container runtimes.
Docker images are pulled from Docker Hub by default, not Github. When using a local docker registry, images are stored in the registry's data volume. The docker registry storage may be customized, by default data is stored in (storage.filesystem.rootdirectory) /var/lib/registry.
You can use tools like skopeo to list images stored in a docker registry, for example:
skopeo list-tags docker://localhost:5000/nginx --tls-verify=false
I am very lost on the steps with gcloud verse docker. I have some gradle code that built a docker image and I see it in images like so
(base) Deans-MacBook-Pro:stockstuff-all dean$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/prod-stock-bot/stockstuff latest b041e2925ee5 27 minutes ago 254MB
I am unclear if I need to run docker push or not or if I can go strait to gcloud run deploy so I try a docker push like so
(base) Deans-MacBook-Pro:stockstuff-all dean$ docker push gcr.io/prod-stockstuff-bot/stockstuff
Using default tag: latest
The push refers to repository [gcr.io/prod-stockstuff-bot/stockstuff]
An image does not exist locally with the tag: gcr.io/prod-stockstuff-bot/stockstuff
I have no idea why it says the image doesn't exist locally when I keep listing the image. I move on to just trying gcloud run deploy like so
(base) Deans-MacBook-Pro:stockstuff-all dean$ gcloud run deploy stockstuff --project prod-stock-bot --region us-west1 --image gcr.io/prod-stockstuff-bot/stockstuff --platform managed
Deploying container to Cloud Run service [stockstuff] in project [prod-stock-bot] region [us-west1]
X Deploying... Image 'gcr.io/prod-stockstuff-bot/stockstuff' not found.
X Creating Revision... Image 'gcr.io/prod-stockstuff-bot/stockstuff' not found.
. Routing traffic...
Deployment failed
ERROR: (gcloud.run.deploy) Image 'gcr.io/prod-stockstuff-bot/stockstuff' not found.
I am doing this all as a playground project and can't seem to even get a cloud run deploy up and running.
I tried the spring example but that didn't even create a docker image and it failed anyways with
ERROR: (gcloud.run.deploy) Missing required argument [--image]: Requires a container image to deploy (e.g. `gcr.io/cloudrun/hello:latest`) if no build source is provided.
This error occurs when an image is not tagged locally/correctly. Steps you can try on your side.
Create image locally with name stockstuff (do not prefix it with gcr and project name while creating).
Tag image with gcr repo detail
$ docker tag stockstuff:latest gcr.io/prod-stockstuff-bot/stockstuff:latest
Check if your image is available in GCR (must see your image here, before deploying on cloudrun).
$ gcloud container images list --repository gcr.io/prod-stockstuff-bot
If you can see your image in list, next you can try to deploy gcloud run with below command (as yours).
gcloud run deploy stockstuff --project prod-stock-bot --region us-west1 --image gcr.io/prod-stockstuff-bot/stockstuff --platform managed
There are 3 contexts that you need to be aware.
Your local station, with your own docker.
The cloud based Google Container Registry: https://console.cloud.google.com/gcr/
Cloud Run product from GCP
So the steps would be:
Build your container either locally or using Cloud Build
Push the container to the GCR registry,
if you built locally
docker tag busybox gcr.io/my-project/busybox
docker push gcr.io/my-project/busybox
Deploy to Cloud Run a container from Google Cloud Repository.
I don't see this image gcr.io/prod-stockstuff-bot/stockstuff when you list images in the local system. You can create a new image by tagging that image with this image gcr.io/prod-stock-bot/stockstuff and re-run the gcloud run command.
for the context I am using Flask (python)
I solved this by
update gcloud-sdk to the latest version
gcloud components update
add .dockerignore, I'm guessing because of the python cache
Dockerfile
README.md
*.pyc
*.pyo
*.pyd
__pycache__
.pytest_cache
expose the port to env $PORT
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 app:app
Just testing on local machine. Windows 7 x64, Minikube 1.14, docker toolbox.
$docker image ls does show the image I would like to use.
REPOSITORY myname/hello-service
TAG 0.0.6
IMAGE ID xxxxxxxxxxx
In my Pod yaml:
spec:
containers:
-name: my-pod
image: myname/hello-service:0.0.6
After running $kubectl create -f pod.yaml. It failed
Error: ImagePullBackOff
Failed to pull image "xxxxx" rpc error: code = ... manifest for myname/hello-service:0.0.6 not found
But the previous version :0.0.5 works just fine.
Both image are build on my machine and store in "default" of docker.
Can it be that myname/hello-service:0.0.6 is only on your windows host? If so, minikube cannot find it.
You have a few options to access in Minikube. One of them is building your local image with minikube's Docker daemon. Another is running a private local Docker registry.
A few examples for this and more I found are [well described here].(https://www.edureka.co/community/17481/local-docker-image-on-minikube)
Try to push it on DockerHub first
docker tag <imageid> <usrDockerHub>/<image_name>:<version>
docker push <usernameDockerHub>/<nome immagine>:<tag>
and try again kubectl create -f pod.yaml
It is easy to work with Openshift as a Container As A Service, see the detailed steps. So, via the docker client I can work with Openshift.
I would like to work on my laptop with Minishift. That's the local version of Openshift on your laptop.
Which docker registry should I use in combination with Minishift? Minishift doesn't have it's own registry - I guess.
So, I would like to do:
$ maven clean install -- building the application
$ oc login to your minishift environment
$ docker build -t myproject/mynewapplication:latest
$ docker tag -- ?? normally to a openshift docker registry entry
$ docker push -- ?? to a local docker registry?
$ on 1st time: $ oc new-app mynewapplication
$ on updates: $ oc rollout latest dc/mynewapplication-n myproject
I use just docker and oc cluster up which is very similar. The internal registry that is deployed has an address in the 172.30.0.0/16 space (ie. the default service network).
$ oc login -u system:admin
$ oc get svc -n default | grep registry
docker-registry ClusterIP 172.30.1.1 <none> 5000/TCP 14m
Now, this service IP is internal to the cluster, but it can be exposed on the router:
$oc expose svc docker-registry -n default
$oc get route -n default | grep registry
docker-registry docker-registry-default.127.0.0.1.nip.io docker-registry 5000-tcp None
In my example, the route was docker-registry-default.127.0.0.1.nip.io
With this route, you can log in with your developer account and your token
$oc login -u developer
$docker login docker-registry-default.127.0.0.1.nip.io -p $(oc whoami -t) -u developer
Login Succeeded
Note: oc cluster up is ephemeral by default; the docs can provide instructions on how to make this setup persistent.
One additional note is that if you want OpenShift to try to use some of it's native builders, you can simply run oc new-app . --name <appname> from within the your source code directory.
$ cat Dockerfile
FROM centos:latest
$ oc new-app . --name=app1
--> Found Docker image 49f7960 (5 days old) from Docker Hub for "centos:latest"
* An image stream will be created as "centos:latest" that will track the source image
* A Docker build using binary input will be created
* The resulting image will be pushed to image stream "app1:latest"
* A binary build was created, use 'start-build --from-dir' to trigger a new build
* This image will be deployed in deployment config "app1"
* The image does not expose any ports - if you want to load balance or send traffic to this component
you will need to create a service with 'expose dc/app1 --port=[port]' later
* WARNING: Image "centos:latest" runs as the 'root' user which may not be permitted by your cluster administrator
--> Creating resources ...
imagestream "centos" created
imagestream "app1" created
buildconfig "app1" created
deploymentconfig "app1" created
--> Success
Build scheduled, use 'oc logs -f bc/app1' to track its progress.
Run 'oc status' to view your app.
There is an internal image registry. You login to it and push images just like you suggest. You just need to know the address and what credentials you need. For details see:
http://cookbook.openshift.org/image-registry-and-image-streams/how-do-i-push-an-image-to-the-internal-image-registry.html
Is there a gcloud API or other command line interface (CLI) to access the list of published container images in the private Google Container Registry? (That is the container registry inside a Google Cloud Platform project)
gcloud container does not seem to help:
$ gcloud container
Usage: gcloud container [optional flags] <group | command>
group may be clusters | operations
command may be get-server-config
Deploy and manage clusters of machines for running containers.
flags:
--zone ZONE, -z ZONE The compute zone (e.g. us-central1-a) for the cluster
global flags:
Run `gcloud -h` for a description of flags available to all commands.
command groups:
clusters Deploy and teardown Google Container Engine clusters.
operations Get and list operations for Google Container Engine
clusters.
commands:
get-server-config Get Container Engine server config.
I also don't want to use gcloud docker to list images because this wants to connect to a particular docker daemon that I don't have. Unless there is a way to tell gcloud docker to connect to a remote public docker daemon that can read the private containers pushed to the registry through my project.
We just released a new command to list the images in your repository! You can try it out with:
gcloud alpha container images list --repository=gcr.io/$MYREPOSITORY
If you want to see the specific tags for an image you can use:
gcloud alpha container images list-tags gcr.io/$MYREPOSITORY/$MYIMAGE
The answer given by Robert Bailey is good for certain tasks, but might be missing what you specifically want to do. Nonetheless, your comments in reply to his answer are not so much faults of his answer as of your own understanding of what the commands which "fail" actually mean to do.
As far as your second comment,
Using docker I get the following error (for the reasons mentioned
above; I also edited the question): Cannot connect to the Docker daemon. Is the docker daemon running on this host?
This is a result of the docker daemon not running. Check if it's running via ps aux | grep docker. You can refer to the Docker documentation to determine how to properly install and run it.
As far as your first comment,
Using curl I get: {"errors":[{"code":"DENIED","message":"Failed to read tags for repository '<my_project>/<my_image>'"}]}. I have to
authenticate somehow to access the images in a private registry. I
don't want to use docker because that means I have to have a docker
daemon available. I only want to see if a container image with a
particular version is in the Container Registry. So what I need is an
API to the Container Registry in the Google Developer Console.
You wouldn't be able to curl the image unless it was public, as mentioned in Robert's latest comment, or unless you somehow provided some great oauth headers during the curl's invocation.
You should use gcloud docker to attempt to list the images in the registry, as you would for other docker registries. The gcloud container command group is the wrong one for your desired task. You can see below an output from gcloud version 96.0.0 (latest as of this comment) for the docker command group:
$ gcloud docker
Usage: docker [OPTIONS] COMMAND [arg...]
docker daemon [ --help | ... ]
docker [ --help | -v | --version ]
A self-sufficient runtime for containers.
Options:
--config=~/.docker Location of client config files
-D, --debug=false Enable debug mode
--disable-legacy-registry=false Do not contact legacy registries
-H, --host=[] Daemon socket(s) to connect to
-h, --help=false Print usage
-l, --log-level=info Set the logging level
--tls=false Use TLS; implied by --tlsverify
--tlscacert=~/.docker/ca.pem Trust certs signed only by this CA
--tlscert=~/.docker/cert.pem Path to TLS certificate file
--tlskey=~/.docker/key.pem Path to TLS key file
--tlsverify=false Use TLS and verify the remote
-v, --version=false Print version information and quit
Commands:
attach Attach to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on a container or image
kill Kill a running container
load Load an image from a tar archive or STDIN
login Register or log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
network Manage Docker networks
pause Pause all processes within a container
port List port mappings or a specific mapping for the CONTAINER
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart a container
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save an image(s) to a tar archive
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop a running container
tag Tag an image into a repository
top Display the running processes of a container
unpause Unpause all processes within a container
version Show the Docker version information
volume Manage Docker volumes
wait Block until a container stops, then print its exit code
Run 'docker COMMAND --help' for more information on a command.
You should use gcloud docker search gcr.io/project-id to check which images are in the repository. gcloud has your credentials, so it can talk to the private registry as long as you're authenticated as an appropriate user on the project.
Finally, as an added resource: The Cloud Platform docs have a whole article about working with Google Container Registry.
If you know the project that is hosting the images (e.g. google-containers) you can list images with
gcloud docker search gcr.io/google_containers
For an individual image (e.g. the pause image in the google-containers project), you can check the versions with
curl https://gcr.io/v2/google-containers/pause/tags/list
I've just found a far simpler way to check for specific images. Once you have authenticated gcloud, use it to generate access tokens for reading from your private registry:
curl -u "oauth2accesstoken:$(gcloud auth print-access-token)" https://gcr.io/v2/<projectName>/<imageName>/tags/list
My best solution so far without having a local docker available and without being able to connect to a remote docker (this would still require at least the local docker client but not the local daemon running), is to SSH into a Container Cluster instance that runs docker and have my search done there and getting the result in my original script:
gcloud compute ssh <container_cluster_instance> -C "sudo gcloud docker search ..."
Of course, to avoid all verbose output (like SSH/Terminal welcome messages) I use some arguments to silent the execution a bit:
gcloud compute ssh --ssh-flag="-q" "$INSTANCE_NAME" -o LogLevel=quiet -C "sudo gcloud docker search ..."