I have a Docker build environment where I build containers locally and test them. When I'm done, I push them to our Dev GitLab container registry to be deployed to Kubernetes.
I've run into a situation where either Docker isn't pushing up the newest layers or GitLab is seeing layers from a previous version and just mounting that layer so when the container is deployed in Kubernetes, the container, despite the new tag, is running the old container image.
I've tried completely wiping my Docker image repository, rebuilding, and repushing and that didn't fix it. I tried using the red trash icon in GitLab to delete the old version of the tag I'm trying to use.
I added some echo's in the console output for the container so I know the new bits aren't being run but I can't figure out if the problem is Docker or GitLab or how to fix it. Anyone have any ideas?
TIA!
Disregard -- my workers had a docker image on them and because my imagePullPolicy in my YAML was not set, it was defaulting to using the cached image in Docker. I just had to clear the image on my worker node (docker rmi -f <image name>) and then I updated my deployment YAML to use an imagePullPolicy: Always.
Related
There is an asp.net core api project, with sources in gitlab.
Created gitlab ci/cd pipeline to build docker image and put the image into gitlab docker registry
(thanks to https://medium.com/faun/building-a-docker-image-with-gitlab-ci-and-net-core-8f59681a86c4).
How to update docker containers on my production system after putting the image to gitlab docker registry?
*by update I mean:
docker-compose down && docker pull && docker-compose up
Best way to do this is to use Image puller, lot of open sources are available, or you can write your own on the Shell. There is one here. We use swarm, and we use this hook concept to be triggered from our CI-CD pipeline. Once our build stage is done, we http the hook url, and the docker pulls the updated image. One disadvantage with this is you need a daemon to watch your hook task, that it doesnt crash or go down. So my suggestion is to run this hook task as a docker container with restart-policy as RestartAlways
I have a docker compose file with a container that uses the latest tag:
code_site:
image: code_site:latest
deploy:
restart_policy:
condition: any
volumes:
- ../../data_to_backup/code_site/drupal_sites:/drupal_www/sites
- drupal_core:/drupal_www/core
- php_fpm_socket:/var/run/php-fpm7
networks:
- main_net
I have a process which will rebuild the container. I use it when I want to make changes to my site.
I am investigating a problem where the docker stack deploy command:
docker stack deploy --compose-file=docker-compose.yml code_site
(The stack name happens to match the image name but this is a coincidence.)
If I go through the following process:
Delete the code_site:latest image (rmi code_site:latest)
Rebuild a new code_site:latest image
Redeploy the stack
It will bring up the OLD version of the container. This is confusing, especially as I have deleted the old version.
I have gone further and I deleted the code_site image then I ran the stack deploy command.
The stack deploys successfully still running the old version of the container.
I can use the docker images command and verify that there is no container named code_site:latest so I have no idea how the stack could possibly deploy.
Can anyone explain how the image is coming back from the dead, and what method I should use to get rid of it permanently and force docker stack to use the real image?
Thanks
Robert
Update 1
code_site is a locally built image
I am running on a swarm but there is only one node in the swarm
Docker stack deploy will pull the latest image from your docker registry, since '--resolve-image always' is set by default, therefore always resolving to the latest image. If you don't want this run
docker stack deploy --resolve-image never [rest of deploy command]
However, to make it easier to maintain changes, I would suggest using version tags for your images in your registry, such as code_site:v1 when code changes push new version tagged code_site:v2 and deploy the new image/version using the normal deploy command without --resolve-image never.
Also if you plan to add nodes to your swarm you will need to change your command to docker stack deploy --with-registry-auth to allow the other nodes to pull the image from your repo.
Update 1
If you are 100% sure you do not have an image in your docker registry with the name code_site:latest then
this should work
Run:
docker rm $(docker ps -aq)
docker volume rm $(docker volume ls --format {{.ID}})
To check for lingering containers/services:
List Existing Services
docker service ls
List Running Stacks
docker stack ls
List All Containers
docker ps -aq
Then redeploy with deploy command
Alternatively to update your service without removing old containers/volumes/images, you can just update your image, then update your service without removing anything.
This will update your service using the new image... no need to stop, remove, then update... Just update.
Docker Service Update Command
Run after new image is built:
docker service update [SERVICE NAME] --image [IMAGE NAME] --force
I followed this tutorial to enable gitlab as a repository of docker images, then I executed docker push of the image and it is loaded in gitlab correctly.
http://clusterfrak.com/sysops/app_installs/gitlab_container_registry/
If I go to the Project registry option in Gitlab, the image appears there, but the problem occurs when I restart the coupler engine or the container where gitlab is located and when I re-enter the option to register the project in gitlab, all the images they are eliminated
That could be happening.
the problem occurs when I restart the coupler engine or the container where gitlab is located and when I re-enter the option to register the project in gitlab, all the images they are eliminated
That means the path where GitLab is storing docker images is part of the container, and is not persistent, ie is not a volume or a bind mount.
From the tutorial, you have the configuration:
################
# Registry #
################
gitlab_rails['registry_enabled'] = true
gitlab_rails['gitlab_default_projects_features_container_registry'] = false
gitlab_rails['registry_path'] = "/mnt/docker_registry"
gitlab_rails['registry_api_url'] = "https://localhost:5000"
You need to make sure, when starting the GitLab container, that it mounts a volume or host local path (which is persistent) to the container internal path /mnt/docker_registry.
Then, restarting GitLab would allow you to find back all the images you might have stored in the GitLAb-managed Docker registry.
I have created the volume where the images are and now it works.
thank you very much
I am a newbie about Docker. But I have looked many guides of that. I am configuring a container that it is running in a base image of jenkins with blue-ocean plugin. I run this one using docker run command and I configured my proxy information and added another plugin, k8s plugin through Jenkins Manage Plugin UI. Then I stop this container and I commit this container to save this state that has the k8s plugin and proxy information that I set already. But I run new docker image that I have made with docker commit command I can't see any proxy information and k8s plugin. It is same image that I started. Is there something I miss?
JENKINS_HOME is set to be a volume in the default Jenkins Docker image (which I'm assuming you're using). Volumes live outside of the Docker container layered filesystem. This means that any changes in those folders will not be persisted in subsequent image commits.
Using Kubernetes for deployment:
Considering I have a Dockerfile, I build, then push to registry.
If I run a container on a host, the image gets pulled and the container is ran.
Now, if i update the Dockerfile, build and push again, without changing its tag then the image is changed in the registry, but the host has the image pulled, and it doesn't seem to go look for updates.
How do i force a pull to get the latest image when running a container?
I can manually pull the image, but I'd like to know if there is a 'formal way' of doing this (in the pod or rc templates?)
Thanks for insight.
Set an imagePullPolicy of Always on the container