Helm fetches different "latest" image different than Docker pull's latest image - docker

Question

A different image is fetched with the same image name Helm does a chart install than if I do an ordinary docker pull. My values.yaml has
image:
repository: gcr.io/rsi-api-test/rsi-api
tag: latest
pullPolicy: IfNotPresent
The only image in the container registry is
gcr.io/rsi-api-test/rsi-
api#sha256:473bd9e31df8fd5d5424e11a57cabfd308179c8507683d6b9219c804bb315116
But helm somehow finds this image with this code:
gcr.io/rsi-api-test/rsi-api#sha256:c5cc78caa54ac4cf855c5fdb6a3448ff74ab641581fcda35d3e4e245c3154766
I believe it found some old version and for some reason refuses to get the latest version. Where is the "cached" or local collection of repositories used by helm and how can I force it cleaned?
The registry is a GCP Artifact Registry
>helm version
Client: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
(base)
UPDATE, CHANGED TAGGING
I edited the tags in the image in GCP, removing both the "latest" and "v0.1.0" tag and adding tag "v1.0.0". I changed the respective values in the Values.yaml file (above) to
image:
repository: gcr.io/rsi-api-test/rsi-api
tag: 1.0.0
pullPolicy: IfNotPresent
Then, I did
docker image pull gcr.io/rsi-api-test/rsi-api:latest
and confirmed there was no more latest (I'm not sure where it's getting these images from).
then
docker run --detached gcr.io/rsi-api-test/rsi-api:1.0.0
Docker fetched the 1.0.0 version, and it ran as I expected/wanted.
As for helm, I re-ran helm install, and it didn't work due to an image error (kubectl logs shown below).
helm install rsapi
>kubectl logs pod/sullen-lemur-rsiapi-7b8d6d656c-7xmxh
Error from server (BadRequest): container "rsiapi" in pod "sullen-lemur-rsiapi-7b8d6d656c-7xmxh" is waiting to start: trying and failing to pull image
(base)
id=woodsman pwd= ~/rsi/backend/rsiapi git=(master) history-id=1494$
>^Cep -R "\/v2" * |less
(base)
id=woodsman pwd= ~/rsi/backend/rsiapi git=(master) history-id=1494$
>kubectl describe pod/sullen-lemur-rsiapi-7b8d6d656c-7xmxh
Name: sullen-lemur-rsiapi-7b8d6d656c-7xmxh
Namespace: default
Priority: 0
Node: gke-helm-cluster-default-pool-7c542461-sdpv/10.128.0.6
Start Time: Thu, 26 May 2022 23:19:00 -0400
Labels: app.kubernetes.io/instance=sullen-lemur
app.kubernetes.io/name=rsiapi
pod-template-hash=7b8d6d656c
Annotations: <none>
Status: Pending
IP: 10.72.1.13
IPs:
IP: 10.72.1.13
Controlled By: ReplicaSet/sullen-lemur-rsiapi-7b8d6d656c
Containers:
rsiapi:
Container ID:
Image: gcr.io/rsi-api-test/rsi-api:1.0.0
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Liveness: http-get http://:http/ delay=60s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=20s timeout=1s period=10s #success=1 #failure=3
Environment:
SPRING_CLOUD_KUBERNETES_CONFIG_NAME: sullen-lemur-rsiapi
MANAGEMENT_ENDPOINT_RESTART_ENABLED: true
SPRING_CLOUD_KUBERNETES_RELOAD_ENABLED: true
SPRING_CLOUD_KUBERNETES_RELOAD_STRATEGY: refresh
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fvgg8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-fvgg8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned default/sullen-lemur-rsiapi-7b8d6d656c-7xmxh to gke-helm-cluster-default-pool-7c542461-sdpv
Normal Pulling 12m (x4 over 13m) kubelet Pulling image "gcr.io/rsi-api-test/rsi-api:1.0.0"
Warning Failed 12m (x4 over 13m) kubelet Failed to pull image "gcr.io/rsi-api-test/rsi-api:1.0.0": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/rsi-api-test/rsi-api:1.0.0": failed to resolve reference "gcr.io/rsi-api-test/rsi-api:1.0.0": gcr.io/rsi-api-test/rsi-api:1.0.0: not found
Warning Failed 12m (x4 over 13m) kubelet Error: ErrImagePull
Warning Failed 11m (x6 over 13m) kubelet Error: ImagePullBackOff
Normal BackOff 3m32s (x42 over 13m) kubelet Back-off pulling image "gcr.io/rsi-api-test/rsi-api:1.0.0"
(base)
id=woodsman pwd= ~/rsi/backend/rsiapi git=(master) history-id=1495$
>helm ls
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
sullen-lemur 1 Thu May 26 23:19:00 2022 DEPLOYED rsiapi-0.1.0 1.0 default
The same image id was tied up in two SHA codes:
> docker image rm 8d0bfa85e8f9
Error response from daemon: conflict: unable to delete 8d0bfa85e8f9
(must be forced) - image is referenced in multiple repositories
>docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/rsi-api-test/rsi-api latest 8d0bfa85e8f9 10 hours ago 379MB
gcr.io/rsi-api-test/rsi-api v1.0.0 8d0bfa85e8f9 10 hours ago 379MB
>docker image rm -f 8d0bfa85e8f9
Untagged: gcr.io/rsi-api-test/rsi-api:latest
Untagged: gcr.io/rsi-api-test/rsi-api:v1.0.0
Untagged: gcr.io/rsi-api-test/rsi-api#sha256:473bd9e31df8fd5d5424e11a57cabfd308179c8507683d6b9219c804bb315116
Deleted: sha256:8d0bfa85e8f929221a7c6b66e5fd6008151e496407ed9e74072dd3e02314ad12
BONUS points to suggest a way/policy to increment the version tag in for helm. This is a Github Workflow Maven build of a spring boot application.
Also, I'm running Helm on my personal Linux machine, but want it to target a GCP cluster. However, I also tried installing helm and using it on a Minikube installation. What do I need to do to make sure I fully switch?