GCP instance group doesn't start containers - docker

Question

I have an instance template that is supposed to run my app in a container running on Google Cloud's Container-Optimized OS. When I create a single VM from this template it runs just fine, but when I use it to create an instance group the containers don't start.
According to the logs the machine didn't even try to start them.
I tried to compare the output from gcloud compute instances describe <instance-name> for the instance that works OK against one of the instances in the MIG, but other than some differences in the network interfaces and some that are due to the fact that one instance is managed by an instance group and the other one isn't I don't see anything unusual.
I also noticed that when I SSH to the instance that works, I get this message:
########################[ Welcome ]########################
# You have logged in to the guest OS. #
# To access your containers use 'docker attach' command #
###########################################################
but when I SSH to one of the instances in the MIG, I don't see it.
Is there a problem with using the container-optimized OS in an instance group?
My instance template is defined as follows:
creationTimestamp: '2022-11-09T03:25:29.896-08:00'
description: ''
id: '757769630202081478'
kind: compute#instanceTemplate
name: server-using-docker-hub-1
properties:
canIpForward: false
confidentialInstanceConfig:
enableConfidentialCompute: false
description: ''
disks:
- autoDelete: true
boot: true
deviceName: server-using-docker-hub
index: 0
initializeParams:
diskSizeGb: '10'
diskType: pd-balanced
sourceImage: projects/cos-cloud/global/images/cos-stable-101-17162-40-20
kind: compute#attachedDisk
mode: READ_WRITE
type: PERSISTENT
keyRevocationActionType: NONE
labels:
container-vm: cos-stable-101-17162-40-20
machineType: e2-micro
metadata:
fingerprint: 76mZ3i--POo=
items:
- key: gce-container-declaration
value: |-
spec:
containers:
- name: server-using-docker-hub-1
image: docker.io/rinbar/kwik-e-mart
env:
- name: AWS_ACCESS_KEY_ID
value: <redacted>
- name: AWS_SECRET_ACCESS_KEY
value: <redacted>
- name: SECRET_FOR_SESSION
value: <redacted>
- name: SECRET_FOR_USER
value: <redacted>
- name: MONGODBURL
value: mongodb+srv://<redacted>#cluster0.<redacted>.mongodb.net/kwik-e-mart
- name: DEBUG
value: server:*
- name: PORT
value: '80'
stdin: false
tty: false
restartPolicy: Always
# This container declaration format is not public API and may change without notice. Please
# use gcloud command-line tool or Google Cloud Console to run Containers on Google Compute Engine.
kind: compute#metadata
networkInterfaces:
- kind: compute#networkInterface
name: nic0
network: https://www.googleapis.com/compute/v1/projects/rons-project-364411/global/networks/default
stackType: IPV4_ONLY
subnetwork: https://www.googleapis.com/compute/v1/projects/rons-project-364411/regions/me-west1/subnetworks/default
reservationAffinity:
consumeReservationType: ANY_RESERVATION
scheduling:
automaticRestart: true
onHostMaintenance: MIGRATE
preemptible: false
provisioningModel: STANDARD
serviceAccounts:
- email: 629139871582-compute#developer.gserviceaccount.com
scopes:
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/trace.append
shieldedInstanceConfig:
enableIntegrityMonitoring: true
enableSecureBoot: false
enableVtpm: true
tags:
items:
- http-server
selfLink: https://www.googleapis.com/compute/v1/projects/rons-project-364411/global/instanceTemplates/server-using-docker-hub-1

Answer 1

I'm unable to replicate your issue; it worked for me.
I wonder whether your issue is container registry permissions? I don't use MIGs but assume a MIG runs as a service account and that perhaps yours doesn't have appropriate permission (to access the container registry)?
The one caveat is that gcloud compute instance-templates create-with-container is confusing and I am unable to resolve how to use --create-disk and --disk flags. I ended up using the Console to create the template. The Console's tool to generate the equivalent gcloud command is also incorrect (submitted feedback).
Q="74331370"
PROJECT="$(whoami)-$(date +%y%m%d)-${Q}"
ZONE="us-west1-c"
TEMPLATE="tmpl"
GROUP="group"
IMAGE="gcr.io/kuar-demo/kuard-amd64:blue"
SIZE="2"
MIN=${SIZE}
MAX=${MIN}
# This command is confusing
# Ultimately I used the console to save time
gcloud compute instance-templates create-with-container ${TEMPLATE} \
--project=${PROJECT} \
--machine-type=f1-micro \
--tags=http-server \
--container-image=${IMAGE} \
--create-disk=image-project=cos-cloud,image-family=cos-stable,mode=rw,size=10,type=pd-balanced \
--disk=auto-delete=yes,boot=yes,device-name=${TEMPLATE}
gcloud beta compute instance-groups managed create ${GROUP} \
--project=${PROJECT} \
--base-instance-name=${GROUP} \
--size=${SIZE} \
--template=${TEMPLATE} \
--zone=${ZONE} \
--list-managed-instances-results=PAGELESS
gcloud beta compute instance-groups managed set-autoscaling ${GROUP} \
--project=${PROJECT} \
--zone=${ZONE} \
--min-num-replicas=${MIN} \
--max-num-replicas=${MAX} \
--mode=off
INSTANCES=$(\
gcloud compute instance-groups managed list-instances ${GROUP} \
--project=${PROJECT} \
--zone=${ZONE} \
--format="value(instance)")
for INSTANCE in ${INSTANCES}
do
gcloud compute ssh ${INSTANCE} \
--project=${PROJECT} \
--zone=${ZONE} \
--command="docker container ls"
done
Yields (edited for clarity):
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
dd902f2d5e29 ${IMAGE} "/kuard" 4 minutes ago Up 4 minutes klt-tmpl-rqhp
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
0182f3e7f3dc ${IMAGE} "/kuard" 4 minutes ago Up 4 minutes klt-tmpl-azxs

Answer 2

Since the instances in the group have no external IP addresses, you need to enable Private Google Access or Cloud NAT to allow the instances to pull the container image from Container Registry / Artifact Registry / Docker Hub / any other container registry.