What is kubeflow gpu resource node allocation criteria? - kubeflow

Question

I’m curious about the Kubeflow GPU Resource. I’m running the job below.
The only part where I specified the GPU Resource is on first container with only 1 GPU. However, the event message tells me 0/4 nodes are available: 4 Insufficient nvidia.com/gpu.
Why is this job searching for 4 nodes though I specified only 1 GPU resource? Does my interpretation have a problem? Thanks much in advance.
FYI) I have 3 worker nodes with each 1 gpu.
apiVersion: batch/v1
kind: Job
metadata:
name: saint-train-3
annotations:
sidecar.istio.io/inject: "false"
spec:
template:
spec:
initContainers:
- name: dataloader
image: <AWS CLI Image>
command: ["/bin/sh", "-c", "aws s3 cp s3://<Kubeflow Bucket>/kubeflowdata.tar.gz /s3-data; cd /s3-data; tar -xvzf kubeflowdata.tar.gz; cd kubeflow_data; ls"]
volumeMounts:
- mountPath: /s3-data
name: s3-data
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef: {key: AWS_ACCESS_KEY_ID, name: aws-secret}
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef: {key: AWS_SECRET_ACCESS_KEY, name: aws-secret}
containers:
- name: trainer
image: <Our Model Image>
command: ["/bin/sh", "-c", "wandb login <ID>; python /opt/ml/src/main.py --base_path='/s3-data/kubeflow_data' --debug_mode='0' --project='kubeflow-test' --name='test2' --gpu=0 --num_epochs=1 --num_workers=4"]
volumeMounts:
- mountPath: /s3-data
name: s3-data
resources:
limits:
nvidia.com/gpu: "1"
- name: gpu-watcher
image: pytorch/pytorch:latest
command: ["/bin/sh", "-c", "--"]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- mountPath: /s3-data
name: s3-data
volumes:
- name: s3-data
persistentVolumeClaim:
claimName: test-claim
restartPolicy: OnFailure
backoffLimit: 6

Answer 1

0/4 nodes are available: 4 Insufficient nvidia.com/gpu
This is mean you haven't nodes with label nvidia.com/gpu