Kubernetes Ambassador ingress docker image rpc timeout - docker

Question

I’m trying to setup ingress using ambassador for my local cluster and working off this guide here https://kind.sigs.k8s.io/docs/user/ingress
but I receive an rpc and timeout error. Some things i’ve attempted in trying to fix include: Using the Ingress Nginx instead but I received same error, amending the dockerhub path by just having the username and image in the url and deleting and recreating the pods. I've also seen other previous questions and the solutions do not seem to work.
NAMESPACE NAME READY STATUS RESTARTS AGE
ambassador ambassador-operator-67668967b8-w28b2 0/1 ImagePullBackOff 0 28m
default bar-app 0/1 ErrImagePull 0 9m19s
default foo-app 0/1 ImagePullBackOff 0 9m19s
kube-system coredns-74ff55c5b-m7s8r 1/1 Running 0 38m
kube-system coredns-74ff55c5b-tgcdg 1/1 Running 0 38m
kube-system etcd-kind8-control-plane 1/1 Running 0 38m
kube-system kindnet-dch9w 1/1 Running 0 37m
kube-system kindnet-dm5gn 1/1 Running 0 38m
kube-system kindnet-sxxdk 1/1 Running 0 37m
kube-system kube-apiserver-kind8-control-plane 1/1 Running 0 38m
kube-system kube-controller-manager-kind8-control-plane 1/1 Running 0 38m
kube-system kube-proxy-n84kf 1/1 Running 0 38m
kube-system kube-proxy-twtsf 1/1 Running 0 37m
kube-system kube-proxy-zjq6t 1/1 Running 0 37m
kube-system kube-scheduler-kind8-control-plane 1/1 Running 0 38m
local-path-storage local-path-provisioner-78776bfc44-kkrht 1/1 Running 0 38m
This is the error log. kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'
kube-system 28m Normal Created pod/coredns-74ff55c5b-tgcdg Created container coredns
kube-system 28m Normal Created pod/coredns-74ff55c5b-m7s8r Created container coredns
kube-system 28m Normal Started pod/coredns-74ff55c5b-m7s8r Started container coredns
local-path-storage 28m Normal Started pod/local-path-provisioner-78776bfc44-kkrht Started container local-path-provisioner
local-path-storage 28m Normal Created pod/local-path-provisioner-78776bfc44-kkrht Created container local-path-provisioner
local-path-storage 28m Normal LeaderElection endpoints/rancher.io-local-path local-path-provisioner-78776bfc44-kkrht_c5300431-393d-4ce5-bee6-9fa03b2567e8 became leader
kube-system 28m Normal Started pod/coredns-74ff55c5b-tgcdg Started container coredns
ambassador 20m Normal ScalingReplicaSet deployment/ambassador-operator Scaled up replica set ambassador-operator-67668967b8 to 1
ambassador 20m Normal SuccessfulCreate replicaset/ambassador-operator-67668967b8 Created pod: ambassador-operator-67668967b8-w28b2
ambassador 20m Normal Scheduled pod/ambassador-operator-67668967b8-w28b2 Successfully assigned ambassador/ambassador-operator-67668967b8-w28b2 to kind8-worker
ambassador 15m Normal Pulling pod/ambassador-operator-67668967b8-w28b2 Pulling image "docker.io/datawire/ambassador-operator:v1.2.9"
ambassador 3s Warning Failed pod/ambassador-operator-67668967b8-w28b2 Error: ImagePullBackOff
ambassador 5m1s Normal BackOff pod/ambassador-operator-67668967b8-w28b2 Back-off pulling image "docker.io/datawire/ambassador-operator:v1.2.9"
ambassador 14m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Error: ErrImagePull
ambassador 19m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 18.214.230.110:443: i/o timeout
ambassador 17m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 3.211.199.249:443: i/o timeout
ambassador 16m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 54.236.165.68:443: i/o timeout
ambassador 14m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 54.236.131.166:443: i/o timeout
default 38s Normal Scheduled pod/foo-app Successfully assigned default/foo-app to kind8-worker
default 38s Normal Scheduled pod/bar-app Successfully assigned default/bar-app to kind8-worker
default 37s Normal Pulling pod/bar-app Pulling image "hashicorp/http-echo:0.2.3"
This is the yaml.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ambassadorinstallations.getambassador.io
spec:
additionalPrinterColumns:
- JSONPath: .spec.version
name: VERSION
type: string
- JSONPath: .spec.updateWindow
name: UPDATE-WINDOW
type: integer
- JSONPath: .status.lastCheckTime
description: Last time checked
name: LAST-CHECK
type: string
- JSONPath: .status.conditions[?(#.type=='Deployed')].status
description: Indicates if deployment has completed
name: DEPLOYED
type: string
- JSONPath: .status.conditions[?(#.type=='Deployed')].reason
description: Reason for deployment completed
name: REASON
priority: 1
type: string
- JSONPath: .status.conditions[?(#.type=='Deployed')].message
description: Message for deployment completed
name: MESSAGE
priority: 1
type: string
- JSONPath: .status.deployedRelease.appVersion
description: Deployed version of Ambassador
name: DEPLOYED-VERSION
type: string
- JSONPath: .status.deployedRelease.flavor
description: Deployed flavor of Ambassador (OSS or AES)
name: DEPLOYED-FLAVOR
type: string
group: getambassador.io
names:
kind: AmbassadorInstallation
listKind: AmbassadorInstallationList
plural: ambassadorinstallations
singular: ambassadorinstallation
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: AmbassadorInstallation is the Schema for the ambassadorinstallations
API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: AmbassadorInstallationSpec defines the desired state of AmbassadorInstallation
properties:
baseImage:
description: An (optional) image to use instead of the image specified
in the Helm chart.
type: string
helmRepo:
description: An (optional) Helm repository.
type: string
installOSS:
description: 'Installs [Ambassador OSS](https://www.getambassador.io/docs/latest/topics/install/install-ambassador-oss/)
instead of [AES](https://www.getambassador.io/docs/latest/topics/install/).
Default is false which means it installs AES by default. TODO: 1.
AES/AOSS is not installed and the user installs using `installOSS:
true`, then we straightaway install AOSS. 2. AOSS is installed via
operator and the user sets `installOSS: false`, then we perform the
migration as detailed here - https://www.getambassador.io/docs/latest/topics/install/upgrade-to-edge-stack/
3. AES is installed and the user sets `installOSS: true`, then we
point users to the docs which gives them pointers on how to do
that themselves.'
type: boolean
logLevel:
description: 'An (optional) log level: debug, info...'
enum:
- info
- debug
- warn
- warning
- error
- critical
- fatal
type: string
updateWindow:
description: "`updateWindow` is an optional item that will control when
the updates can take place. This is used to force system updates to
happen late at night if that’s what the sysadmins want. \n * There
can be any number of `updateWindow` entries (separated by commas).
\ * `Never` turns off automatic updates even if there are other entries
in the comma-separated list. `Never` is used by sysadmins to disable
all updates during blackout periods by doing a `kubectl apply`
or using our Edge Policy Console to set this. * Each `updateWindow`
is in crontab format (see https://crontab.guru/) Some examples of
`updateWindows` are: - `* 0-6 * * * SUN`: every Sunday, from _0am_
to _6am_ - `* 5 1 * * *`: every first day of the month, at _5am_
* The Operator cannot guarantee minute time granularity, so specifying
\ a minute in the crontab expression can lead to some updates happening
\ sooner/later than expected."
type: string
version:
description: "We are using SemVer for the version number and it can
be specified with any level of precision and can optionally end in
`*`. These are interpreted as: \n * `1.0` = exactly version 1.0 *
`1.1` = exactly version 1.1 * `1.1.*` = version 1.1 and any bug fix
versions `1.1.1`, `1.1.2`, `1.1.3`, etc. * `2.*` = version 2.0 and
any incremental and bug fix versions `2.0`, `2.0.1`, `2.0.2`, `2.1`,
`2.2`, `2.2.1`, etc. * `*` = all versions. * `3.0-ea` = version `3.0-ea1`
and any subsequent EA releases on `3.0`. Also selects the final
3.0 once the final GA version is released. * `4.*-ea` = version `4.0-ea1`
and any subsequent EA release on `4.0`. Also selects the final GA
`4.0`. Also selects any incremental and bug fix versions `4.*` and
`4.*.*`. Also selects the most recent `4.*` EA release i.e., if
`4.0.5` is the last GA version and there is a `4.1-EA3`, then this
\ selects `4.1-EA3` over the `4.0.5` GA. \n You can find the reference
docs about the SemVer syntax accepted [here](https://github.com/Masterminds/semver#basic-comparisons)."
type: string
type: object
status:
description: AmbassadorInstallationStatus defines the observed state of
AmbassadorInstallation
properties:
conditions:
description: List of conditions the installation has experienced.
items:
description: AmbInsCondition defines an Ambassador installation condition,
as well as the last time there was a transition to this condition..
properties:
lastTransitionTime:
format: date-time
type: string
message:
type: string
reason:
type: string
status:
type: string
type:
type: string
required:
- status
- type
type: object
type: array
deployedRelease:
description: the currently deployed Helm chart
nullable: true
properties:
appVersion:
type: string
flavor:
type: string
manifest:
type: string
name:
type: string
version:
type: string
type: object
lastCheckTime:
description: Last time a successful update check was performed.
format: date-time
nullable: true
type: string
required:
- conditions
type: object
type: object
version: v2
versions:
- name: v2
served: true
storage: true

Answer 1

I figured this out. I'd previously disabled docker bridge0 using this entry under
/etc/docker/daemon.json
{
"iptables": false,
"bridge": "none"
}
To fixed i simply deleted and restarted docker:
systemctl restart docker