I am trying to deploy a container using Cloud Run on GKE with Anthos enabled. I was following the codelab https://codelabs.developers.google.com/codelabs/cloud-run-gke.
When I create the container in Google Console I see that pods are created but it gets stuck with status "Waiting for Load Balancer to be Ready" on the "Routing traffic" stage. I am including the output below of the events.
Status:
Address:
URL: http://hello-run.default.svc.cluster.local
Conditions:
Last Transition Time: 2020-09-09T02:16:13Z
Status: True
Type: ConfigurationsReady
Last Transition Time: 2020-09-09T02:16:14Z
Message: Waiting for load balancer to be ready
Reason: Uninitialized
Status: Unknown
Type: Ready
Last Transition Time: 2020-09-09T02:16:14Z
Message: Waiting for load balancer to be ready
Reason: Uninitialized
Status: Unknown
Type: RoutesReady
Latest Created Revision Name: hello-run-00001-qaz
Latest Ready Revision Name: hello-run-00001-qaz
Observed Generation: 1
Traffic:
Latest Revision: true
Percent: 100
Revision Name: hello-run-00001-qaz
URL: http://hello-run.default.example.com
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 7m59s service-controller Created Configuration "hello-run"
Normal Created 7m59s service-controller Created Route "hello-run"
Has anyone run into this situation ?
Related
Milvus was working fine for a while and I wanted to insert data. When I released the collection, milvus crashed with the error "StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet". I recreated the docker image several times and even tried to reinstall docker but the error persists .
Details:
Milvus: 2.1.4
ETCD: 3.5.0
minio: RELEASE.2022-03-17T06-34-49Z
System: Ubuntu 22.04LTS
The docker compose file has been taken from: https://github.com/milvus-io/milvus/releases/download/v2.1.4/milvus-standalone-docker-compose.yml
Ran the following command to drop a collection:
utility.drop_collection("Data")
The expected output is to drop a collection without an error
the actual output:
[drop_collection] retry:4, cost: 0.27s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait>
[drop_collection] retry:5, cost: 0.81s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait>
[drop_collection] retry:6, cost: 2.43s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait>
I'm having an Argo workflow with
retryStrategy:
limit: "2"
retryPolicy: Always
If my workflow fails, it will be retried again 2 times, so is there any way by which I can know how many times my workflow was retried before getting passed.
is there any way by which I can know how many times my workflow was retried before getting passed.
retryStrategy doesn't retry Worfklows. It retries tasks/steps within Workflows. If a step/task within a workflow fails, it will be retried according to the template's retryStrategy (or the default retryStrategy defined at the Workflow level if none is defined at the template level).
There are many ways to check retry counts. Here are a few:
Argo CLI
argo submit https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/retry-on-error.yaml -n argo
argo get #latest -n argo
Name: retry-on-error-6dmcq
Namespace: argo
ServiceAccount: default
Status: Succeeded
Conditions:
PodRunning False
Completed True
Created: Fri Dec 10 09:42:36 -0500 (1 minute ago)
Started: Fri Dec 10 09:42:36 -0500 (1 minute ago)
Finished: Fri Dec 10 09:42:56 -0500 (1 minute ago)
Duration: 20 seconds
Progress: 2/2
ResourcesDuration: 4s*(1 cpu),4s*(100Mi memory)
STEP TEMPLATE PODNAME DURATION MESSAGE
✔ retry-on-error-6dmcq error-container
├─✖ retry-on-error-6dmcq(0) error-container retry-on-error-6dmcq-1358267989 3s Error (exit code 3)
└─✔ retry-on-error-6dmcq(1) error-container retry-on-error-6dmcq-1157083656 3s
The step failed once and then succeeded.
Argo UI
The Argo web UI has a similar presentation. The graph will branch, and failed attempts will appear as leaf nodes marked as failed.
Other
You could inspect the Workflow object and analyze the nodes to find failures associated with retries.
kubectl get wf retry-on-error-6dmcq -n argo -ojson
You could also look at app logs for repetition from the step in question.
I am trying to create an OpenLdap instance using image osixia/openldap:1.5.0 as a k8s service. Image works really well and the Ldap instance is also worked fine. However, when I'm trying to create users using user.ldif file, getting error. I created a configmap nameing users and mounted it to the k8s cluster.
openldap.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: open-ldap-pod
labels:
app: open-ldap-pod
spec:
selector:
matchLabels:
app: open-ldap-pod
replicas: 1
template:
metadata:
labels:
app: open-ldap-pod
spec:
containers:
- name: open-ldap-pod
image: osixia/openldap:1.5.0
args: [ "-c", "/container/tool/run --copy-service" ]
ports:
- containerPort: 389
name: openldap
volumeMounts:
- name: users
mountPath: /container/service/slapd/assets/config/bootstrap/ldif/custom/users.ldif
subPath: users.ldif
volumes:
- name: users
configMap:
name: users
items:
- key: users.ldif
path: users.ldif
users.ldif
dn: ou=People,dc=example,dc=org
ou: People
objectClass: organizationalUnit
dn: cn=john,ou=People,dc=example,dc=org
myAttribute1: myAttribute
myAttribute2: myAttribute
sn: john
mail: john#example.org
cn: john
objectClass: personnel
dn: cn=mike,ou=People,dc=example,dc=org
myAttribute1: myAttribute
myAttribute2: myAttribute
sn: mike
mail: mike#example.org
cn: mike
objectClass: personnel
Error Stack:
*** CONTAINER_LOG_LEVEL = 3 (info)
*** Search service in CONTAINER_SERVICE_DIR = /container/service :
*** link /container/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools
*** link /container/service/slapd/startup.sh to /container/run/startup/slapd
*** link /container/service/slapd/process.sh to /container/run/process/slapd/run
*** Set environment for startup files
*** Environment files will be proccessed in this order :
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.startup.yaml
/container/environment/99-default/default.yaml
To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
*** Running '/container/tool/run --copy-service'...
*** CONTAINER_LOG_LEVEL = 3 (info)
*** Copy /container/service to /container/run/service
*** Search service in CONTAINER_SERVICE_DIR = /container/run/service :
*** link /container/run/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools
*** failed to link /container/run/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools: [Errno 17] File exists: '/container/run/service/:ssl-tools/startup.sh' -> '/container/run/startup/:ssl-tools'
*** link /container/run/service/slapd/startup.sh to /container/run/startup/slapd
*** failed to link /container/run/service/slapd/startup.sh to /container/run/startup/slapd: [Errno 17] File exists: '/container/run/service/slapd/startup.sh' -> '/container/run/startup/slapd'
*** link /container/run/service/slapd/process.sh to /container/run/process/slapd/run
*** directory /container/run/process/slapd already exists
*** failed to link /container/run/service/slapd/process.sh to /container/run/process/slapd/run : [Errno 17] File exists: '/container/run/service/slapd/process.sh' -> '/container/run/process/slapd/run'
*** Set environment for startup files
*** Environment files will be proccessed in this order :
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.startup.yaml
/container/environment/99-default/default.yaml
To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
*** Running /container/run/startup/:ssl-tools...
*** Running /container/run/startup/slapd...
openldap user and group adjustments
get current openldap uid/gid info inside container
-------------------------------------
openldap GID/UID
-------------------------------------
User uid: 911
User gid: 911
uid/gid changed: false
-------------------------------------
updating file uid/gid ownership
Database and config directory are empty...
Init new ldap server...
Backing up /etc/ldap/slapd.d in /var/backups/slapd-2.4.50+dfsg-1~bpo10+1... done.
Creating initial configuration... done.
Creating LDAP directory... done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
Start OpenLDAP...
Waiting for OpenLDAP to start...
Add bootstrap schemas...
config file testing succeeded
Add image bootstrap ldif...
Add custom bootstrap ldif...
*** /container/run/startup/slapd failed with status 68
*** Killing all processes...
*** /container/tool/run failed with status 1
*** Killing all processes...
Is there anything that I'm missing here?
PROBLEM
I cannot get serverless offline to run when not connected to internet.
serverless.yml
service: my-app
plugins:
- serverless-offline
# run on port 4000, because client runs on 3000
custom:
serverless-offline:
port: 4000
# app and org for use with dashboard.serverless.com
app: my-app
org: my-org
provider:
name: aws
runtime: nodejs10.x
functions:
getData:
handler: data-service.getData
events:
- http:
path: data/get
method: get
cors: true
isOffline: true
saveData:
handler: data-service.saveData
events:
- http:
path: data/save
method: put
cors: true
isOffline: true
To launch serverless offline, I run serverless offline start in terminal. This works when I am connected to the internet, but when offline, I get the following errors:
Console Error
:4000/data/get:1 Failed to load resource: net::ERR_CONNECTION_REFUSED
20:34:02.820 localhost/:1 Uncaught (in promise) TypeError: Failed to fetch
Terminal Error
FetchError: request to https://api.serverless.com/core/tenants/{tenant}/applications/my-app/profileValue failed, reason: getaddrinfo ENOTFOUND api.serverless.com api.serverless.com:443
Request
I suspect the cause is because I am not sure how to setup offline using instruction: "The event object passed to your λs has one extra key: { isOffline: true }. Also, process.env.IS_OFFLINE is true."
Any assistance on how to debug the issue would be much appreciated.
Probably you already fix it, but the problem is because app and org attribute
# app and org for use with dashboard.serverless.com
app: my-app
org: my-org
When you use it, serverless will use config set on serverless.com, commonly env var.
To use env var, you can use plugin serverless-dotenv-plugin. This way, you don't need to connect on internet.
I can create an application and build it successfully. However, I can't create a new for tag for an image of the application. The following is the command I used to create the tag and the result I got:
oc tag docker-registory.svc:5000/myproject/myapp#sha256:231d3371ad687233f7a6140c67a314c416105b424c19f28b124ca298989998b3 myproject/myapp:test
Tag myapp:test set to docker-registory.svc:5000/myproject/myapp#sha256:231d3371ad687233f7a6140c67a314c416105b424c19f28b124ca298989998b3.
root#018fdb569d8a:/# oc get is
NAME DOCKER REPO TAGS UPDATED
myapp docker-registory.svc:5000/myproject/myapp latest,test 5 minutes ago
root#018fdb569d8a:/# oc describe is
Name: myapp
Namespace: myproject
Created: 7 minutes ago
Labels: app=myapp
Description: Keeps track of changes in the application image
Annotations: openshift.io/generated-by=OpenShiftNewApp
openshift.io/image.dockerRepositoryCheck=2019-01-07T03:44:42Z
Docker Pull Spec: docker-registory.svc:5000/myproject/myapp
Image Lookup: local=false
Unique Images: 1
Tags: 2
latest
no spec tag
docker-registory.svc:5000/myproject/myapp#sha256:231d3371ad687233f7a6140c67a314c416105b424c19f28b124ca298989998b3
5 minutes ago
test
tagged from docker-registory.svc:5000/myproject/myapp#sha256:231d3371ad687233f7a6140c67a314c416105b424c19f28b124ca298989998b3
! error: Import failed (InternalError): Internal error occurred: Get https://docker-registory.svc:5000/v2/: x509: certificate signed by unknown authority