One of our jobs started throwing up the following warning/error:
856573 [main] WARN com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob - There were problems getting current job status:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 429 Too Many Requests
{
"code" : 429,
"errors" : [ {
"domain" : "global",
"message" : "Resource has been exhausted (e.g. check quota).",
"reason" : "rateLimitExceeded"
} ],
"message" : "Resource has been exhausted (e.g. check quota).",
"status" : "RESOURCE_EXHAUSTED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1056)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.getState(DataflowPipelineJob.java:188)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.waitToFinish(DataflowPipelineJob.java:126)
at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:86)
at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
at com.tls.cdf.job.AbstractCloudDataFlowJob.execute(AbstractCloudDataFlowJob.java:100)
at com.tls.cdf.CloudDataFlowJobExecutor.main(CloudDataFlowJobExecutor.java:44)
At first we though this was an error allocating the desired resources (VM's) for the job, but in fact the job ran fine and it was able to scale up as needed. It seems to be when trying to retrieve the job status is the problem.
Interestingly, each time the error was thrown in the application (there were multiple reported while the job ran), the developer console would also bork with this:
The job id is: 2015-05-04_20_49_53-2521857061976321751
What is this warning/error related to?
There was an issue on our end reporting monitoring information. We've rolled back the change, and you should be good now. Please let us know if you're still having issues. Sorry for the trouble!
Related
I'm currently working on migrating from the Google My business V4.9 to the Business Profile Performance API .
For number of credentials the changes are working as expected but for others, there is a problem with making the request as I'm getting this error.
GET https://businessprofileperformance.googleapis.com/v1/locations/****:getDailyMetricsTimeSeries?dailyMetric=CALL_CLICKS&dailyRange.endDate.day=8&dailyRange.endDate.month=1&dailyRange.endDate.year=2023&dailyRange.startDate.day=31&dailyRange.startDate.month=12&dailyRange.startDate.year=2022
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "The caller does not have permission",
"reason" : "forbidden"
} ],
"message" : "The caller does not have permission",
"status" : "PERMISSION_DENIED"
}
Worth to mention that for the same credentials and scope everything is working with the older endpoints.
I'm currently using https://www.googleapis.com/auth/plus.business.manage, can this be the problem as it has been deprecated(But you can still use it) and now there is https://www.googleapis.com/auth/business.manage instead?
UPDATE
It seems that before it simply returned an empty list of reports for this location, and now it's throwing an exception.
I am able to publish messages to DLX when they are not able to process due to insufficient information or any other issues with the message received by binding DLX for the queue through Spring AMPQ.
For instance, invoice received with missing of billable hours and/or no employee id present it it.
{
"invoiceId": "INV1234",
"hourRate": 18.5,
"hoursWorked": 0,
"employeeId": "EMP9900"
}
Due to less size of request body, it's easy to understand what is the issue. But, we have some considerable request body length and 15-20 validation points.
Producer of the message expecting what is the issue when pushing back the message to publishing back them through DLX.
I have the following two thoughts to address this requirements.
Option #1: Append the errors information to the original message.
{
"message": {
"invoiceId": "INV1234",
"hourRate": 18.5,
"hoursWorked": 0,
"employeeId": "EMP9900"
},
"errors": [
{
"errorCode": "001",
"errorMessage": "Invalid no. of hours."
},
{
"errorCode": "002",
"errorMessage": "Employee not found in the system."
}
]
}
Option #2 : Add additional errors object in the headers
Out of these two options,
what is the better way of handling this requirements? And
Is there any in-built solution available in either spring-amqp or any other library?
See the documentation. The framework implements your #2 solution.
Starting with version 1.3, a new RepublishMessageRecoverer is provided, to allow publishing of failed messages after retries are exhausted.
...
The RepublishMessageRecoverer publishes the message with additional information in message headers, such as the exception message, stack trace, original exchange, and routing key. Additional headers can be added by creating a subclass and overriding additionalHeaders().
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
I'm using the google safebrowsing api for getting fullhashes for the prefixes of hashes of url present in threat lists. When i'm using 5 threads to do the job, the job completes but the latency is high, so i tried increasing the threads to 7 and more but i'm getting the following error on doing that:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 504 Gateway Time-out
{
"code" : 504,
"errors" : [ {
"domain" : "global",
"message" : "Deadline expired before operation could complete.",
"reason" : "backendError"
} ],
"message" : "Deadline expired before operation could complete.",
"status" : "DEADLINE_EXCEEDED"
}
But, I'm sure that my daily quota has not exceeded.
By looking at the console, i can see that the number of requests per second is not more than the default quota (3000 req/100sec).
What other reason can be there for the above error ?
I am new to google cloud dataflow. I setup everything on my windows machine and when i tried to run wordcount example using below command:
mvn compile exec:java -Dexec.mainClass=com.nyt.dataflowPoc.WordCount -Dexec.args="--project=cdfpoc-1264 --stagingLocation=gs://poc-location/staging --runner=BlockingDataflowPipelineRunner --output=gs://poc-location"
and it gave me below error.
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid Value",
"reason" : "invalid"
} ],
"message" : "Invalid Value"
}
I looked at other posts and it said that i need to give projectID which shouuld be alpha numberic.I am already giving that but still same error.
Can anyone please help?
Thanks in advance.
-Kapil
The problem comes from:
--output=gs://poc-location
As such you are only specifying a bucket. WordCount actually interprets this as a prefix for your output, and tries to write the file gs://poc-location-00000-of-00001. Since this bucket does not exist you get an error.
Your pipeline will run fine with:
--output=gs://poc-location/
The problem is resolved. I was giving both staging and output location. When I digged more into the code , I found that only one of them was needed. So when I removed --output parameter , I got actual output and i could see results at storage location specified.
Thanks,
Kapil