Request queue was disposed exception - project-reactor

I am having a weird error when using r2dbc (java/reactor/spring/mysql).
I have a bunch of controllers querying the db and once in a while I get these weird errors I can't make sense of.
Here is the mysterious error:
{
"level": "ERROR",
"logger_name": "reactor.core.publisher.Operators",
"thread_name": "reactor-tcp-epoll-2",
"#version": 1,
"exception": {
"exception_message": "Request queue was disposed",
"stacktrace": "java.lang.IllegalStateException: Request queue was disposed\n\tat dev.miku.r2dbc.mysql.client.RequestQueue.requireDisposed(RequestQueue.java:150)\n\tat dev.miku.r2dbc.mysql.client.RequestQueue.dispose(RequestQueue.java:139)\n\tat dev.miku.r2dbc.mysql.client.MessageDuplexCodec.channelInactive(MessageDuplexCodec.java:131)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)\n\tat io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1073)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
"exception_class": "java.lang.IllegalStateException",
"exception_rootCause": {
"exception_message": "Request queue was disposed",
"stacktrace": "java.lang.IllegalStateException: Request queue was disposed\n\tat dev.miku.r2dbc.mysql.client.RequestQueue.requireDisposed(RequestQueue.java:150)\n\tat dev.miku.r2dbc.mysql.client.RequestQueue.dispose(RequestQueue.java:139)\n\tat dev.miku.r2dbc.mysql.client.MessageDuplexCodec.channelInactive(MessageDuplexCodec.java:131)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)\n\tat io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1073)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
"exception_class": "java.lang.IllegalStateException"
}
},
"message": "Operator called default onErrorDropped"
}
No other error is being thrown together with those which makes me wonder if these are inner recoverable errors of the library that I should just hide.
I have been searching all over for a hint on this error with no success, any help would be much appreciated!

Related

AWS SQS SimpleMessageListenerContainer failing while polling queue

I have one SqsListener in my spring boot app as below:
#SqsListener(value = "QUEUE-FQN", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
private void receiveNotifications(String payload, MessageHeaders headers, Acknowledgment acknowledgment)
throws IOException, ParseException, InterruptedException {
try {
// process message here
}
}
And the following two beans defined:
#Bean
public AmazonSQSAsync amazonSQSAsync(AWSCredentialsProvider awsCredentialsProvider) {
return AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(awsCredentialsProvider)
.withRegion(Regions.US_EAST_1.getName())
.build();
}
#Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(AmazonSQSAsync amazonSqs) {
SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
factory.setAmazonSqs(amazonSqs);
factory.setMaxNumberOfMessages(10);
factory.setAutoStartup(false);
return factory;
}
After the simpleMessageListenerContainer.start(QUEUE_NAME), I see below exceptions constantly and the listener is never able to poll for new messages (never makes any progress).
Anything that I am doing wrong here? How to get past this error:
WARN 1297 --- [enerContainer-2] i.a.c.m.l.SimpleMessageListenerContainer : An Exception occurred while polling queue 'https://sqs:us-east-1:amazonaws:com/ACCOUNTID/QUEUE_NAME'. The failing operation will be retried in 10000 milliseconds
org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#3cee2db[Running, pool size = 11, active threads = 11, queued tasks = 0, completed tasks = 1188]] did not accept task: io.awspring.cloud.messaging.listener.SimpleMessageListenerContainer$SignalExecutingRunnable#681b4433
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:363)
at io.awspring.cloud.messaging.listener.SimpleMessageListenerContainer$AsynchronousMessageListener.run(SimpleMessageListenerContainer.java:343)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.RejectedExecutionException: Task io.awspring.cloud.messaging.listener.SimpleMessageListenerContainer$SignalExecutingRunnable#681b4433 rejected from java.util.concurrent.ThreadPoolExecutor#3cee2db[Running, pool size = 11, active threads = 11, queued tasks = 0, completed tasks = 1188]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:360)
... 6 more

NestJs HttpException response to Dart's http request

I would like to parse the response that my NestJs backend is sending to my Dart front end.
If from NestJs controller I reply with
return new HttpException('Already running..', HttpStatus.PROCESSING);
I receive on Dart's Response object:
Response: {
...
statusCode: 201
body {
"response": "Already running..",
"status":102,
"message": "Already running..",
"name": "HttpException"
}
}
Whereas if I return
throw new HttpExcpetion('Already running..', HttpStatus.PROCESSING)
I receive:
Response: {
...
statusCode: 102
body: ""
}
I would have expected to receive something like the below:
Response: {
...
statusCode: 102
message: "Already running..",
}
Any ideas how the two approaches are different and what should be the proper, consistent way so I know how to parse responses from backend?

Aws step function timeout not catched by error handler

I have the following state where the timeout is not catch nevertheless there is the catch with "States.ALL". According to here https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html
it should be. Can you tell me what is wrong?
"PublishIotCmd&WaitTask": {
"Next": "SuccedTask",
"Retry": [
{
[..]
}
],
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error",
"Next": "ErrorHandlerTask"
}
],
"Type": "Task",
"TimeoutSeconds": 600,
"ResultPath": "$.cmdResult",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters": {
"FunctionName": "xx",
"Payload": {
"token.$": "$$.Task.Token",
"request.$": "$.detail"
}
}
},
On the specific case the timeout is due to the task not getting the token with sendTaskSuccess. The error is, of course, this one but "ErrorHandlerTask" is not called, the state machine just hangs.
const publishIot = new tasks.LambdaInvoke(this, 'PublishIotCmd&WaitTask', {
lambdaFunction: iotSendCommandFn,
payload: sfn.TaskInput.fromObject({
token: sfn.JsonPath.taskToken,
//request: sfn.JsonPath.entirePayload,
request: sfn.JsonPath.stringAt('$.detail'),
}),
resultPath: '$.cmdResult',
integrationPattern: sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN,
timeout: Duration.minutes(TIMEOUT_WAIT_REPLY_SECONDS),
Thank you in advance
With task tokens, I believe you're supposed to use the Heartbeat timeout rather than a general timeout.
In the docs it calls out "The "HeartbeatSeconds": 600 field sets the heartbeat timeout interval to 10 minutes." and that "If the waiting task doesn't receive a valid task token within that 10-minute period, the task fails with a States.Timeout error name."
I think since it's a different service integration Heartbeat works here.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token

Receive validationToken but Subscription validation request timed out

Anyone can help me?.
I want register subscription for user, i use ngrok.
I try use postman it success but in my web it error.
this is my log
[2020-07-20 11:12:52] local.INFO: Error: Client error: `POST https://graph.microsoft.com/v1.0/subscriptions` resulted in a `400 Bad Request` response:
{
"error": {
"code": "InvalidRequest",
"message": "Subscription validation request timed out.",
"inner (truncated...)
this is my validationToken i receive:
array (
'validationToken' => 'Validation: Testing client application reachability for subscription Request-Id: d11e795b-9b06-46ff-b2ba-0df49a6e1c5c',
)
and my code
try {
$graph = new Graph();
$graph->setAccessToken($this->token);
$sub = new Model\Subscription();
$sub->setChangeType("created");
$sub->setNotificationUrl($this->domain. '/api/receive-notification');
$sub->setResource( "users/" . $outlook_id . "/events");
$sub->setClientState('SecretClientState');
$dateTime = new Carbon();
$dateTime->addDays(3);
$sub->setExpirationDateTime($dateTime);
$subResult = $graph->createRequest("POST", "/subscriptions")
->attachBody($sub)
->setReturnType(Model\Subscription::class)
->execute();
dd($subResult);
} catch (\Exception $e) {
dd($e->getMessage());
}
Graph API only allows 4230 minutes into the future.
Change
$dateTime->addDays(3);
to
$dateTime->modify('+4230 minutes');

How to throw an exception from a server API code in Dart?

I'm developing a client-server application in Dart and have been following the tutorial. My server code is roughly based on it.
In my server API code, when something goes wrong, I want to throw an exception, for example:
void checkEverything() {
if(somethingWrong)
throw new RpcError(400, "Something Wrong", "Something went wrong!");
}
#ApiMethod(path: 'myservice/{arg}')
Future<String> myservice(String arg) async {
checkEverything();
// ...
return myServiceResponse;
}
and that exception should be processed in the main server, e.g.
// ...
var apiResponse;
try {
var apiRequest = new HttpApiRequest.fromHttpRequest(request);
apiResponse = await _apiServer.handleHttpApiRequest(apiRequest);
} catch (error, stack) {
var exception = error is Error ? new Exception(error.toString()) : error;
if((error is RpcError && error.statusCode==400) {
// My code for creating the HTTP response
apiResponse = new HttpApiResponse.error(
HttpStatus.BAD_REQUEST, "Something went wrong", exception, stack);
}
else {
// standard error processing from the Dart tutorial
apiResponse = new HttpApiResponse.error(
HttpStatus.INTERNAL_SERVER_ERROR, exception.toString(),
exception, stack);
}
}
(snippet, see the tutorial for the complete code sans my error handling).
However, my exception never reaches the above catch clause. Instead, it seems to get caught in _apiServer.handleHttpApiRequest(apiRequest);, which, in turns, throws INTERNAL_SERVER_ERROR (500):
[WARNING] rpc: Method myservice returned null instead of valid return value
[WARNING] rpc:
Response
Status Code: 500
Headers:
access-control-allow-credentials: true
access-control-allow-origin: *
cache-control: no-cache, no-store, must-revalidate
content-type: application/json; charset=utf-8
expires: 0
pragma: no-cache
Exception:
RPC Error with status: 500 and message: Method with non-void return type returned 'null'
Unhandled exception:
RPC Error with status: 400 and message: Something went wrong!
#0 MyApi.myservice (package:mypackage/server/myapi.dart:204:24)
[...]
This is not very specific for the client. I'd like to communicate that an error has happened, not to return a good-looking response. So what is the proper way of handling server-side exceptions in Dart and passing that information to the client?
OK, I think I solved the problem. The throw clause apparently has to be in the API method itself, and not in a subordinate method. I.e.:
#ApiMethod(path: 'myservice/{arg}')
Future<String> myservice(String arg) async {
if(somethingWrong)
throw new RpcError(400, "Something Wrong", "Something went wrong!");
// ...
return myServiceResponse;
}
and not:
void checkEverything() {
if(somethingWrong)
throw new RpcError(400, "Something Wrong", "Something went wrong!");
}
#ApiMethod(path: 'myservice/{arg}')
Future<String> myservice(String arg) async {
checkEverything();
// ...
return myServiceResponse;
}

Resources