On test and prod environment getting lot of socketTimeOut Exception when the timeout limit was 2sec the error rate was a lot 37% but there were successful calls as well not all of them failed.
As a trail we tried increasing timeout limit to 10sec the error rate is 0.9% with still lot of SocketTimeOUtException in the logs , here again there are many successful calls as well.
So my question is, if the service get response in 2Sec for few request does it mean the server is capable of serving the request within 2 sec? and there could be a bottle neck at the server side so it is not responding.
Recently it has started occurring. Also I could see this generally happens between 12 pm to 4 pm when there is lots of request.
Can I assume if there are few requests which are successful, does it means all the connection related majors are in place correctly for eg. the port it is trying to connect to
Will it be a good try to see by creating an extra kubernetes pod when there is load on the service during peek time.
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:232)
at okio.AsyncTimeout.exit(AsyncTimeout.java:286)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242)
at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213)
at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at datadog.trace.instrumentation.okhttp3.TracingInterceptor.intercept(TracingInterceptor.java:35)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:223)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at com.product.composition.client.config.ExceptionTranslatorInterceptor.intercept(ExceptionTranslatorInterceptor.java:15)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall.execute(RealCall.java:81)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:186)
at com.product.billing.platform.retrofit.adapter.UnwrappedCallAdapterFactory$BodyCallAdapter.adapt(UnwrappedCallAdapterFactory.java:51)
at retrofit2.HttpServiceMethod.invoke(HttpServiceMethod.java:89)
at retrofit2.Retrofit$1.invoke(Retrofit.java:147)
at com.sun.proxy.$Proxy134.generatePDF(Unknown Source)
at com.product.main.composition.service.PdfGeneratorServiceImpl.lambda$null$0(PdfGeneratorServiceImpl.java:56)
at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:56)
at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.resubscribe(FluxRetryPredicate.java:124)
at reactor.core.publisher.MonoRetryPredicate.subscribeOrReturn(MonoRetryPredicate.java:51)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoCompletionStage.lambda$subscribe$0(MonoCompletionStage.java:86)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:110)
at reactor.core.publisher.MonoCompletionStage.subscribe(MonoCompletionStage.java:61)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:61)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:82)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:61)
at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.resubscribe(FluxRetryPredicate.java:124)
at reactor.core.publisher.MonoRetryPredicate.subscribeOrReturn(MonoRetryPredicate.java:51)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoCompletionStage.lambda$subscribe$0(MonoCompletionStage.java:86)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:110)
Related
I am using rsocket starter in Spring Boot (2.5.3). I have a requester and responder set up and its working quite well (for all kind of rsocket communication). The issue is when I am trying to implement resumability option on the requester side.
Here is the code for requester
#Bean
public RSocketRequester rSocketRequester(RSocketRequester.Builder builder) {
Resume resume = new Resume()
.sessionDuration(Duration.ofMinutes(15))
.streamTimeout(Duration.ofMinutes(15))
.retry(Retry.fixedDelay(Long.MAX_VALUE, Duration.ofSeconds(5))
.doBeforeRetry(s -> System.out.println("Resume: Before Retry : "+s))
.doAfterRetry(s -> System.out.println("Resume: After Retry : "+s))
);
return builder
.dataMimeType(MimeType.valueOf("application/json"))
.rsocketConnector(rSocketConnector -> {
rSocketConnector.reconnect(Retry.fixedDelay(10, Duration.ofSeconds(20)).doBeforeRetry(s -> {
System.out.println("Reconnect : Disconnected. Trying to reconnect..."+s);
}));
rSocketConnector.resume(resume);
})
.websocket(URI.create("ws://localhost:9190/api/workflow/realtime"));
}
On the responder side:
#Bean
RSocketServerCustomizer rSocketResume() {
Resume resume = new Resume()
.sessionDuration(Duration.ofMinutes(15))
.streamTimeout(Duration.ofMinutes(15))
.retry(Retry.fixedDelay(Long.MAX_VALUE, Duration.ofSeconds(5))
.doBeforeRetry(s -> System.out.println("Resume: Before Retry : "+s))
.doAfterRetry(s -> System.out.println("Resume: After Retry : "+s))
);
return rSocketServer -> rSocketServer.resume(resume );
}
But when responder goes down and comes back alive, I get below error on requester console:
**[----> Responder dead at this point]**
Resume: Before Retry : attempt #1 (1 in a row), last failure={io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:9190}
Resume: After Retry : attempt #1 (1 in a row), last failure={io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:9190}
Resume: Before Retry : attempt #2 (2 in a row), last failure={io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:9190}
Resume: After Retry : attempt #2 (2 in a row), last failure={io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:9190}
Resume: Before Retry : attempt #3 (3 in a row), last failure={io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:9190}
Resume: After Retry : attempt #3 (3 in a row), last failure={io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:9190}
**[<---- Responder back to live at this point]**
2021-08-08 04:48:39.404 ERROR 24748 --- [actor-tcp-nio-5] reactor.core.publisher.Operators : Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: RejectedResumeException (0x4): unknown resume token
Caused by: io.rsocket.exceptions.RejectedResumeException: unknown resume token
at io.rsocket.exceptions.Exceptions.from(Exceptions.java:64) ~[rsocket-core-1.1.1.jar:na]
at io.rsocket.resume.ClientRSocketSession.tryReestablishSession(ClientRSocketSession.java:271) ~[rsocket-core-1.1.1.jar:na]
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:196) ~[reactor-core-3.4.8.jar:3.4.8]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) ~[reactor-core-3.4.8.jar:3.4.8]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249) ~[reactor-core-3.4.8.jar:3.4.8]
at reactor.core.publisher.FluxFirstWithSignal$FirstEmittingSubscriber.onNext(FluxFirstWithSignal.java:330) ~[reactor-core-3.4.8.jar:3.4.8]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:160) ~[reactor-core-3.4.8.jar:3.4.8]
at io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:114) ~[rsocket-core-1.1.1.jar:na]
at io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19) ~[rsocket-core-1.1.1.jar:na]
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.8.jar:3.4.8]
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:279) ~[reactor-netty-core-1.0.9.jar:1.0.9]
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:388) ~[reactor-netty-core-1.0.9.jar:1.0.9]
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404) ~[reactor-netty-core-1.0.9.jar:1.0.9]
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:706) ~[reactor-netty-http-1.0.9.jar:1.0.9]
at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:159) ~[reactor-netty-http-1.0.9.jar:1.0.9]
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93) ~[reactor-netty-core-1.0.9.jar:1.0.9]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.66.Final.jar:4.1.66.Final]
at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_261]
2021-08-08 04:48:39.407 ERROR 24748 --- [actor-tcp-nio-5] reactor.Flux.Map.1 : onError(RejectedResumeException (0x4): unknown resume token)
Can anyone suggest why this behavior? Also I want to understand what is this token on resume?
Thanks.
To give you the background, we are running kaa server on ubuntu in combination with MongoDB and have Endpoints attached to it (max 10). Apparently I am seeing following error way too frequently on our server and this makes our Kaa client (C client) throw a segment fault error.
The work around we have found out is restarting kaa-node service on server.
We are planning to go live with product with IoT enabled features and we are worried about this.
Can someone throw light as to what might be going wrong?
I understand the problem is with zookeeper and looks like it does not find operations servers
2017-05-11 12:31:23,808 [EPS-core-dispatcher-208419] DEBUG o.k.k.s.o.s.a.a.core.TenantActor - [f8d64123-6861-446f-b937-1e1d398159c8] Processing status request
2017-05-11 12:31:23,808 [EPS-core-dispatcher-208419] TRACE o.k.k.s.o.s.a.a.c.ApplicationActor - [26309946599256320307] Received: org.kaaproject.kaa.server.operations.service.akka.messages.core.stats.StatusRequestMessage#fb460fe
2017-05-11 12:31:23,808 [EPS-core-dispatcher-208419] DEBUG o.k.k.s.o.s.a.a.c.ApplicationActor - [f8d64123-6861-446f-b937-1e1d398159c8] Processing status request
2017-05-11 12:31:23,808 [EPS-core-dispatcher-208419] TRACE o.k.k.s.o.s.a.a.core.TenantActor - [1] Received: org.kaaproject.kaa.server.operations.service.akka.messages.core.stats.ApplicationActorStatusResponse#4ab6ff6c
2017-05-11 12:31:23,808 [EPS-core-dispatcher-208419] DEBUG o.k.k.s.o.s.a.a.c.OperationsServerActor - Received: org.kaaproject.kaa.server.operations.service.akka.messages.core.stats.TenantActorStatusResponse#2fc38dcf
2017-05-11 12:31:23,813 [pool-5-thread-1] ERROR o.k.k.s.common.zk.ControlNodeTracker - Unknown Error
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /operationsServerNodes/operationsServerNodes0000000021
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) ~[curator-framework-2.9.0.jar:na]
at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) ~[curator-framework-2.9.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.0.jar:na]
at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:267) ~[curator-framework-2.9.0.jar:na]
at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) ~[curator-framework-2.9.0.jar:na]
at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) ~[curator-framework-2.9.0.jar:na]
at org.kaaproject.kaa.server.common.zk.operations.OperationsNode$1.doWithZkClient(OperationsNode.java:78) ~[zk-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.zk.ControlNodeTracker.doZkClientAction(ControlNodeTracker.java:294) [zk-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.zk.ControlNodeTracker.doZkClientAction(ControlNodeTracker.java:280) [zk-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.zk.operations.OperationsNode.updateNodeData(OperationsNode.java:75) [zk-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.loadbalance.DefaultLoadBalancingService.onStatusUpdate(DefaultLoadBalancingService.java:98) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.loadbalance.DefaultLoadBalancingService$1$1.run(DefaultLoadBalancingService.java:72) [kaa-node-0.10.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
Below is the frequency of this particular error(filename: no of occurrences of error)
kaa-node-2017-05-10.0.log:0
kaa-node-2017-05-10.1.log:0
kaa-node-2017-05-10.2.log:0
kaa-node-2017-05-10.3.log:0
kaa-node-2017-05-10.4.log:0
kaa-node-2017-05-10.5.log:0
kaa-node-2017-05-10.6.log:0
kaa-node-2017-05-10.7.log:0
kaa-node-2017-05-11.0.log:0
kaa-node-2017-05-11.1.log:0
kaa-node-2017-05-11.2.log:0
kaa-node-2017-05-11.3.log:0
kaa-node-2017-05-11.4.log:520
kaa-node-2017-05-11.5.log:0
kaa-node-2017-05-11.6.log:0
kaa-node-2017-05-11.7.log:0
kaa-node-2017-05-11.8.log:0
kaa-node-2017-05-12.0.log:0
kaa-node-2017-05-12.1.log:0
kaa-node-2017-05-12.2.log:0
kaa-node-2017-05-12.3.log:0
kaa-node-2017-05-12.4.log:0
kaa-node-2017-05-12.5.log:0
kaa-node-2017-05-12.6.log:0
kaa-node-2017-05-13.0.log:0
kaa-node-2017-05-13.1.log:0
kaa-node-2017-05-13.2.log:0
kaa-node-2017-05-13.3.log:0
kaa-node-2017-05-13.4.log:0
kaa-node-2017-05-13.5.log:0
kaa-node-2017-05-13.6.log:0
kaa-node-2017-05-13.7.log:0
kaa-node-2017-05-14.0.log:0
kaa-node-2017-05-14.1.log:0
kaa-node-2017-05-14.2.log:0
kaa-node-2017-05-14.3.log:0
kaa-node-2017-05-14.4.log:0
kaa-node-2017-05-14.5.log:0
kaa-node-2017-05-14.6.log:0
kaa-node-2017-05-14.7.log:0
kaa-node-2017-05-14.8.log:0
kaa-node-2017-05-14.9.log:0
kaa-node-2017-05-15.0.log:0
kaa-node-2017-05-15.1.log:0
kaa-node-2017-05-15.2.log:0
kaa-node-2017-05-15.3.log:0
kaa-node-2017-05-15.4.log:0
kaa-node-2017-05-15.5.log:0
kaa-node-2017-05-15.6.log:276
kaa-node-2017-05-15.7.log:875
kaa-node-2017-05-15.8.log:873
kaa-node-2017-05-15.9.log:539
kaa-node-2017-05-16.0.log:879
kaa-node-2017-05-16.1.log:878
kaa-node-2017-05-16.2.log:878
kaa-node-2017-05-16.3.log:878
kaa-node-2017-05-16.4.log:879
kaa-node-2017-05-16.5.log:880
kaa-node-2017-05-16.6.log:877
kaa-node-2017-05-16.7.log:878
kaa-node-2017-05-16.8.log:878
kaa-node-2017-05-16.9.log:735
kaa-node-2017-05-17.0.log:877
kaa-node-2017-05-17.1.log:878
kaa-node-2017-05-17.2.log:878
kaa-node-2017-05-17.3.log:878
kaa-node-2017-05-17.4.log:877
kaa-node-2017-05-17.5.log:876
kaa-node-2017-05-17.6.log:871
kaa-node-2017-05-17.7.log:868
kaa-node-2017-05-17.8.log:877
kaa-node-2017-05-17.9.log:760
kaa-node-2017-05-18.0.log:876
kaa-node-2017-05-18.1.log:876
kaa-node-2017-05-18.2.log:876
kaa-node-2017-05-18.3.log:876
kaa-node-2017-05-18.4.log:877
kaa-node-2017-05-18.5.log:880
kaa-node-2017-05-18.6.log:879
kaa-node-2017-05-18.7.log:878
kaa-node-2017-05-18.8.log:876
kaa-node-2017-05-18.9.log:746
kaa-node-2017-05-19.0.log:876
kaa-node-2017-05-19.1.log:877
kaa-node-2017-05-19.2.log:877
kaa-node-2017-05-19.3.log:879
kaa-node-2017-05-19.4.log:878
kaa-node-2017-05-19.5.log:879
kaa-node-2017-05-19.6.log:1037
kaa-node-2017-05-19.7.log:1285
kaa-node-2017-05-19.8.log:1051
kaa-node-2017-05-20.0.log:1887
kaa-node-2017-05-20.1.log:2400
kaa-node-2017-05-20.2.log:2396
kaa-node-2017-05-20.3.log:1957
kaa-node-2017-05-21.0.log:8640
kaa-node-2017-05-22.0.log:5326
kaa-node-2017-05-22.1.log:0
kaa-node-2017-05-22.2.log:0
kaa-node-2017-05-22.3.log:0
kaa-node-2017-05-22.4.log:0
kaa-node-2017-05-23.0.log:0
kaa-node-2017-05-23.1.log:0
While creating Neo4j HA cluster using 2 machines, after making necessary changes and starting neo4j service, an authentication card pops up in the slave neo4j web UI, asking for username and password, even though security is disabled in the neo4j-server.properties file of both instances.
For entering a wrong password, it shows the error invalid username or password, however throws an error 'HTTP Status: 500 - Server Error' for entering the right password.
The console.log shows a different picture though:
ERROR Failed to start transaction. Timeout waiting for database to become available and allow new transactions. Waited 2m. 1 reasons for blocking: High Availability member state not ready.
org.neo4j.graphdb.TransactionFailureException: Timeout waiting for database to become available and allow new transactions. Waited 2m. 1 reasons for blocking: High Availability member state not ready.
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.checkAvailability(GraphDatabaseFacade.java:349)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:298)
at org.neo4j.server.rest.transactional.TransitionalPeriodTransactionMessContainer.newTransaction(TransitionalPeriodTransactionMessContainer.java:39)
at org.neo4j.server.rest.transactional.TransactionHandle.ensureActiveTransaction(TransactionHandle.java:242)
at org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:144)
at org.neo4j.server.rest.web.TransactionalService$2.write(TransactionalService.java:211)
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.neo4j.server.rest.dbms.AuthorizationFilter.doFilter(AuthorizationFilter.java:116)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
at java.lang.Thread.run(Thread.java:745)
What the message says, the cluster is not ready yet.
Timeout waiting for database to become available and allow new
transactions. Waited 2m. 1 reasons for blocking: High Availability
member state not ready. org.neo4j.graphdb.TransactionFailureException:
Timeout waiting for database to become available and allow new
transactions. Waited 2m. 1 reasons for blocking: High Availability
member state not ready.
You need a quorum of at least 3 machines to run a cluster.
I am trying to get the current state of my job. The following is the code.
Job creation
final DataflowPipelineJob job =DataflowPipelineRunner.fromOptions(configuration.getOptions())
.run(pipe);
My code to get job state
DataflowPipelineJob job=new DataflowPipelineJob(Utils.getProjectId(), jobId, getDataflowClient());
job.getState();
But it always throws the following warning
com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob getState: There were problems getting current job status:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 OK
Not Found
at
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1049)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.getState(DataflowPipelineJob.java:188)
at com.pwc.google.dataflow.factory.DataFlowFactory.getStatus(DataFlowFactory.java:89)
at com.pwc.google.dataflow.controllers.JobMonitor.doPost(JobMonitor.java:28)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:482)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:230)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:234)
at java.lang.Thread.run(Thread.java:745)
And it returns UNKNOWN as state.
Please help to fix this.
You should create your Dataflow client object like this:
DataflowPipelineOptions options = PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
Dataflow client = com.google.cloud.dataflow.sdk.util.Transport.newDataflowClient(options);
This will initialize credentials properly and automatically handle retries.
I am new to gcm and I tried to connect to Cloud Connection Server(XMPP) of GCM using Smack API. It was alright at the start,
My code:
uid = "123456789";
apiKey = "A**************B";
XMPPTCPConnectionConfiguration.Builder config;
config.setSocketFactory(SSLSocketFactory.getDefault());
config = XMPPTCPConnectionConfiguration.builder();
config.setUsernameAndPassword(uid,apiKey);
config.setServiceName("gcm.googleapis.com");
config.setHost("gcm.googleapis.com");
config.setPort(5235);
config.setDebuggerEnabled(true);
mConnection = new XMPPTCPConnection(config.build());
mConnection.setPacketReplyTimeout(10000);
try {
mConnection.connect();
mConnection.login();
}
catch (SmackException | IOException | XMPPException e) {
System.out.println("Exception at SmackCcsClient.init()");
e.printStackTrace();
}
But I couldnt get past the initial handshaking process. I used some dummy random GCMIDs to test downstream messaging at first and it was showing up in the smack debug window but later on, the same code shows nothing after the following xml feed as Raw sent packets:
<stream:stream xmlns='jabber:client' to='gcm.googleapis.com' xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xml:lang='en'>
and i tried
mConnection.login(uid+"#gcm.googleapis.com",apiKey);//even though i assume its next step of the handshake.
Console prints the following errors:
org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 10000ms (~10s). Used filter: No filter used or filter was 'null'.
at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:106)
at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:85)
at org.jivesoftware.smack.SynchronizationPoint.checkForResponse(SynchronizationPoint.java:253)
at org.jivesoftware.smack.SynchronizationPoint.checkIfSuccessOrWait(SynchronizationPoint.java:146)
at org.jivesoftware.smack.SynchronizationPoint.checkIfSuccessOrWaitOrThrow(SynchronizationPoint.java:125)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:837)
at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:360)
at psdc.gcm.SmackCcsClient.init(SmackCcsClient.java:64)
at psdc.gcm.GCMServer.activate(GCMServer.java:44)
at psdc.servlets.Mapper.selectIds(Mapper.java:191)
at psdc.servlets.Mapper.doPost(Mapper.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Jun 25, 2015 5:36:18 PM org.jivesoftware.smack.AbstractXMPPConnection callConnectionClosedOnErrorListener
WARNING: Connection closed with error
java.io.EOFException: input contained no data
at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:2965)
at org.xmlpull.mxp1.MXParser.more(MXParser.java:3003)
at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1409)
at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1394)
at org.xmlpull.mxp1.MXParser.next(MXParser.java:1092)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1151)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$200(XMPPTCPConnection.java:937)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:952)
at java.lang.Thread.run(Unknown Source)
Please help me to solve this as i am really stuck with this and nowhere to go.Please tell me a way to check if my xml request reaches google or not.
Am using the SMACK library version 4.1.1
Using config.setSecurityMode(ConnectionConfiguration.SecurityMode.ifpossible); solved my problem. It defines the configuration of the connection to be used.It Turns on TLS if the server supports the same.
refer : http://www.igniterealtime.org/builds/smack/docs/latest/javadoc/org/jivesoftware/smack/ConnectionConfiguration.html
config = XMPPTCPConnectionConfiguration.builder();
config.setSecurityMode(ConnectionConfiguration.SecurityMode.ifpossible);
config.setSocketFactory(SSLSocketFactory.getDefault());
config.setUsernameAndPassword(uid,apiKey);
config.setServiceName("gcm.googleapis.com");
config.setHost("gcm.googleapis.com");
config.setPort(5235);
config.setDebuggerEnabled(true);
mConnection = new XMPPTCPConnection(config.build());
mConnection.setPacketReplyTimeout(10000);
try {
mConnection.connect();
Hope it helps someone.