Hi am working on Wso2ESB 4.9.0 and recently while monitoring JMX attributes of the ESB server and observed there are some fault receiving and fault sending and timeouts are happening in the ESB.
Below are the the attributes of the following objects in JMX:
org.apache.synapse:Type=Transport,Name=passthru-http-sender
org.apache.synapse:Type=Transport,Name=passthru-http-receiver
[EsbMonitoring] ***************** Transport http Attributes *********************
[EsbMonitoring] Sender Stats---------------------
[EsbMonitoring] Active threads :0
[EsbMonitoring] FaultsReceiving :3
[EsbMonitoring] FaultsSending :3
[EsbMonitoring] TimeoutsReceiving :34
[EsbMonitoring] TimeoutsSending :0
[EsbMonitoring] QueueSize :0
[EsbMonitoring] MessagesSent :28786806
[EsbMonitoring] MessagesReceived :28786728
[EsbMonitoring] Receiver Stats-------------------
[EsbMonitoring] Active threads :0
[EsbMonitoring] FaultsReceiving :69
[EsbMonitoring] FaultsSending :0
[EsbMonitoring] TimeoutsReceiving :0
[EsbMonitoring] TimeoutsSending :0
[EsbMonitoring] QueueSize :0
[EsbMonitoring] MessagesSent :12210777
[EsbMonitoring] MessagesReceived :12228538
They are some fault receiving and fault sending connections between Wso2ESB and Wso2DSS,There are around 34 timeouts receiving and 3 each on fault sending and receiving in the passthru-http-sender and 69 fault receiving on passthru-http-reveived side,what is the exact reason this faults happen. How to know the reason why this happened and how to avoid such things.
Related
We implemented connection pooling in our client code to invoke a server which closes(sends Connection:close in response headers) a connection after 2.5mins. Due to server behaviour we sometimes/intermittently get NoHttpResponseException. And this may occur at high TPS or at low TPS as well.
We are using apache http client version 4.5.11. And there is one validateAfterInactivity setting in PoolingHttpClientConnectionManager which is by-default set to 2000ms. But i think we may get same exception if we try to get the connection in 2000ms period.
We can choose to set aggressive value for validateAfterInactivity but i heard that it can degrade the performance by ~20 to 30ms for each request.
is retrying this exception a good solution ?
And also align to same context, can we retry in case of java.net.SocketException: Connection reset ?
#ok2c any suggestion here ?
Thanks in advance.
NoHttpResponseException is considered safe to retry for idempotent methods.
In your particular case however I would consider limiting the TTL (total to live) of client connections to 2.5 minutes to match that of the server endpoints.
I'm trying to get SRTP going in my iOS application using PJSIP. I have TLS working and without SRTP I can make and receive calls. However with SRTP I'm getting this odd 488 error on the INVITE. It's not able to initialize the media.
I've read other articles mentioning about the codecs. But I've insured that the codes being used by my Asterisk server and those compiled with PJSIP library on my iOS app are the same. The only thing I'm seeing here is that I'm enabled the crypto and PJSIP isn't liking it. Any thoughts?
INVITE sip:[REDACTED]#[REDACTED]:47229;transport=TLS;ob SIP/2.0
Via: SIP/2.0/TLS [REDACTED]:5161;rport;branch=z9hG4bKPj8ea1a332-0748-438f-ae74-5d17b038891d;alias
From: "Test" <sip:asterisk#172.31.18.138>;tag=7c3663cb-b5f5-4762-8526-8425d18b2466
To: <sip:[REDACTED]#[REDACTED];ob>
Contact: <sip:asterisk#[REDACTED]:5161;transport=TLS>
Call-ID: f454ef36-01ea-4f29-9482-4a10768bf1b7
CSeq: 24942 INVITE
Allow: OPTIONS, SUBSCRIBE, NOTIFY, PUBLISH, INVITE, ACK, BYE, CANCEL, UPDATE, PRACK, REGISTER, MESSAGE, REFER
Supported: 100rel, timer, replaces, norefersub, path
Session-Expires: 1800
Min-SE: 90
Max-Forwards: 70
User-Agent: FPBX-AsteriskNOW-13.0.190.12(13.13.1)
Content-Type: application/sdp
Content-Length: 398
v=0
o=- 1582453973 1582453973 IN IP4 172.31.18.138
s=Asterisk
c=IN IP4 [REDACTED]
t=0 0
m=audio 11410 RTP/AVP 3 110 9 97 101
a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:84m7hqGvMjTU21xzkhBS3RQpQQjJ+aep0VwSlhx+
a=rtpmap:3 GSM/8000
a=rtpmap:110 speex/8000
a=rtpmap:9 G722/8000
a=rtpmap:97 iLBC/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=ptime:20
a=maxptime:60
a=sendrecv
--end msg--
19:10:11.601 pjsua_call.c .Incoming Request msg INVITE/cseq=24942 (rdata0x1421f0540)
19:10:11.603 tsx0x1421fe0a8 ...Transaction created for Request msg INVITE/cseq=24942 (rdata0x1421f0540)
19:10:11.603 tsx0x1421fe0a8 ..Incoming Request msg INVITE/cseq=24942 (rdata0x1421f0540) in state Null
19:10:11.603 tsx0x1421fe0a8 ...State changed from Null to Trying, event=RX_MSG
19:10:11.603 dlg0x1421fd8a8 ....Transaction tsx0x1421fe0a8 state changed to Trying
19:10:11.603 dlg0x1421fd8a8 ..UAS dialog created
19:10:11.603 dlg0x1421fd8a8 ..Module mod-invite added as dialog usage, data=0x141de7588
19:10:11.603 dlg0x1421fd8a8 ...Session count inc to 3 by mod-invite
19:10:11.603 inv0x1421fd8a8 ..UAS invite session created for dialog dlg0x1421fd8a8
19:10:11.603 dlg0x1421fd8a8 ...Session count inc to 3 by mod-pjsua
19:10:11.603 pjsua_media.c ..Call 0: initializing media..
19:10:11.603 pjsua_call.c ..Error initializing media channel: Not Acceptable Here [status=170488]
19:10:11.604 endpoint ..Response msg 488/INVITE/cseq=24942 (tdta0x1421fe800) created
19:10:11.604 dlg0x1421fd8a8 ...Sending Response msg 488/INVITE/cseq=24942 (tdta0x1421fe800)
19:10:11.606 tsx0x1421fe0a8 ...Sending Response msg 488/INVITE/cseq=24942 (tdta0x1421fe800) in state Trying
19:10:11.606 pjsua_core.c ....TX 429 bytes Response msg 488/INVITE/cseq=24942 (tdta0x1421fe800) to TLS [REDACTED]:5161:
SIP/2.0 488 Not Acceptable Here
Via: SIP/2.0/TLS [REDACTED]:5161;rport=5161;received=[REDACTED];branch=z9hG4bKPj8ea1a332-0748-438f-ae74-5d17b038891d;alias
Call-ID: f454ef36-01ea-4f29-9482-4a10768bf1b7
From: "Test" <sip:asterisk#172.31.18.138>;tag=7c3663cb-b5f5-4762-8526-8425d18b2466
To: <sip:[REDACTED]#[REDACTED];ob>;tag=5oFGceZO4ZaKpLFEg7piOrM7IV2yeDLT
CSeq: 24942 INVITE
Content-Length: 0
--end msg--
In case anyone else has this issue. I'll tell you what solved this for me.
On Asterisk in my endpoint (pjsip show endpoint myendpoint) setting I had media_encryption_optimistic set to true. When I set this to false everything started working.
I'm not sure why as the how to on asterisk stated to turn this on. But I confirmed at all traffic was indeed encrypted by using wireshark to check the actual voice data.
If anyone know why this would be the case that this needed to be set to false it would help me better understand this. But for now I'm up and running.
I had this error on PJSIP ios and I solved on my end after DISABLING "SRTP" or "S-RTP" or "Secure RTP" or "Secure Transport". I removed this for TLS configuration.
//acc_cfg.srtp_secure_signaling = 1;
//acc_cfg.use_srtp = PJMEDIA_SRTP_MANDATORY;
488 / Not Acceptable Here
I also get the same problem with my server, atlast i find the solution
by getting to know that it is because codec, use right codec or
disable codec works great for you.
I'm getting this EOF error when attempting to register my iOS project with my Asterisk server. Without TLS everything is working. So I'm now going through the process of enabling TLS. I've recompiled the PJSIP libraries and configured the Asterisk server. I believe that is all correct. I can see the traffic occurring in an encrypted form over port 5161 so that appears to be correct as well.
1) What I'm curious about is what could cause this error? Could it be the certificates that I generated? Based on the fact that I see encrypted traffic on wireshark but not in the Asterisk pjsip logger is making me thing it's the cert?
2) Second question is how do you create a certificate for a client who's hostname will be unknown (i.e. their IP will obviously change as they roam on cellular or to different WIFI networks). When I created them according to the Asterisk tutorial (https://wiki.asterisk.org/wiki/display/AST/Secure+Calling+Tutorial) it required a value for the -C parameter to specify the client hostname. Which again this would assumed to be dynamic.
Here's the log output (personal info [REDACTED])...
15:26:15.857 os_core_unix.c !pjlib 2.6 for POSIX initialized
15:26:15.858 sip_endpoint.c .Creating endpoint instance...
15:26:15.859 pjlib .select() I/O Queue created (0x11e809228)
15:26:15.859 sip_endpoint.c .Module "mod-msg-print" registered
15:26:15.859 sip_transport. .Transport manager created.
15:26:15.859 pjsua_core.c .PJSUA state changed: NULL --> CREATED
15:26:15.859 sip_endpoint.c .Module "mod-pjsua-log" registered
15:26:15.859 sip_endpoint.c .Module "mod-tsx-layer" registered
15:26:15.859 sip_endpoint.c .Module "mod-stateful-util" registered
15:26:15.859 sip_endpoint.c .Module "mod-ua" registered
15:26:15.859 sip_endpoint.c .Module "mod-100rel" registered
15:26:15.859 sip_endpoint.c .Module "mod-pjsua" registered
15:26:15.860 sip_endpoint.c .Module "mod-invite" registered
15:26:15.888 coreaudio_dev. .. dev_id 0: iPhone IO device (in=1, out=1) 8000Hz
15:26:15.888 coreaudio_dev. ..core audio initialized
15:26:15.888 pjlib ..select() I/O Queue created (0x11e16fc28)
15:26:15.888 speex_codec.c ..Adjusting quality to 5 for uwb
15:26:15.889 conference.c ..Creating conference bridge with 12 ports
15:26:15.889 Master/sound ..Using delay buffer with WSOLA.
15:26:15.907 sip_endpoint.c .Module "mod-evsub" registered
15:26:15.907 sip_endpoint.c .Module "mod-presence" registered
15:26:15.907 evsub.c .Event pkg "presence" registered by mod-presence
15:26:15.907 sip_endpoint.c .Module "mod-mwi" registered
15:26:15.907 evsub.c .Event pkg "message-summary" registered by mod-mwi
15:26:15.907 sip_endpoint.c .Module "mod-refer" registered
15:26:15.907 evsub.c .Event pkg "refer" registered by mod-refer
15:26:15.907 sip_endpoint.c .Module "mod-pjsua-pres" registered
15:26:15.907 sip_endpoint.c .Module "mod-pjsua-im" registered
15:26:15.907 sip_endpoint.c .Module "mod-pjsua-options" registered
15:26:15.908 pjsua_core.c .1 SIP worker threads created
15:26:15.908 pjsua_core.c .pjsua version 2.6 for iOS-10.2.1/arm-iPad6,7/iOS-SDK initialized
15:26:15.908 pjsua_core.c .PJSUA state changed: CREATED --> INIT
15:26:15.910 tlstp:5161 SIP TLS listener is ready for incoming connections at 10.200.154.118:5161
15:26:15.910 pjsua_core.c PJSUA state changed: INIT --> STARTING
15:26:15.910 sip_endpoint.c .Module "mod-unsolicited-mwi" registered
15:26:15.910 pjsua_core.c .PJSUA state changed: STARTING --> RUNNING
15:26:15.910 pjsua_acc.c Adding account: id=sip:[REDACTED]#[REDACTED]
15:26:15.910 pjsua_acc.c .Account sip:[REDACTED]#[REDACTED] added with id 0
15:26:15.910 pjsua_acc.c .Acc 0: setting registration..
15:26:15.912 tlsc0x11e18882 ..TLS client transport created
15:26:15.912 tlsc0x11e18882 ..TLS transport 10.200.154.118:58635 is connecting to [REDACTED]:5161...
15:26:15.912 pjsua_acc.c ..Contact for acc 0 updated: <sip:[REDACTED]#10.200.154.118:58635;transport=TLS;ob>;reg-id=1;+sip.instance="<urn:uuid:00000000-0000-0000-0000-00009c3fe9b2>"
15:26:15.912 endpoint ..Request msg REGISTER/cseq=10712 (tdta0x11e950400) created.
15:26:15.913 tsx0x11e9524a8 ...Transaction created for Request msg REGISTER/cseq=10713 (tdta0x11e950400)
15:26:15.913 tsx0x11e9524a8 ..Sending Request msg REGISTER/cseq=10713 (tdta0x11e950400) in state Null
15:26:15.913 sip_resolve.c ...Target '[REDACTED]:5161' type=TLS resolved to '[REDACTED]:5161' type=TLS (TLS transport)
15:26:15.913 pjsua_core.c ...TX 730 bytes Request msg REGISTER/cseq=10713 (tdta0x11e950400) to TLS [REDACTED]:5161:
REGISTER sip:[REDACTED]:5161;transport=tls SIP/2.0
Via: SIP/2.0/TLS 10.200.154.118:58635;rport;branch=z9hG4bKPjYEl90guBh1gvKz2dDJZlgxzyy7AzLMpx;alias
Max-Forwards: 70
From: <sip:[REDACTED]#[REDACTED]>;tag=V6bPvLX5S0jBnpssQQBhG1W2xhXz01gj
To: <sip:[REDACTED]#[REDACTED]>
Call-ID: 5h.iiZFsYAX2vtMCNx.ZmH9NsicZoQEX
CSeq: 10713 REGISTER
User-Agent: [REDACTED]
Supported: outbound, path
Contact: <sip:[REDACTED]#10.200.154.118:58635;transport=TLS;ob>;reg-id=1;+sip.instance="<urn:uuid:00000000-0000-0000-0000-00009c3fe9b2>"
Expires: 300
Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS
Content-Length: 0
--end msg--
15:26:15.914 tsx0x11e9524a8 ...State changed from Null to Calling, event=TX_MSG
15:26:15.914 pjsua_acc.c ..Acc 0: Registration sent
15:26:16.001 tlsc0x11e18882 !TLS connect() error: End of file (PJ_EEOF) [code=70016]
15:26:16.001 tsx0x11e9524a8 Failed to send Request msg REGISTER/cseq=10713 (tdta0x11e950400)! err=70016 (End of file (PJ_EEOF))
15:26:16.001 tsx0x11e9524a8 State changed from Calling to Terminated, event=TRANSPORT_ERROR
15:26:16.001 pjsua_acc.c ..SIP registration failed, status=503 (End of file (PJ_EEOF))
15:26:16.001 pjsua_acc.c ..Scheduling re-registration retry for acc 0 in 2 seconds..
15:26:16.001 PJSIPInterface ..AccountID 0 reg_state=503
15:26:16.001 tlsc0x11e18882 TLS send() error, sent=-70016
15:26:16.001 pjsua_acc.c Disconnected notification for transport tlsc0x11e188828
15:26:16.001 tsx0x11e9524a8 Timeout timer event
15:26:16.001 tsx0x11e9524a8 .State changed from Terminated to Destroyed, event=TIMER
15:26:16.001 tdta0x11e95040 ..Destroying txdata Request msg REGISTER/cseq=10713 (tdta0x11e950400)
15:26:16.001 tsx0x11e9524a8 Transaction destroyed!
15:26:16.001 tlsc0x11e18882 TLS transport destroyed with reason 70016: End of file (PJ_EEOF)
Here's my transport setup code in my iOS app...
pjsua_transport_config tlsTransportConfig;
pjsua_transport_config_default(&tlsTransportConfig);
tlsTransportConfig.port = 5161;
tlsTransportConfig.tls_setting.ca_list_file = pj_str((char*)[[[NSBundle mainBundle] pathForResource:#"ca.crt" ofType:#"pem"] cStringUsingEncoding:NSUTF8StringEncoding]);
tlsTransportConfig.tls_setting.cert_file = pj_str((char*)[[[NSBundle mainBundle] pathForResource:#"myapp.crt" ofType:#"pem"] cStringUsingEncoding:NSUTF8StringEncoding]);
tlsTransportConfig.tls_setting.privkey_file = pj_str((char*)[[[NSBundle mainBundle] pathForResource:#"myapp.key" ofType:#"pem"] cStringUsingEncoding:NSUTF8StringEncoding]);
tlsTransportConfig.tls_setting.password = pj_str("[REDACTED]");
tlsTransportConfig.tls_setting.verify_client = PJ_TRUE;
tlsTransportConfig.tls_setting.verify_server = PJ_TRUE;
tlsTransportConfig.tls_setting.method = PJSIP_SSLV2_METHOD;
status = pjsua_transport_create(PJSIP_TRANSPORT_TLS, &tlsTransportConfig, NULL);
In case anyone else runs into this issue. As per a tutorial I was following I had set the tls method to PJSIP_SSLV2_METHOD as you can see in my code above. However when I changed this to PJSIP_TLSV1_METHOD everything started working. This also requires this to be changed in the PJSIP settings for the SSL method as well.
I'm not sure why SSLV2 didn't work but tlsv1 works just fine.
I think this also might relate to the server side. If the server you are using is Opensips it might be that SIP message is too fragmented and OpenSIPS closes the connection to prevent packets fragmentation attacks, default value is 4.
Changing the .cfg file on the server option should fix the issue
modparam("proto_tls", "tls_max_msg_chunks", 8)
While creating Neo4j HA cluster using 2 machines, after making necessary changes and starting neo4j service, an authentication card pops up in the slave neo4j web UI, asking for username and password, even though security is disabled in the neo4j-server.properties file of both instances.
For entering a wrong password, it shows the error invalid username or password, however throws an error 'HTTP Status: 500 - Server Error' for entering the right password.
The console.log shows a different picture though:
ERROR Failed to start transaction. Timeout waiting for database to become available and allow new transactions. Waited 2m. 1 reasons for blocking: High Availability member state not ready.
org.neo4j.graphdb.TransactionFailureException: Timeout waiting for database to become available and allow new transactions. Waited 2m. 1 reasons for blocking: High Availability member state not ready.
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.checkAvailability(GraphDatabaseFacade.java:349)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:298)
at org.neo4j.server.rest.transactional.TransitionalPeriodTransactionMessContainer.newTransaction(TransitionalPeriodTransactionMessContainer.java:39)
at org.neo4j.server.rest.transactional.TransactionHandle.ensureActiveTransaction(TransactionHandle.java:242)
at org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:144)
at org.neo4j.server.rest.web.TransactionalService$2.write(TransactionalService.java:211)
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.neo4j.server.rest.dbms.AuthorizationFilter.doFilter(AuthorizationFilter.java:116)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
at java.lang.Thread.run(Thread.java:745)
What the message says, the cluster is not ready yet.
Timeout waiting for database to become available and allow new
transactions. Waited 2m. 1 reasons for blocking: High Availability
member state not ready. org.neo4j.graphdb.TransactionFailureException:
Timeout waiting for database to become available and allow new
transactions. Waited 2m. 1 reasons for blocking: High Availability
member state not ready.
You need a quorum of at least 3 machines to run a cluster.
I am trying to insert 10000 records in redis from erlang using gen_server. However, i get following exception
exception exit: {connection_error,{connection_error,eaddrnotavail}}
Note:-
Ports range on redis server is sufficient
Redis is configured to accept 10000 connections at once
I also tried using timer:sleep to rule out the possibility of connections are getting full.
I am starting connection , firing query and closing connection immediately
Call from gen_server to redis is synchronous
I am using eredis as a library
I get this error approximately 200 to 500 insertions in redis
Got It worked :) Posting answer so that it can help others...
Problem was kernal TIME_WAIT.
Eredis was using gen_tcp and as I was inside the fast network and generating 10000 connections Many of the connections was in TIME_WAIT state and eredis was using resuse addr to true so Although I close the connection in my code OS was the port which were in TIME_WAIT state and erlang was again trying to connect to that port.
Thanks for the posting.
Posting some of my changes as well.
change the eredis.hrl from
-define(SOCKET_OPTS, [binary, {active, once}, {packet, raw}, {reuseaddr, true}]).
to
-define(SOCKET_OPTS, [binary, {active, once}, {packet, raw}, {reuseaddr, false}]).