Getting Timeout from wso2 to endpoint going through http proxy - timeout

Question

I am getting an HTTP Timeout (101504) on an Endpoint, but only going through the ESB. This is a Soap1.1 HTTP/1.0 Endpoint. The connection to the esb and the endpoint are both secured and are both via ssl. In addition I am connecting via a proxy host and am also behind a virtual private network.
I have configured the axis.xml to utilize the proxy host and it works for other Proxies that I have set up.
I am able to hit the endpoint directly from SoapUI with no issues it returns in about 180ms. I can also curl from the esb host command line going thru the http proxy host and it works perfectly. So I know the proxy server isn't preventing the request from passing through.
Originally I was using the HttpPassthruSender and was getting a Connect Cancel 101507 error. I thought that may be an issue so I switched it to the NIO Sender/Receiver. Then it started producing a 101504 HTTP Transport Error.
I have quadruple checked the endpoint address and even copy/pasted it into soapUI and ran it successfully to make sure it was correct
I have included below my proxy configuration and my endpoint configuration. Note that I have included the HTTP_FORCE_1.0 and POST_TO_URI Properties. I am also certain that my credentials for the endpoint are correct, because I have purposely changed them and get the expected result when authentication fails.
Proxy Configuration
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="TestRemedy"
transports="https"
statistics="disable"
trace="enable"
startOnLoad="true">
<target faultSequence="fault" endpoint="RemedyQueryIncidents">
<inSequence>
<call-template target="SetTriadBasicAuthHeader">
<with-param name="username" value="mouser"/>
<with-param name="password" value="my password"/>
</call-template>
<property name="POST_TO_URI" value="true" scope="axis2" type="STRING"/>
<property name="FORCE_HTTP_1.0" value="true" scope="axis2" type="STRING"/>
</inSequence>
<outSequence>
<send/>
</outSequence>
</target>
<description/>
</proxy>
Endpoint Configuration
<endpoint xmlns="http://ws.apache.org/ns/synapse" name="RemedyQueryIncidents">
<address uri="https://myendpoint.com/IncidentQueryList" format="soap11">
<suspendOnFailure>
<progressionFactor>1.0</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
<retryDelay>0</retryDelay>
</markForSuspension>
</address>
</endpoint>
Log (With Log Level TRACE) To remain within the 30K Character limit this is not complete. I can send the complete log if necessary. I attempted to keep it to the relevant parts.
...
[2013-11-19 21:54:45,125] DEBUG - SoapAction is (urn:IncidentQueryList/Incident_QueryList_Service) {org.apache.axis2.context.MessageContext}
[2013-11-19 21:54:45,125] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][r:w][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Set attribute ENDPOINT_PREFIX {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,125] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][r:w][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Set attribute HTTP_REQ_METHOD {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,125] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][r:w][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Set attribute http.request {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,125] DEBUG - I/O session http-incoming-53-89 myesbhost.com:8243<->myesbhost.com:34746[ACTIVE][r:r][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Set attribute REQ_DEPARTURE_TIME {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37: POST https://myendpoint.com/IncidentQueryList HTTP/1.0 {org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> POST https://myendpoint.com/IncidentQueryList HTTP/1.0 {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> Content-Type: text/xml; charset=UTF-8 {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> Accept-Encoding: gzip,deflate {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> Authorization: Basic UldfVFJJQUQuZ2VuOmszcW5OUn5QR3Axdw== {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> X-Forwarded-For: myvpnhost.com {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> X-Forwarded-Host: myesbhost.com {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> X-Forwarded-Server: myesbhost.com {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> SOAPAction: "urn:IncidentQueryList/Incident_QueryList_Service" {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> Content-Length: 422 {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> Host: myendpoint.com:443 {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> Connection: Keep-Alive {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - http-outgoing-37 >> User-Agent: Synapse-HttpComponents-NIO {org.apache.synapse.transport.http.headers}
[2013-11-19 21:54:45,127] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][r:w][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Set event [w] {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,128] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][rw:w][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Set event [w] {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,128] DEBUG - http-outgoing-37: Produce output {org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection}
[2013-11-19 21:54:45,128] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][rw:w][ACTIVE][NOT_HANDSHAKING][0][0][576][0]: 555 bytes written {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,128] DEBUG - << "POST myendpoint.com/IncidentQueryList HTTP/1.0[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,128] DEBUG - << "Content-Type: text/xml; charset=UTF-8[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,128] DEBUG - << "Accept-Encoding: gzip,deflate[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,128] DEBUG - << "Authorization: Basic myBlahBlahBlahBase64endodedPassword==[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,128] DEBUG - << "X-Forwarded-For: myvpnhost.com[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "X-Forwarded-Host: myesbhost.com[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "X-Forwarded-Server: myesbhost.com[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "SOAPAction: "urn:IncidentQueryList/Incident_QueryList_Service"[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "Content-Length: 422[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "Host: myendpoint.com:443[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "Connection: Keep-Alive[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "User-Agent: Synapse-HttpComponents-NIO[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - << "[\r][\n]" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][rw:w][ACTIVE][NOT_HANDSHAKING][0][0][576][0]: 0 bytes written {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,129] DEBUG - http-outgoing-37: Produce output {org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection}
[2013-11-19 21:54:45,129] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][rw:w][ACTIVE][NOT_HANDSHAKING][0][0][443][0]: 422 bytes written {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,129] DEBUG - << "<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:IncidentQueryList"><soapenv:Body><urn:Incident_QueryList_Service><urn:Qualification>'Status'="Assigned" And 'Assigned Group'="MYGROUP-IT"</urn:Qualification><urn:startRecord>1</urn:startRecord><urn:maxLimit>1</urn:maxLimit></urn:Incident_QueryList_Service></soapenv:Body></soapenv:Envelope>" {org.apache.synapse.transport.http.wire}
[2013-11-19 21:54:45,129] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][rw:w][ACTIVE][NOT_HANDSHAKING][0][0][443][0]: Clear event [w] {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:54:45,129] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][rw:w][ACTIVE][NOT_HANDSHAKING][0][0][443][0]: Remove attribute synapse.axis2-attachment {org.apache.http.nio.reactor.ssl.SSLIOSession}
...
[2013-11-19 21:55:09,047] DEBUG - myesbhost.com:8243 - - - "POST /services/TestRemedy HTTP/1.1" - - "-" "Apache-HttpClient/4.1.1 (java 1.5)" {org.apache.synapse.transport.nhttp.access}
[2013-11-19 21:55:09,047] DEBUG - - - - "- - " 200 - "-" "-" {org.apache.synapse.transport.nhttp.access}
[2013-11-19 21:55:09,048] DEBUG - - - - - "CONNECT myendpoint.com:443 HTTP/1.1" - - "-" "-" {org.apache.synapse.transport.nhttp.access}
[2013-11-19 21:55:09,048] DEBUG - myendpoint.com:443 - - - "POST myendpoint.com/IncidentQueryList HTTP/1.0" - - "-" "Synapse-HttpComponents-NIO" {org.apache.synapse.transport.nhttp.access}
[2013-11-19 21:55:20,776] DEBUG - Running deployment synchronizer update... tenant : carbon.super {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}
[2013-11-19 21:55:20,778] DEBUG - Running deployment synchronizer commit... tenant : carbon.super {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}
[2013-11-19 21:55:20,778] TRACE - Retrieving user manager components from the realm. {org.wso2.carbon.registry.core.jdbc.realm.RegistryRealm}
[2013-11-19 21:55:20,778] TRACE - Setting the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234, callerTenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,778] TRACE - Saving managed registry connection to map. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:20,779] TRACE - Checking the existence of the root collection of the Registry. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,779] TRACE - Committed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:20,779] TRACE - Closed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:20,779] TRACE - adding system collections. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,779] TRACE - Setting the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234, callerTenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,779] TRACE - Checking the existence of '/_system' collection of the Registry. {org.wso2.carbon.registry.core.utils.RegistryUtils}
[2013-11-19 21:55:20,780] TRACE - Clearing the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,780] TRACE - Clearing the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,780] TRACE - Preparing operation resource exists, path: repository/components/org.wso2.carbon.deployment.synchronizer/CarbonRepository. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,780] TRACE - Setting the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234, callerTenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,780] TRACE - Deriving the absolute path, chroot-base: /_system/local, path: repository/components/org.wso2.carbon.deployment.synchronizer/CarbonRepository. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:20,780] TRACE - Saving managed registry connection to map. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:20,780] TRACE - REG_PATH_CACHECache: REG_PATH_CACHEMemoryStore miss for org.wso2.carbon.caching.core.registry.RegistryCacheKey#b20989ce {net.sf.ehcache.store.MemoryStore}
[2013-11-19 21:55:20,780] TRACE - REG_PATH_CACHE cache - Miss {net.sf.ehcache.Cache}
[2013-11-19 21:55:20,781] TRACE - REG_PATH_CACHECache: REG_PATH_CACHEMemoryStore miss for org.wso2.carbon.caching.core.registry.RegistryCacheKey#3b52efd6 {net.sf.ehcache.store.MemoryStore}
[2013-11-19 21:55:20,781] TRACE - REG_PATH_CACHE cache - Miss {net.sf.ehcache.Cache}
[2013-11-19 21:55:20,781] TRACE - Committed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:20,781] TRACE - Closed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:20,781] TRACE - Clearing the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,781] DEBUG - Running deployment synchronizer update... tenant : carbon.super {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}
[2013-11-19 21:55:35,783] DEBUG - Running deployment synchronizer commit... tenant : carbon.super {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}
[2013-11-19 21:55:35,784] TRACE - Retrieving user manager components from the realm. {org.wso2.carbon.registry.core.jdbc.realm.RegistryRealm}
[2013-11-19 21:55:35,784] TRACE - Setting the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234, callerTenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,784] TRACE - Saving managed registry connection to map. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:35,784] TRACE - Checking the existence of the root collection of the Registry. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,785] TRACE - Committed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:35,785] TRACE - Closed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:35,785] TRACE - adding system collections. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,785] TRACE - Setting the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234, callerTenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,785] TRACE - Checking the existence of '/_system' collection of the Registry. {org.wso2.carbon.registry.core.utils.RegistryUtils}
[2013-11-19 21:55:35,785] TRACE - Clearing the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,786] TRACE - Clearing the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,786] TRACE - Preparing operation resource exists, path: repository/components/org.wso2.carbon.deployment.synchronizer/CarbonRepository. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,786] TRACE - Setting the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234, callerTenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,786] TRACE - Deriving the absolute path, chroot-base: /_system/local, path: repository/components/org.wso2.carbon.deployment.synchronizer/CarbonRepository. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:35,786] TRACE - Saving managed registry connection to map. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:35,786] TRACE - REG_PATH_CACHECache: REG_PATH_CACHEMemoryStore miss for org.wso2.carbon.caching.core.registry.RegistryCacheKey#b20989ce {net.sf.ehcache.store.MemoryStore}
[2013-11-19 21:55:35,786] TRACE - REG_PATH_CACHE cache - Miss {net.sf.ehcache.Cache}
[2013-11-19 21:55:35,787] TRACE - REG_PATH_CACHECache: REG_PATH_CACHEMemoryStore miss for org.wso2.carbon.caching.core.registry.RegistryCacheKey#3b52efd6 {net.sf.ehcache.store.MemoryStore}
[2013-11-19 21:55:35,787] TRACE - REG_PATH_CACHE cache - Miss {net.sf.ehcache.Cache}
[2013-11-19 21:55:35,787] TRACE - Committed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:35,787] TRACE - Closed all transactions. {org.wso2.carbon.registry.core.jdbc.utils.Transaction}
[2013-11-19 21:55:35,787] TRACE - Clearing the session for registry operation, chroot: /_system/local, username: wso2.system.user, tenantId: -1234. {org.wso2.carbon.registry.core.session.UserRegistry}
[2013-11-19 21:55:45,115] DEBUG - http-incoming-53: Consume input {org.apache.synapse.transport.http.conn.LoggingNHttpServerConnection}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 myesbhost.com:8243<->myesbhost.com:34746[ACTIVE][r:r][ACTIVE][NOT_HANDSHAKING][EOF][][0][0][0][0]: -1 bytes read {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] DEBUG - http-incoming-53: Close connection {org.apache.synapse.transport.http.conn.LoggingNHttpServerConnection}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 myesbhost.com:8243<->myesbhost.com:34746[ACTIVE][r:r][ACTIVE][NOT_HANDSHAKING][EOF][][0][0][0][0]: Close {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] DEBUG - http-incoming-53: Removing the connection : http-incoming-53 from pool of size : 0 {org.apache.synapse.transport.nhttp.ServerHandler}
[2013-11-19 21:55:45,115] DEBUG - http-incoming-53: Shutdown connection {org.apache.synapse.transport.http.conn.LoggingNHttpServerConnection}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 0.0.0.0:8243<->myesbhost.com:34746[CLOSED][][CLOSED][NEED_UNWRAP][EOF][][0][0][29][0]: Shutdown {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 0.0.0.0:8243<->myesbhost.com:34746[CLOSED][][CLOSED][NEED_UNWRAP][EOF][][0][0][29][0]: Remove attribute synapse.request-sink-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 0.0.0.0:8243<->myesbhost.com:34746[CLOSED][][CLOSED][NEED_UNWRAP][EOF][][0][0][29][0]: Remove attribute synapse.response-source-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 0.0.0.0:8243<->myesbhost.com:34746[CLOSED][][CLOSED][NEED_UNWRAP][EOF][][0][0][29][0]: Remove attribute synapse.connectionCreationTime {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] DEBUG - I/O session http-incoming-53-89 0.0.0.0:8243<->myesbhost.com:34746[CLOSED][][CLOSED][NEED_UNWRAP][EOF][][0][0][29][0]: Remove attribute synapse.server-connection-debug {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,115] TRACE - http-incoming-53: Connection closed {org.apache.synapse.transport.nhttp.ServerHandler}
[2013-11-19 21:55:45,195] DEBUG - http-outgoing-37: Connection timeout For : myproxyhostipaddr:3128 For Request : Axis2Request [Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687] [Status Completed : false] [Status SendingCompleted : true] {org.apache.synapse.transport.nhttp.ClientHandler}
[2013-11-19 21:55:45,195] DEBUG - http-outgoing-37: Connection to remote address : myproxyhost.com/myproxyhostipaddr:3128 from local address : /myesbhost.com:50695 is closed! {org.apache.synapse.transport.nhttp.ClientHandler}
[2013-11-19 21:55:45,195] DEBUG - http-outgoing-37: Shutdown connection {org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection}
[2013-11-19 21:55:45,195] DEBUG - getAction (null) from org.apache.axis2.client.Options#30e41c7e {org.apache.axis2.client.Options}
[2013-11-19 21:55:45,195] DEBUG - Old WSAAction is (null) {org.apache.axis2.context.MessageContext}
[2013-11-19 21:55:45,195] DEBUG - New WSAAction is (http://www.w3.org/2005/08/addressing/soap/fault) {org.apache.axis2.context.MessageContext}
[2013-11-19 21:55:45,195] DEBUG - setAction Old action is (null) {org.apache.axis2.client.Options}
[2013-11-19 21:55:45,195] DEBUG - setAction New action is (http://www.w3.org/2005/08/addressing/soap/fault) {org.apache.axis2.client.Options}
[2013-11-19 21:55:45,195] DEBUG - I/O session http-outgoing-37-91 myesbhost.com:50695<->myproxyhostipaddr:3128[ACTIVE][r:w][ACTIVE][NOT_HANDSHAKING][0][0][0][0]: Shutdown {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,195] TRACE - hasAnonymousAddress: http://www.w3.org/2005/08/addressing/anonymous is Anonymous: true {org.apache.axis2.addressing.EndpointReference}
[2013-11-19 21:55:45,195] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.response-sink-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,195] DEBUG - Unsetting USE_ASYNC_OPERATIONS and DISABLE_RESPONSE_ACK for unreliable message {org.apache.sandesha2.SandeshaModule}
[2013-11-19 21:55:45,196] DEBUG - Entry: SandeshaModule::resolveTarget {org.apache.sandesha2.SandeshaModule}
[2013-11-19 21:55:45,196] DEBUG - Entry: SandeshaUtil::isMessageUnreliable {org.apache.sandesha2.util.SandeshaUtil}
[2013-11-19 21:55:45,196] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.request-source-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,196] DEBUG - Exit: SandeshaUtil::isMessageUnreliable, false {org.apache.sandesha2.util.SandeshaUtil}
[2013-11-19 21:55:45,196] DEBUG - Exit: SandeshaModule::resolveTarget {org.apache.sandesha2.SandeshaModule}
[2013-11-19 21:55:45,196] DEBUG - start createFaultEnvelope() {org.apache.axis2.util.MessageContextBuilder}
[2013-11-19 21:55:45,196] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.client-connection-debug {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,196] DEBUG - axisFault != null {org.apache.axis2.util.MessageContextBuilder}
[2013-11-19 21:55:45,196] DEBUG - faultCode == null {org.apache.axis2.util.MessageContextBuilder}
[2013-11-19 21:55:45,196] DEBUG - context.isSOAP11() = true {org.apache.axis2.util.MessageContextBuilder}
[2013-11-19 21:55:45,196] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.connectionCreationTime {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,196] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.response-sink-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,196] DEBUG - End createFaultEnvelope() {org.apache.axis2.util.MessageContextBuilder}
[2013-11-19 21:55:45,196] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.request-source-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,196] DEBUG - Sending Fault for Request with Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687 {org.apache.synapse.transport.nhttp.ClientHandler}
[2013-11-19 21:55:45,196] TRACE - http-outgoing-37: Connection close For : myproxyhostipaddr:3128 For Request : Axis2Request [Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687] [Status Completed : true] [Status SendingCompleted : true] {org.apache.synapse.transport.nhttp.ClientHandler}
[2013-11-19 21:55:45,196] DEBUG - Callback removed for request message id : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687. Pending callbacks count : 0 {org.apache.synapse.core.axis2.SynapseCallbackReceiver}
[2013-11-19 21:55:45,197] DEBUG - http-outgoing-37: Keep-alive connection closed For : myproxyhostipaddr:3128 For Request : Axis2Request [Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687] [Status Completed : true] [Status SendingCompleted : true] {org.apache.synapse.transport.nhttp.ClientHandler}
[2013-11-19 21:55:45,197] DEBUG - Created a Error Log : ErrorLog{errorCode='101504'} {org.apache.synapse.aspects.statistics.ErrorLogFactory}
[2013-11-19 21:55:45,197] DEBUG - http-outgoing-37: Connection to remote address : myproxyhost.com/myproxyhostipaddr:3128 from local address : 0.0.0.0/0.0.0.0:50695 is closed! {org.apache.synapse.transport.nhttp.ClientHandler}
[2013-11-19 21:55:45,198] DEBUG - [Failed Request Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687] [New to be Retried Request Message ID : urn:uuid:97a91179-1350-47c4-af2c-3d746ecc57b9] {org.apache.synapse.core.axis2.SynapseCallbackReceiver}
[2013-11-19 21:55:45,198] DEBUG - http-outgoing-37: Shutdown connection {org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection}
[2013-11-19 21:55:45,198] WARN - ERROR_CODE : 101504 {org.apache.synapse.FaultHandler}
[2013-11-19 21:55:45,198] WARN - ERROR_MESSAGE : Connection timeout For : myproxyhostipaddr:3128 For Request : Axis2Request [Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687] [Status Completed : false] [Status SendingCompleted : true] {org.apache.synapse.FaultHandler}
[2013-11-19 21:55:45,198] WARN - ERROR_DETAIL : Connection timeout For : myproxyhostipaddr:3128 For Request : Axis2Request [Message ID : urn:uuid:ed377766-eb0a-4be7-92d9-f2b752289687] [Status Completed : false] [Status SendingCompleted : true] {org.apache.synapse.FaultHandler}
[2013-11-19 21:55:45,198] WARN - ERROR_EXCEPTION : null {org.apache.synapse.FaultHandler}
[2013-11-19 21:55:45,198] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Shutdown {org.apache.http.nio.reactor.ssl.SSLIOSession}
[2013-11-19 21:55:45,198] WARN - FaultHandler : Endpoint [RemedyQueryIncidents] {org.apache.synapse.FaultHandler}
[2013-11-19 21:55:45,198] DEBUG - Encountered a default HTTP connection timeout error : 101504 {org.apache.synapse.endpoints.AddressEndpoint}
[2013-11-19 21:55:45,198] DEBUG - Endpoint : RemedyQueryIncidents will be marked for SUSPENSION due to the occurrence of one of the configured errors {org.apache.synapse.endpoints.EndpointContext}
[2013-11-19 21:55:45,198] DEBUG - I/O session http-outgoing-37-91 0.0.0.0:50695<->myproxyhostipaddr:3128[CLOSED][][CLOSED][NOT_HANDSHAKING][0][0][0][0]: Remove attribute synapse.response-sink-buffer {org.apache.http.nio.reactor.ssl.SSLIOSession}