Http connection using apache.DefaultHttpClient goes into TIME_WAIT not getting reused - connection-pooling

Question

We have a pool(GenericObjectPool) of HttpClient(apache.DefaultHttpClient) objects. HttpPost objects are put into for execution through these clients may Post request are sent simultaneously. The protocol used is: HTTP/1.1 with keepalive.
Observed during load testing through netstat that new socket connections are indiscriminately created and old connections goes to TIME_WAIT.
Log Excerpts:-
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "POST /INBOX/4504a09e-13c0-3853-a285-9e2b9a22f65e/1e1e5a20-a8c1-11e2-99b8-7c19e9129271 HTTP/1.1[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "Content-Length: 117[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "Host: rwcdtgxb0402:15010[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "Connection: Keep-Alive[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "User-Agent: Apache-HttpClient/4.2.1 (java 1.5)[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - >> "[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - << "HTTP/1.1 200 OK[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - << "Content-Length: 0[\r][\n]"
[Worker-2] org.apache.http.impl.conn.Wire 63 - << "[\r][\n]"
[Worker-2] org.apache.http.impl.conn.DefaultClientConnection 254 - Receiving response: HTTP/1.1 200 OK
[Worker-2] org.apache.http.impl.conn.DefaultClientConnection 257 - << HTTP/1.1 200 OK
[Worker-2] org.apache.http.impl.conn.DefaultClientConnection 260 - << Content-Length: 0
[Worker-2] org.apache.http.impl.client.DefaultRequestDirector 540 - Connection can be kept alive indefinitely
**[Worker-2] org.apache.http.impl.conn.DefaultClientConnection 154 - Connection 0.0.0.0:51211<->192.168.88.172:15010 shut down**
[Worker-2] org.apache.http.impl.conn.BasicClientConnectionManager 189 - Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl#12f65ce5
Is DefaultClientConnection.shutdown (Connection 0.0.0.0:51210<->192.168.88.172:15010 shut down) closing the connection from Client end? How is it getting invoked?
In code after receiving the response (200OK) from server httpPost.releaseConnection() is only executed in client side code.
What should I do to keep the connections in ESTABLISHED state and reuse them instead of creating the connection for every request and they going into TIME_WAIT.
Any help will be highly appreciated. Thank you.

Answer 1

Note calling EntityUtils.consume(entity) does a bit more and should be used:
/**
* Ensures that the entity content is fully consumed and
* the content stream, if exists, is closed.
*
* #param entity the entity to consume.
* #throws IOException if an error occurs reading the input stream
*
* #since 4.1
*/
public static void consume(final HttpEntity entity) throws IOException {
if (entity == null) {
return;
}
if (entity.isStreaming()) {
final InputStream inStream = entity.getContent();
if (inStream != null) {
inStream.close();
}
}
}

Answer 2

Got the problem resolved few days back.
Mistake: calling releaseConnection() on the HttpClient object. BUT, avoiding it did not solve the problem either.
Correction:
The action to be taken for keeping the connections alive (in state ESTABLISHED) was through following sample code:
HttpClient client = new org.apache.http.impl.client.DefaultHttpClient();
...
org.apache.http.HttpResponse response = client.executed(HttpPost) ;
org.apache.http.HttpEntity en = response.getEntity();
if (en != null) {
en.getContent().close();
}
Once the HttpResponse is received, in order to reuse and maintain the created connection it is necessary to close only the InputStream (en.getContent()) on the http-response. Do not call any other release method on HttpPost or Client objects.
Tested this handling with JMeter load and found it working. Haven't observed any side effects yet!