HttpURLConnection implementation - network-programming

I have read that HttpURLConnection supports persistent connections, so that a connection can be reused for multiple requests. I tried it and the only way to send a second POST was by calling openConnection for a second time. Otherwise I got a IllegalStateException("Already connected");
I used the following:
try{
URL url = new URL("http://someconection.com");
}
catch(Exception e){}
HttpURLConnection con = (HttpURLConnection) url.openConnection();
//set output, input etc
//send POST
//Receive response
//Read whole response
//close input stream
con.disconnect();//have also tested commenting this out
con = (HttpURLConnection) url.openConnection();
//Send new POST
The second request is send over the same TCP connection (verified it with wireshark) but I can not understand why (although this is what I want) since I have called disconnect.
I checked the source code for the HttpURLConnection and the implementation does keep a keepalive cache of connections to the same destinations. My problem is that I can not see how the connection is placed back in the cache after I have send the first request. The disconnect closes the connection and without the disconnect, still I can not see how the connection is placed back in the cache. I saw that the cache has a run method to go through over all idle connections (I am not sure how it is called), but I can not find how the connection is placed back in the cache. The only place that seems to happen is in the finished method of httpClient but this is not called for a POST with a response.
Can anyone help me on this?
EDIT
My interest is, what is the proper handling of an HttpUrlConnection object for tcp connection reuse. Should input/output stream be closed followed by a url.openConnection(); each time to send the new request (avoiding disconnect())? If yes, I can not see how the connection is being reused when I call url.openConnection() for the second time, since the connection has been removed from the cache for the first request and can not find how it is returned back.
Is it possible that the connection is not returned back to the keepalive cache (bug?), but the OS has not released the tcp connection yet and on new connection, the OS returns the buffered connection (not yet released) or something similar?
EDIT2
The only related i found was from JDK_KeepAlive
...when the application calls close()
on the InputStream returned by
URLConnection.getInputStream(), the
JDK's HTTP protocol handler will try
to clean up the connection and if
successful, put the connection into a
connection cache for reuse by future
HTTP requests.
But I am not sure which handler is this. sun.net.www.protocol.http.Handler does not do any caching as I saw
Thanks!

Should input/output stream be closed
followed by a url.openConnection();
each time to send the new request
(avoiding disconnect())?
Yes.
If yes, I can not see how the connection is being
reused when I call
url.openConnection() for the second
time, since the connection has been
removed from the cache for the first
request and can not find how it is
returned back.
You are confusing the HttpURLConnection with the underlying Socket and its underlying TCP connection. They aren't the same. The HttpURLConnection instances are GC'd, the underlying Socket is pooled, unless you call disconnect().

From the javadoc for HttpURLConnection (my emphasis):
Each HttpURLConnection instance is
used to make a single request but the
underlying network connection to the
HTTP server may be transparently
shared by other instances. Calling the
close() methods on the InputStream or
OutputStream of an HttpURLConnection
after a request may free network
resources associated with this
instance but has no effect on any
shared persistent connection. Calling
the disconnect() method may close the
underlying socket if a persistent
connection is otherwise idle at that
time.

I found that the connection is indeed cached when the InputStream is closed. Once the inputStream has been closed the underlying connection is buffered. The HttpURLConnection object is unusable for further requests though, since the object is considered still "connected", i.e. its boolean connected is set to true and is not cleared once the connection is placed back in the buffer. So each time a new HttpUrlConnection should be instantiated for a new POST, but the underlying TCP connection will be reused, if it has not timed out.
So EJP answer's was the correct description. May be the behavior I saw, (reuse of the TCP connection) despite explicitly calling disconnect() was due to caching done by the OS? I do not know. I hope someone who knows can explain.
Thanks.

How do you "force use of HTTP1.0" using the HttpUrlConnection of JDK?
According to the section ā€˛Persistent Connectionsā€¯ of the Java 1.5 guide support for HTTP1.1 connections can be turned off or on using the java property http.keepAlive (default is true). Furthermore, the java property http.maxConnections indicates the maximum number of (concurrent) connections per destination to be kept alive at any given time.
Therefore, a "force use of HTTP1.0" could be applied for the whole application at once by setting the java property http.keepAlive to false.

Hmmh. I may be missing something here (since this is an old question), but as far as I know, there are 2 well-known ways to force closing of the underlying TCP connection:
Force use of HTTP 1.0 (1.1 introduced persistent connections) -- this as indicated by the http request line
Send 'Connection' header with value 'close'; this will force closing as well.

Abandoning streams will cause idle TCP connections. The response stream should be read completely. Another thing I overlooked initially, and have seen overlooked in most answers on this topic is forgetting to deal with the error stream in case of exceptions. Code similar to this fixed one of my apps that wasn't releasing resources properly:
HttpURLConnection connection = (HttpURLConnection)new URL(uri).openConnection();
InputStream stream = null;
BufferedReader reader = null;
try {
stream = connection.getInputStream();
reader = new BufferedReader(new InputStreamReader(stream, Charset.forName("UTF-8")));
// do work on part of the input stream
} catch (IOException e) {
// read the error stream
InputStream es = connection.getErrorStream();
if (es != null) {
BufferedReader esReader = null;
esReader = new BufferedReader(new InputStreamReader(es, Charset.forName("UTF-8")));
while (esReader.ready() && esReader.readLine() != null) {
}
if (esReader != null)
esReader.close();
}
// do something with the IOException
} finally {
// finish reading the input stream if it was not read completely in the try block, then close
if (reader != null) {
while (reader.readLine() != null) {
}
reader.close();
}
// Not sure if this is necessary, closing the buffered reader may close the input stream?
if (stream != null) {
stream.close();
}
// disconnect
if (connection != null) {
connection.disconnect();
}
}
The buffered reader isn't strictly necessary, I chose it because my use case required reading one line at a time.
See also: http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html

Related

How does the connection manager discard unconsumed connection?

I am trying to debug a connection leak issue in my app where the connection manager is using more connections than I would have wanted. One lead I have is the following:
According to Apache HttpClient quick start, if response content is not fully consumed for any reason, the pooling connection manager can not safely reuse the underlying connection and will discard it.
Can anyone point me to the code block that does the checking of unconsumed content in a connection and the discarding of the connection?
// Please note that if response content is not fully consumed the underlying
// connection cannot be safely re-used and will be shut down and discarded
// by the connection manager.
try (CloseableHttpResponse response1 = httpclient.execute(httpGet)) {
System.out.println(response1.getCode() + " " + response1.getReasonPhrase());
HttpEntity entity1 = response1.getEntity();
// do something useful with the response body
// and ensure it is fully consumed
EntityUtils.consume(entity1);
}
HttpClient version 4.x and 5.x wrap HTTP response entity with a proxy that releases the underlying connection back to the pool upon reaching the end of the message stream. In all other cases HttpClient assumes the message has not been fully consumed and the underlying connection cannot be re-used.
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/classic/ResponseEntityProxy.java

TIdHTTP - Get only Responsecode

I am using the TIdHTTP component and it's GET function.
The GET function sends a complete request, which is fine.
However I would like to spare/save some traffic from a GET response and only want to receive the Responsecode which is in the first "line" of a HTTP response.
Is there a possibility of disconnecting the connection in order to save traffic from any further content?
As mentioned, I only need the responsecode from a website.
I alternatively thought about using Indy's TCP component (with SSL IOHandler) and craft an own HTTP Request Header and then receive the responsecode and disconnect on success - but I don't know how to do that.
TIdHTTP has an OnHeadersAvailable event that is intended for this very task. It is triggered after the response headers have been read and before the body content is read, if any. It has a VContinue output parameter that you can set to False to cancel any further reading.
Update: Something I just discovered: When setting VContinue=False in the OnHeadersAvailable event, TIdHTTP will set Response.KeepAlive=False and skip reading the response body (OK so far), but after the response is done being processed, TIdHTTP checks the KeepAlive value, and the property getter returns True if the socket hasn't been closed on the server's end (HTTP 1.1 uses keep-alives by default). This causes TIdHTTP to not close its end of the socket, and will leave any response body unread. If you then re-use the same TIdHTTP object for a new HTTP request, it will end up processing any unread body data from the previous response before it sees thee response headers of the new request.
You can work around this issue by setting the Request.Connection property to 'close' before calling TIdHTTP.Get(). That tells the server to close its end of the socket connection after sending the response (although, I just found that when requesting an HTTPS url, especially after an HTTP request directs to HTTPS, TIdHTTP clears the Request.Connection value!). Or, simply call TIdHTTP.Disconnect() after TIdHTTP.Get() exits.
I have now updated TIdHTTP to:
no longer clear the Request.Connection when preparing an HTTPS request.
close its end of the socket connection if either:
OnHeadersAvailable returns VContinue=False
the Request.Connection property (or, if connected to a proxy, the Request.ProxyConnection property) has been set to 'close', regardless of the server's response.
Usually you would use TIdHttp.Head, because HEAD requests are intended for doing just that.
If the server does not accept HEAD requests like in OP's case, you can assign the OnWorkBegin event of your TIdHttp instance, and call TIdHttp(Sender).Disconnect; there. This immediately closes the connection, the download does not continue, but you still have the meta data like response code, content length etc.

BizTalk: Analyze binary blob hiding in XmlDocument?

I'm using BizTalk 2013 R1 to download a binary blob from a website via http. When I receive the blob, I'm just storing the message in an XmlDocument. However, sometimes that site returns the files I want, and sometimes it returns errors in the form of http pages containing error information.
I've attempted to screen for this by trying to run xpath on my return message. In particular, I'm looking for occurrences of "Error" in /html/head/title. My thinking is that if it find that text, or if it parses as Xml at all, I've gotten an error and I should throw an exception.
In practice though, I get this when I attempt to run that xpath:
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 128.30.52.100:80
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at System.Xml.XmlDownloadManager.GetNonFileStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
at System.Xml.XmlTextReaderImpl.OpenAndPush(Uri uri)
at System.Xml.XmlTextReaderImpl.PushExternalEntityOrSubset(String publicId, String systemId, Uri baseUri, String entityName)
at System.Xml.XmlTextReaderImpl.DtdParserProxy_PushExternalSubset(String systemId, String publicId)
at System.Xml.DtdParser.ParseExternalSubset()
at System.Xml.DtdParser.Parse(Boolean saveInternalSubset)
at System.Xml.DtdParser.System.Xml.IDtdParser.ParseInternalDtd(IDtdParserAdapter adapter, Boolean saveInternalSubset)
at System.Xml.XmlTextReaderImpl.ParseDtd()
at System.Xml.XmlTextReaderImpl.ParseDoctypeDecl()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.Load(TextReader txtReader)
at Microsoft.XLANGs.Core.Value.GetXmlDocument()
at Microsoft.XLANGs.Core.Value.RetrieveAs(Type t)
at Microsoft.XLANGs.Core.Part.get_XmlDocument()
at Microsoft.XLANGs.Core.Part.XPathLoad(Part sourcePart, String xpath, Type dstType)
at QTC.BizTalk.LSPDispatchIMNL.SendCommercialInvoice.segment3(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)
Upon seeing this, it makes sense since I believe that BizTalk handles messages as streams in the background. Suddenly, the technique of hiding binary in XmlDocuments makes sense. So perhaps my test itself is causing a different problem.
I would like to be able to validate my response in some way, however. Is there anything I can do to analyze the response I get from the site, without causing the message to load? There's nothing all that useful in the context properties so I'm curious what I can do.
I'm not really sure how to make sense of your error (especially without seeing the code you're actually using to check the message), but either way I think you should do this in a custom pipeline component, for a few reasons.
Loading the XmlDocument into the orchestration is going to be prohibitive if you're dealing with large binary objects.
Trying to use XPath on binary data won't work
Trying to use XPath on HTML won't always work
You could very easily check the message size in a pipeline component (pInMsg.BodyPart.GetOriginalDataStream().Length for example). You could also try to read the first few bytes of the stream and check those for certain conditions more efficiently.

mina server response port different from port incoming messages

I set up a server that receives messages over port xxx, but I want to respond to port yyy.
Is there a simple way to achieve this?
My server:
IoAcceptor acceptor = new NioSocketAcceptor();
acceptor.setHandler(new MessageHandler());
acceptor.getFilterChain().addLast("logger", new LoggingFilter());
acceptor.getFilterChain().addLast("codec", new protocolCodecFilter(codecFactory));
acceptor.getSessionConfig().setReadBufferSize(bufferSize);
acceptor.bind(new InetSocketAddress(port));
The encode method of my encoder:
public void encode(IoSession session, Object message, ProtocolEncoderOutput out) throws Exception {
byte[] writeBytes = (byte[]) message;
IoBuffer buffer = IoBuffer.allocate(writeBytes.length).setAutoExpand(false);
buffer.put(writeBytes);
buffer.flip();
out.write(buffer);
writeMessage(session,writeBytes);
}
The msessage should be written to a different port. How do I achieve this?
If you want to response message using different tcp port, you must make another other tcp connection first, which means you have two servers and tow clients.
request
client1---------->server1
reponse
server2---------->client2

Sending data in HTTP request's body fails

I am using the HttpConnection class of J2ME in my BlackBerry app to send data to a web server. I need to send the contents of an image in the body of the HTTP request.
This is what I do
Get the bytes of the file in an array
Open HTTP connection
Set content type header as image/jpeg
Get output stream of the connection
Write the bytes to the output stream
Close the output stream and connection
But the image is not uploaded to the server. What could be the problem?
Thanks.
EDIT - Adding code
HttpConnection conn = null;
OutputStream out = null;
try{
conn = new HttpConnection(Connector.open(myURL));
conn.setRequestProperty("Content-Type", "image/jpeg");
conn.setRequestMethod(HttpConnection.POST);
conn.setRequestProperty("Content-Disposition", "form-data");
conn.setRequestProperty("Connection", "Keep-Alive");
out = conn.openOutputStream;
out.write(buffer, 0, buffer.length);
conn.setRequestProperty("Content-Length", buffer.length);
out.flush();
}
catch(Exception e){
e.printStackTrace();
}
finally{
if(out != null)
out.close();
if(conn != null){
System.out.println("" + conn.getResponseCode());
conn.close();
}
}
EDIT
The same code, when I try it with a string, works fine and sends the string to the server. But it is still a problem with the image bytes.
A few things that may be missing from your list:
use HttpConnection.setRequestMethod(HttpConnection.POST) between 2 and 3.
set content length with HttpConnection.setRequestProperty("Content-Length",...) between 5 and 6.
knowing the HTTP request response code can help debug your issues: call HttpConnection.getResponseCode() after you've closed the OutputStream but before you close the HttpConnection.
conn = new HttpConnection(Connector.open(myURL));
This line is wrong. Connection is a factory class that creates new Connection objects by looking it's appropriate protocol implementation.
HttpConnection conn = (HttpConnection) Connector.open(myURL);
The rest of the code seems ok. When you're POSTing, at minimun you need to define the content-type and content-length.
You most definitely need to set all headers before sending the POST data, including the Content-Length header.
Also, make sure you are sending headers valid for requests, and not response-only headers.
You'll need to encode the bytes (preferably Base-64) and send that string. The raw bytes aren't likely to be http safe.
Then on the server, you'll need to decode them back into a byte array and do whatever you were going to do with it (save as file, stuff into database, etc.)

Resources