This question already has answers here:
httpclient exception "org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection"
(10 answers)
Closed 8 years ago.
I use Multi-thread to scan the different URL in the same time in java. There was the bug,if the sum of request time exceed 100,000. I have already close which i should close. Here the code from my servlet
private String proyGetHttp(String url) throws ParseException, IOException,
InterruptedException {
String content = "";
getMethod = new HttpGet(url);
HttpResponse response = null;
HttpEntity httpEntity = null;
boolean success = false;
while (!success) {
System.out.println("url：" + url + "，connect...");
try {
response = client.execute(getMethod);
httpEntity = response.getEntity();
StringBuffer sb = new StringBuffer();
if (httpEntity != null) {
BufferedReader in = null;
InputStream instream = httpEntity.getContent();
try {
in = new BufferedReader(new InputStreamReader(instream));
String lineContent = "";
while(lineContent != null){
sb.append(lineContent);
lineContent = in.readLine();
}
} catch (Exception ex)
getMethod.abort();
throw ex;
} finally {
// Closing the input stream will trigger connection release
try { instream.close(); in.close();} catch (Exception ignore) {}
}
}
content = sb.toString();
success = true;
System.out.println("connect successfully...");
} catch (Exception e) {
e.printStackTrace();
getMethod.abort();
System.out.println("connect fail, please waitting...");
Thread.sleep(sleepTime);
}finally{
getMethod.releaseConnection();
}
}
return content;
}
Here code create the default client
PoolingClientConnectionManager cm = new PoolingClientConnectionManager();
cm.setMaxTotal(100);
DefaultHttpClient client = null;
client = new DefaultHttpClient(cm);
client.getParams().setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY);
client.getParams().setParameter(HttpConnectionParams.CONNECTION_TIMEOUT, 5000);
I have the same problem and I found the fix. This timeout is because of a connection leak. In my case, I'm using httpDelete method and not consuming the response. Instead, I'm checking the status of the response.
The fix is, the response entity need to be consumed. In order to ensure the proper release of system resources, one must close the content stream associated with the entity.
So I used EntityUtils.consumeQuietly(response.getEntity()); which ensures that the entity content is fully consumed and the content stream, if exists, is closed.
Related
My question is, behind the scene, for element-wise Beam DoFn (ParDo), how does the Cloud Dataflow parallel workload? For example, in my ParDO, I send out one http request to an external server for one element. And I use 30 workers, each has 4vCPU.
Does that mean on each worker, there will be 4 threads at maximum?
Does that mean from each worker, only 4 http connections are necessary or can be established if I keep them alive to get the best performance?
How can I adjust the level of parallelism other than using more cores or more workers?
with my current setting (30*4vCPU worker), I can establish around 120 http connections on the http server. But both server and worker has very low resource usage. basically I want to make them work much harder by sending out more requests out per second. What should I do...
Code Snippet to illustrate my work:
public class NewCallServerDoFn extends DoFn<PreparedRequest,KV<PreparedRequest,String>> {
private static final Logger Logger = LoggerFactory.getLogger(ProcessReponseDoFn.class);
private static PoolingHttpClientConnectionManager _ConnManager = null;
private static CloseableHttpClient _HttpClient = null;
private static HttpRequestRetryHandler _RetryHandler = null;
private static String[] _MapServers = MapServerBatchBeamApplication.CONFIG.getString("mapserver.client.config.server_host").split(",");
#Setup
public void setupHttpClient(){
Logger.info("Setting up HttpClient");
//Question: the value of maxConnection below is actually 10, but with 30 worker machines, I can only see 115 TCP connections established on the server side. So this setting doesn't really take effect as I expected.....
int maxConnection = MapServerBatchBeamApplication.CONFIG.getInt("mapserver.client.config.max_connection");
int timeout = MapServerBatchBeamApplication.CONFIG.getInt("mapserver.client.config.timeout");
_ConnManager = new PoolingHttpClientConnectionManager();
for (String mapServer : _MapServers) {
HttpHost serverHost = new HttpHost(mapServer,80);
_ConnManager.setMaxPerRoute(new HttpRoute(serverHost),maxConnection);
}
// config timeout
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(timeout)
.setConnectionRequestTimeout(timeout)
.setSocketTimeout(timeout).build();
// config retry
_RetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(
IOException exception,
int executionCount,
HttpContext context) {
Logger.info(exception.toString());
Logger.info("try request: " + executionCount);
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof InterruptedIOException) {
// Timeout
return false;
}
if (exception instanceof UnknownHostException) {
// Unknown host
return false;
}
if (exception instanceof ConnectTimeoutException) {
// Connection refused
return false;
}
if (exception instanceof SSLException) {
// SSL handshake exception
return false;
}
return true;
}
};
_HttpClient = HttpClients.custom()
.setConnectionManager(_ConnManager)
.setDefaultRequestConfig(requestConfig)
.setRetryHandler(_RetryHandler)
.build();
Logger.info("Setting up HttpClient is done.");
}
#Teardown
public void tearDown(){
Logger.info("Tearing down HttpClient and Connection Manager.");
try {
_HttpClient.close();
_ConnManager.close();
}catch (Exception e){
Logger.warn(e.toString());
}
Logger.info("HttpClient and Connection Manager have been teared down.");
}
#ProcessElement
public void processElement(ProcessContext c) {
PreparedRequest request = c.element();
if(request == null)
return;
String response="{\"my_error\":\"failed to get response from map server with retries\"}";
String chosenServer = _MapServers[request.getHardwareId() % _MapServers.length];
String parameter;
try {
parameter = URLEncoder.encode(request.getRequest(),"UTF-8");
} catch (UnsupportedEncodingException e) {
Logger.error(e.toString());
return;
}
StringBuilder sb = new StringBuilder().append(MapServerBatchBeamApplication.CONFIG.getString("mapserver.client.config.api_path"))
.append("?coordinates=")
.append(parameter);
HttpGet getRequest = new HttpGet(sb.toString());
HttpHost host = new HttpHost(chosenServer,80,"http");
CloseableHttpResponse httpRes;
try {
httpRes = _HttpClient.execute(host,getRequest);
HttpEntity entity = httpRes.getEntity();
if(entity != null){
try
{
response = EntityUtils.toString(entity);
}finally{
EntityUtils.consume(entity);
httpRes.close();
}
}
}catch(Exception e){
Logger.warn("failed by get response from map server with retries for " + request.getRequest());
}
c.output(KV.of(request, response));
}
}
Yes, based on this answer.
No, you can establish more connections. Based on my answer, you can use a async http client to have more concurrent requests. As this answer also describes, you need to collect the results from these asynchronous calls and output it synchronously in any #ProcessElement or #FinishBundle.
See 2.
Since your resource usage is low, it indicates that the worker spends most of its time waiting for a response. I think with the described approach above, you can utilize your resources far better and you can achieve the same performance with far less workers.
I am getting an error
javax.net.ssl.SSLHandshakeException: Connection closed by peer
when i am trying to access a HTTPS url.
my code is:
private void executeHTTRequestVerifyingLogin(String userid, String pwd, String key) throws Exception {
String strReturn = "";
BufferedReader in = null;
HttpClient hc = CUtils.getNewHttpClient();
HttpPost hp = new HttpPost(CGlobalVariables.VERIFYING_LOGIN);
try {
hp.setEntity(new UrlEncodedFormEntity(getNameValuePairs_Login(userid, pwd, key), HTTP.UTF_8));
HttpResponse response = hc.execute(hp);
in = new BufferedReader(new InputStreamReader(response.getEntity()
.getContent()));
StringBuffer sb = new StringBuffer("");
String data = "";
while ((data = in.readLine()) != null)
sb.append(data);
in.close();
setVerifyingLoginValue(sb.toString());
} catch (Exception e) {
throw e;
}
}
My code is working upto api level 23. I don't know why this exception is thrown in Nougat(Android 7.0) only.
I've implemented timer task on background application.
I've collected current lat and long. and send to server each 30 seconds.
I've used below code to send the information to server. It sends successfully..
My problem is, after i've checked 10 minutes, I'm unable to send. it throws a No Network error. I've checked browser too but no network.
If reset the device, its working again well. But the same problem occurs after 5 or 10 mins.
How to resolve this?
My code is,
try
{
StreamConnection connection = (StreamConnection) Connector.open(url+suffix);
((HttpConnection) connection).setRequestMethod(HttpConnection.GET);
int responseCode = ((HttpConnection) connection).getResponseCode();
if (responseCode != HttpConnection.HTTP_OK) {
showDialog("Unexpected response code :"+ responseCode);
connection.close();
return;
}
((HttpConnection) connection).getHeaderField("Content-type");
ByteArrayOutputStream baos = new ByteArrayOutputStream();
InputStream responseData = connection.openInputStream();
byte[] buffer = new byte[1000];
int bytesRead = responseData.read(buffer);
while (bytesRead > 0) {
baos.write(buffer, 0, bytesRead);
bytesRead = responseData.read(buffer);
}
baos.close();
connection.close();
String s = new String(baos.toByteArray());
showDialog("Responce from server "+s);
}
catch (IOException e)
{
}
Usually, when you have some problem where it works a few times, and then stops working, and you need to reset the device, you've done something that has used up all available resources, without releasing them when you're done.
When performing repeated network operations, you should clean up your streams and connections after each use.
Normally, the proper way to write network code is to declare network variables outside a try block, assign and use them inside the try, while catching any IOExceptions thrown. Then, you use a finally block to clean up your resources, no matter whether the code finished successfully or not.
I'll also note that when debugging network problems, you don't want to have a catch() handler that simply traps exceptions and does nothing with them. Print out a message to the console (for testing) or log the error to a file.
Finally, I can't see your showDialog() method, but if it's displaying a UI to the user/tester, you need to do that on the UI thread. But, the network code that you show above should be run on a background thread to keep the UI responsive. So, inside showDialog(), just make sure you use code to modify the UI on the UI thread.
So, a better implementation might be this:
private void requestFromServer() {
StreamConnection connection = null;
ByteArrayOutputStream baos = null;
InputStream responseData = null;
try
{
connection = (StreamConnection) Connector.open(url+suffix);
((HttpConnection) connection).setRequestMethod(HttpConnection.GET);
int responseCode = ((HttpConnection) connection).getResponseCode();
if (responseCode != HttpConnection.HTTP_OK) {
showDialog("Unexpected response code :"+ responseCode);
return;
}
((HttpConnection) connection).getHeaderField("Content-type");
baos = new ByteArrayOutputStream();
responseData = connection.openInputStream();
byte[] buffer = new byte[1000];
int bytesRead = responseData.read(buffer);
while (bytesRead > 0) {
baos.write(buffer, 0, bytesRead);
bytesRead = responseData.read(buffer);
}
String s = new String(baos.toByteArray());
showDialog("Responce from server "+s);
}
catch (IOException e)
{
System.out.println("Network error: " + e.getMessage());
}
finally
{
try {
if (connection != null) {
connection.close();
}
if (baos != null) {
baos.close();
}
if (responseData != null) {
responseData.close();
}
} catch (IOException e) {
// nothing to do here
}
}
}
private void showDialog(final String msg) {
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run() {
Dialog.alert(msg);
}
});
}
I am working on an BB app in which I need to maintain a HTTP connection and with a name of image which is stored on server to get the text written in that image document.
I am getting the response in RTF format.
When I directly hit the server on open browser Chrome, I RTF file get downloaded.
Now I needs to perform that programetically,
1) Either convert the bytes which are coming in response in a simple string format so that I can read that.
or
2) Download the file as its happening on the browser manually so that by reading that file I read the information written in the document.
please suggest me how can I read the data from server by hitting any URL?
Currently I am working with this code:
try {
byte []b = send("new_image.JPG");
String s = new String(b, "UTF-8");
System.out.println(s);
} catch (Exception e) {
e.printStackTrace();
}
public byte[] send(String Imagename) throws Exception
{
HttpConnection hc = null;
String imageName = "BasicExp_1345619462234.jpg";
InputStream is = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] res = null;
try
{
hc = (HttpConnection) Connector.open("http://webservice.tvdevphp.com/basisexpdemo/webservices/ocr.php?imgname="+imageName);
hc.setRequestProperty("Content-Type", "multipart/form-data;");
hc.setRequestMethod(HttpConnection.GET);
int ch;
StringBuffer sb= new StringBuffer();
is = hc.openInputStream();
while ((ch = is.read()) != -1)
{
bos.write(ch);
sb.append(ch);
}
System.out.println(sb.toString());
res = bos.toByteArray();
}
catch(Exception e){
e.printStackTrace();
}
finally
{
try
{
if(bos != null)
bos.close();
if(is != null)
is.close();
if(hc != null)
hc.close();
}
catch(Exception e2)
{
e2.printStackTrace();
}
}
return res;
}
The response is like:
{\rtf1\ansi\ansicpg1252\uc1\deflang1033\adeflang1033...................
I can read the data but its not formatted, so that i can read that programetically too.
I have done with this task....
Actually the mistake was on server side.
When they were performing OCR, the format parameter was not corrected that was reason.
i am developing and app for blackberry and i need to send a Http Post Request to my server. I'm using the simulator in order to test my app and i found this code in order to send request:
http://vasudevkamath.techfiz.com/general/posting-data-via-http-from-blackberry/
But i can't get it work, because it fails in this line:
int rc = _httpConnection.getResponseCode();
Any idea?
thanks
Here is a sample code on how to send a POST request:
HttpConnection c = (HttpConnection) Connector.open(url, Connector.READ_WRITE);
c.setRequestMethod(HttpConnection.POST);
OutputStream os = c.openOutputStream();
os.write(request.getBytes("UTF-8"));
os.flush();
os.close();
InputStream is = c.openInputStream();
Just make sure you use this code in a separate thread.
public static ResponseBean sendRequestAndReceiveResponse(String method, String absoluteURL, String bodyData, boolean readResponseBody)
throws IOException
{
ResponseBean responseBean = new ResponseBean();
HttpConnection httpConnection = null;
try
{
String formattedURL = absoluteURL + "deviceside=true;interface=wifi"; // If you are using WiFi
//String formattedURL = absoluteURL + "deviceside=false"; // If you are using BES
//String formattedURL = absoluteURL + "deviceside=true"; // If you are using TCP
if(DeviceInfo.isSimulator()) // if you are using simulator
formattedURL = absoluteURL;
httpConnection = (HttpConnection) Connector.open(formattedURL);
httpConnection.setRequestMethod(method);
if (bodyData != null && bodyData.length() > 0)
{
OutputStream os = httpConnection.openOutputStream();
os.write(bodyData.getBytes("UTF-8"));
}
int responseCode = httpConnection.getResponseCode();
responseBean.setResponseCode(responseCode);
if (readResponseBody)
{
responseBean.setBodyData(readBodyData(httpConnection));
}
}
catch (IOException ex)
{
System.out.println("!!!!!!!!!!!!!!! IOException in NetworkUtil::sendRequestAndReceiveResponse(): " + ex);
throw ex;
}
catch(Exception ex)
{
System.out.println("!!!!!!!!!!!!!!! Exception in NetworkUtil::sendRequestAndReceiveResponse(): " + ex);
throw new IOException(ex.toString());
}
finally
{
if (httpConnection != null)
httpConnection.close();
}
return responseBean;
}
public static StringBuffer readBodyData(HttpConnection httpConnection) throws UnsupportedEncodingException, IOException
{
if(httpConnection == null)
return null;
StringBuffer bodyData = new StringBuffer(256);
InputStream inputStream = httpConnection.openDataInputStream();
byte[] data = new byte[256];
int len = 0;
int size = 0;
while ( -1 != (len = inputStream.read(data)) )
{
bodyData.append(new String(data, 0, len,"UTF-8"));
size += len;
}
if (inputStream != null)
{
inputStream.close();
}
return bodyData;
}
I know this question is pretty old and OP probably solved it by now, but I've just run into the same problem and managed to fix it!
You need to append ;deviceside=true to your URL.
So for example, your URL will change from "http://example.com/directory/submitpost.php" to "http://example.com/directory/submitpost.php;deviceside=true".
I found this here: http://supportforums.blackberry.com/t5/Java-Development/Different-ways-to-make-an-HTTP-or-socket-connection/ta-p/445879
My POST request was timing out after 3 minutes when I did not have this (See My Comment), but it works fine with this appended to the url.
I would also recommend using ConnectionFactory. Here's some of my code:
Network.httpPost("http://example.com/directory/submitpost.php;deviceside=true", paramNamesArray, paramValsArray)
public static void httpPost(String urlStr, String[] paramName, String[] paramVal) throws Exception {
ConnectionFactory conFactory = new ConnectionFactory();
conFactory.setTimeLimit(1000);
HttpConnection conn = (HttpConnection) conFactory.getConnection(urlStr).getConnection();
conn.setRequestMethod(HttpConnection.POST);
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
StringBuffer sb = new StringBuffer();
for (int i = 0; i < paramName.length; i++) {
sb.append(paramName[i]);
sb.append("=");
sb.append(paramVal[i]);
sb.append("&");
}
byte[] postData = sb.toString().getBytes("UTF-8");
conn.setRequestProperty("Content-Length",new Integer(postData.length).toString());
OutputStream out = conn.openOutputStream();
out.write(postData);
//out.flush(); //Throws an Exception for some reason/Doesn't do anything anyways
out.close();
//This writes to our connection and waits for a response
if (conn.getResponseCode() != 200) {
throw new Exception(conn.getResponseMessage());
}
}
Not sure about the site you posted, but I've successfully used the sample ConnectionFactory code provided on the blackberry site.
http://supportforums.blackberry.com/t5/Java-Development/Sample-Code-Using-the-ConnectionFactory-class-in-a-BrowserField/ta-p/532860
Just make sure not to invoke the connection on the EventThread.
That's how you add parameters, Full answer is here:
StringBuffer postData = new StringBuffer();
httpConn = (HttpConnection) Connector.open("https://surveys2.kenexa.com/feedbacksurveyapi/login?");
httpConn.setRequestMethod(HttpConnection.POST);
postData.append("username="+username);
postData.append("&password="+pass);
postData.append("&projectcode="+projectid);
String encodedData = postData.toString();
httpConn.setRequestProperty("Content-Language", "en-US");
httpConn.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
httpConn.setRequestProperty("Content-Length",(new Integer(encodedData.length())).toString());
byte[] postDataByte = postData.toString().getBytes("UTF-8");
OutputStream out = httpConn.openOutputStream();
out.write(postDataByte);
out.close();
httpConn.getResponseCode();