I am using ibrowse to make POST and GET requests to the external server but I have issues when it comes to supporting frequent and simultaneous requests.
I start ibrowse with default settings:
application:start(ibrowse).
Then I use ibrowse to carry out POST requests:
ibrowse:send_req("http://somelink.com/api/capi/send.php?" ++ Data,[],get,[]).
or:
ibrowse:send_req("http://somelink.com/cgi-bin/send",[{"Content-Type", "application/x-www-form-urlencoded"}],post,Data).
When I call that function, it succeeds on the first try with this result:
[{http_code,"202"},
{http_headers,[{"Server","CCSRouter/0.1"},
{"Content-Length","34"},
{"Content-type","text/html"},
{"Pragma","no-cache"},
{"Cache-Control","no-cache"}]},
{result,"status=1"}],
But when I call the function repeatedly, it fails returning the following error:
{error,{conn_failed,{error,econnrefused}}}
How can I ensure that whenever I make high frequency connections with ibrowse, it succeeds?
Related
I'm using java.net.http.HttpClient.newHttpClient() under Java 19 (Temurin) and perform sendAsync(...) requests from different treads on the same instance. I assume this is ok, as the javadoc states:
Once built, an HttpClient is immutable...
However, some requests fail with:
java.io.IOException: HTTP/1.1 header parser received no bytes
The weird thing is, it depends on the speed of my requests:
Requests every 5 seconds: 30% failure
Requests every 3 seconds: 0% failure
I've written a test for it:
private final HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("https://..."))
.setHeader("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofByteArray("[]".getBytes()))
.build();
#ParameterizedTest
#ValueSource(ints = {3, 5})
void httpClientTest(int intervalSeconds) throws Exception {
HttpClient httpClient = HttpClient.newHttpClient();
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
}
I've already tried the following:
Doing the same with curl on the command line. No requests fail whatever interval I try. So it's probably not a problem with the server.
Running the tests multiple times in parallel. Still the 5-second-intervals fail (then multiple times in parallel). So it's probably not a problem with the server.
Creating an HttpClient.newHttpClient() for every request. No requests fail whatever interval. So it's probably not a problem with the server but with an internal state of the HttpClient (although it claims to be immutable?).
Do you have an idea what I could do, without needing to create a new HttpClient for every request?
Here is the answer for the record: the java.net.HttpClient has a long default HTTP/1.1 keepAlive time, which is longer than what usual servers are configured with. This often results in the server closing idle HTTP/1.1 connections before the client does. If the server closes the connection at about the same time than the client tries to reuse it, some IOException might get raised.
If such exceptions are observed too frequently applications should consider adapting the default keepAlive time in the client to some value shorter than what the servers it connects to are using.
A default value for the HttpClient HTTP/1.1 keepAlive time can be specified on the command line with: -Djdk.httpclient.keepalive.timeout=duration-in-seconds
So for instance - if a server is configured with a keepAlive time of 5s, you could consider supplying -Djdk.httpclient.keepalive.timeout=3 or -Djdk.httpclient.keepalive.timeout=4 on the client's java command line.
I have an application that needs 2-way communication to external daemon (bitcoind). There is functionality in bitcoind that allows to call my application whenever new block or transaction of interest occurs ('--walletnotify' and '--blocknotify'). For that I'm using CURL to request "http://myapp/walletnotify" and so on:
walletnotify = /usr/bin/curl --max-time 60 http://myapp/walletnotify/%s
I'm trying to create integration tests for this callback behavior. Unfortunately when running integration tests, I'm receiving errors on daemon side, as it is not able to perform requests to "http://myapp/walletnotify" - obviously Rails server cannot be reached (or the connection is interrupted?). Of course tests fail as appropriate actions are not called.
My question is: how to properly test such scenario? Is there any way to allow for direct external requests to application during integration tests? Is there a way to make sure that Rails server is running during integration tests? Or maybe I should listen to such requests inside integration test and then proxy them to application?
Update 2018-06-03: I'm using minitest. The test that I'm trying to run is here:
https://github.com/cryptogopher/token_voting/blob/master/test/integration/token_votes_notify_test.rb
After calling
#rpc.generate(101)
bitcoind daemon in regtest mode should generate 101 blocks and call 'blocknotify' callbacks. The problem is it cannot send HTTP request to application during test.
Ok, I resolved this one.
Looks like 'minitest' does not start application server no matter which driver you choose. Still you can start your own HTTP server to listen for notifications from external sources and forward them to tested application.
For this to work you need:
require 'webrick'
Setup HTTP server (logging disabled to avoid clutter):
server = WEBrick::HTTPServer.new(
Port: 3000,
Logger: WEBrick::Log.new("/dev/null"),
AccessLog: []
)
Specify how to handle incoming HTTP requests. In my case there will be only GET requests, but it is important to forward them with original headers:
server.mount_proc '/' do |req, resp|
headers = {}
req.header.each { |k,v| v.each { |a| headers[k] = a } }
resp = get req.path, {}, headers
end
Start HTTP server in separate thread (it is blocking the thread where it is running):
#t = Thread.new {
server.start
}
Minitest.after_run do
#t.kill
#t.join
end
Timeout.timeout(5) do
sleep 0.1 until server.status == :Running
end
I'm runing parse on azure (Parse Server on managed Azure services),
I'ts include DocumentDB as database and have limit for requests per seconds.
Some parse cloud functions are large and the speed of requests is too high (even for S3 tier) so i'm getting this error (seen using Visual Studio Team Services (was Visual Studio Online) and Streaming logs).
error: Uncaught internal server error. { [MongoError: Message: {"Errors":["Request rate is large"]}
ActivityId: a4f1e8eb-0000-0000-0000-000000000000, Request URI: rntbd://10.100.99.69:14000/apps/f8a35ed9-3dea-410f-a89a-28650ff41381/services/2d8e5320-89e6-4750-a06f-174c12013c69/partitions/53e8a085-9fed-4880-bd90-f6191765f625/replicas/131091039101528218s]
name: 'MongoError',
message: 'Message: {"Errors":["Request rate is large"]}\r\nActivityId: a4f1e8eb-0000-0000-0000-000000000000, Request URI: rntbd://10.100.99.69:14000/apps/f8a35ed9-3dea-410f-a89a-28650ff41381/services/2d8e5320-89e6-4750-a06f-174c12013c69/partitions/53e8a085-9fed-4880-bd90-f6191765f625/replicas/131091039101528218s' } MongoError: Message: {"Errors":["Request rate is large"]}
ActivityId: a4f1e8eb-0000-0000-0000-000000000000, Request URI: rntbd://10.100.99.69:14000/apps/f8a35ed9-3dea-410f-a89a-28650ff41381/services/2d8e5320-89e6-4750-a06f-174c12013c69/partitions/53e8a085-9fed-4880-bd90-f6191765f625/replicas/131091039101528218s
at D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:673:34
at handleCallback (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:159:5)
at setCursorDeadAndNotified (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:501:3)
at nextFunction (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:672:14)
at D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:585:7
at queryCallback (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:241:5)
at Callbacks.emit (D:\home\site\wwwroot\node_modules\mongodb-core\lib\topologies\server.js:119:3)
at null.messageHandler (D:\home\site\wwwroot\node_modules\mongodb-core\lib\topologies\server.js:397:23)
at TLSSocket.<anonymous> (D:\home\site\wwwroot\node_modules\mongodb-core\lib\connection\connection.js:302:22)
at emitOne (events.js:77:13)
How to handle this error?
TL;DR;
Upgrade the old S3 collection to a new single collection under the new pricing scheme. This can support up to 10K RU (up from 2500 RU)
Delete the old S3 collection and create a new partitioned collection. Will require support for partitioned collection in parse.
Implement a backoff strategy in line with the x-ms-retry-after-ms response header.
Long answer:
Each request to DocumentDB returns a HTTP header with the Request charge for that operation. The number of request units is configured per collection. As per my understanding you have 1 collection of size S3, so this collection can only handle 2500 Request Units per second.
DocumentDB scales by adding multiple collections. With the old configuration using S1 -> S3 you must do this manually, i.e. you must distribute your data over the collections using an algorithm such as consistent hashing, a map or perhapse date. With the new pricing in DocumentDB you can use partitioned collections, by defining a partition key, DocumentDB will shard your data for you. If you see sustained rates of RequestRateTooLarge errors I recommend scaling out the partitions. However, you will need to investigate if Parse supports partitined collections.
When you receive a HTTP 429 RequestRateTooLarge there's also a header called x-ms-retry-after-ms :### where ### denotes the number of milliseconds to wait before you retry the operation. What you can do is to implement a back-off strategy which retries the operation. Do note that if you have clients hanging on the server during retries, you may build up request queues and clog the server. I recommend adding a Queue to handle such burst. For short burst of requests this is a nice way to handled it without scaling up the collections.
i used Mlab as external mongoDB database and configure the parse app in azure to use it instead of documentDB.
I have to will to pay so much for "performance" increase.
I have a cowboy websocket server. Many clients send message over the websocket. I need to do processing on the message. I can do that in websocket_handle, However as it's realtime I would like to avoid it instead I want to send the message to a Global Process where all the processing can be done.
As Each cowboy has it's own process How to run a process where every user can send message and processing can be done in that process.
Just to clarify, each websocket connection will have its own erlang process in Cowboy, so messages from different websocket clients will be processed in different processes.
If you need to move the processing from the websocket you can simply start a new handler/server process when your app starts (e.g. when you start Cowboy) that listens for process commands and data. Sample processing code:
-module(my_processor).
-export([start/0]).
start() ->
spawn(fun process_loop/0).
process_loop() ->
receive
{process_cmd, Data} ->
process(Data)
end,
process_loop().
When you start it, also register the process with a global name. That way we can reference it from the websocket handlers later.
Pid=my_processor:start().
register(processor, Pid).
Now you can send the data from Cowboy's websocket_handle/3 function to the handling process:
websocket_handle(Data, Req, State) ->
...,
processor ! {process_cmd, Data},
...,
{ok,Req,State}.
Note that the my_processor process will handle the processing requests from all connections. If you want to have a separate process for each websocket connection you could start my_processor in Cowboy's websocket_init/3 function, store the Pid of the my_processorprocess in the State parameter returned from websocket_init and use that pid instead of the processor global name.
I have written a web-service using Erlang and Mochiweb. The web service returns a lot of results and takes some time to finish the computation.
I'd like to return results as soon as the program finds it, instead of returning them when it found them all.
edit:
i found that i can use a chunked request to stream result, but seems that i can't find a way to close the connection. so any idea on how to close a mochiweb request?
To stream data of yet unknown size with HTTP 1.1 you can use HTPP chunked transfer encoding. In this encoding each chunk of data prepended by its size in hexadecimal. Last chunk is a zero-length chunk, with the chunk size coded as 0, but without any data.
If client doesn't support HTTP 1.1 server can send data as binary chunks and close connection at the end of the stream.
In MochiWeb it's all works as following:
HTTP response should be started with Response = Request:respond({Code, ResponseHeaders, chunked}) function. (By the way, look at the code comments);
Then chunks can be send to client with Response:write_chunk(Data) function. To indicate client the end of the stream chunk of zero length should be sent: Response:write_chunk(<<>>).
When handling of current request is over MochiWeb decides should connection be closed or can be reused by HTTP persistent connection.