Suave stops responding with "Socket failed to accept a client" error - f#

I have an app running on AWS through ECS (as a Docker container) for which I used Suave to receive some REST commands.
The code is very primitive:
let conf =
{ defaultConfig with bindings = [ HttpBinding.createSimple HTTP "0.0.0.0" 80] }
let app =
choose
[
POST >=> choose
[
path "/" >=> request (fun m ->
// find if this is a subscription confirmation
let isConfirmation =
m.headers |> List.exists (fun kvp -> snd kvp = "SubscriptionConfirmation")
... more stuff here but that's it for the Suave code
and it is started with:
let listening, server = startWebServerAsync conf app
server |> Async.Start
listening |> Async.RunSynchronously |> ignore // wait for the server to start listening
At least once a day, Suave stops replying to any request and I get the following in the log:
(this is a screenshot because I couldn't get the text log out of the (super annoying) AWS's logging UI)
The error is always the same, and this is an app in testing with very little traffic. However, it gets a ping message every 10s from another app.
Since this is my first Suave project, how can I troubleshoot this? is this a known problem? is there more logging I can do to help troubleshoot it? could the issue be due to docker itself?
Edit:
I decided to print the connections used every 1 min:
there are 9 connections open
there are 15 connections open
there are 21 connections open
there are 26 connections open
there are 32 connections open
there are 37 connections open
there are 43 connections open
there are 49 connections open
there are 55 connections open
there are 61 connections open
there are 67 connections open
There is a ping received, as a POST message, every 10 seconds.
I track it this way:
let p = IPGlobalProperties.GetIPGlobalProperties().GetTcpIPv4Statistics()
terminal.BroadcastMessage $"there are {p.CurrentConnections} connections open"
It looks like the connections never gets closed.

This is just a guess, but I'm now able to reproduce this using PowerShell Invoke-WebRequest across multiple terminals. Each new terminal I open causes the number of open connections to increase. It turns out that Invoke-WebRequest uses a Keep-Alive header by default, so I was able to fix the problem by disabling this feature.
Thus, I'm wondering if perhaps your client (the thing sending the ping) is using a keep-alive header, but failing to reuse the connections it opens. If so, you could fix this by disabling keep-alive in the client.

Related

Flickering HttpClient sometimes throwing IOException

I'm using java.net.http.HttpClient.newHttpClient() under Java 19 (Temurin) and perform sendAsync(...) requests from different treads on the same instance. I assume this is ok, as the javadoc states:
Once built, an HttpClient is immutable...
However, some requests fail with:
java.io.IOException: HTTP/1.1 header parser received no bytes
The weird thing is, it depends on the speed of my requests:
Requests every 5 seconds: 30% failure
Requests every 3 seconds: 0% failure
I've written a test for it:
private final HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("https://..."))
.setHeader("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofByteArray("[]".getBytes()))
.build();
#ParameterizedTest
#ValueSource(ints = {3, 5})
void httpClientTest(int intervalSeconds) throws Exception {
HttpClient httpClient = HttpClient.newHttpClient();
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
Thread.sleep(Duration.ofSeconds(intervalSeconds));
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).get();
}
I've already tried the following:
Doing the same with curl on the command line. No requests fail whatever interval I try. So it's probably not a problem with the server.
Running the tests multiple times in parallel. Still the 5-second-intervals fail (then multiple times in parallel). So it's probably not a problem with the server.
Creating an HttpClient.newHttpClient() for every request. No requests fail whatever interval. So it's probably not a problem with the server but with an internal state of the HttpClient (although it claims to be immutable?).
Do you have an idea what I could do, without needing to create a new HttpClient for every request?
Here is the answer for the record: the java.net.HttpClient has a long default HTTP/1.1 keepAlive time, which is longer than what usual servers are configured with. This often results in the server closing idle HTTP/1.1 connections before the client does. If the server closes the connection at about the same time than the client tries to reuse it, some IOException might get raised.
If such exceptions are observed too frequently applications should consider adapting the default keepAlive time in the client to some value shorter than what the servers it connects to are using.
A default value for the HttpClient HTTP/1.1 keepAlive time can be specified on the command line with: -Djdk.httpclient.keepalive.timeout=duration-in-seconds
So for instance - if a server is configured with a keepAlive time of 5s, you could consider supplying -Djdk.httpclient.keepalive.timeout=3 or -Djdk.httpclient.keepalive.timeout=4 on the client's java command line.

Erlang Cowboy, Server Side Event, eventsource close and reconnects about every 60 seconds

I have the cowboy example eventsource running on a local Debian server. For the code please see --> https://github.com/ninenines/cowboy/tree/master/examples/eventsource
After about 60 seconds there is always a 'eventsource was closed' then a 'eventsource connected' message.
I am testing on the latest chrome browser on win10.
I cannot see any reason for this and I wondered if this something built into the SSE standard, that the connection is dropped at regular intervals.
The chrome debugger shows the following error message:
GET http://192.168.1.100:8080/eventsource net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)
Any thoughts?
MPC
You're most likely hitting the idle timeout.
You can change it with the following snippet in the cowboy example:
{ok, _} = cowboy:start_clear(http, [{port, 8080}], #{
idle_timeout => 15000,
env => #{dispatch => Dispatch}
}),
You have more information about ProtocolOpts in the cowboy doc

Erlang TCP sockets get closed

To learn Erlang I am trying to implement a tiny web server based on gen_tcp. Unfortunately, my code seems to trigger some wired behaviour. To demonstrate the problem I have attached a minimised version of my implementation which is sufficient to reproduce the problem. It is just delivering a static 200 OK, no matter what the HTTP request was.
The problem arises when I try to run ab (Apache HTTP server benchmarking) against my web server (using loopback interface). Without any concurrent requests (-c) everything is running just fine. However, if I use -c 8 or -c 16, the call to gen_tcp:accept/1 seems to fail on some sockets as I see a number of request: closed lines in the shell.
What makes the whole story even weirder is, that I see different behaviours on different operating systems:
OS X+Erlang/OTP 18: ab reports "Connection reset by peer" almost immediately after starting.
Debian+Erlang R15B01: All but two of the HTTP requests seem to run through. But then, ab hangs for a few seconds and reports "The timeout specified has expired, Total of 4998 requests completed", when i run ab with -n 5000. Similarly, 14998 is reported when I run 15000 tests.
This one does not seem to be the problem. I am honestly quite lost and therefore appreciate any help! :) Thanks!
server(Port) ->
Opt = [list, {active, false}, {reuseaddr, true}],
case gen_tcp:listen(Port, Opt) of
{ok, Listen} ->
handler(Listen),
gen_tcp:close(Listen),
ok;
{error, Error} ->
io:format("init: ~w~n", [Error])
end.
handler(Listen) ->
case gen_tcp:accept(Listen) of
{ok, Client} ->
request(Client),
handler(Listen);
{error, Error} ->
io:format("request: ~w~n", [Error])
end.
request(Client) ->
Recv = gen_tcp:recv(Client, 0),
case Recv of
{ok, _} ->
Response = reply(),
gen_tcp:send(Client, Response);
{error, Error} ->
io:format("request: ~w~n", [Error])
end,
gen_tcp:close(Client).
reply() ->
"HTTP/1.0 200 OK\r\n" ++
"Content-Length: 7\r\n\r\n"
"static\n".
When you increase the number of concurrent requests sent with ab -c N it will immediately open multiple TCP sockets to the server.
By default a socket opened with gen_tcp:listen/2 will support only five outstanding connection requests. Increase the number of connection requests outstanding with the {backlog, N} option to gen_tcp:listen/2.
I tested your code on OS X with ab and saw this resolve the prolem with "Connection reset by peer".

Connection in RabbitMQ server auto lost after 600s

I'm using rabbitMQ server with amq.
I am having a difficult problem. After leaving the server alone for about 10 min, the connection is lost.
What could be causing this?
If you look at the Erlang client documentation http://www.rabbitmq.com/erlang-client-user-guide.html you will see a section titled Connecting To A Broker
This gives you a few different options that you can specify when setting up your connection to the RabbitMQ server, one of the options is the heartbeat, as you can see the default is 0 so no heartbeat is specified.
I don't know the exact Erlang notation, but you will need to do something like:
{ok, Connection} = amqp_connection:start(#amqp_params_network{heartbeat = 5})
The heartbeat timeout is specified in seconds. So this would cause your consumer to heartbeat back to the server every 5seconds.
Also take a look at this discussion: https://groups.google.com/forum/?fromgroups=#!topic/rabbitmq-discuss/u227xzvqOr8
The default connection timeout for the RabbitMQ connection factory is 600 seconds (at least in the Java client API), hence your 10 minutes. You can change this by specifying to the connection factory your timeout of choice.
It is good practice to ensure your connection is release and recreated after a specific amount of time, to prevent eventual leaks and excessive resournces. Your code should ensure that it seeks a valid connection that is not close to be timed-out, and re-establish a new connection on the ones that did time-out. Overall, adopt a connection-pooling approach.
- Java example:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(this.serverName);
factory.setPort(this.serverPort);
factory.setUsername(this.userName);
factory.setPassword(this.userPassword);
factory.setConnectionTimeout( YOUR-TIMEOUT-IN-SECONDS );
Connection = factory.newConnection();

How to check server connection

i want to check my server connection to know if its available or not to inform the user..
so how to send a pkg or msg to the server (it's not SQL server; it's a server contains some serviecs) ...
thnx in adcvance ..
With all the possibilities for firewalls blocking ICMP packets or specific ports, the only way to guarantee that a service is running is to do something that uses that service.
For instance, if it were a JDBC server, you could execute a non-destructive SQL query, such as select * from sysibm.sysdummy1 for DB2. If it's a HTTP server, you could create a GET packet for index.htm.
If you actually have control over the service, it's a simple matter to create a special sub-service to handle these requests (such as you send through a CHECK packet and get back an OKAY response).
That way, you avoid all the possible firewall issues and the test is a true end-to-end one. PINGs and traceroutes will be able to tell if you can get to the machine (firewalls permitting) but they won't tell you if your service is functioning.
Take this from someone who's had to battle the network gods in a corporate environment where machines are locked up as tight as the proverbial fishes ...
If you can open a port but don't want to use ping (i dont know why but hey) you could use something like this:
import socket
host = ''
port = 55555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host, port))
s.listen(1)
while 1:
try:
clientsock, clientaddr = s.accept()
clientsock.sendall('alive')
clientsock.close()
except:
pass
which is nothing more then a simple python socket server listening on 55555 and returning alive

Resources