I tested a simulation disconnect of multiple clients by cutting their internet connection. I found that TIdTCPServer did not discharge their threads, it did not detect their disconnect. By comparison, when I closed a client manually, the server detected the disconnection and discharged its thread.
Abnormal disconnects are not detected by the OS in a timely manner. It can take a considerable amount of time for a lost socket connection to timeout internally so the OS can invalidate it. Until the OS does that, Indy has no way of knowing that the client connection is gone.
To account for that, you should either:
implement a timeout in your application-layer data protocol. If you are expecting a client to send something to your server, and it does not do so for a certain amount of time, assume the client is gone and close the connection. During periods of idle activity, require clients to send a heartbeat command to your server at regular intervals to keep their connections alive. You can use the AContext.Connection.IOHandler.CheckForDataOnSource() method to wait for data to arrive, or you can use the AContext.Binding.SetSockOpt() method to specify an SO_RCVTIMEO timeout on blocking reads.
if you cannot change your data protocol, you can at least enable TCP-level keep-alives on the socket itself. In the server's OnConnect event, you can call the AContext.Binding.SetKeepAliveValues() method to enable keep-alives. The OS will then handle the keep-alives for you, and will invalidate the connection if the timeout elapses.
With that said, also make sure that your server event handlers are not swallowing Indy exceptions (derived from EIdException). That can also cause the server to not terminate threads correctly, if a connection is lost and Indy raises an exception about it but you are not allowing the server to process it. If you need to catch exceptions (for logging, etc), make sure to re-raise any EIdException-derived exception and let the server handle it.
Related
I want to download a file from FTP. If the file is small (usually under 1000MB) it works. However, if the file is big I get an EIdReadTimeout. Why? Should I keep the connection alive? From what I know reading data has its own channel so I don't have to keep the connection alive.
What is odd is that the exception appears at the end of the Get (after Get successfully downloads the whole file): FTP.Get(Name, TempGzFile, TRUE, FALSE) !!!!
Documentation:
TIdFTP.ReadTimeout - Number of milliseconds to wait for an FTP protocol response.
TIdFTP.TransferTimeout - Timeout value for read operations on the data channel for the FTP
client.
By default ReadTimeout is set to 60sec and TransferTimeout to 10sec.
I a using Delphi XE7 (which I guess uses Indy 10). The Passive property for my IdFTP is set to false.
The FTP protocol uses multiple TCP/IP connections - one for the main command/response connection, and separate connections for data transfers. While a data transfer is in progress, the main command connection sits idle. Once the transfer is finished, the command connection receives a response.
If you are passing through a router/firewall that is not FTP-aware, the command connection is likely to get killed if it sits idle for too long during a large transfer. The connection is usually not killed "gracefully", so even the OS does not know the connection is gone. When TIdFTP then tries to read a transfer response that never arrives, it times out.
To account for that, use the TIdFTP.NATKeepAlive property to enable TCP/IP level keep-alives on the command connection during transfers. Set NATKeepAlive.UseKeepAlive to True, and set NATKeepAlive.IdleTimeMS (the idle timeout before keepalives start sending) and NATKeepAlive.IntervalMS (the interval between each keepalive) to suitable values.
Note, however, that IdleTimeMS and IntervalMS are only implemented for Windows 2000+, Linux, and BSD at this time. Other platforms use defaults provided by the OS (which tend to be very large). If you need to customize the values on those platforms, you can use the TIdFTP.OnDataChannelCreate and TIdFTP.OnDataChannelDestroy events to call TIdFTP.Socket.Binding.SetSocketOption() directly as needed.
I have a Windows Service that works with an advantage database and occasionally makes some http calls. On rare occasions these calls can be very long. To the tune that my database connection times out. I'm not using a Data Module or anything. Just creating the connection manually.
My primary question is what usually prevents the connection from timing out if I just haven't used it in a while? Do the TAdsComponents send a keep alive message that gets called in the background somehow? Is that dependent on the vcl so I don't have that in my service? Somehow I feel like creating a thread to make my http call, and in the main thread checking for it to finish every few seconds would prevent the connection from dying. Is that ever true?
Yes, there is a keepalive mechanism as you expect. The client (for all communication types, TCP, UDP, Shared memory) sends a "ping" to the server every so often to let the server know that connection is still alive. The frequency of that keepalive ping is based on the server configuration parameter CLIENT_TIMEOUT. With the default settings, I believe the keepalive ping is sent every 30 seconds.
The keepalive logic runs in a separate thread that is started by the code that handles the communication. In other words, it does not depend on any of the VCL components; if you have a connection to the server, then that thread should be running.
One way to check if your connections are timing out is to look in the Advantage error log. There should be 7020 errors corresponding to timed out connections.
Some things that come to mind that might result in timed out connections include:
The client process being suspended for some reason so that the keepalive thread could not run. This seems unlikely.
The keepalive thread was killed for some reason. This also seems unlikely; you would have to go out of your way to make this happen.
A firewall may close the connection if there is no activity for a time. I would think, though, that a 30 second interval would be sufficient to prevent that.
A firewall may disallow the UDP keepalive packets. Firewalls, by nature, are "suspicious" of UDP packets. You might make sure you are using TCP/IP.
If a client connects to a server over a normal tcp connection, and then later on the client's connection cuts out, the server will get (assuming active mode) {tcp_closed,Socket}. But there are cases where the server won't know that the client has disconnected, such as power failure or crashing and such (I believe, I could be wrong). In these cases, the client is gone but the server still believes it's connected. If the server attempts to send the client a message in these cases, will it assume that the client gets the message or will the tcp stack sort that out on the low level and the server gets back some kind of error?
I know this is a simplistic question, but I've been having trouble testing it myself, as I can't get a client to catastrophically fail like I need it to (even kill -9 isn't doing it). Does anyone have any experience with this?
The answer depends. When you try to send out data, the kernels TCP window will slowly fill until it can't take any more data. Then your send will block because the internal kernel buffer is full. TCP has some timers which will trigger after some time. When that happens, the kernel will error the send request, Erlangs VM runtime will transform it into {error, Reason}, where Reason is the posix() error message from the underlying system.
If you want to be sure the data got through, you have to acknowledge it on the stream the other way. Or you can make the data idempotent so you can resend it without trouble. It is especially important if the other endpoint, the client, is a device like a mobile phone where disconnects will happen all the time.
To test it, you can block the communication with a firewall rule on lo.
im using indy10 for my communications, and sometimes when a client disconnects it raises an exception, i was wondering whats the safest way to disconnect a connection (TIdContext) ?
and what should i do on the OnDisconnect even and similar?
thanks.
Raising an exception is normal behavior. Indy is designed to make heavy use of exceptions, not only for error handling but also for internal notifications and such. OnDisconnect is fired when TIdTCPServer detects that the connection is finished, either because the client disconnected (and TIdTCPServer handled the exception for you) or because an uncaught exception occured in your OnExecute handler code. Either way, use OnDisconnect to perform any cleanup you need. TIdTCPServer will close the socket for you after the OnDisconnect event handler exits.
I just want to add something about sockets internal work (TCP) that I know:
All that server and client does, they sends pieces of data to each other. Server differs from the client only so that he is passive until any client don't send a connection request first. But if client want or forced to break the connection, all is need to do is stop send data to server. To gracefully close a connection client may send special data about this event, like saying "goodbay" by phone, but this is not absolutely required. Simply imagine phone call from you (client) to any service (server). You start conversation with "hello" and service worker responds. If you accidently press reset on your phone, call will lost. But service still continue his work. And you may make call it again. Nothing bad happens from that.
All what you need to care about is stable and correct client and server work by itself. Check incorrect sending and receiving data. Try to reconnect when it needed from client. If some exception throws inside client it must be processed as needed and its normal situation when current connection was lost by such forced events.
Everything else has already answered by Remy Lebeau.
I'm looking to detect local connection loss. Is there a mean to do that, as with the events on the Corelabs components ?
Thanks
EDIT:
Sorry, I'm going to try to be more specific:
I'm currently designing a prototype using datasnap 2009. So I've got a thin client, a stateless server app and a database server.
What I would be able to do is to detect and handle connection loss (internet connectivity) between the client and the server app to handle it appropriately, ie: Display an informative error message to the user or to detect a server shutdown to silently redirect on another app server.
In 2-tier I used to manage that with ODAC components, the TOraSession have some events to handle this issues.
Normally there is no event fired when a connection is broken, unless a statement is fired against the database. This is because there is no way of knowing a connection loss unless there is some sort of is-alive pinging going on.
Many frameworks check if a connection is still valid by doing a very small query against the server. Could be getting the time from a server. Especially in a connection pooling environment.
You can implement a connection checking function in your application in some of the database events (beforeexecute?). Or make a timer that checks every 10 seconds.
Spawn a thread on the client which periodically sends some RPC 'Ping' or 'Heartbeat' commands to the server.
if this fails, the client knows that something happened to the connection
if the server does not hear the client anymore for some time period (for example, two times the heartbeat interval), he can conclude that the client disconnected, however this requires a stateful server (and your design is stateless so it would require event processing in a secondary system, which could be fed through a message queue)