I want to download a file from FTP. If the file is small (usually under 1000MB) it works. However, if the file is big I get an EIdReadTimeout. Why? Should I keep the connection alive? From what I know reading data has its own channel so I don't have to keep the connection alive.
What is odd is that the exception appears at the end of the Get (after Get successfully downloads the whole file): FTP.Get(Name, TempGzFile, TRUE, FALSE) !!!!
Documentation:
TIdFTP.ReadTimeout - Number of milliseconds to wait for an FTP protocol response.
TIdFTP.TransferTimeout - Timeout value for read operations on the data channel for the FTP
client.
By default ReadTimeout is set to 60sec and TransferTimeout to 10sec.
I a using Delphi XE7 (which I guess uses Indy 10). The Passive property for my IdFTP is set to false.
The FTP protocol uses multiple TCP/IP connections - one for the main command/response connection, and separate connections for data transfers. While a data transfer is in progress, the main command connection sits idle. Once the transfer is finished, the command connection receives a response.
If you are passing through a router/firewall that is not FTP-aware, the command connection is likely to get killed if it sits idle for too long during a large transfer. The connection is usually not killed "gracefully", so even the OS does not know the connection is gone. When TIdFTP then tries to read a transfer response that never arrives, it times out.
To account for that, use the TIdFTP.NATKeepAlive property to enable TCP/IP level keep-alives on the command connection during transfers. Set NATKeepAlive.UseKeepAlive to True, and set NATKeepAlive.IdleTimeMS (the idle timeout before keepalives start sending) and NATKeepAlive.IntervalMS (the interval between each keepalive) to suitable values.
Note, however, that IdleTimeMS and IntervalMS are only implemented for Windows 2000+, Linux, and BSD at this time. Other platforms use defaults provided by the OS (which tend to be very large). If you need to customize the values on those platforms, you can use the TIdFTP.OnDataChannelCreate and TIdFTP.OnDataChannelDestroy events to call TIdFTP.Socket.Binding.SetSocketOption() directly as needed.
Related
I use a TIdFTP control to connect to an FTP server. It seems that my FTP connection is terminated by something (ftp server/firewall?) after 60 seconds of inactivity. Once the connection was terminated I cannot use the FTP control (TIdFTP) anymore because its status is undefined. For example, every time I try to REconnect I get the same (10038) error. However, FTP.Connected shows True. Trying FTP.Disconnect gives me a 'connection closed gracefully' error and the control remains connected. The only solution is to end the program and start it again.
The NAT Keep Alive is already set to 15000.
How to reset the status of the FTP control (how do I disconnect)?
I use a TIdFTP control to connect to an FTP server. It seems that my FTP connection is terminated by something (ftp server/firewall?) after 60 seconds of inactivity
TIdFTP has a NATKeepAlive property to avoid that exact issue from happening when connected through an FTP-unaware router/firewall. During a transfer, TCP keepalives are temporarily enabled on the control connection so it does not get closed prematurely.
However, FTP.Connected shows True.
That means the IOHandler.InputBuffer likely has unread data in it. Connected() is designed to return True if read operations can still be satisfied even if the socket has been closed.
Trying FTP.Disconnect gives me a 'connection closed gracefully' error
By default, Disconnect() sends a QUIT command to the server before closing the socket. Since the connection is already gone, that send fails.
Disconnect() has an optional ANotifyPeer parameter that you can set to False to skip the QUIT command.
You should also Clear() the IOHandler.InputBuffer after an unexpected disconnect to clear out any unread data.
The only solution is to end the program and start it again.
That is never the solution. Worse case, you could simply destroy the TIdFTP object and create a new one. But if you follow the steps above, you should not need to resort to even that.
You cannot keep TIdFTP open. The connection will be closed from server side for every timed out time.
As you use this component to fetch or put file to file server for a time interval, You can connect just before using the TIDFTP, and disconnect it after the usage. Then you do not need to bother about freeing the component and connection failures.
I tested a simulation disconnect of multiple clients by cutting their internet connection. I found that TIdTCPServer did not discharge their threads, it did not detect their disconnect. By comparison, when I closed a client manually, the server detected the disconnection and discharged its thread.
Abnormal disconnects are not detected by the OS in a timely manner. It can take a considerable amount of time for a lost socket connection to timeout internally so the OS can invalidate it. Until the OS does that, Indy has no way of knowing that the client connection is gone.
To account for that, you should either:
implement a timeout in your application-layer data protocol. If you are expecting a client to send something to your server, and it does not do so for a certain amount of time, assume the client is gone and close the connection. During periods of idle activity, require clients to send a heartbeat command to your server at regular intervals to keep their connections alive. You can use the AContext.Connection.IOHandler.CheckForDataOnSource() method to wait for data to arrive, or you can use the AContext.Binding.SetSockOpt() method to specify an SO_RCVTIMEO timeout on blocking reads.
if you cannot change your data protocol, you can at least enable TCP-level keep-alives on the socket itself. In the server's OnConnect event, you can call the AContext.Binding.SetKeepAliveValues() method to enable keep-alives. The OS will then handle the keep-alives for you, and will invalidate the connection if the timeout elapses.
With that said, also make sure that your server event handlers are not swallowing Indy exceptions (derived from EIdException). That can also cause the server to not terminate threads correctly, if a connection is lost and Indy raises an exception about it but you are not allowing the server to process it. If you need to catch exceptions (for logging, etc), make sure to re-raise any EIdException-derived exception and let the server handle it.
i have a client server application (TCP) that's designed with indy delphi.
i want to have a queue and flow control in my server side application.
my server should not lose any clients data when server traffic is full.
for example , in my server side application i want determine maximum of bandwidth for server is 10Mbps and then if server bandwidth (this 10Mbps) was full the other clients be on queue until bandwidth get free .
so i want to know how can i design this with delphi ?
thanks
best regard
The client should not send the message directly to the server. Put the message in a local store (f.i. sqlite-db) and in a thread you read the first message from the local store and try to send it to the server.
If the message was delivered to the server (no exception raised) delete the message from the local store and process the next "first" message in the local store.
Within the TIdTCPServer.OnExecute method which receives the client data, it is possible to 'delay' processing of the incoming request with a simple Sleep command. The client data will stay in the TCP socket until the Sleep command finished.
If your server keeps track of the current 'global' bandwidth usage for all clients, it is possible to set the Sleep time dynamically. You could even set different priorities for different clients.
So you would need a simple but thread safe bandwidth usage monitor, an algorithm which calculates sensible Sleep time values, and a way to assign this Sleep time to the individual client connection contexts.
See also:
https://en.wikipedia.org/wiki/Token_bucket
https://github.com/bandwidth-throttle/token-bucket for an example implementation in PHP
http://www.nurkiewicz.com/2011/03/tenfold-increase-in-server-throughput.html
I have a Windows Service that works with an advantage database and occasionally makes some http calls. On rare occasions these calls can be very long. To the tune that my database connection times out. I'm not using a Data Module or anything. Just creating the connection manually.
My primary question is what usually prevents the connection from timing out if I just haven't used it in a while? Do the TAdsComponents send a keep alive message that gets called in the background somehow? Is that dependent on the vcl so I don't have that in my service? Somehow I feel like creating a thread to make my http call, and in the main thread checking for it to finish every few seconds would prevent the connection from dying. Is that ever true?
Yes, there is a keepalive mechanism as you expect. The client (for all communication types, TCP, UDP, Shared memory) sends a "ping" to the server every so often to let the server know that connection is still alive. The frequency of that keepalive ping is based on the server configuration parameter CLIENT_TIMEOUT. With the default settings, I believe the keepalive ping is sent every 30 seconds.
The keepalive logic runs in a separate thread that is started by the code that handles the communication. In other words, it does not depend on any of the VCL components; if you have a connection to the server, then that thread should be running.
One way to check if your connections are timing out is to look in the Advantage error log. There should be 7020 errors corresponding to timed out connections.
Some things that come to mind that might result in timed out connections include:
The client process being suspended for some reason so that the keepalive thread could not run. This seems unlikely.
The keepalive thread was killed for some reason. This also seems unlikely; you would have to go out of your way to make this happen.
A firewall may close the connection if there is no activity for a time. I would think, though, that a 30 second interval would be sufficient to prevent that.
A firewall may disallow the UDP keepalive packets. Firewalls, by nature, are "suspicious" of UDP packets. You might make sure you are using TCP/IP.
I developped an application that uses indy component to download updates from a remote server.
The problem is that if the FTP server is down or the IP address is not correct, the idFTP.connect() takes too long to give the result (connection failure).
What is the best way to accelerate the connection answer, or may be checking ip address before connection to idFTP.
Thanks in advance.
You should set ReadTimeout property, by default it is set to one minute.
By default, Indy clients wait as long as it takes for the OS to report whether the connection was successful or not. Yes, that can take a long time, if the OS has to look up the hostname with DNS, do network checks, deal with network latency, etc. If you do not want to wait that long, you can use the Timeout parameter of Connect() in Indy 9 and earlier, or the ConnectTimeout property in Indy 10, to reduce the amount of time waited on. HOWEVER, that only applies to the actual socket connect attempt once the server IP has been determined. If you set the Host property to a non-IP hostname, Indy asks the OS to perform a DNS lookup to get the hostname's IP, and there is no logic available in Connect() to control the time it takes to do that lookup. If you need that much control, then use TIdDNSResolver to get the IP manually and then assign it to the Host property before calling Connect().
Well, native connect() API timeouts are notoriously lengthy by design, (to accommodate high latency links like modems). Artificially shortening the timeout may result in premature failure notification, (though as many developers have never seen a modem, it's not that much of a problem today:).
FTP is a reasonably complex transfer requiring two TCP connections and perhaps a DNS lookup - any of these could conceivably generate long connection delays. TidFTP has an inherited 'ReadTimeout' property and a connect() overload with a timeout parameter, but I'm not sure how effective they are.
Historically, I have always timed out such operations myself using a TTimer or similar - if the FTP thread does not respond with a suitable signal, (eg. TThread.Sychronize or user-defined Windows message SendMessage()'d to the GUI), in time, a 'FTP failed' actions are taken and a flag is set in the FTP thread that tells it to ignore any replies and self-terminate. Don't use PostMessage - if you do, there is a small window of time in which a posted response my be queued up while the TTimer is firing - a race.
Oh - and if you are just plonking a TidFTP onto the form, (or creating one in TForm.FormCreate), and trying to run it from the main GUI thread, (with, or without, TidAntiFreeze), stop doing it and thread off the FTP.