I developped an application that uses indy component to download updates from a remote server.
The problem is that if the FTP server is down or the IP address is not correct, the idFTP.connect() takes too long to give the result (connection failure).
What is the best way to accelerate the connection answer, or may be checking ip address before connection to idFTP.
Thanks in advance.
You should set ReadTimeout property, by default it is set to one minute.
By default, Indy clients wait as long as it takes for the OS to report whether the connection was successful or not. Yes, that can take a long time, if the OS has to look up the hostname with DNS, do network checks, deal with network latency, etc. If you do not want to wait that long, you can use the Timeout parameter of Connect() in Indy 9 and earlier, or the ConnectTimeout property in Indy 10, to reduce the amount of time waited on. HOWEVER, that only applies to the actual socket connect attempt once the server IP has been determined. If you set the Host property to a non-IP hostname, Indy asks the OS to perform a DNS lookup to get the hostname's IP, and there is no logic available in Connect() to control the time it takes to do that lookup. If you need that much control, then use TIdDNSResolver to get the IP manually and then assign it to the Host property before calling Connect().
Well, native connect() API timeouts are notoriously lengthy by design, (to accommodate high latency links like modems). Artificially shortening the timeout may result in premature failure notification, (though as many developers have never seen a modem, it's not that much of a problem today:).
FTP is a reasonably complex transfer requiring two TCP connections and perhaps a DNS lookup - any of these could conceivably generate long connection delays. TidFTP has an inherited 'ReadTimeout' property and a connect() overload with a timeout parameter, but I'm not sure how effective they are.
Historically, I have always timed out such operations myself using a TTimer or similar - if the FTP thread does not respond with a suitable signal, (eg. TThread.Sychronize or user-defined Windows message SendMessage()'d to the GUI), in time, a 'FTP failed' actions are taken and a flag is set in the FTP thread that tells it to ignore any replies and self-terminate. Don't use PostMessage - if you do, there is a small window of time in which a posted response my be queued up while the TTimer is firing - a race.
Oh - and if you are just plonking a TidFTP onto the form, (or creating one in TForm.FormCreate), and trying to run it from the main GUI thread, (with, or without, TidAntiFreeze), stop doing it and thread off the FTP.
Related
I want to download a file from FTP. If the file is small (usually under 1000MB) it works. However, if the file is big I get an EIdReadTimeout. Why? Should I keep the connection alive? From what I know reading data has its own channel so I don't have to keep the connection alive.
What is odd is that the exception appears at the end of the Get (after Get successfully downloads the whole file): FTP.Get(Name, TempGzFile, TRUE, FALSE) !!!!
Documentation:
TIdFTP.ReadTimeout - Number of milliseconds to wait for an FTP protocol response.
TIdFTP.TransferTimeout - Timeout value for read operations on the data channel for the FTP
client.
By default ReadTimeout is set to 60sec and TransferTimeout to 10sec.
I a using Delphi XE7 (which I guess uses Indy 10). The Passive property for my IdFTP is set to false.
The FTP protocol uses multiple TCP/IP connections - one for the main command/response connection, and separate connections for data transfers. While a data transfer is in progress, the main command connection sits idle. Once the transfer is finished, the command connection receives a response.
If you are passing through a router/firewall that is not FTP-aware, the command connection is likely to get killed if it sits idle for too long during a large transfer. The connection is usually not killed "gracefully", so even the OS does not know the connection is gone. When TIdFTP then tries to read a transfer response that never arrives, it times out.
To account for that, use the TIdFTP.NATKeepAlive property to enable TCP/IP level keep-alives on the command connection during transfers. Set NATKeepAlive.UseKeepAlive to True, and set NATKeepAlive.IdleTimeMS (the idle timeout before keepalives start sending) and NATKeepAlive.IntervalMS (the interval between each keepalive) to suitable values.
Note, however, that IdleTimeMS and IntervalMS are only implemented for Windows 2000+, Linux, and BSD at this time. Other platforms use defaults provided by the OS (which tend to be very large). If you need to customize the values on those platforms, you can use the TIdFTP.OnDataChannelCreate and TIdFTP.OnDataChannelDestroy events to call TIdFTP.Socket.Binding.SetSocketOption() directly as needed.
I have a server application which runs on a Linux machine. I can connect this application from Windows/Linux machines and can send/recieve data. After a few hours, something occurs and I get following error on the client side.
On Windows: An existing connection was forcibly closed by the remote host
On Linux: Connection timed out
I have made a search on the web and found some posts which suggest to increase/decrease OS's keep alive time. However, it didin't work for me.
Can I found a soultion to this problem or should I simply try to reconnect to the server when the connection is forcibly closed?
EDIT: I have tracked the situation. I sent a data to the remote node and sent another data after waiting 5 hours. Sending side sent the first data, but whet the sender sent the second data it didn't response. TCP/IP stack of the sender repeated this 5 times by incrementing the times between retries. Finally, sender reset the connection. I can't be sure why this is happening (Maybe because of a firewall or NAT - see Section 2.4) but I applied two different approach to solve this problem:
Use TCP/IP keep alive using setsockopt (Section 4.2)
Make an application level keep alive. This is more reliable since the first approach is OS related.
It depends on what your application is supposed to do. A little more information and perhaps the code you use for listening and handling connections could be of help.
Regardless, technically a longer keep alive time, should prevent the OS from cutting you off. So perhaps it is something else causing the trouble.
Such a thing could be router malfunction or traffic causing your keep-alive packet to get lost.
If you aren't already testing it on a LAN (without heavy trafic) I suggest doing so.
It might also be due to how your socket is handled (which I can't determine from your question)
This article might help.
Non blocking socket with timeout
I'm not used to how connections are handled on Linux, but I expect the OS won't cut off a connection unnecessary.
You can re-establish connection as a recovery, but you need to take into account that not all disconnects are gentle, and therefore you could end up making recovery on a connection you actually wish to be closed.
Since it is TCP, it will do its best to make a gentle disconnect, but you can send a custom message telling the server or client not to re-establish the connection right before disconnecting. That way you be absolutely sure, despite that it should be unnecessary to do so.
I have a Windows Service that works with an advantage database and occasionally makes some http calls. On rare occasions these calls can be very long. To the tune that my database connection times out. I'm not using a Data Module or anything. Just creating the connection manually.
My primary question is what usually prevents the connection from timing out if I just haven't used it in a while? Do the TAdsComponents send a keep alive message that gets called in the background somehow? Is that dependent on the vcl so I don't have that in my service? Somehow I feel like creating a thread to make my http call, and in the main thread checking for it to finish every few seconds would prevent the connection from dying. Is that ever true?
Yes, there is a keepalive mechanism as you expect. The client (for all communication types, TCP, UDP, Shared memory) sends a "ping" to the server every so often to let the server know that connection is still alive. The frequency of that keepalive ping is based on the server configuration parameter CLIENT_TIMEOUT. With the default settings, I believe the keepalive ping is sent every 30 seconds.
The keepalive logic runs in a separate thread that is started by the code that handles the communication. In other words, it does not depend on any of the VCL components; if you have a connection to the server, then that thread should be running.
One way to check if your connections are timing out is to look in the Advantage error log. There should be 7020 errors corresponding to timed out connections.
Some things that come to mind that might result in timed out connections include:
The client process being suspended for some reason so that the keepalive thread could not run. This seems unlikely.
The keepalive thread was killed for some reason. This also seems unlikely; you would have to go out of your way to make this happen.
A firewall may close the connection if there is no activity for a time. I would think, though, that a 30 second interval would be sufficient to prevent that.
A firewall may disallow the UDP keepalive packets. Firewalls, by nature, are "suspicious" of UDP packets. You might make sure you are using TCP/IP.
I was given 10 new PCs, all (supposedly) with Windows 7 Pro freshly installed and nothing else done to them.
I have a program, coded in Delphi XE2, using Indy 10 components for the networking. I set the "connect timeout" and "read timeout" properties of my TIdTcpCleint to 500ms, set "resuse socket" to 'o/s dependant'" (I also tried a build with it set to No) and leave "use Nagle" (whatever that is set to True (I also tried with false).
Here's the problem: when I run the same .EXE on these PCs and test the case where I pull the network cable, my debug trace shows the connect attempt / connect timeout happening in the same second or the next second (with a granularity of 1 second) - but on others it is 20 or 21 seconds before I see the conenction timeout.
It would seem some of that the PCs are not totally "fresh install" as claimed, although I see no aps installed. Maybe some one installed somethign then removed it, maybe they tried to tweak performance.
Before I reinstall Windows on 10 PCs, can anyone suggest where to look? Does 20 (or 21) seconds ring a bell with regard to TCP Client connect timeout?
[update] I am attempting to connect directly to a specific IP Address, so I am not sure if #Nikolai suggestion to check DNS is relevant. Sorry for not mentioning this originally.
[upperdate] the program does not attempt to keep the socket open. It connects, sends some data & disconnects - repeatedly, for each new piece of data.
Sadly, this is working as intended. The connect did already timeout. Indy made the determination that the connect would fail in the 500 milliseconds that you asked it to. However, that does not guarantee the function will return.
After the connect times out, Indy spins down the connection to release all of its resources. It does this synchronously. This means that you wind up waiting for the underlying TCP operation to fail. This typically takes 20 seconds.
The solution is to call connect in a thread. Believe it or not, this is what Indy already does to implement the timeout. However, when it times out waiting for the thread, it tries to shut down the connection in the main thread. You need to defer that to a worker thread.
As for why it happens immediately on some systems and in 20 seconds on others, it depends on the precise networking configuration. For example, if IPv6 is enabled, the stack may attempt to use an IPv6-to-IPv4 connection, and that may not report down even if the physical interface is down. Immediate detection of connection impossibility is never guaranteed and you shouldn't rely on it.
I've had same problems with INDY in the past (while using D6, year 1998-2000). I changed the component to IP*Works. At that time it was an external component, but as far as I know it is included in XE2. Ip*Works is a bit hard to understand at the beginning but the way they approach to the communication structure is a lot different.
I think that it would be worth to give it a try.
So i'm using tidhttp to make post requests. when i try to set connectTimeout whatever period of time it has no effect. I still wait ~5 sec before i get exception.
Thanks in advance!
If you are connecting to a hostname instead of an IP address, the hostname has to be resolved to an IP via a DNS lookup, which can take time, especially if the hostname has not been cached by the OS yet. The ConnectTimeout property does not account for that time.
Also, when ConnectTimeout is not zero, or if TIdAntiFreeze is being used, Indy has to use an internal worker thread to perform the actual socket API connect() call to the server. Starting a new thread can take some time, depending on available system resources at that moment. ConnectTimeout does not account for that time, either.
The current implementation of ConnectTimeout applies only to the actual socket API connect() call itself, not to any of the extra work needed to reach that point. As such, if you have a 5 second ConnectTimeout value set, it is possible to hanve more than 5 seconds elapse before TIdTCPClient.Connect() exits.