I want to download a file from internet and I imagine this should be a simple task. Trying several different approaches I have found that each one has its own drawback.
The main issues are:
Application freezes until it downloads the file
Application freezes forever if the Internet connection is lost/server does not respond.
(details:
How to retrieve a file from Internet via HTTP?
The connection does not timeout while downloading file from internet )
So, finally I used the suggestions I got from several people to use "pro" libraries such as Indy. However, Indy is not much better than the pieces of code I have tried (but it is way much larger and difficult to maintain). While using Indy the application does not freezes only for short periods so it is still (somehow) usable. However, the application cannot be shut down until the download finishes (never if the Internet connections gets broken).
Other people reported the same problem: http://borland.newsgroups.archived.at/public.delphi.internet.winsock/200609/0609079112.html
https://forums.embarcadero.com/thread.jspa?threadID=25199&tstart=90
So, there is some hacking I had to do to TIDAntiFreeze in order to make it work?
Also, the ConnectTimeout property is not recognized.
fIDHTTP := TIDHTTP.Create(NIL);
fIDHTTP.ConnectTimeout:=5000;
Should I drop Indy and return to original idea of downloading the file in a separate thread and end the thread when it does not respond (at least this way I get rid of 3rd party libraries)? There will be unforeseen side effects if I do this?
Using: Delphi 7, Indy 10.1.5 10.5 (probably).
Thanks
You probably need to use Indy the Indy way: using threads. Indy was specifically designed to work in blocking mode, because that's how most internet protocols work (example: with HTTP, at protocol level, you send a request, then you read the response. You don't send and receive at the same time). TIdAntiFreeze is supposed to help you use some Indy functionality without dealing with threads; I never used that because, at least conceptually, it's an ugly hack.
If you don't want to deal with threads then you should take a look at ICS - it was designed to be used in async mode, without threading. It doesn't need the equivalent of TIdAntiFreeze because it's not blocking. You start a download and you handle some events to get progress and completion notifications. ICS is just as well-known, professional and wildly used as Indy.
It's not too difficult to solve these sorts of problems. The first thing you have to do is make sure that you have properly handled error handling. If something fails then make sure everything cleans up properly. Beyond that make sure the downloading code is part of a separate thread. If there is any problem you can always terminate the thread from your main program. Here's the code (for downloading only, not the threading) which is working fine for me.
with TDownloadURL.Create(nil) do
try
URL := 'myurltodownload.com';
filename := 'locationtosaveto';
try
ExecuteTarget(nil);
except
result := false;
end;
if not FileExists(filename) then
result := false;
finally
clear;
free;
end;
Related
I use TIdHTTPServer in Delphi 11 to run a simple web server on a VPS.
It works great, except from time to time my app will start to use 100% of the CPU and keep this way forever, and I can't identify what is causing this.
When this happens, the server is still active and replying to requests, but very slowly. The only way to fix this is to force close the application and open it again.
I don't have any code to show, as my code is just generic responses using the OnCommandGet event of the TIdHTTPServer. This event will handle GET parameters on the URL and return something in the AResponseInfo.ContentText.
I know this is difficult, but any ideas about what I should hunt for to fix this?
We use TIdHttpServer quite a lot and have no problems with it. We use it in Delphi 10.3-10.4.2, but it’s not the reason for the problem. Programs work a few months without restarting.
From our experience we can say that problem of such unexpected behavior can be (in order of probability):
Code is not threadsafe. Event OnCommandGet run not in a main thread, so all access to global object/resources/etc must be done thru some kind of synchronization mechanism (locks, TEvent, synchronize, mutex, semaphore or other). If code does not use synchronization – it can broke logic, throw exceptions or do some other unexpected actions (like high CPU usage).
Connections count go over the limit. TIdHttpServer has properties like ListenQueue and MaxConnections. Can be that you make more requests that the server can handle. In this case your new requests wait until they can be handled by your code and it can make some additional CPU usage. To diagnose this – try to increment some internal variable at the start of your event and decrement it at the end. Make some service request to return this variable and you will know if all work correctly. Other similar situation – connection does not close after using the inside event and stay in memory, then you can go over limits too. Try to workaround properties TIdHttpServer.KeepAlive := false and TIdHttpServer.ReuseSocket := rsFalse.
Memory leaks. Try to set variable ReportMemoryLeaksOnShutdown := true and start the application, make a few requests and close it. If you’ll see a message with leaks – then you do something wrong, try to handle these objects in the right way. In production these small leaks can take a lot of RAM and Windows will dump part of memory into a swap-file, so your new requests will take more time to be processed.
Without an example, we can't say more.
I have trouble with Delphi XE2 app. Sometimes WinInet call to ASMX service blocks and never returns - user must terminate process from task manager to close app.
To connect to ASMX service app uses service generated by WSDLImp tool.
During its work, app makes a lot of calls to web service (~1000-2000). And at some moment (last time it was 782 request item, first time it was near the end) app freezes. After some digging, logging I find out that app blocks on
WinInetResult := HttpSendRequest(Request, nil, 0, DatStr.Bytes, DatStr.Size);
In Soap.SOAPHTTPTrans unit
First guess was it is some server-side problem – server hangs on request processing. But on trials server was processing requests from other clients, while target one was blocked. And, when you use Fiddler to debug http traffic from app everything works as expected, no locks. Also, WinInet’s SendTimeout, ReceiveTimeout, ConnectTimeout has no effect – there are no timeout errors. One more point, app blocks not on specific method call, but on different ones.
After googling, I find out that HttpSendRequest can block on max parallel connections exceeded. But there are no parallel execution in app – each action is performed in main GUI thread.
My next try was to use Indy for HTTP communication instead of WinInet. And with Indy, app does its work as it should, no locks. But downside is performance degradation – app’s work takes two times longer with Indy.
This is not very good. So, I want to go back to WinInet. But for this I need to find reason of blocking. Does anybody know why HttpSendRequest can block?
P.S.
It is strange that with Indy we have such performance degradation. Maybe there are some properties, parameters to increase performance?
So, I have finally fixed this issue.
After all trials with no success, I've re-implemented SOAP calls using WinHTTP instead of WinInet.
With WinHTTP everything works normally.
A few years back for one of my applications I moved my serial handling to a thread when a certain app had to respond very quickly to certain serial events.
It was in BDS2006, with an older version (some 3.x?) It was done by having code like this in the tthread.execute:
while ...
begin
events:=[evRxChar];
{ Prepare a stop event for killing a waiting communication wait. }
try
comport1.WaitForEvent(events,stopevent.handle,500);
if evRxChar in events then
ComPort1RxChar(comport1,comport1.InputCount); // method of thread.
on e: exception do {only logs} ;
end;
Initialization was like
procedure TSerThread.initcomport(comportname:string='COM1');
begin
comport1:=tcomport.create(nil);
ComPort1.BaudRate:=br115200;
ComPort1.DataBits:=dbEight;
ComPort1.Port:=comportname;
ComPort1.StopBits:=sbOneStopBit;
ComPort1.Events:=[];
ComPort1.Connected:=true;
StopEvent := TEvent.Create(nil,{ManualReset}false,{InitialState}false,'StopEvent');
end;
The rxchar method reads using
comport1.readstr()
I recently had to dig this up, and noticed it didn't work in my Delphi XE, which has tcomport4. Looking in the source I saw remarks that
comport4 has changed the way it deals with its internal threads ("overlapped" property), but the default seems to be "true" has a comment "classic", and I
assumed that means compatible with older versions.
Note that the protocols are binary, and all string,char<-> ansistring,ansichar changes have been done like I did in the normal mainthread version
Now the real questions:
does anybody have tcomport4 working in a thread ?
Are there obvious mistakes in the above?
or do I need to migrate to a different component?
I'm still debugging what is going on, but am in a hurry, which is while I post this in the hope of quickly getting pointers.
Update
I reinstalled an old turbo delphi copy, and verified it worked there with v3. I fixed a small bug though in the codepath that went a bit different from what I thought (not in the above code)
This allows me to describe the behaviour between dxe/comportv4 and bds2006/comportv3 better; the v4 code generates much more read events (hundreds/second). It seems that the read doesn't delete the read chars from the incoming queue or so.
Update 2
I did a quick test with the newest version, and did the necessary rearranging (kill string function use for an essentially binary protocol). I got stuck for a while because the application crashed on startup, but that is because I use gnugettext, and Comport packages a different (non unicode even?) version. But after that it works.
Unfortunately the changes are a bit to risky to propagate back to the production version (completely written protocol decoding), so I'll have to test with inbetween versions (between 4.0 and 4.11f). I'll do that in due time, any suggestion which version was the last readstr(ansistring) one?
Update 3
In the end I simply renamed the 4.11f units and use them in parallel to the old version, using the 4.11f for the threaded codebases and the old one to maintain existing code.
Long term I might simply take the waitforevent code and make an own version. The main problem with it is that it is afaik not possible to see if the wait terminated on timeout, stopevent or whatever. This means you need another timer if you want to e.g. send on regular intervals.
You need to upgrade to the latest stable 4.11 release here.
Can any one tell me which is more stable? I know each has their own advantages and disadvantages. But which one is better for http, etc?
In my previous application I used indy9 but I wasn't satisfied with it, as I would sometimes get strange errors.
Can anyone recommend one?
I use Indy in a lot of projects. I used both 9 and 10 mainly as HTTP server and proxy. The projects get very intense traffic at times (HTTP). Indy never did let me down. It works very stable.
But I also had some "strange" situations where I had to dig deep to find the underlying problem. I also do not like the way Indy tends to handle a lot of things through exceptions. In general I like the ICS coding style more. But let me go to ICS.
ICS uses non-blocking sockets, while indy uses blocking. While non-blocking is ok and seems to be better at first sight, I found it irritating in a lot of situations. The problem is that the natural flow of the code gets lost because of the callback functions. This makes it harder to write procedural type of libraries. Furthermore I do not like how everything is handled through messages. For me it gets messy real quick when mixed with multithreading. And multithreading is mainstream these days.
So while I like the coding style and quality of the code in ICS, I prefer the simplicity of use and blocking mode of Indy. What you like more is up to you, but both libraries are mature and stable.
These are my two cents.
I also use both Indy and ICS.
Most of the time I prefer Indy because implementing sequential type of protocols with it is very easy (the request runs in it's own thread so you simply Read/Write to the connection, really easy). Using Indy requires solid knowledge of threading and synchronization. Unlike Runner I like how Indy uses Exceptions to handle "exceptional" stuff because it allows me to concentrate on the normal flow of the protocol (I use try-finally blocks to make sure I deallocate resources).
I also used ICS in a application where Indy simply failed: I used it for an application that implements an TCP/IP proxy. Using ICS was simpler because of it's non-blocking nature. I was able to "proxy" TCP/IP protocols that I know nothing about, so I have no idea how bytes would flow from one end to the other. Indy failed in that scenario because in Indy you're ether reading or you're writing, you can't do both at the same time. Using ICS to implement an sequential-type protocol is a bit of pain: you essentially need to use state-machine logic, brake the protocol in small bits, keep flags laying around so you know where you are in the protocol. An big plus: François Piette, the author of ICS, is active and very helpful on a number of forums and mailing list, and is very prompt to help with anything related to ICS.
For me, if I need to do something with TCP/IP, the decision path is very simple: Can it be done with Indy? Then it's Indy. If it can't be done with Indy then it'll be done with ICS!
I've used Indy 9 and 10 for TCP, HTTP and FTP with very few problems. ICS is a good choice, too. It's non blocking, which will change how you use it.
I haven't used it, but I've heard good things about Synapse, which is also blocking.
We test Indy10 IdTCPClient to receive video stream from remote server, it's OK. But when it's receiveing stream, the same time use it send some data to the server, after some minute, the received stream data began lost data bytes. We use sniffer tool to trace this problem, confirmed that IdTCPClient lost some bytes in receiveing stream.
So, we test Indy9.018, the same problem happend but a few times VS. Indy 10.
Remember that Indy is always in beta stage. Sometimes you need to work with night builds.
Which one is better really depends on specific use case, but I was unhappy with indy as an http client and ICS ended up being exactly what I needed it to be, with a lot less random quirks.
Note though that it is non blocking, so it's not just a drop in replacement.
I use Indy 9 for stable, released, production code to over 1 million users, and have never received any strange errors.
I'd say the answer depends on what you want to do with the internet. Indy is fine if you are prepared to get involved with understanding how it works, and is very capable. ICS is a different take on things, and I've used it effectively for many-connection systems. But for day to day "grab a file or send an email" type stuff, where you want to do a basic task, I pretty much always use Clever Components Internet Suite as you just create the component,
set the options, and it works. The suite is quite comprehensive, and gets useful updates.
I have a simple application, which should send a single byte to a serial port once a minute. But sometimes, from some strange reason, it freezes somewhere in the WriteFile() function. Both sw and hw flow controls are turned off. I've googled some stuff about pending read operations performed from another threads, but I believe this is not a problem, because my app has single thread. Also, handle from CreateFile looks valid, so the port should not be used by any other applications. Have anybody suffered this?
If you google for the words writefile hangs, you'll find a number of discussions on this problem. Some leads are buffer overruns, sizing your buffer correctly, a defective COM port, clearing the status on error... Seems like there are plenty of things to try.
Another thing I would suggest is to use a communications library instead of calling the API directly, something like Async Professional (http://sourceforge.net/projects/tpapro/). Even if they add some overhead to your application, they might simplify your work and avoid a number of potential pitfalls...
Well, I'm using this library: http://lhdelphi.ic.cz/uploader/storage/ComDrv32.pas in Delphi 7, on Windows XP, but the component inside is just a wrapper around some Win API calls, CreateFile, WriteFile, etc.
Have you tried setting CommPortDriver.CheckLineStatus to true ("to prevent hangs when not device connected or device is OFF")? The source for the comdrv32.pas library contains that suggestion.
You can also try ComPort, which was neglected for some time but is now actively developed again.