I use TIdHTTPServer in Delphi 11 to run a simple web server on a VPS.
It works great, except from time to time my app will start to use 100% of the CPU and keep this way forever, and I can't identify what is causing this.
When this happens, the server is still active and replying to requests, but very slowly. The only way to fix this is to force close the application and open it again.
I don't have any code to show, as my code is just generic responses using the OnCommandGet event of the TIdHTTPServer. This event will handle GET parameters on the URL and return something in the AResponseInfo.ContentText.
I know this is difficult, but any ideas about what I should hunt for to fix this?
We use TIdHttpServer quite a lot and have no problems with it. We use it in Delphi 10.3-10.4.2, but it’s not the reason for the problem. Programs work a few months without restarting.
From our experience we can say that problem of such unexpected behavior can be (in order of probability):
Code is not threadsafe. Event OnCommandGet run not in a main thread, so all access to global object/resources/etc must be done thru some kind of synchronization mechanism (locks, TEvent, synchronize, mutex, semaphore or other). If code does not use synchronization – it can broke logic, throw exceptions or do some other unexpected actions (like high CPU usage).
Connections count go over the limit. TIdHttpServer has properties like ListenQueue and MaxConnections. Can be that you make more requests that the server can handle. In this case your new requests wait until they can be handled by your code and it can make some additional CPU usage. To diagnose this – try to increment some internal variable at the start of your event and decrement it at the end. Make some service request to return this variable and you will know if all work correctly. Other similar situation – connection does not close after using the inside event and stay in memory, then you can go over limits too. Try to workaround properties TIdHttpServer.KeepAlive := false and TIdHttpServer.ReuseSocket := rsFalse.
Memory leaks. Try to set variable ReportMemoryLeaksOnShutdown := true and start the application, make a few requests and close it. If you’ll see a message with leaks – then you do something wrong, try to handle these objects in the right way. In production these small leaks can take a lot of RAM and Windows will dump part of memory into a swap-file, so your new requests will take more time to be processed.
Without an example, we can't say more.
Related
I have trouble with Delphi XE2 app. Sometimes WinInet call to ASMX service blocks and never returns - user must terminate process from task manager to close app.
To connect to ASMX service app uses service generated by WSDLImp tool.
During its work, app makes a lot of calls to web service (~1000-2000). And at some moment (last time it was 782 request item, first time it was near the end) app freezes. After some digging, logging I find out that app blocks on
WinInetResult := HttpSendRequest(Request, nil, 0, DatStr.Bytes, DatStr.Size);
In Soap.SOAPHTTPTrans unit
First guess was it is some server-side problem – server hangs on request processing. But on trials server was processing requests from other clients, while target one was blocked. And, when you use Fiddler to debug http traffic from app everything works as expected, no locks. Also, WinInet’s SendTimeout, ReceiveTimeout, ConnectTimeout has no effect – there are no timeout errors. One more point, app blocks not on specific method call, but on different ones.
After googling, I find out that HttpSendRequest can block on max parallel connections exceeded. But there are no parallel execution in app – each action is performed in main GUI thread.
My next try was to use Indy for HTTP communication instead of WinInet. And with Indy, app does its work as it should, no locks. But downside is performance degradation – app’s work takes two times longer with Indy.
This is not very good. So, I want to go back to WinInet. But for this I need to find reason of blocking. Does anybody know why HttpSendRequest can block?
P.S.
It is strange that with Indy we have such performance degradation. Maybe there are some properties, parameters to increase performance?
So, I have finally fixed this issue.
After all trials with no success, I've re-implemented SOAP calls using WinHTTP instead of WinInet.
With WinHTTP everything works normally.
I am doing a delphi application that will run on my pc 24/7 in the background and will check if it has to do some actions or not, wait 30 minutes and check again, and so on.
How can I make sure the application will not overload cpu or memory because of being running all the time.
Create a timer to run every 30 minutes, and call your checks/actions from there. Then your application can just sit idle when there is nothing to do.
Alternatively you could create a Scheduled Task that just runs periodically to do this.
The answers about timers are good solutions, and I add this:
Make sure that the timer event, or subsequent procedure called, checks for busy. i.e. if you wake up, make sure that the last batch is done before starting a new batch. This is easy to miss when things are flowing well, and then you have a situation where things are backed up and the whole system logjams at 8 in the morning because something bad happened at midnight and now there are now 16 calls stacked up (or threads, processes, etc..).
So write your timer event like this:
OnMyTimer...
begin
MyTimer.Enabled := false;
try
DoSomethingForALongTime; // Usually runs in 45 seconds, but sometimes takes 45 minutes!
finally
MyTimer.Enabled := true; // and (SomeAbortRequest = False) and (SomeHorribleErrorCount = 0);
end;
end;
The answers about timers are pretty much exactly what you're looking for. As for your question about not overloading the CPU or memory, take a look at your program in the Task Manager. When it's not doing anything, it should sit at a "steady state" of memory, not allocating any more, and using 1% or less of CPU time.
It's safe to let most programs idle for long periods. The VCL knows how to handle the idling without hogging CPU resources for you, and you just need to use a timer to make sure it wakes up and activates its event at the right time.
Most programming languages have a "sleep" function that you can call to make the program stop doing stuff.
You are in control of the memory usage; you can deallocate your memory before going to sleep and reallocate it when coming awake, but...
It might be better to just set up some recurring job. I don't know how to do this, but I suspect there's a way with Windows Scripting Host to just launch your application on whatever schedule you want.
If you want to enforce an absolute state of inactivity, I guess you could use the "sleep" function. Though I'm not sure how it would behave on a reboot. (I guess Windows would report the application as being unresponsive.)
If the application has no main form and just sitting in the tray (Or being totally invisible), it won't do much. The main message loop will handle all message it receive from the OS, but it shouldn't receive many. And the few message it will receive, it should process them (Shutdown messages, System parameters change notification, etc)
So, I think you could just set up a timer and forget about setting code to force your program to stay idle.
If you really want to limit that process activity to a maximum, you could set the thread priority when you enter/leave the timer's event. So you would set the priority to "normal" when you enter the event, and set it to "Low" when getting out of it.
You didn't tell, but if your application uses more than one thread, this could add to the amount of CPU the OS spends on your process (read up on time slices and thread-switches for example).
The OS may also swap out some of your memory pages, thus using less memory and/or reducing memory-accesses in the "wait-time" helps too.
So, if you use only one thread and have no additional message-loops either, just calling Sleep() could be a good way, as that will signal the OS that you don't need a time slice at all for a long while to come.
Avoid YieldThread()/SwitchToThread() and your own time-keeping (using Now() for example) as that would mean lots of thread-switching is taking place, to do .... nothing!
Another method could be to use WaitForMultipleObjects with a large timeout, so your application can still respond to messages.
I would suggest creating a service application (set startup type as automatic) and use CreateTimerQueueTimer as your timer.
Memory leaks can be mitigated by reserving memory requirements/pooling classes.
I have written an application (using Delphi 2009) that allows a user to select a series of queries which can be run across a number of different systems. In order to allow queries to be run concurrently, each query is run in its own thread, using a TADOQuery object. This all works fine.
The problem that I have is when I try to close the application when a query is still running (and therefore a separate thread is active). When I create each thread, I record the thread's THandle in an array. When I try to close the application, if any threads are still running, I retrieve the thread's handle and pass it to TerminateThread, which should theoretically terminate the thread and allow the application to close. However, this doesn't happen. The main form's onClose event is triggered and it looks like the application is closing, but the process remains active and my Delphi interface appears as though the application is still running (i.e. "Run" button greyed out, debug view active etc.). I don't get control back to Delphi until I manually end the process (either Ctrl-F2 in Delphi or via Task Manager).
I am using TerminateThread because the query can take a long time to run (a few minutes in cases where we're dealing with a million or so records, which in the end user environment is perfectly possible) and while it is running, unless I'm mistaken, the thread won't be able to check the Terminated property and therefore won't be able to end itself if this were set to True until the query had returned, so I can't terminate the thread in the usual way (i.e. by checking the Terminated property). It may be the case that the user wants to exit the application while a large query is running, and in that case, I need the application to immediately end (i.e. all running threads immediately terminate) rather than forcing them to wait until all queries have finished running, so TerminateThread would be ideal but it isn't actually terminating the thread!
Can anyone help out here? Does anyone know why TerminateThread doesn't work properly? Can anyone suggest anything to get the threads running large ADO queries to immediately terminate?
You should be able to cancel an ADO Query by hooking the OnFetchProgress event, and setting the Eventstatus variable to esCancel. This should cause your query to terminate and allow the thread to close gracefully without having to resort to using TerminateThread.
Instead of using threads with TADOQuery, maybe you should consider using the async options of ADO.
ADOQuery1.ExecuteOptions := [eoAsyncExecute, eoAsyncFetch, eoAsyncFetch];
Then when your application close, you can call :
ADOQuery1.cancel;
As you can read in the msdn using TerminateThread is dangerous.
TerminateThread is a dangerous
function that should only be used in
the most extreme cases. You should
call TerminateThread only if you know
exactly what the target thread is
doing, and you control all of the code
that the target thread could possibly
be running at the time of the
termination.
But it also is very effective in killing threads. Are you sure you are right in your conclusions? Maybe the thread is killed, but another thread is still running? Maybe your handles are not thread handles? Could you show us some code? Or even better: A small working example we could try for ourselves?
I've got a program that's having some trouble during shutdown, raising exceptions that I can't trace back to their source. It appears to be timing-related and non-deterministic. This is occurring after all shared resources have been released, and since it's shutdown, memory leaks are not an issue, so that makes me wonder if there's any way to just tell the program to terminate immediately and silently after releasing the shared resources, instead of continuing with the shutdown sequence and giving an exception message box.
Does anyone know how to do that?
After looking at the Delphi Run Time Library source code, and at the Microsoft documentation; I can corroborate Mason and Paul-Jan comments.
The hierarchy of shutdown is as follows
Application.Terminate()
performs some unidentified housekeeping of application
calls Halt()
Halt()
calls ExitProc if set
alerts the user in case of runtime error
get rid of PackageLoad call contexts that might be pending
finalize all units
clear all exception handlers
call ExitprocessProc if set
and finally, call ExitProcess() from 'kernel32.dll'
ExitProcess()
unloads all DLLs
uses TerminateProcess() to kill the process
ExitProcess(0) ?
Halt(0) used to be the good old fashioned way of telling the program to end with immediate effect. There's probably a more Delphi-friendly way of doing that now, but I'm 95% sure halt(0) still works. :-)
In case HeartWare's suggestion of using ExitProcess() fails, it might be you are using some DLL's that do not respond well to the DLL_PROCESS_DETACH. In that case, try using a TerminateProcess( getCurrentProcess, 0 );
Once you resort to such measures, one might wonder if the "cleanly" part of the topic title still holds up to scrutiny.
The last time I had to hunt a problem like this was the shutdown was a causing an event (resize? It's been a while.) to fire on the dying window causing an attempt to redraw something that needed stuff that had already been disposed of.
I created a class that handles serial port asynchronously. I use it to communicate with a modem. I have no idea why, but sometimes, when I close my application, I get the Blue Screen and my computer restarts. I logged my code step by step, but when the BSOD appeared, and my computer restarted, the file into which I was logging data contained only white spaces. Therefore I have no idea, what the reason of the BSOD could be.
I looked through my code carefully and I found several possible reasons of the problem (I was looking for all that could lead to accessing unallocated memory and causing AV exceptions).
When I rethought the idea of asynchronous operations, a few things came to my mind. Please verify whether these are right:
1) WaitCommEvent() takes a pointer to the overlapped structure. Therefore, if I call WaitCommEvent() inside a function and then leave the function, the overlapped structure cannot be a local variable, right? The event mask variable and event handle too, right?
2) ReadFile() and WriteFile() also take references or pointers to variables. Therefore all these variables have to be accessible until the overlapped read or write operations finish, right?
3) I call WaitCommEvent() only once and check for its result in a loop, in the mean time doing other things. Because I have no idea how to terminate asynchronous operations (is it possible?), when I destroy my class that keeps a handle to a serial port, I first close the handle, and then wait for the event in the overlapped structure that was used when calling the WaitCommEvent() function. I do this to be sure that the thread that waits asynchronously for a comm event does not access any fields of my class which is destroyed. Is it a good idea or is it stupid?
try
CloseHandle(FSerialPortHandle);
if Assigned(FWaitCommEvent) then
FWaitCommEvent.WaitFor(INFINITE);
finally
FSerialPortHandle := INVALID_HANDLE_VALUE;
FreeAndNil(FWaitCommEvent);
end;
Before I noticed all these, most of the variables mentioned in point one and two were local variables of the functions that called the three methods above. Could it be the reason of the BSOD or should I look for some other mistakes in my code?
When I corrected the code, the BSOD stopped occuring, but It might be a coincidence. How do you think?
Any ideas will be appreciated. Thanks in advance.
I read the CancelIo() function documentation and it states that this method cancells all I/O operations issued by the calling thread. Is it OK to wait for the FWaitCommEvent after calling CancelIo() if I know that WaitCommEvent() was issued by a different thread than the one that calls CancelIo()?
if Assigned(FWaitCommEvent) and CancelIo(FSerialPortHandle) then
begin
FWaitCommEvent.WaitFor(INFINITE);
FreeAndNil(FWaitCommEvent);
end;
I checked what happens in such case and the thread calling this piece of code didn't get deadlocked even though it did not issue WaitCommEvent(). I tested in on Windows 7 (if it matters). May I leave the code as is or is it dangerous? Maybe I misunderstood the documentation and this is the reason of my question. I apologize for asking so many questions, but I really need to be sure about that.
Thanks.
An application running as a standard user should never be able to cause a bug check (a.k.a. BSOD). (And an application running as an Administrator should have to go well out of its way to do so.) Either you ran into a driver bug or you have bad hardware.
By default, Windows is configured to save a minidump in %SystemRoot%\minidump whenever a bug check occurs. You may be able to determine more information about the crash by loading the minidump file in WinDbg, configuring WinDbg to use the Microsoft public symbol store, and running the !analyze -v command in WinDbg. At the very least, this should identify what driver is probably at fault (though I would guess it's your modem driver).
Yes, you do need to keep the TOverlapped structure available for the duration of the overlapped operation. You're going to call GetOverlappedResult at some point, and GetOverlappedResult says it should receive a pointer to a structure that was used when starting the overlapped operation. The event mask and handle can be stored in local variables if you want; you're going to have a copy of them in the TOverlapped structure anyway.
Yes, the buffers that ReadFile and WriteFile use must remain valid. They do not make their own local copies to use internally. The documentation for ReadFile even says so:
This buffer must remain valid for the duration of the read operation. The caller must not use this buffer until the read operation is completed.
If you weren't obeying that rule, then you were likely reading into unreserved stack space, which could easily cause all sorts of unexpected behavior.
To cancel an overlapped I/O operation, use CancelIo. It's essential that you not free the memory of your TOverlapped record until you're sure the associated operation has terminated. Likewise for the buffer you're reading or writing. CancelIo does not cancel the operation immediately, so your buffers might still be in use even after you call it.