MSDN Thread
Hi, all.
First of all, please excuse any english language mistakes in the following description, because, I'm not a native speaker and well, I can't write it perfectly.
I'm trying to create a .NET (4.0) service for remote/transactional/asynchronous reception of recoverable messages from several queues. So, first, I use BeginPeek method and then Receive method in a TransactionScope (which implicitly uses MSDTC).
The problem is the mqsvc.exe of the host machine (win7/2k8r2 sp1) running my service, which does nothing else (and certainly nothing related to the reception/hosting of messages, MSMQ is empty and clean). mqsvc.exe memory allocation grows and it never releases any memory. All MSMQ registry keys about cache cleaning interval have a short time value (about 1 minute).
I tried several options :
with local and remote MSDTC (remote with obviously the host machine of messages).
with the COM library mqoa.dll instead of .NET to use explicit MSDTC transactions for MSMQ.
with several different machines (all win7/2k8r2 sp1).
There are no exceptions at the execution of my service, and all resources that I can close or/and dispose are closed/disposed as soon as possible. The memory allocation of my service is stable.
In all cases, it's the same problem. How to solve it?
Thanks in advance.
Vincent.
Problem solved on MSDN.
MSDN Thread
The following hotfix addresses this problem:
High memory usage by the Message Queuing service when you perform a remote transactional read on a Message Queuing 5.0 queue in Windows 7 or in Windows Server 2008 R2
Related
Environment:
Delphi 2009 client applications (and one Java), running on Windows 2003 server
connecting to InterBase 7.5.1 (another Windows 2003 Server) over dbExpress
The Delphi applications log the time to open the TSQLConnection using the AfterConnect event handler of the TSQLConnection object.
In random intervals, the connect need a three-minutes "extra time". I first suspected it could be a problem with the SQL query, but more detailed logging today showed that it is the SQLConnection.Connect which hangs.
I am not sure if this could be a problem with network, the InterBase server, or the Delphi / dbExpress layer.
Has anybody experienced a similar three-minutes "hang"?
p.s. the Java application does not log connect time so I can not say wheter it is affected by this problem.
This phenomenon appeared in the log files since we started with logging in 2012, but the rate has sharply increased last month. The only environment change has been the addition of new Windows servers (for RDP services, Mail, and Fax) so it could be a network-related problem.
Aside of a possible network problem, the cause of the delay can be that, from time to time, your query triggers a garbage collection in one of the table(s) that it is querying.
I don't know in detail Interbase 7.5 internals, but in my experience (with Firebird), this usually happens when a select is made on a table from which many records have been deleted/updated recently.
This doc at IBExpert.net explains it:
A garbage collection is only performed during a database sweep, database backup or when a SELECT query is made on a table (and not by INSERT, ALTER or DELETE). Whenever Firebird/InterBase® touches a row, such as during a SELECT operation, the versioning engine sweeps out any versions of the row where the transaction number is older than the Oldest Interesting Transaction (OIT). This helps to keep the version history small and manageable and also keeps performance reasonable.
A periodic sweep or backup made at low usage hours, can increase performance and minimize the risk of being hitted by an inconvenient garbage collection. See Sweep interval and automated housekeeping (page 6-20) and Facilitating garbage collection (page 11-19) at the Interbase 7.5 Operations Guide for more info on this.
Please check if hard disk power saving is activated on any disk on mentioned servers. That would explain if you have a delay in first connect and then no delay in following connections. Then, after a while power saving gets activated and the same problem raises again.
Since the rate has increased with the additions of new servers on the network you could have a packet loss and a long timeout to retry. For test that hypothesis you can change the connection timeout to a small value. You also can monitor the network traffic between the servers using wireshark or tcpdump.
Monitoring
To monitor the TCP handshake only you can use:
tcpdump -i eth0 'tcp[13] & 2 = 2
I have written a program in Delphi 7 (includes a ModBus component that uses Indy). On my machine it uses Indy 9 and works fine. It communicates well with other machines via a ModBus protocol. However, when the program is run on a different machine, I get a CPU 90-100% load. Unfortunately this machine is not in my office but "on the other side of the world". How can I find out whether this machine is using Indy 9 or Indy 10? And, further, If it is running Indy 10, could that be the problem or is this very unlikely?
Definitive answer is No
If you compile your program with indy 9, even if using packages, it shall use INDY 9 to run. AFAIK, there's no way to compile the executable using INDY 9 and use INDY 10 at runtime, even if you want, and no way it happen by accident.
To find out whats causing the high CPU load you might try a profiler like AQTime or SamplingProfiler.
That will get you the method(s) that are running most of the time. Then you will be able to find out whats causing the problem.
Alternatively you could add some logging to your application.
To find the root cause you could prepare a test application which will go through a sequence of actions like opening / closing connections. If it asks the user for confirmation ("Continue ? y/n") before proceeding, the user can check the CPU load for every step to detect the critical operation.
Thanks for answers. I do not think this is an Indy issue though. On my Quad CPU PC the CPU load also goes up from 1-2 % to aprox. 25%. This happens if I keep the line open (connected). If I, however, disconnect the ModBus Server after every poll from the ModBus CLient side and let that PC reconnect, the CPU load is always low. WHat is normal? Having the line open all time, or connect and disconnect for every poll? The polling frequency is: in Idle mode : 2000ms, in active mode 500ms.
you need to add logs to ensure you know whats going on.
is it the connection itself that is causing you the issue? or is it the work performed while connected?
Logs will help you narrow this down and you may be able to alter you code to be less processor hungry.
using AQTime or SamplingProfiler as also suggest earlier will help you.
personally i always add logging to every application by default, alot of them require turning on but its there. Once the software it on site you never know what may change and simply turning the logs on can save you alot of time
We have a C/S application all written in Delphi (Client and Server-or middleware if you want)
For the client part we use Indy.
For the server we use DXSock.
Since DXSock is dead for a while we are investigating alternatives for the sever part.
I want to hear some comments about the best Server Socket alternative component for Delphi.
The current system usually have tens of permanent connections working each one on its own thread but could be hundreads in the future (this should be improved to a thread pool if possible)
If you want to have the best possible performance, you'd have to use sockets in non blocking mode, or using completion ports. IPWorks is implemented like that, as well as iocp. As far as I can tell, Indy or Synapse don't implement them (at least officially).
We used completion ports and a thread pool in our open source SynCrtSock unit, used in our Synopse SQLite3 framework.
Here are some benchmarks of this solution, working from Delphi 6 up to Delphi XE. I don't tell this is the "best component", but it's a working and speedy one (every request is about 4 KB of JSON data):
Http client keep alive (i.e. one HTTP/1.1 client connection kept alive during requests):
first in 7.87ms, done in 153.37ms i.e. 6520/s, average 153us
Http client multi connect (i.e. one new HTTP/1.0 client connection created for each request - this one uses completion ports and a thread pool):
first in 151us, done in 305.98ms i.e. 3268/s, average 305us
For speed comparison, here are other communication protocols available in our framework:
Named pipe access:
first in 78.67ms, done in 187.15ms i.e. 5343/s, average 187us
Local window messages:
first in 148us, done in 112.90ms i.e. 8857/s, average 112us
Direct in process access:
first in 44us, done in 41.69ms i.e. 23981/s, average 41us
We use HTTP/1.1 protocol over TCP/IP, because there is very little overhead over plain TCP/IP, and this is a well handled protocol for firewalls and such, and allows our framework to be used by an AJAX application, whereas its main purpose is to serve Delphi clients.
IMHO there is no "best Server Socket alternative component for Delphi", it depends what is the purpose of your server application. The main bottleneck will be in the Windows kernel itself. Perhaps direct access to the HTTP Kernel-Mode Driver (Http.sys) of Windows could help.
Consider using a dedicated optimized Server instead of a Delphi server, like lighttpd or Cherokee using FastCGI to handle the requests via a Free Pascal (or CrossKylix) application, under Linux. I guess this will be the best performance possible.
I use Indy components for commercial server-side work and the component set is pretty solid (9 or 10). My servers have millions of connections per day with no issues.
I used DXSock many moons ago. He was always optimizing, but never seemed to finish it. He does seem to have another version out.
If you want commercial support, then I'd recommend IPWorks from nSoftware.
Actually DXSock is not dead, v6.1 was just released. The web hosting company we used to use in Tennessee lost the domain - so only customers who have kept their subscription renewed annually have received DXSock 5.0, 6.0 and 6.1.
Indy CANNOT support more than 2,000 concurrent connections on 32bit Windows - as Chad and crew use TThread, which implements the defacto 1MB per thread/socket connection - 2000x1MB = >2.5GB of RAM which 32bit OSes do not support. DXSock implements a 0b per connection model (unless you define otherwise) and can handle over 50,000 concurrent on Windows, Linux, Mac, Pi, etc.
Ozz Nixon - ozznixon#bpdx.com if you want more details on 6.1
Author of DXSock
Co-Author of Winshoes which became INDY.
HI,
We have a device on the field which sends TCP packets to our server once a day. I have a Windows service which constantly listens for those packets. The code in the service is pretty much a carbon copy from the MSDN example (Asynchronous Server Socket Example) – the only difference being our implementation doesn't send anything back. It just receives, processes the data and closes the socket. The service just starts a thread which immediately addresses the code linked above.
The problem is that when I goto the Task Manager of the server on which it is running, the service seems be to using all of the CPU (it says 99) all the time. I was notified of this by IT. But I don't understand what those CPU cycles are being used for, the thread just blocks on allDone.WaitOne() doesn't it?
I also made a Console Application with the same code, and that works just fine i.e. using CPU only when data is being processed. The task in each case is completed successfully each time, but the service implementation, from the looks of it seems very inefficient. What could I be doing wrong here?
Thanks.
Use a profiler to find out where your CPU is spent. That should have been yourfirst throught - profilers are one of the main tools for programmers.
It will pretty much exactly tell you what part of the code blows the CPU.
The code, btw., looks terrible. Like an example how to use async cockets, not like a good architecture for a multi connection server. Sorry to say, you possibly have to rewrite thse.
Is it possible to emulate incoming messages using Indy (if it's of any importance: I'm using Indy 10 and Delphi 2009)? I want to be able to create these messages locally and I want Indy to believe that they come from specific clients in the network. All the internal Indy handling (choice of the thread in which the message is received and stuff like that) should be exactly the same as if the message would have arrived over the network.
Any ideas on that? Thanks in advance for any tips.
What you want to do has nothing to do with Indy, as you would need to do this on a much lower level. The easiest way to make Indy believe that messages come from a specific client is to inject properly prepared packets into the network stack. Read up on TCP Packet Injection on Google or Wikipedia. EtterCap is one such tool that allows to inject packets into established connections. However, this is definitely going into gray areas, as some of the tools are illegal in some countries.
Anyway, all of this is IMHO much too complicated. I don't know what exactly you want to do, but a specially prepared client or server is a much better tool to emulate certain behaviour while developing server or client applications. You can run them locally, or if you need to have different IP addresses or subnets you can do a lot with virtual machines.
Indy doesn't have any built-in mechanisms for this but thinking off the top of my head I would recommend building a small test application (or a suite) that runs locally on your development machine and connects to your Indy server application to replay messages.
It should be irrelevant to your Indy server applications if a TCP connection is made either locally or from a remote host as the mechanisms by which a server thread is created and a command processed is identical to both scenarios.
My last gig involved using Indy and all our testing was done with a similar Resender type application that would load local message files and send these to the Indy server app.
HTH and good luck!
One thing you can do would be to create virtual machines to run your test clients, that way they will not be seen as "local machine", and its fairly simple to create a complex network with VMS -- provided you have enough memory and disk space. The other advantage of testing with VM's is you can eliminate the development environment completely when its time to focus on deployment. Amazing how much time that saves alone.
VirtualPC is a free download from Microsoft and works fairly well. VMWare has another option, but costs a little more to get started. For development purposes, I prefer the desktop versions but the server versions also work well. You will still need to have a license to install the virtual OS. MSDN membership is probably the cheapest way to go, and allows you to build test environments for other flavors of the OS.
Indy has abstract stack mechanism for crossplatform support (IDStack.pas) I think u can hack the stack for windows (IdStackWindows.pas). It is a class. U can even consider to derivate it and override some functions to do the hack.