we are developing a SOAP webservice (Apache/PHP). All run well for small size calls, but with a 1Mb soap call (the HTTPS call size is 1MB) our Delphi Soap client stop with a timeout on all PC but one, and our PHP clients run well with a default_socket_timeout=300, but stop with a "Error Fetching http headers" with default_socket_timeout=60.
How can we change the timeout for Delphi? In fact this timeout seem to be in a Windows XP network library (wininet.dll called by soaphttptrans.pas)
Thanks
Cédric
In fact it was a problem with IE7 installation : it change all the network timeout.
PC with IE6 have a 3600 secondes timeout, IE7 change it to 30 secondes.
Use of InternetQueryOption() show this, and InternetSetOption help to change this.
Big thanks to my work mate wich hunt the bug for hours.
There's a MaxSinglePostSize in SOAPHttpTrans. I seem to recall having issues with that. This isn't a limit, but it behaves differently (breaks up into chunks for sending) if you're over, or under that limit. (basically 32768 by default). I expect you'll hit that size sooner if you're on D2009/D2010 due to widestrings. It would be interesting to see if you run into trouble around that size. You can use Fiddler to capture some output (or hook into the OnBeforePost event and dump the serialized XML to a file yourself) and see if that's where you run into trouble, instead of the previously observed 1MB.
But anyway, the THTTPReqResp class has options for SendTimeout and ReceiveTimeout. Try adjusting those.
Also... if you are using Delphi prior to Delphi2007, you should update your soap libraries. There's a download somewhere... many bug fixes, including a nasty memory issue that will cause your app to be halted by DEP.
Related
I use TIdHTTPServer in Delphi 11 to run a simple web server on a VPS.
It works great, except from time to time my app will start to use 100% of the CPU and keep this way forever, and I can't identify what is causing this.
When this happens, the server is still active and replying to requests, but very slowly. The only way to fix this is to force close the application and open it again.
I don't have any code to show, as my code is just generic responses using the OnCommandGet event of the TIdHTTPServer. This event will handle GET parameters on the URL and return something in the AResponseInfo.ContentText.
I know this is difficult, but any ideas about what I should hunt for to fix this?
We use TIdHttpServer quite a lot and have no problems with it. We use it in Delphi 10.3-10.4.2, but it’s not the reason for the problem. Programs work a few months without restarting.
From our experience we can say that problem of such unexpected behavior can be (in order of probability):
Code is not threadsafe. Event OnCommandGet run not in a main thread, so all access to global object/resources/etc must be done thru some kind of synchronization mechanism (locks, TEvent, synchronize, mutex, semaphore or other). If code does not use synchronization – it can broke logic, throw exceptions or do some other unexpected actions (like high CPU usage).
Connections count go over the limit. TIdHttpServer has properties like ListenQueue and MaxConnections. Can be that you make more requests that the server can handle. In this case your new requests wait until they can be handled by your code and it can make some additional CPU usage. To diagnose this – try to increment some internal variable at the start of your event and decrement it at the end. Make some service request to return this variable and you will know if all work correctly. Other similar situation – connection does not close after using the inside event and stay in memory, then you can go over limits too. Try to workaround properties TIdHttpServer.KeepAlive := false and TIdHttpServer.ReuseSocket := rsFalse.
Memory leaks. Try to set variable ReportMemoryLeaksOnShutdown := true and start the application, make a few requests and close it. If you’ll see a message with leaks – then you do something wrong, try to handle these objects in the right way. In production these small leaks can take a lot of RAM and Windows will dump part of memory into a swap-file, so your new requests will take more time to be processed.
Without an example, we can't say more.
I have trouble with Delphi XE2 app. Sometimes WinInet call to ASMX service blocks and never returns - user must terminate process from task manager to close app.
To connect to ASMX service app uses service generated by WSDLImp tool.
During its work, app makes a lot of calls to web service (~1000-2000). And at some moment (last time it was 782 request item, first time it was near the end) app freezes. After some digging, logging I find out that app blocks on
WinInetResult := HttpSendRequest(Request, nil, 0, DatStr.Bytes, DatStr.Size);
In Soap.SOAPHTTPTrans unit
First guess was it is some server-side problem – server hangs on request processing. But on trials server was processing requests from other clients, while target one was blocked. And, when you use Fiddler to debug http traffic from app everything works as expected, no locks. Also, WinInet’s SendTimeout, ReceiveTimeout, ConnectTimeout has no effect – there are no timeout errors. One more point, app blocks not on specific method call, but on different ones.
After googling, I find out that HttpSendRequest can block on max parallel connections exceeded. But there are no parallel execution in app – each action is performed in main GUI thread.
My next try was to use Indy for HTTP communication instead of WinInet. And with Indy, app does its work as it should, no locks. But downside is performance degradation – app’s work takes two times longer with Indy.
This is not very good. So, I want to go back to WinInet. But for this I need to find reason of blocking. Does anybody know why HttpSendRequest can block?
P.S.
It is strange that with Indy we have such performance degradation. Maybe there are some properties, parameters to increase performance?
So, I have finally fixed this issue.
After all trials with no success, I've re-implemented SOAP calls using WinHTTP instead of WinInet.
With WinHTTP everything works normally.
So my application allows for users to upload video, convert it using FFMPEG and then transfer it over to the Flash Media Server. Lately, I've run into an issue.
If ever there is an error when converting video, I automatically generate cfcatch report PDF. This time I came across a "Cannot allocate memory" error. This massively concerns me because I'm about to promote my system out and I can't afford for the scripts to stop running within the first few hours.
Is there a way to clean up the memory issues with ColdFusion? I mean, once the job has been done, can I essentially "reset" the memory that the server was using?
If you understand the potential disaster, I'm sure you'll understand why I've got to find out how to make sure my scripts are executing properly. The physical fix is to restart the server, but I obviously cannot be restarting the server every single time a user uploads a video...
Great answer but I couldn't get your script to work so re-touched and what a difference!!!
<cfloop collection="#REQUEST#" item="mydex">
<cfset StructDelete(REQUEST, "#mydex#", "True")>
</cfloop>
<cfloop collection="#VARIABLES#" item="mydex">
<cfset StructDelete(VARIABLES, "#mydex#", "True")>
</cfloop>
I remember reading that some server versions don't properly dispose of COM objects and the like when a page request is finished. If any of this is being done via a CFC or Java class that's being set to a variable, you can put this in OnRequestEnd.cfm:
<cfset StructDelete(variables)>
<cfset StructDelete(request)>
This will get rid of any variables set on the page which are no longer needed. Shouldn't have any negative side effects, and should clear up any memory that might be still lurking in one of the variables set during the processing of that page.
You might also look into using something other than <cfexecute> to process the videos. Maybe have a background process that routinely checks a folder for videos and then converts them in the background? ColdFusion isn't necessarily efficient when it comes to batch processing.
I have come across some instances where it's helpful to actually manually run JVM garbage collection from within CF, usually when there is a long running thread doing long-term queue management and the request is very long running.
It might be worth a shot in your case.
To run the garbage collector from within CF you call the following:
<cfset runtime = CreateObject("java", "java.lang.Runtime").getRuntime()>
<cfset runtime.gc()>
Hope it helps!
If you are on windows, I recommend calling a batch file to do the conversion and file transfer. You can execute the Batch file from CF. This will prevent CF from consuming the entire memory for the conversion and the task can continue running in the background. If you want to wait to get the status, add a "timer" using a CF Java object instantiation to check the status after X seconds.
or you can call a cmd window to run it - http://www.forta.com/blog/index.cfm/2006/7/31/Using-CFEXECUTE-To-Execute-Command-Line-Utilities
I don't understand why in the world you're trying to reinvent the wheel when I wrote a DSL wrapper for FFMPEG with the fix for the memory leak included in it:
https://github.com/rip747/cfffmpeg
Fork and submit any enhancements or fixes that you'd like.
BTW, if you want to see how to really handle the memory issues that you are having, then read the article by CFSEARCHING:
http://cfsearching.blogspot.com/2007/12/using-ffmpeg-to-convert-video-files-to.html
Again, this approach is included in the DSL.
Can any one tell me which is more stable? I know each has their own advantages and disadvantages. But which one is better for http, etc?
In my previous application I used indy9 but I wasn't satisfied with it, as I would sometimes get strange errors.
Can anyone recommend one?
I use Indy in a lot of projects. I used both 9 and 10 mainly as HTTP server and proxy. The projects get very intense traffic at times (HTTP). Indy never did let me down. It works very stable.
But I also had some "strange" situations where I had to dig deep to find the underlying problem. I also do not like the way Indy tends to handle a lot of things through exceptions. In general I like the ICS coding style more. But let me go to ICS.
ICS uses non-blocking sockets, while indy uses blocking. While non-blocking is ok and seems to be better at first sight, I found it irritating in a lot of situations. The problem is that the natural flow of the code gets lost because of the callback functions. This makes it harder to write procedural type of libraries. Furthermore I do not like how everything is handled through messages. For me it gets messy real quick when mixed with multithreading. And multithreading is mainstream these days.
So while I like the coding style and quality of the code in ICS, I prefer the simplicity of use and blocking mode of Indy. What you like more is up to you, but both libraries are mature and stable.
These are my two cents.
I also use both Indy and ICS.
Most of the time I prefer Indy because implementing sequential type of protocols with it is very easy (the request runs in it's own thread so you simply Read/Write to the connection, really easy). Using Indy requires solid knowledge of threading and synchronization. Unlike Runner I like how Indy uses Exceptions to handle "exceptional" stuff because it allows me to concentrate on the normal flow of the protocol (I use try-finally blocks to make sure I deallocate resources).
I also used ICS in a application where Indy simply failed: I used it for an application that implements an TCP/IP proxy. Using ICS was simpler because of it's non-blocking nature. I was able to "proxy" TCP/IP protocols that I know nothing about, so I have no idea how bytes would flow from one end to the other. Indy failed in that scenario because in Indy you're ether reading or you're writing, you can't do both at the same time. Using ICS to implement an sequential-type protocol is a bit of pain: you essentially need to use state-machine logic, brake the protocol in small bits, keep flags laying around so you know where you are in the protocol. An big plus: François Piette, the author of ICS, is active and very helpful on a number of forums and mailing list, and is very prompt to help with anything related to ICS.
For me, if I need to do something with TCP/IP, the decision path is very simple: Can it be done with Indy? Then it's Indy. If it can't be done with Indy then it'll be done with ICS!
I've used Indy 9 and 10 for TCP, HTTP and FTP with very few problems. ICS is a good choice, too. It's non blocking, which will change how you use it.
I haven't used it, but I've heard good things about Synapse, which is also blocking.
We test Indy10 IdTCPClient to receive video stream from remote server, it's OK. But when it's receiveing stream, the same time use it send some data to the server, after some minute, the received stream data began lost data bytes. We use sniffer tool to trace this problem, confirmed that IdTCPClient lost some bytes in receiveing stream.
So, we test Indy9.018, the same problem happend but a few times VS. Indy 10.
Remember that Indy is always in beta stage. Sometimes you need to work with night builds.
Which one is better really depends on specific use case, but I was unhappy with indy as an http client and ICS ended up being exactly what I needed it to be, with a lot less random quirks.
Note though that it is non blocking, so it's not just a drop in replacement.
I use Indy 9 for stable, released, production code to over 1 million users, and have never received any strange errors.
I'd say the answer depends on what you want to do with the internet. Indy is fine if you are prepared to get involved with understanding how it works, and is very capable. ICS is a different take on things, and I've used it effectively for many-connection systems. But for day to day "grab a file or send an email" type stuff, where you want to do a basic task, I pretty much always use Clever Components Internet Suite as you just create the component,
set the options, and it works. The suite is quite comprehensive, and gets useful updates.
I have a simple application, which should send a single byte to a serial port once a minute. But sometimes, from some strange reason, it freezes somewhere in the WriteFile() function. Both sw and hw flow controls are turned off. I've googled some stuff about pending read operations performed from another threads, but I believe this is not a problem, because my app has single thread. Also, handle from CreateFile looks valid, so the port should not be used by any other applications. Have anybody suffered this?
If you google for the words writefile hangs, you'll find a number of discussions on this problem. Some leads are buffer overruns, sizing your buffer correctly, a defective COM port, clearing the status on error... Seems like there are plenty of things to try.
Another thing I would suggest is to use a communications library instead of calling the API directly, something like Async Professional (http://sourceforge.net/projects/tpapro/). Even if they add some overhead to your application, they might simplify your work and avoid a number of potential pitfalls...
Well, I'm using this library: http://lhdelphi.ic.cz/uploader/storage/ComDrv32.pas in Delphi 7, on Windows XP, but the component inside is just a wrapper around some Win API calls, CreateFile, WriteFile, etc.
Have you tried setting CommPortDriver.CheckLineStatus to true ("to prevent hangs when not device connected or device is OFF")? The source for the comdrv32.pas library contains that suggestion.
You can also try ComPort, which was neglected for some time but is now actively developed again.