I'm having some trouble reading files with Indy from a site that has WordPress installed.
It appears that the site is configured to redirect all hits to sitename/com/wordpress.
Can I use HandleRedirect to turn that off so I can read files from the root folder?
What is the normal setting for this property? Any downsides to using it for this purpose?
(Edit: it appears that my problem may be caused by Windows cacheing of a file I've accessed before through Indy. I'm using fIDHTTP.Request.CacheControl := 'no-cache'; is that adequate?
When the server sends a 3xx result for a request, the HandleRedirects property controls whether Indy will immediately turn around and issue a new request using the new location. The alternative is that Indy will return the response code to your program. You're welcome to handle it yourself with the OnRedirect event, but if the server bothers to send anything in addition to the response code, it's unlikely to be of much use to your program. It's not as though there are hidden files that the redirection is preventing you from downloading. Set the property to true and let Indy take care of the redirection for you.
It's probably not the case that Windows is caching anything for your program. Indy doesn't use the OS cache. The Cache-Control header is an instruction to a proxy or the so-called origin server that it should not satisfy your request using a cached response without validating it with the origin server. Maybe WordPress has a cache of its own that you're by-passing.
Related
This question is maybe more a tip for people to search a solution if they have the same problem (as I found the solution eventually).
I had an application that does some HTTP requests with a local server (a mix of GET/POST with JSON content in the request/response bodies). The server is a third-party application, and after I upgraded it to a recent version, my Delphi app was no longer working.
It turned out that it was now hanging on the statement:
IdHTTP.Post("URL", "Payload", "BytesStreamResult");
As a manual POSTMAN request was still working, it had to be on the Delphi client side.
Further isolating the issue showed that the HTTP POST request did get an HTTP 200 response with valid HTTP response headers, but then was getting stuck reading the response body. It was hanging on:
IOHandler.ReadLn
When I compared the headers with the POSTMAN response, I noticed that 'Transfer-Encoding: chunked' was missing in the Delphi response.
Finally, I noticed the code related to TIdHTTP's hoKeepOrigProtocol option, which is not set by default.
So, my POST request was "downgraded" to an HTTP 1.0 request, and I guess this now made the (updated) server to respond differently (I'm not an RFC expert, but I guess 'chunked' is maybe an HTTP 1.1 option only).
After setting this option, everything worked like before (and indeed, the response was now read as "chunked" in Delphi).
Summary:
Shouldn't hoKeepOrigProtocol be the default option? (why punish good citizens for those that are not...)
Can we intercept this? Now my POST is assuming upfront a streamed response and thus it hangs because the server doesn't write anything to the buffer.
What would that high-level code look like? As it seems a mix of interpreting the header response headers and then deciding if more response reading is required.
(it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes...)
TIdHTTP supports non-chunked responses (which yes, is an HTTP 1.1 feature), so the hanging would have to be caused by the server sending a malformed response (a bug that should be reported to the server author).
When reading a non-chunked and non-MIME response, TIdHTTP does not use IOHandler.ReadLn to read the response's body, as you claim. Only when reading the response's headers.
But, since you did not show what the response actually looks like, nobody can explain for sure exactly why the hang occurs.
Shouldn't hoKeepOrigProtocol be the default option?
At the time the option was first introduced, no. There were plenty of buggy HTTP 1.1 servers around that downgrading to HTTP 1.0 was warranted.
However, that was many years ago. Nowadays, HTTP 1.1 is much more mature, and such buggy servers are rare. So, feel free to submit a change/pull request to Indy's GitHub repo if you feel the default behavior should be changed.
Can we intercept this?
No. The behavior you describe is most likely caused by a bug in the HTTP server. Either it is not sending all of the data it should be, or else the response is likely malformed in a way that makes TIdHTTP expect more data than is actually being sent. Either way, all you can do is assign a non-infinite timeout to TIdHTTP.
it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes.
Indy is designed to use infinite timeouts by default. You can assign custom timeouts to TIdHTTP's ConnectTimeout and ReadTimeout properties.
Setting this prevent the HTTP protocol downgrade:
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoKeepOrigProtocol];
This is, of course, dependant upon how the server processes the protocol specification, and if it results in issues or not.
For months, I have used the IdWebSocketSimpleClient unit with Delphi 10.3 provided here to setup a Websocket connection to the discord gateway API. Now, that I changed to Delphi 11, the same code shows another behaviour.
Previously, the component connected to the server, initiated an HTTP Upgrade to a websocket connection, received the 101 - switching protocols response and started to communicate. Now, with Delphi 11, I receive a 400 - Bad request. The inner error message shows "The plain HTTP request was sent to HTTPS port". In both scenarios, the same OpenSSL dll files are sitting in the applications executable folder and an TIdSSLIOHandlerSocketOpenSSL component is used.
At this point, I am really confused why this is happening, as the programs code hasn't been changed. This makes me wonder if there have been any behaviour changes to the indy units involved. I have no clue where to even start my investigation. Can someone help?
Delphi 10.3 was released in late 2018. There was a change made to the TIdSSLIOHandlerSocketBase class in late 2019, to fix a bug where its PassThrough property was being initialized to False when it should have been initialized to True instead (which is now the current behavior).
So, that could easily account for the behavior you are seeing, as the error message is complaining about an unsecured HTTP message being sent to a secure HTTPS port (because PassThrough is likely True).
That change did not affect most Indy components, as internally they explicitly set PassThrough as needed (typically based on a UseTLS property, or a specific URL protocol/port being requested). But, the change does affect end-user code that uses TIdTCPClient directly (as this WebSocket code is doing). In that case, the user is (and always has been) responsible for setting PassThrough as they need, but this WebSocket code is not doing that. When connecting to a secure URL, it assumes that PassThrough has been set to False by the user before TIdSimpleWebSocketClient.Connect() is called, rather than forcing it to False. And PassThrough is certainly not being set to False if TIdSimpleWebSocketClient needs to auto-create its own SSLIOHandler object.
TIdSimpleWebSocketClient.Connect() should internally be setting PassThrough := not lSecure; before calling inherited Connect. I have reported this as a bug to the author for you:
#10: TIdSimpleWebSocketClient TLS error - 400 Bad request, the plain HTTP request was sent to HTTPS port
In the meantime, until the author fixes this bug, you can simply assign your own SSLIOHandler component to the TIdSimpleWebSocketClient.IOHandler property and set it to PassThrough=False before calling TIdSimpleWebSocketClient.Connect() with a wss: URL.
I have a SOAP server/client application written in Delphi XE working fine for some time until a user who runs it on Windows 7 x64 behind a corporate proxy/firewall. The application sends and receives TSOAPAttachment object in the request.
The Problem:
Once the first request from this user is received and processed, the server could not process any request (from any user) successfully coming after this.
The server still response to the request, but the SOAPAttachment of the request
seems corrupted after the first one from this user, that's why it couldn't process the request successfully.
After putting may debug logs to the server, I noticed the TSOAPAttachment.SourceStream in the request's parameter become inaccessible (or empty), and TSOAPAttachment.CacheFile also empty. Therefore whenever trying to use the SourceStream, it will return Access Violation error.
Further investigation found that the BorlandSoapAttachment(n) file generated in the temp folder by the first request still exist and locked (which should be deleted when a request is completed normally), and BorlandSoapAttachment(n+1) files of the following request are piling up.
The SOAP server will work again after restarting IIS or recycle the application pool.
It is quite certain that it is caused by the proxy or the user’s networks because when the same machine runs outside this networks, it will work fine.
To add more mystery to the problem, running the application on WinXP behind the same proxy have no problem AT ALL!
Any help or recommendation is very appreciated as we have stuck in this situation for some time.
Big thanks in advance.
If you are really sure that you debugged all your server logic that handles the attachments to attempt do discover any piece of code that could failed specifically on Windows 7, I would suggest:
1) Use some network sniffer Wireshark is good for this task, make two subsequent requests with the same data/parameters values, and compare the HTTP contents. This analyze should be done both in the client (to see if the data is always leaving the client machine with the same content) and also in the server, to analyze the incoming data;
2) I faced a similar situation in the past, and my attempts to really understand the problem was not well succeed. I did workaround the problem sending files as Base64 encoded strings parameters, and not using the SOAP attachments. The side affect of using Base64 its an increase of ~30% in the data size to be sent, and this could be significant if you are transferring large files.
Remember that SOAP attachments create temp files in the server, and Windows 7 has different file access rules than Windows XP. I don't know if this could explain the first call being processed ant the others not, but maybe there are something related with file access.
Maybe it is UAC (User Access Control) problem under Win 7. Try running the client in win 7 "As Administrator" and see if it is working properly.
I'm looking for a cache implementation for the Indy IdHTTP component that would be used for checking of a specific resource is already cached before doing the actual GET and returning the cached resource instead (if appropriate)?
If the component would be a derived class from TIdHTTP and handle the cache-features automatically it would be the easiest way for me, so I could just replace my existing IdHTTP objects with the new component.
Has anyone ever seen such a component for D2010?
I dont think there is a component to do that, but you can implement your own based on response header Cache-Control and Expires-Date (i am not sure if it is the corrent name). You should save the page on a file with the Expires date on it, so you can check if you already have the page on cache before make the request to the server. I guess it is what some proxies do, like Squid (of course not as simple as that).
Check this out: Caching in HTTP
I was testing an app from Marco Cantu's Delphi 2009 Handbook. This app, called AnonAjax (page 200), has some interesting features to recognize certain elements from a Web page and list them. This app uses anonymous methods with an internal Indy HTTP client component used to access to a given URL. As part of this app's functionality, it loads the first image listed. That's great, but with a Web page using cookies it does not load the image properly. I've tried with AllowCookies=true but it does not work. How can I enable cookies for this app?
you can attach a cookie manager (indy component) to the indy http client component.
Indy 10's cookie management is currently undergoing a major rewrite to address many issues that it has. It is likely that you are encountering a situation where the version of Indy you are using is either rejecting the server's cookies incorrectly, or is not sending them back to the server correctly. Unfortunately, I do not have an exact ETA as to when the new cookie management code will be ready for public use, but it is very close, possibly by the end of this month if time permits.