For months, I have used the IdWebSocketSimpleClient unit with Delphi 10.3 provided here to setup a Websocket connection to the discord gateway API. Now, that I changed to Delphi 11, the same code shows another behaviour.
Previously, the component connected to the server, initiated an HTTP Upgrade to a websocket connection, received the 101 - switching protocols response and started to communicate. Now, with Delphi 11, I receive a 400 - Bad request. The inner error message shows "The plain HTTP request was sent to HTTPS port". In both scenarios, the same OpenSSL dll files are sitting in the applications executable folder and an TIdSSLIOHandlerSocketOpenSSL component is used.
At this point, I am really confused why this is happening, as the programs code hasn't been changed. This makes me wonder if there have been any behaviour changes to the indy units involved. I have no clue where to even start my investigation. Can someone help?
Delphi 10.3 was released in late 2018. There was a change made to the TIdSSLIOHandlerSocketBase class in late 2019, to fix a bug where its PassThrough property was being initialized to False when it should have been initialized to True instead (which is now the current behavior).
So, that could easily account for the behavior you are seeing, as the error message is complaining about an unsecured HTTP message being sent to a secure HTTPS port (because PassThrough is likely True).
That change did not affect most Indy components, as internally they explicitly set PassThrough as needed (typically based on a UseTLS property, or a specific URL protocol/port being requested). But, the change does affect end-user code that uses TIdTCPClient directly (as this WebSocket code is doing). In that case, the user is (and always has been) responsible for setting PassThrough as they need, but this WebSocket code is not doing that. When connecting to a secure URL, it assumes that PassThrough has been set to False by the user before TIdSimpleWebSocketClient.Connect() is called, rather than forcing it to False. And PassThrough is certainly not being set to False if TIdSimpleWebSocketClient needs to auto-create its own SSLIOHandler object.
TIdSimpleWebSocketClient.Connect() should internally be setting PassThrough := not lSecure; before calling inherited Connect. I have reported this as a bug to the author for you:
#10: TIdSimpleWebSocketClient TLS error - 400 Bad request, the plain HTTP request was sent to HTTPS port
In the meantime, until the author fixes this bug, you can simply assign your own SSLIOHandler component to the TIdSimpleWebSocketClient.IOHandler property and set it to PassThrough=False before calling TIdSimpleWebSocketClient.Connect() with a wss: URL.
Related
This question is maybe more a tip for people to search a solution if they have the same problem (as I found the solution eventually).
I had an application that does some HTTP requests with a local server (a mix of GET/POST with JSON content in the request/response bodies). The server is a third-party application, and after I upgraded it to a recent version, my Delphi app was no longer working.
It turned out that it was now hanging on the statement:
IdHTTP.Post("URL", "Payload", "BytesStreamResult");
As a manual POSTMAN request was still working, it had to be on the Delphi client side.
Further isolating the issue showed that the HTTP POST request did get an HTTP 200 response with valid HTTP response headers, but then was getting stuck reading the response body. It was hanging on:
IOHandler.ReadLn
When I compared the headers with the POSTMAN response, I noticed that 'Transfer-Encoding: chunked' was missing in the Delphi response.
Finally, I noticed the code related to TIdHTTP's hoKeepOrigProtocol option, which is not set by default.
So, my POST request was "downgraded" to an HTTP 1.0 request, and I guess this now made the (updated) server to respond differently (I'm not an RFC expert, but I guess 'chunked' is maybe an HTTP 1.1 option only).
After setting this option, everything worked like before (and indeed, the response was now read as "chunked" in Delphi).
Summary:
Shouldn't hoKeepOrigProtocol be the default option? (why punish good citizens for those that are not...)
Can we intercept this? Now my POST is assuming upfront a streamed response and thus it hangs because the server doesn't write anything to the buffer.
What would that high-level code look like? As it seems a mix of interpreting the header response headers and then deciding if more response reading is required.
(it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes...)
TIdHTTP supports non-chunked responses (which yes, is an HTTP 1.1 feature), so the hanging would have to be caused by the server sending a malformed response (a bug that should be reported to the server author).
When reading a non-chunked and non-MIME response, TIdHTTP does not use IOHandler.ReadLn to read the response's body, as you claim. Only when reading the response's headers.
But, since you did not show what the response actually looks like, nobody can explain for sure exactly why the hang occurs.
Shouldn't hoKeepOrigProtocol be the default option?
At the time the option was first introduced, no. There were plenty of buggy HTTP 1.1 servers around that downgrading to HTTP 1.0 was warranted.
However, that was many years ago. Nowadays, HTTP 1.1 is much more mature, and such buggy servers are rare. So, feel free to submit a change/pull request to Indy's GitHub repo if you feel the default behavior should be changed.
Can we intercept this?
No. The behavior you describe is most likely caused by a bug in the HTTP server. Either it is not sending all of the data it should be, or else the response is likely malformed in a way that makes TIdHTTP expect more data than is actually being sent. Either way, all you can do is assign a non-infinite timeout to TIdHTTP.
it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes.
Indy is designed to use infinite timeouts by default. You can assign custom timeouts to TIdHTTP's ConnectTimeout and ReadTimeout properties.
Setting this prevent the HTTP protocol downgrade:
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoKeepOrigProtocol];
This is, of course, dependant upon how the server processes the protocol specification, and if it results in issues or not.
As far as I can tell, this is the process to create an HTTPS request using Indy:
Create a TIdHTTP object
Use a TIdSSLIOHandlerSocketOpenSSL object as its IOHandler
Set up this TIdSSLIOHandlerSocketOpenSSL object's SSLOptions and SSLContext to get the proper behaviour before starting the request
However, Indy's documentation is quite minimal as for the possible values for these two SSLOptions and SSLContext objects, even to achieve what seems to me to be pretty standard behaviour.
In particular, I would like to know how to do the following:
Validate the certificate against either (depending on what is more straightforward) :
The local system's trust store
A list of root certificates provided with the application
Drop the connection if the certificate has not been correctly validated.
It seems to me to be the most basic behaviour for an application that needs to call base once in a while: you want to make sure you're really speaking to your own back-end, but still leave you the possibility of changing CAs if you ever need it.
I guess the SSLContext's field rootCertFile should be used, however:
Nowhere is it said in what format the rootCertFile should be provided (pem? der? pkcs something?)
It is in no way obvious how one should process to configure several alternatives root certificates.
Can someone provide the method, and if possible, some sample code on how this behaviour can be achieved?
I am not good with delphi yet, but based on some examples I have managed to create simple http server with no more than 10 users.
There are 2 main problems I don't know how to solve yet.
proper way to authenticate, manage users - sessions
main problem, connection must be secure, so SSL encryption is needed, how to implement it?
Any example I found in relation with idhttpserver and openssl, was not quite complete or with older version of Indy.
I am currently working with Delphi XE2 with Indy 10 components.
proper way to authenticate, manage users - sessions
TIdHTTPServer manages HTTP sessions for you if you set the TIdHTTPServer.SessionState property is true (it is false by default). TIdHTTPServer uses cookies for session management, so your clients need to have cookies enabled.
Authentication has to be performed manually, but how you do that depends on whether your clients are using HTTP-based or HTML-based authentication.
For HTTP authentication, there are ARequestInfo.UserName and ARequestInfo.Password properties available. If not valid, send an appropriate 401 response back to the client (if you set the AResponseInfo.AuthRealm property to a non-blank string, TIdHTTPServer will send a 401 response for you). By default, TIdHTTPServer only supports BASIC authentication. If you want to support other authentication schemes, you will have to use the TIdHTTPServer.OnParseAuthentication event, and send the 401 reply manually so you can send back appropriate WWW-Authenticate headers. Either way, if the client is validated, you can use HTTP sessions to keep the client logged in between requests. The AResponseInfo.Session and AResponseInfo.Session properties point at the current session. If TIdHTTPServer.AutoStartSession is true (it is false by default), TIdHTTPServer creates new sessions automatically. Otherwise, you can call TIdHTTPServer.CreateSession() yourself when needed. TIdHTTPSession has a Content property that you can store session-specific data in. Or you can derive a new class from TIdHTTPSession and then use the TIdHTTPServer.OnCreateSession event to create instances of that class.
For HTML authentication, you have two choices, depending on how you configure your HTML:
if your HTML <form> tag does not have an enctype attribute, or it is set to application/x-www-webform-urlencoded, TIdHTTPServer will store the raw webform data in the ARequestInfo.FormParams property, and if TIdHTTPServer.ParseParams is true (which it is by default), the data will also be parsed into the ARequestInfo.Params property for you.
if your HTML <form> tag has an enctype attribute set to multipart/form-data, you will have to parse the content of the ARequestInfo.PostStream manually, as TIdHTTPServer does not yet parse that data for you (examples have been posted many times before on many different forums on how to parse that data manually using Indy's TIdMessageDecoderMIME class). By default, ARequestInfo.PostStream points at a TMemoryStream object. You can use the TIdHTTPServer.OnCreatePostStream event to create an instance of a different TStream-derived class, if desired.
main problem, connection must be secure, so SSL encryption is needed, how to implement it?
Before activating the server:
assign a TIdServerIOHandlerSSLBase-derived component, such as TIdServerIOHandlerSSLOpenSSL, to the TIdHTTPServer.IOHandler property and configure it as needed (certificate, peer validation, SSL version(s), etc). In the case of OpenSSL, you will have to deploy the 2 OpenSSL library binaries libeay32.dll and ssleay32.dll (or non-Windows platform equivalents) with your app if they are not already pre-installed on the target OS, or if you want to ensure your app uses a specific version of OpenSSL. At this time, OpenSSL is the only encryption that Indy supports natively, but there are third-party solutions available that are compatible with Indy, such as EldoS SecureBlackbox.
fill in the TIdHTTPServer.Binding property with a binding for your desired HTTPS port (443 is the default HTTPS port). Typically you should create 2 bindings, one for HTTP port 80 and one for HTTPS port 443. Inside your OnCommand... handlers, if you receive a request that requires SSL/TLS encryption, you can check the port that the request was made on (AContext.Binding.Port) and if not HTTPS then redirect (AResponseInfo.Redirect()) the client to retry the request on the HTTPS port.
assign a handler to the TIdHTTPServer.OnQuerySSLPort event and have it set its VUseSSL parameter to True when its APort parameter matches your HTTPS port. UPDATE starting with SVN rev 5461, an OnQuerySSLPort handler is no longer needed if your only HTTPS port is 443.
We have a delphi XE application that uses SOAP (THTTPRIO etc) communications which (in delphi) works over WinInet.dll, by default. We fixed the authentication code so that it works, using https authentication, and when the user name and password for https are correct, everything is fine.
The problem is that when authentication details are incorrect, you get a message box from Windows, that is probably being popped up by WinInet.dll itself. I want to make that dialog box go away. I can't figure out how to change my Delphi SOAP so the password won't come up.
The situation is different than this question in the following ways:
I am doing all the things that he is doing, including calling InternetSetOption(...) to set the user name and password.
I am not using a server with a self-signed certificate, so the soIgnoreInvalidCerts flag is not applicable to my case.
Somehow, I think I need to get some API calls into WinInet to tell it not to pop up the InternetErrorDlg that it has (some versions of windows say Windows Security Options) that pops up to ask the user.
In my case the user name and password we have in our configuration file is being used, it is wrong (out of date) and so we want the WinInet code to simply return an error instead of popping up the dialog box.
Perhaps the other question the guy really did figure out how to do this, but the detail on that question is insufficient to see how he did it. The accepted answer does not work for me.
Some dead ends I've followed:
WinInet MSDN docs for PLUGIN_AUTH_FLAGS_CAN_HANDLE_UI - that doesn't appear to be applicable to a WinInet user, rather to a plugin.
WinInet MSDN docs discuss InternetSetOption, and some newsgroups have lead me to the following on-before-post event handler code:
procedure TMyDevice.HTTPWebNodeOnBeforePost(
const HTTPReqResp: SOAPHTTPTrans.THTTPReqResp; Data: Pointer);
var
SecurityFlagsLen:DWORD;
SecurityFlags:DWORD;
begin
{ authentication, NTLM+HTTPS, WinInet authentication set via WinInet SET INTERNET OPTION API.
This approach recommended on newsgroups for https basic authentication. }
if fUserName<>'' then
if not InternetSetOption(Data,
INTERNET_OPTION_USERNAME,
PChar(fUserName),
Length(fUserName)) then
raise EWebServiceAuthException.Create(SysErrorMessage(Windows.GetLastError));
if fPassword<>'' then
if not InternetSetOption(Data,
INTERNET_OPTION_PASSWORD,
PChar(fPassword),
Length (fPassword)) then
raise EWebServiceAuthException.Create(SysErrorMessage(Windows.GetLastError));
{ possible type of hackage: WinInet Security option flags to stop password box? }
SecurityFlagsLen := SizeOf(SecurityFlags);
InternetQueryOption({Request}data, INTERNET_OPTION_SECURITY_FLAGS,
Pointer(#SecurityFlags), SecurityFlagsLen);
SecurityFlags := SecurityFlags or SECURITY_FLAG_something;
InternetSetOption({Request}data, INTERNET_OPTION_SECURITY_FLAGS,
Pointer(#SecurityFlags), SecurityFlagsLen);
end;
This code makes the password work, but when the user's entered password is wrong, how do I get the SOAP call to fail, or raise an exception, instead of popping up a message box?
Replace WinINet by WinHTTP component. Both have very close APIs, and the 2nd does not create any UI interaction, but will return error codes, just like any other API. The UI part of WinINet may be a good idea for some software, but it sounds like if does not fit your needs.
See http://msdn.microsoft.com/en-us/library/windows/desktop/aa384068(v=vs.85).aspx
Of course, HTTPS and authentication will be handled in a similar manner. But you'll have to prompt for the user name and password, and update the HTTP headers as requested. See this link.
From our tests, WinHTTP is much faster than WinINet (certainly because it does not implement any UI part, and is not linked to Internet Explorer libraries).
You can take a look at our Open Source classes to guess how small is the difference in the API between WinINet and WinHTTP (most code is shared in the linked unit).
Try modifying SOAPHTTPTrans to handle the error silently.
In THTTPReqResp.HandleWinInetError, there is ultimately a call to the error dialog:
Result := CallInternetErrorDlg
You can probably detect your particular error, you should be able to return a 0 from HandleWinInetError, or at least NOT make a call to CallInternetErrorDlg. See if that helps.
I'm having some trouble reading files with Indy from a site that has WordPress installed.
It appears that the site is configured to redirect all hits to sitename/com/wordpress.
Can I use HandleRedirect to turn that off so I can read files from the root folder?
What is the normal setting for this property? Any downsides to using it for this purpose?
(Edit: it appears that my problem may be caused by Windows cacheing of a file I've accessed before through Indy. I'm using fIDHTTP.Request.CacheControl := 'no-cache'; is that adequate?
When the server sends a 3xx result for a request, the HandleRedirects property controls whether Indy will immediately turn around and issue a new request using the new location. The alternative is that Indy will return the response code to your program. You're welcome to handle it yourself with the OnRedirect event, but if the server bothers to send anything in addition to the response code, it's unlikely to be of much use to your program. It's not as though there are hidden files that the redirection is preventing you from downloading. Set the property to true and let Indy take care of the redirection for you.
It's probably not the case that Windows is caching anything for your program. Indy doesn't use the OS cache. The Cache-Control header is an instruction to a proxy or the so-called origin server that it should not satisfy your request using a cached response without validating it with the origin server. Maybe WordPress has a cache of its own that you're by-passing.