Delphi soap https authentication failure pops up a dialog box - delphi

We have a delphi XE application that uses SOAP (THTTPRIO etc) communications which (in delphi) works over WinInet.dll, by default. We fixed the authentication code so that it works, using https authentication, and when the user name and password for https are correct, everything is fine.
The problem is that when authentication details are incorrect, you get a message box from Windows, that is probably being popped up by WinInet.dll itself. I want to make that dialog box go away. I can't figure out how to change my Delphi SOAP so the password won't come up.
The situation is different than this question in the following ways:
I am doing all the things that he is doing, including calling InternetSetOption(...) to set the user name and password.
I am not using a server with a self-signed certificate, so the soIgnoreInvalidCerts flag is not applicable to my case.
Somehow, I think I need to get some API calls into WinInet to tell it not to pop up the InternetErrorDlg that it has (some versions of windows say Windows Security Options) that pops up to ask the user.
In my case the user name and password we have in our configuration file is being used, it is wrong (out of date) and so we want the WinInet code to simply return an error instead of popping up the dialog box.
Perhaps the other question the guy really did figure out how to do this, but the detail on that question is insufficient to see how he did it. The accepted answer does not work for me.
Some dead ends I've followed:
WinInet MSDN docs for PLUGIN_AUTH_FLAGS_CAN_HANDLE_UI - that doesn't appear to be applicable to a WinInet user, rather to a plugin.
WinInet MSDN docs discuss InternetSetOption, and some newsgroups have lead me to the following on-before-post event handler code:
procedure TMyDevice.HTTPWebNodeOnBeforePost(
const HTTPReqResp: SOAPHTTPTrans.THTTPReqResp; Data: Pointer);
var
SecurityFlagsLen:DWORD;
SecurityFlags:DWORD;
begin
{ authentication, NTLM+HTTPS, WinInet authentication set via WinInet SET INTERNET OPTION API.
This approach recommended on newsgroups for https basic authentication. }
if fUserName<>'' then
if not InternetSetOption(Data,
INTERNET_OPTION_USERNAME,
PChar(fUserName),
Length(fUserName)) then
raise EWebServiceAuthException.Create(SysErrorMessage(Windows.GetLastError));
if fPassword<>'' then
if not InternetSetOption(Data,
INTERNET_OPTION_PASSWORD,
PChar(fPassword),
Length (fPassword)) then
raise EWebServiceAuthException.Create(SysErrorMessage(Windows.GetLastError));
{ possible type of hackage: WinInet Security option flags to stop password box? }
SecurityFlagsLen := SizeOf(SecurityFlags);
InternetQueryOption({Request}data, INTERNET_OPTION_SECURITY_FLAGS,
Pointer(#SecurityFlags), SecurityFlagsLen);
SecurityFlags := SecurityFlags or SECURITY_FLAG_something;
InternetSetOption({Request}data, INTERNET_OPTION_SECURITY_FLAGS,
Pointer(#SecurityFlags), SecurityFlagsLen);
end;
This code makes the password work, but when the user's entered password is wrong, how do I get the SOAP call to fail, or raise an exception, instead of popping up a message box?

Replace WinINet by WinHTTP component. Both have very close APIs, and the 2nd does not create any UI interaction, but will return error codes, just like any other API. The UI part of WinINet may be a good idea for some software, but it sounds like if does not fit your needs.
See http://msdn.microsoft.com/en-us/library/windows/desktop/aa384068(v=vs.85).aspx
Of course, HTTPS and authentication will be handled in a similar manner. But you'll have to prompt for the user name and password, and update the HTTP headers as requested. See this link.
From our tests, WinHTTP is much faster than WinINet (certainly because it does not implement any UI part, and is not linked to Internet Explorer libraries).
You can take a look at our Open Source classes to guess how small is the difference in the API between WinINet and WinHTTP (most code is shared in the linked unit).

Try modifying SOAPHTTPTrans to handle the error silently.
In THTTPReqResp.HandleWinInetError, there is ultimately a call to the error dialog:
Result := CallInternetErrorDlg
You can probably detect your particular error, you should be able to return a 0 from HandleWinInetError, or at least NOT make a call to CallInternetErrorDlg. See if that helps.

Related

Are there SSL behaviour changes in Delphi 11?

For months, I have used the IdWebSocketSimpleClient unit with Delphi 10.3 provided here to setup a Websocket connection to the discord gateway API. Now, that I changed to Delphi 11, the same code shows another behaviour.
Previously, the component connected to the server, initiated an HTTP Upgrade to a websocket connection, received the 101 - switching protocols response and started to communicate. Now, with Delphi 11, I receive a 400 - Bad request. The inner error message shows "The plain HTTP request was sent to HTTPS port". In both scenarios, the same OpenSSL dll files are sitting in the applications executable folder and an TIdSSLIOHandlerSocketOpenSSL component is used.
At this point, I am really confused why this is happening, as the programs code hasn't been changed. This makes me wonder if there have been any behaviour changes to the indy units involved. I have no clue where to even start my investigation. Can someone help?
Delphi 10.3 was released in late 2018. There was a change made to the TIdSSLIOHandlerSocketBase class in late 2019, to fix a bug where its PassThrough property was being initialized to False when it should have been initialized to True instead (which is now the current behavior).
So, that could easily account for the behavior you are seeing, as the error message is complaining about an unsecured HTTP message being sent to a secure HTTPS port (because PassThrough is likely True).
That change did not affect most Indy components, as internally they explicitly set PassThrough as needed (typically based on a UseTLS property, or a specific URL protocol/port being requested). But, the change does affect end-user code that uses TIdTCPClient directly (as this WebSocket code is doing). In that case, the user is (and always has been) responsible for setting PassThrough as they need, but this WebSocket code is not doing that. When connecting to a secure URL, it assumes that PassThrough has been set to False by the user before TIdSimpleWebSocketClient.Connect() is called, rather than forcing it to False. And PassThrough is certainly not being set to False if TIdSimpleWebSocketClient needs to auto-create its own SSLIOHandler object.
TIdSimpleWebSocketClient.Connect() should internally be setting PassThrough := not lSecure; before calling inherited Connect. I have reported this as a bug to the author for you:
#10: TIdSimpleWebSocketClient TLS error - 400 Bad request, the plain HTTP request was sent to HTTPS port
In the meantime, until the author fixes this bug, you can simply assign your own SSLIOHandler component to the TIdSimpleWebSocketClient.IOHandler property and set it to PassThrough=False before calling TIdSimpleWebSocketClient.Connect() with a wss: URL.

How to validate server certificate against list of CAs during HTTP request using Delphi + Indy

As far as I can tell, this is the process to create an HTTPS request using Indy:
Create a TIdHTTP object
Use a TIdSSLIOHandlerSocketOpenSSL object as its IOHandler
Set up this TIdSSLIOHandlerSocketOpenSSL object's SSLOptions and SSLContext to get the proper behaviour before starting the request
However, Indy's documentation is quite minimal as for the possible values for these two SSLOptions and SSLContext objects, even to achieve what seems to me to be pretty standard behaviour.
In particular, I would like to know how to do the following:
Validate the certificate against either (depending on what is more straightforward) :
The local system's trust store
A list of root certificates provided with the application
Drop the connection if the certificate has not been correctly validated.
It seems to me to be the most basic behaviour for an application that needs to call base once in a while: you want to make sure you're really speaking to your own back-end, but still leave you the possibility of changing CAs if you ever need it.
I guess the SSLContext's field rootCertFile should be used, however:
Nowhere is it said in what format the rootCertFile should be provided (pem? der? pkcs something?)
It is in no way obvious how one should process to configure several alternatives root certificates.
Can someone provide the method, and if possible, some sample code on how this behaviour can be achieved?

Delphi soap requester - Problems accessing TLS1.1/TLS1.2

I have asked this question of Embacadero Tech Support, raised a case there, but wondered also if anyone here knew an answer to this.
I work for a retail company and have written their Mail Order program in Delphi.
One thing the programme relies heavily on is processing credit card data. Since day one, this has never been much of an issue. Our gateway provider is Verifone in the UK and up until now I have used a SOAP requester to talk to the Verifone processing servers. I didn't have too many issues getting this working and it it has worked ever since. I don't profess to being an expert in SOAP calls etc though. This software, for component legacy reasons, resides still in D2007 and it is fine that way. I can still develop it as needed.
Until now......
Yesterday I received an e-mail from Verifone stating the following:
"On Tuesday 4th October 2016, Verifone will be making the following changes:
1. Upgrade security certificates from SHA-1 to the more secure, SHA-256 algorithm.
2. Retire support for SSL v3 and TLS v1.0."
They provided a link to this page:
http://www.verifone.co.uk/support/ssl
To begin with, I wasn't overly concerned, we have had some similar changes in the past (although mainly certificate changes) and this hasn't effected us. But the more I read it, the more I became a little concerned.
So I decided I had better run some tests. On that page on the second tab, there are some test URLs. I swapped our existing test URL (the test one we use for training on the system) with the new test one and...... it failed. I get the error:
"A connection with the server could not be established - URL:http://xml-cst.cxmlpg.com/gateway/gateway.asmx - SOAPAction:https://www.commidea.webservices.com/GetServerStatus"
Or similar to this screenshot which is an actual test credit card check rather than a server status check:
This was just doing a simple 'GetServerStatus' call which requires no account or credentials. I know the server is working using this URL in Chrome:
https://xml-cst.cxmlpg.com/gateway/gateway.asmx?op=GetServerStatus
I began to wonder if this was an issue in the components for the SOAP call in D2007.
So last evening I tried it at home in Seatlle. My thinking was that if I could at least getting it going there then I could write a little utility in Seattle that just sat alongside the D2007 Mail Order suite just to process the credit cards. I imported the WSDL from:
https://xml-cst.cxmlpg.com/gateway/gateway.asmx?wsdl
using the WSDL importer and added a button to a form to call the GetServerStatus function (in the same way that I got the original working in D2007):
procedure TForm1.Button1Click(Sender: TObject);
var
gs : GatewaySoap;
begin
gs := (HTTPRIO1 as GatewaySoap);
gs.GetServerStatus
end;
I kept my fingers crossed and clicked the button.
Alas, I got the same error message as above. So now I assume it is nothing to do with the Delphi version at all.
This is a very scary issue as basically it totally stops the automated functionality of the mail order software of which this is part. If I can't get the simple GetServerfStatus to work, then I am at a total loss.
Has anyone any ideas what is happening here? Is it that I can't do non-SSL/TLS1.0 via the SOAP requester? It looks like it has to be TLS1.1/1.2. How can i resolve this issue? I am guessing now that if it can resolved for Seattle then the same solution might apply to the existing Mail Order code in D2007.
Really hoping someone can help me out on this one! I was up verfy late last night trying different things, but can't get to grips with it. It could literally be somethhing I have just overlooked, something simple - which is my hope. Then again, I suspect it may be something a lot more technical.
Trevor
It seems the SOAP server is returning http URLs for its services in the published WSDL (whereas they should be https).
If you control the SOAP server you should fix the WSDL generator. (If it's written in Delphi it's quite probably just missing poPublishLocationAsSecure option in their TWSDLHTMLPublish.PublishOptions property.
A possible workaround on the client might be to use a (https) URL directly and not rely on the published WSDL.

Exchange Web Services with Delphi

I want to create an Exchange Web Services (EWS) client application using Delphi XE6.
I am using a THttpRio component with a wsdl. How do I set the user credentials? In other languages, the equivalent of the THttpRio component has a Credentials property (example).
But this is missing from the Delphi component.
The authentication mechanism (apart from impersonation) is not part of the ews wsdl. It is native to the SOAP layer.
Listing 1:
procedure TForm1.Button1Click( Sender: TObject);
var
lESB : ExchangeServicePortType;
request : GetServiceConfiguration;
Impersonation : ExchangeImpersonation;
RequestVersion: RequestServerVersion;
MailboxCulture1: MailboxCulture;
GetServiceConfigurationResult: GetServiceConfigurationResponse;
ServerVersion : ServerVersionInfo;
begin
lESB := HTTPRIO1 as ExchangeServicePortType;
request := GetServiceConfiguration.Create;
request.RequestedConfiguration := ArrayOfServiceConfigurationType.Create( 'UnifiedMessagingConfiguration');
Impersonation := ExchangeImpersonation.Create;
RequestVersion := RequestServerVersion.Create;
MailboxCulture1 := MailboxCulture.Create;
GetServiceConfigurationResult:= GetServiceConfigurationResponse.Create;
ServerVersion := ServerVersionInfo.Create;
try
lESB.GetServiceConfiguration(
request, Impersonation, RequestVersion, MailboxCulture1,
GetServiceConfigurationResult, ServerVersion)
finally
request.Free;
Impersonation.Free;
RequestVersion.Free;
MailboxCulture1.Free;
GetServiceConfigurationResult.Free;
ServerVersion.Free
end
end;
Listing 1 above, shows some sample code, that I have tried so far. The purpose of the function is to get the version information about the server. HTTPRIO1 is a THTTPRIO component with default properties, and hooked up to the standard wsdl for EWS. This doesn't work because user credentials are not set.
How to set the user credentials?
After a lot of trial and error, this is the solution that I came up with...
(1) Authentication
If using the THTTPRio component, the UserName/Password pair of properties of the HTTPWebNode property of the THHPRio can be used to identifiy the credentials of the user, from the perspective of establishing an internet connection through a proxy server.
If using the THTTPReqResp component, the same UserName/Password properties are directly owned by the component.
If using the TIdHTTP, there are proxy related properties to use.
If you want to use the credentials of the currently logged on, with explicitly passing the password onto the component, you can achieve this by leaving blank, the UserName and Password properties of the THTTPRio and THTTPReqResp components. Proxy information will be automatically picked up from the system registry. However TIdHTTP is different. This component required proxy configuration to be set up explicitly, including the credentials of the internet user.
(2) What component to use for EWS Soap transactions?
I could not get the standard soap solution, the THTTPRio component, to work. The problem was that the THTTPRio component did not produce the correct shape of request envelope. Without going into detail, there were numerous issues such as elements that should have been placed in the body, would appear in the header instead, and schema violations. If you can get this (THTTPRio based on the published wsdl) to work, (but I doubt it), it is worth noting that the wsdl needs to be tweeked to include the service node. Microsoft deliberately excluded the service node for security reasons, but it is needed for correct operation of the Delphi soap wizard.
Developing a solution based on TIdHTTP is a non-starter, if your application sits behind a firewall and must negotiate with a proxy server.
The simplest and best solution is to use the THTTPReqResp component. You don't even need the wisdl for this. You must craft your own xml request envelopes, but with Microsoft's excellent documentation, showing simple envelopes, this task is a doddle.
(3) On using THTTPReqResp to access EWS
EWS uses UTF-8, so set UseUTF8InHeader := True
Set InvokeOptions = [soIgnoreInvalidCerts, soAutoCheckAccessPointViaUDDI]
Set the URL to the service server URL, as required.
Set SoapAction to http://schemas.microsoft.com/exchange/services/2006/messages
Set UserName and Password as required, or leave empty to use the credentials of the logged-on user.
EWS uses soap 1.1, not 1.2, so exclude wnoSOAP12 from WebNodeOptions.
Leverage MSDN online. It's a superb resource with plenty of examples.
(4) On building XML documents.
Soap envelopes are XML documents following a specific published format, with your request data embedded. How to build these envelopes?
Here two possible ways that worked well for me:
Use a template software design pattern. For example I use a component very similar to the TPageProducer. The source your basic template, and you use tag replacement to achieve your desired result. My version of the PageProducer can define optional fragments, or fragments that are repeated n times (with different data per iteration), marked up within the one template.
Use XSLT. This is the simplest solution. It is almost code free.
Here are two possible ways, that you might use, but I recommend against. It is not that they do not work, but just that the techniques are too clumsy and take too much coding:
Manually construct the document as a string (or use string builder) part-by-part.
Build the document, part-by-part using the standard XML interfaces (IXMLNode etc), and then stream out the result.

Indy TIdHTTP URI with login/password not working

I'm writing an HTTP API wrapper to integrate with particular IP Cameras. Part of the process requires login credentials in the URI, for example:
http://user:pass_with_#_sign#address:port/somespecificpath/
I'm using Indy's TIdHTTP.Get to send the request to the device. However, the password consists of a '#' character. If this URI is placed in any browser with the plain password, the # character throws it off. Therefore, I need to encode the password's # to %23...
http://user:pass_with_%23_sign#address:port/somespecificpath/
When I paste this URI into any browser, it successfully logs in and does what it needs. However, when I pass the exact same URI into TIdHTTP.Get, it does not successfully log in, and therefore I cannot do anything as long as the password contains # (or %23). Changing the password to not include a # is far too sloppy of a solution. Indy must be messing something up with this URI/password.
Is this a bug in Indy, or is there something else I need to do to make Indy accept such an encoded password?
UPDATE
I added a new account on one of the cameras with username and password without any special characters which need encoding, and authentication still does not work. It seems as if Indy is stripping out the login credentials completely from the URI, and doesn't even send these credentials. Next thing I need to do is monitor the URI which is actually sent.
UPDATE 2
I did a packet capture via WireShark and have verified that the credentials ...user:pass#... are not even sent - they're stripped from the URI that Indy actually sends.
UPDATE 3
TLama suggested that I get a capture of what's sent when using a browser. I tried this, and sure enough even when using a browser the capture doesn't show these login credentials either... even though it works from a browser. So I have to figure out another way to identify whether or not these credentials are sent.
UPDATE 4
I tried (before seeing Remy's answer) to provide these credentials in Request.Username and Request.Password instead of in the URI, and I still have no success. I keep getting "Unauthorized" back from the device.
UPDATE 5
The documentation for this API mentions nothing relevant to how users are authenticated other than this paragraph:
Grandstream Video Surveillance API (Application Programming Interface) supports HTTP 1.0 protocol (RFC1945). This document explains in detail the parameter of functions in client side, via the supported GET/POST method. Users will require administrator privilege to retrieve or set the parameters.
And on that note, I did switch the TIdHTTP protocol version to 1.0.
UPDATE 6
Hopefully the last update needed... I did another comparison with the packet captures between a browser (Chrome) and TIdHTTP. Chrome actually sends two requests, the first one does not have any credentials, but in the second request there's a node in the header Authorization: Basic... and Credentials: User:Pass, whereas using TIdHTTP only sends 1 single request without these credentials.
UPDATE 7
7 is a lucky number :-) I just realized, the very first request I make to the device returns "Unauthorized", but all following requests I make (using the same TIdHTTP instance) are successful! So going back to my prior update, just like I see in the browser capture, it takes that second repetitive request for it to work.
# is an illegal character in a URL prior to the fragment portion of the URL, that is why it has to be encoded as %23 when used in other areas of the URL.
A username/password is not actually part of a real URL, that is why they get stripped off when TIdHTTP sends the URL to a server (monitor the traffic of any web browser and you will see the same thing happen).
To use HTTP authentication with TIdHTTP, you need to use the TIdHTTP.Request.Username and TIdHTTP.Request.Password properties instead (and you do not need to URL encode the values), eg:
IdHTTP1.Request.Username := 'user';
IdHTTP1.Request.Password := 'pass_with_#_sign';
IdHTTP1.Get('http://address:port/somespecificpath/');
If you pass a URL that has an encoded username/password in it, TIdHTTP will strip off the values and move them to the Request.Username and Request.Password properties for you, but they will remain in their original encoded format, eg:
IdHTTP1.Get('http://user:pass_with_%23_sign#address:port/somespecificpath/')
// this will set Request.Username to 'user',
// Request.Password to 'pass_with_%23_sign', and
// send a request for 'http://address:port/somespecificpath/'
If you are being given an encoded URL to start with, you can use the TIdURI class to manually decode it prior to then calling TIdHTTP.Get(), eg:
var
RequestUrl: string;
Uri: TIdURI;
begin
RequestUrl := 'http://user:pass_with_%23_sign#address:port/somespecificpath/';
...
Uri := TIdURI.Create(RequestURL);
try
IdHTTP1.Request.Username := TIdURI.URLDecode(Uri.UserName);
IdHTTP1.Request.Password := TIdURI.URLDecode(Uri.Password);
RequestURL := Uri.URI;
finally
Uri.Free;
end;
IdHTTP1.Get(RequestUrl);
...
end;
Update: either way, make sure you have appropriate IdAuthentication... units, or the IdAllAuthentications unit, in your uses clause to enable Indy's HTTP authentication classes for TIdHTTP to use.
I have solved the issue by sending two sequential Get requests. After observing the packet captures between a browser and Indy, I noticed browsers would always send one request without credentials, and then another identical request with credentials. So it only sends the credentials when it needs to. Indy was only sending one request, but if I send another request right afterward, I have success.
So, the request now looks like...
FWeb.Get(U);
FWeb.Get(U, R);
Of course it really should be in the order of "If the first request is unauthorized, then send another request with credentials".

Resources