Trying to download a file using indy,(post to asp save the excel response) but running into errors, using wireshark the request is missing cookies.
Trying to grab the cookie out of a Twebbrowser window and save it.
procedure TForm1.WebBrowser1DownloadComplete(Sender: TObject);
var
document: IHTMLDocument2;
cookies:tstringlist;
begin
cookies:=tstringlist.Create;
document := WebBrowser1.Document as IHTMLDocument2;
cookies.Add(document.cookie);
//do stuff with them
end;
returns nothing, whats the best way to extract a cookie(or 2) out of twebbrowser, or is there something better i'm missing?
TWebBrowser is a wrapper around Internet Explorer, which itself is a wrapper around WinInet. Indy does not share cookies with IE/WinInet, so you have to copy the cookie details manually into Indy's TIdCookieManager component. However, the IHTMLDocument2.cookie property is just a delimited string of name=value pairs (if it returns anything at all, due to security restrictions), which does not contain enough information for Indy's use, such as a cookie's source URL and target domain/path, so you have to get that information from somewhere else, such as by parsing IE's cookie files that are stored in Windows' Cookies folder.
Because of a history with malware (ab)using JavaScript and the document.cookie property, web-servers may request on the HTTP level that the cookie only be returned over HTTP and not be available over JavaScript.
https://www.owasp.org/index.php/HTTPOnly
I'm not sure this is the case in your instance, but you mention you use Indy to fetch a file. If you put the data from it into WebBrowser1 yourself, it would make sense the cookie data is not copied along, since it is a property of the HTTP transaction. See the Indy documentation about cookies:
http://www.indyproject.org/docsite/html/TIdHTTP_CookieManager.html
Related
I want to create an Exchange Web Services (EWS) client application using Delphi XE6.
I am using a THttpRio component with a wsdl. How do I set the user credentials? In other languages, the equivalent of the THttpRio component has a Credentials property (example).
But this is missing from the Delphi component.
The authentication mechanism (apart from impersonation) is not part of the ews wsdl. It is native to the SOAP layer.
Listing 1:
procedure TForm1.Button1Click( Sender: TObject);
var
lESB : ExchangeServicePortType;
request : GetServiceConfiguration;
Impersonation : ExchangeImpersonation;
RequestVersion: RequestServerVersion;
MailboxCulture1: MailboxCulture;
GetServiceConfigurationResult: GetServiceConfigurationResponse;
ServerVersion : ServerVersionInfo;
begin
lESB := HTTPRIO1 as ExchangeServicePortType;
request := GetServiceConfiguration.Create;
request.RequestedConfiguration := ArrayOfServiceConfigurationType.Create( 'UnifiedMessagingConfiguration');
Impersonation := ExchangeImpersonation.Create;
RequestVersion := RequestServerVersion.Create;
MailboxCulture1 := MailboxCulture.Create;
GetServiceConfigurationResult:= GetServiceConfigurationResponse.Create;
ServerVersion := ServerVersionInfo.Create;
try
lESB.GetServiceConfiguration(
request, Impersonation, RequestVersion, MailboxCulture1,
GetServiceConfigurationResult, ServerVersion)
finally
request.Free;
Impersonation.Free;
RequestVersion.Free;
MailboxCulture1.Free;
GetServiceConfigurationResult.Free;
ServerVersion.Free
end
end;
Listing 1 above, shows some sample code, that I have tried so far. The purpose of the function is to get the version information about the server. HTTPRIO1 is a THTTPRIO component with default properties, and hooked up to the standard wsdl for EWS. This doesn't work because user credentials are not set.
How to set the user credentials?
After a lot of trial and error, this is the solution that I came up with...
(1) Authentication
If using the THTTPRio component, the UserName/Password pair of properties of the HTTPWebNode property of the THHPRio can be used to identifiy the credentials of the user, from the perspective of establishing an internet connection through a proxy server.
If using the THTTPReqResp component, the same UserName/Password properties are directly owned by the component.
If using the TIdHTTP, there are proxy related properties to use.
If you want to use the credentials of the currently logged on, with explicitly passing the password onto the component, you can achieve this by leaving blank, the UserName and Password properties of the THTTPRio and THTTPReqResp components. Proxy information will be automatically picked up from the system registry. However TIdHTTP is different. This component required proxy configuration to be set up explicitly, including the credentials of the internet user.
(2) What component to use for EWS Soap transactions?
I could not get the standard soap solution, the THTTPRio component, to work. The problem was that the THTTPRio component did not produce the correct shape of request envelope. Without going into detail, there were numerous issues such as elements that should have been placed in the body, would appear in the header instead, and schema violations. If you can get this (THTTPRio based on the published wsdl) to work, (but I doubt it), it is worth noting that the wsdl needs to be tweeked to include the service node. Microsoft deliberately excluded the service node for security reasons, but it is needed for correct operation of the Delphi soap wizard.
Developing a solution based on TIdHTTP is a non-starter, if your application sits behind a firewall and must negotiate with a proxy server.
The simplest and best solution is to use the THTTPReqResp component. You don't even need the wisdl for this. You must craft your own xml request envelopes, but with Microsoft's excellent documentation, showing simple envelopes, this task is a doddle.
(3) On using THTTPReqResp to access EWS
EWS uses UTF-8, so set UseUTF8InHeader := True
Set InvokeOptions = [soIgnoreInvalidCerts, soAutoCheckAccessPointViaUDDI]
Set the URL to the service server URL, as required.
Set SoapAction to http://schemas.microsoft.com/exchange/services/2006/messages
Set UserName and Password as required, or leave empty to use the credentials of the logged-on user.
EWS uses soap 1.1, not 1.2, so exclude wnoSOAP12 from WebNodeOptions.
Leverage MSDN online. It's a superb resource with plenty of examples.
(4) On building XML documents.
Soap envelopes are XML documents following a specific published format, with your request data embedded. How to build these envelopes?
Here two possible ways that worked well for me:
Use a template software design pattern. For example I use a component very similar to the TPageProducer. The source your basic template, and you use tag replacement to achieve your desired result. My version of the PageProducer can define optional fragments, or fragments that are repeated n times (with different data per iteration), marked up within the one template.
Use XSLT. This is the simplest solution. It is almost code free.
Here are two possible ways, that you might use, but I recommend against. It is not that they do not work, but just that the techniques are too clumsy and take too much coding:
Manually construct the document as a string (or use string builder) part-by-part.
Build the document, part-by-part using the standard XML interfaces (IXMLNode etc), and then stream out the result.
I'm writing an HTTP API wrapper to integrate with particular IP Cameras. Part of the process requires login credentials in the URI, for example:
http://user:pass_with_#_sign#address:port/somespecificpath/
I'm using Indy's TIdHTTP.Get to send the request to the device. However, the password consists of a '#' character. If this URI is placed in any browser with the plain password, the # character throws it off. Therefore, I need to encode the password's # to %23...
http://user:pass_with_%23_sign#address:port/somespecificpath/
When I paste this URI into any browser, it successfully logs in and does what it needs. However, when I pass the exact same URI into TIdHTTP.Get, it does not successfully log in, and therefore I cannot do anything as long as the password contains # (or %23). Changing the password to not include a # is far too sloppy of a solution. Indy must be messing something up with this URI/password.
Is this a bug in Indy, or is there something else I need to do to make Indy accept such an encoded password?
UPDATE
I added a new account on one of the cameras with username and password without any special characters which need encoding, and authentication still does not work. It seems as if Indy is stripping out the login credentials completely from the URI, and doesn't even send these credentials. Next thing I need to do is monitor the URI which is actually sent.
UPDATE 2
I did a packet capture via WireShark and have verified that the credentials ...user:pass#... are not even sent - they're stripped from the URI that Indy actually sends.
UPDATE 3
TLama suggested that I get a capture of what's sent when using a browser. I tried this, and sure enough even when using a browser the capture doesn't show these login credentials either... even though it works from a browser. So I have to figure out another way to identify whether or not these credentials are sent.
UPDATE 4
I tried (before seeing Remy's answer) to provide these credentials in Request.Username and Request.Password instead of in the URI, and I still have no success. I keep getting "Unauthorized" back from the device.
UPDATE 5
The documentation for this API mentions nothing relevant to how users are authenticated other than this paragraph:
Grandstream Video Surveillance API (Application Programming Interface) supports HTTP 1.0 protocol (RFC1945). This document explains in detail the parameter of functions in client side, via the supported GET/POST method. Users will require administrator privilege to retrieve or set the parameters.
And on that note, I did switch the TIdHTTP protocol version to 1.0.
UPDATE 6
Hopefully the last update needed... I did another comparison with the packet captures between a browser (Chrome) and TIdHTTP. Chrome actually sends two requests, the first one does not have any credentials, but in the second request there's a node in the header Authorization: Basic... and Credentials: User:Pass, whereas using TIdHTTP only sends 1 single request without these credentials.
UPDATE 7
7 is a lucky number :-) I just realized, the very first request I make to the device returns "Unauthorized", but all following requests I make (using the same TIdHTTP instance) are successful! So going back to my prior update, just like I see in the browser capture, it takes that second repetitive request for it to work.
# is an illegal character in a URL prior to the fragment portion of the URL, that is why it has to be encoded as %23 when used in other areas of the URL.
A username/password is not actually part of a real URL, that is why they get stripped off when TIdHTTP sends the URL to a server (monitor the traffic of any web browser and you will see the same thing happen).
To use HTTP authentication with TIdHTTP, you need to use the TIdHTTP.Request.Username and TIdHTTP.Request.Password properties instead (and you do not need to URL encode the values), eg:
IdHTTP1.Request.Username := 'user';
IdHTTP1.Request.Password := 'pass_with_#_sign';
IdHTTP1.Get('http://address:port/somespecificpath/');
If you pass a URL that has an encoded username/password in it, TIdHTTP will strip off the values and move them to the Request.Username and Request.Password properties for you, but they will remain in their original encoded format, eg:
IdHTTP1.Get('http://user:pass_with_%23_sign#address:port/somespecificpath/')
// this will set Request.Username to 'user',
// Request.Password to 'pass_with_%23_sign', and
// send a request for 'http://address:port/somespecificpath/'
If you are being given an encoded URL to start with, you can use the TIdURI class to manually decode it prior to then calling TIdHTTP.Get(), eg:
var
RequestUrl: string;
Uri: TIdURI;
begin
RequestUrl := 'http://user:pass_with_%23_sign#address:port/somespecificpath/';
...
Uri := TIdURI.Create(RequestURL);
try
IdHTTP1.Request.Username := TIdURI.URLDecode(Uri.UserName);
IdHTTP1.Request.Password := TIdURI.URLDecode(Uri.Password);
RequestURL := Uri.URI;
finally
Uri.Free;
end;
IdHTTP1.Get(RequestUrl);
...
end;
Update: either way, make sure you have appropriate IdAuthentication... units, or the IdAllAuthentications unit, in your uses clause to enable Indy's HTTP authentication classes for TIdHTTP to use.
I have solved the issue by sending two sequential Get requests. After observing the packet captures between a browser and Indy, I noticed browsers would always send one request without credentials, and then another identical request with credentials. So it only sends the credentials when it needs to. Indy was only sending one request, but if I send another request right afterward, I have success.
So, the request now looks like...
FWeb.Get(U);
FWeb.Get(U, R);
Of course it really should be in the order of "If the first request is unauthorized, then send another request with credentials".
We have a delphi XE application that uses SOAP (THTTPRIO etc) communications which (in delphi) works over WinInet.dll, by default. We fixed the authentication code so that it works, using https authentication, and when the user name and password for https are correct, everything is fine.
The problem is that when authentication details are incorrect, you get a message box from Windows, that is probably being popped up by WinInet.dll itself. I want to make that dialog box go away. I can't figure out how to change my Delphi SOAP so the password won't come up.
The situation is different than this question in the following ways:
I am doing all the things that he is doing, including calling InternetSetOption(...) to set the user name and password.
I am not using a server with a self-signed certificate, so the soIgnoreInvalidCerts flag is not applicable to my case.
Somehow, I think I need to get some API calls into WinInet to tell it not to pop up the InternetErrorDlg that it has (some versions of windows say Windows Security Options) that pops up to ask the user.
In my case the user name and password we have in our configuration file is being used, it is wrong (out of date) and so we want the WinInet code to simply return an error instead of popping up the dialog box.
Perhaps the other question the guy really did figure out how to do this, but the detail on that question is insufficient to see how he did it. The accepted answer does not work for me.
Some dead ends I've followed:
WinInet MSDN docs for PLUGIN_AUTH_FLAGS_CAN_HANDLE_UI - that doesn't appear to be applicable to a WinInet user, rather to a plugin.
WinInet MSDN docs discuss InternetSetOption, and some newsgroups have lead me to the following on-before-post event handler code:
procedure TMyDevice.HTTPWebNodeOnBeforePost(
const HTTPReqResp: SOAPHTTPTrans.THTTPReqResp; Data: Pointer);
var
SecurityFlagsLen:DWORD;
SecurityFlags:DWORD;
begin
{ authentication, NTLM+HTTPS, WinInet authentication set via WinInet SET INTERNET OPTION API.
This approach recommended on newsgroups for https basic authentication. }
if fUserName<>'' then
if not InternetSetOption(Data,
INTERNET_OPTION_USERNAME,
PChar(fUserName),
Length(fUserName)) then
raise EWebServiceAuthException.Create(SysErrorMessage(Windows.GetLastError));
if fPassword<>'' then
if not InternetSetOption(Data,
INTERNET_OPTION_PASSWORD,
PChar(fPassword),
Length (fPassword)) then
raise EWebServiceAuthException.Create(SysErrorMessage(Windows.GetLastError));
{ possible type of hackage: WinInet Security option flags to stop password box? }
SecurityFlagsLen := SizeOf(SecurityFlags);
InternetQueryOption({Request}data, INTERNET_OPTION_SECURITY_FLAGS,
Pointer(#SecurityFlags), SecurityFlagsLen);
SecurityFlags := SecurityFlags or SECURITY_FLAG_something;
InternetSetOption({Request}data, INTERNET_OPTION_SECURITY_FLAGS,
Pointer(#SecurityFlags), SecurityFlagsLen);
end;
This code makes the password work, but when the user's entered password is wrong, how do I get the SOAP call to fail, or raise an exception, instead of popping up a message box?
Replace WinINet by WinHTTP component. Both have very close APIs, and the 2nd does not create any UI interaction, but will return error codes, just like any other API. The UI part of WinINet may be a good idea for some software, but it sounds like if does not fit your needs.
See http://msdn.microsoft.com/en-us/library/windows/desktop/aa384068(v=vs.85).aspx
Of course, HTTPS and authentication will be handled in a similar manner. But you'll have to prompt for the user name and password, and update the HTTP headers as requested. See this link.
From our tests, WinHTTP is much faster than WinINet (certainly because it does not implement any UI part, and is not linked to Internet Explorer libraries).
You can take a look at our Open Source classes to guess how small is the difference in the API between WinINet and WinHTTP (most code is shared in the linked unit).
Try modifying SOAPHTTPTrans to handle the error silently.
In THTTPReqResp.HandleWinInetError, there is ultimately a call to the error dialog:
Result := CallInternetErrorDlg
You can probably detect your particular error, you should be able to return a 0 from HandleWinInetError, or at least NOT make a call to CallInternetErrorDlg. See if that helps.
I'm having some trouble reading files with Indy from a site that has WordPress installed.
It appears that the site is configured to redirect all hits to sitename/com/wordpress.
Can I use HandleRedirect to turn that off so I can read files from the root folder?
What is the normal setting for this property? Any downsides to using it for this purpose?
(Edit: it appears that my problem may be caused by Windows cacheing of a file I've accessed before through Indy. I'm using fIDHTTP.Request.CacheControl := 'no-cache'; is that adequate?
When the server sends a 3xx result for a request, the HandleRedirects property controls whether Indy will immediately turn around and issue a new request using the new location. The alternative is that Indy will return the response code to your program. You're welcome to handle it yourself with the OnRedirect event, but if the server bothers to send anything in addition to the response code, it's unlikely to be of much use to your program. It's not as though there are hidden files that the redirection is preventing you from downloading. Set the property to true and let Indy take care of the redirection for you.
It's probably not the case that Windows is caching anything for your program. Indy doesn't use the OS cache. The Cache-Control header is an instruction to a proxy or the so-called origin server that it should not satisfy your request using a cached response without validating it with the origin server. Maybe WordPress has a cache of its own that you're by-passing.
I'm looking for a cache implementation for the Indy IdHTTP component that would be used for checking of a specific resource is already cached before doing the actual GET and returning the cached resource instead (if appropriate)?
If the component would be a derived class from TIdHTTP and handle the cache-features automatically it would be the easiest way for me, so I could just replace my existing IdHTTP objects with the new component.
Has anyone ever seen such a component for D2010?
I dont think there is a component to do that, but you can implement your own based on response header Cache-Control and Expires-Date (i am not sure if it is the corrent name). You should save the page on a file with the Expires date on it, so you can check if you already have the page on cache before make the request to the server. I guess it is what some proxies do, like Squid (of course not as simple as that).
Check this out: Caching in HTTP