I am trying to develop a CORS proxy server in Delphi XE2 using Indy 10 so that I can get around the issue of embedding sites into an IFrame where sites have added X-Frame-Options to the response headers.
Can anyone give me some example code as to how I can use IdHTTPProxyServer to do this?
When using TIdHTTPProxyServer, you can modify HTTP headers for GET/POST/HEAD requests only, in the following events:
OnHTTPBeforeCommand event (client headers)
OnHTTPResponse event (server headers)
OnHTTPDocument event (client or server headers, depending on the TIdHTTPProxyServerContext.TransferSource property) when the proxy's DefaultTransferMode property is set to tmFullDocument.
The headers are stored in the Headers property of the TIdHTTPProxyServerContext object provided to each event.
For example, using the OnHTTPResponse event, you can easily remove an X-Frame-Options header, eg:
procedure TMyForm.IdHTTPProxyServer1HTTPResponse(AContext: TIdHTTPProxyServerContext);
var
I: Integer;
begin
I := AContext.Headers.IndexOfName('X-Frame-Options');
if I <> -1 then
AContext.Headers.Delete(I);
end;
Or:
procedure TMyForm.IdHTTPProxyServer1HTTPResponse(AContext: TIdHTTPProxyServerContext);
begin
AContext.Headers.Values['X-Frame-Options'] := '';
end;
For the ProcessHTTPRequest - Resp: TIdHTTPResponseInfo - you can add this line to set the http header response:
Resp.CustomHeaders.Values['X-Frame-Options'] := 'Deny';
Works with the normal indy http server, should work with the proxy as well.
In the same way you can set it to empty, or other values.
I came upon your post looking for a way to "add" it - i'm hoping this answer helps others who are looking for similar information, even if the actual answer isn't exactly what you're looking for in the OP.
Related
I have several URLs which work just fine in all browsers, but if I try to get the page content using Get() of the Indy Http client, it returns error code 500, internal server error. This is with the latest Indy SVN build (4981).
Here is my example code. All that is needed for this is Delphi with Indy components and a form with a button and a memo.
procedure TForm1.Button1Click(Sender: TObject);
var HTTPCLIENT1: TIdHTTP;
begin
try
try
HTTPCLIENT1 := TIdHTTP.Create(nil);
Memo1.Clear;
with HTTPCLIENT1 do
begin
HandleRedirects := True;
Request.UserAgent := 'Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)';
Memo1.Text := Get('http://www.laredoute.fr/vente-machine-a-coudre-bernette-20-kit-couture--garantie-2-ans.aspx?productid=401225048&documentid=999999&categoryid=22918417&customertarget=0&offertype=0&prodcolor=1#pos=33_n_n_n_n_n_n&numberpage=2');
Caption := ResponseText;
end;
except
On e: Exception do
begin
Memo1.Lines.Add('Exception: '+e.Message);
end;
end;
finally
HTTPCLIENT1.Free;
end;
end;
It's not a connection problem on my side, since 99% of URLs return 200 or 404, only few return 500, but every browser opens them fine in a second.
That kind of failure usually suggests the GET request is malformed in some way, causing the server code to fail on its end. But without seeing what the webbrowser requests actually look like for comparison to TIdHTTP's requests, there is no way to know for sure what the server is not liking.
Update: what I see happening is that when a webbrowser requests the URL, the server sends back a 200 response immediately, however when TIdHTTP requests the URL, the server sends a 301 redirect to a new URL, which then sends a 302 redirect to an error page when TIdHTTP requests that URL, which then sends the 500 response when TIdHTTP requests that URL.
The two differences between a webbrowser request and the initial TIdHTTP request that would have an effect on a webserver are:
the URL you are requesting with TIdHTTP includes an anchor tag at the end (everything after the # character - #pos=33_n_n_n_n_n_n&numberpage=2) which webbrowsers would normally strip out. Anchors are not actually part of URLs. They are meant for webbrowsers to use when locating spots within data that is retrieved from a URL.
the user agent. Some web servers are sensitive to different user agents, and can send different responses to different types of user agents.
When I remove the anchor from the URL, TIdHTTP.Get() no longer crashes:
Memo1.Text := Get('http://www.laredoute.fr/vente-machine-a-coudre-bernette-20-kit-couture--garantie-2-ans.aspx?productid=401225048&documentid=999999&categoryid=22918417&customertarget=0&offertype=0&prodcolor=1');
I am including these in uses clause
IdAuthentication
,
IdAuthenticationDigest
,
IdAuthenticationNTLM
,
IdAuthenticationSSPI
Currently I have code that does this:
W.Request.BasicAuthentication := True;
W.Request.Username := AOptionsPtr^.AuthUsername;
W.Request.Password := AOptionsPtr^.AuthPassword;
And if I have access to OpenSSL:
TmpOpenSSL := TIdSSLIOHandlerSocketOpenSSL.Create;
TmpOpenSSL.SSLOptions.Method := sslvSSLv23;
TmpOpenSSL.SSLOptions.Mode := sslmClient;
TmpOpenSSL.SSLOptions.VerifyMode := [];
TmpOpenSSL.SSLOptions.VerifyDepth := 0;
//--
W.IOHandler := TmpOpenSSL;
From skimming the documentation for WinINet (yes, I know it is not Indy) it seems persistent connections is also required for authentication. I suppose this also goes for Indy? URL:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa384220(v=vs.85).aspx
I need to get this to work with SharePoint. The problem, however, is hat I have no intranet SharePoint server to test against. Thus I can not step through the code and see what works and what does not. However, I have a potential customer that can test it for me.
What more do I need to do to get above code working with SharePoint using Windows Authentication (NTML? SPPI?)
Will Indy automatically test and use proper auhentication?
do I need to set W.Request.BasicAuthentication := False; for auto authentication/detection to work?
If multiple requests are necessary (with first response being 401) I assume I need to add support for this in my own code when making a GET request? (To set authentication mode and make a new request?)
You can request a persistent connection by setting the Request.Connection property to 'keep-alive'.
TIdHTTP will check the server's WWW-Authorization header and compare it to the TIdAthentication classes you have included in your uses clause. The TIdHTTP.OnSelectAuthorization event will tell you which class was picked, and allow you to override it if needed. The TIdHTTP.OnAuthorization event will be triggered if authentication fails and different credentials are needed.
The BasicAuthrnication property simply allows TIdHTTP to fall back to TIdBasicAuthentication if no other TIdAuthentication class is assigned.
No, you do not need to handle multi-request authentications manually, like NTLM. TIdHTTP and TIdAuthentication handle those details for you.
I'm trying to get data from a site using the Indy components. (This is in Delphi 7 but happy to use anything that works.)
If you go into a normal browser and put in the path:
http://inventory.data.xyz.com/provide_data.aspx?ID=41100&Mixed=no?fc=true&lang=en
it makes you tick a disclaimer before redirecting you to the actual site. This creates a cookie, which if I look at it in Firefox is like this:
http://inventory.data.xyz.com
Name: ASP.NET_SessionId
Content: vm4l0w033cdng5mevz5bkzzq
Path: /
Send For: Any type of connection
Expires: At end of session
I can't get through the disclaimer part using programming but I thought if I manually sign the disclaimer, I can then enter the details of the cookie into my code and connect directly to the data page. I have tried to do this with the code below but it only returns the html for the disclaimer page which tends to imply it's not using the cookie data I've given it. What am I doing wrong?
procedure TfmMain.GetWebpageData;
var
http: TIdHTTP;
cookie: TIdCookieManager;
sResponse: String;
begin
try
http := TIdHTTP.Create(nil);
http.AllowCookies := True;
http.HandleRedirects := True;
cookie := TIdCookieManager.Create(nil);
cookie.AddCookie('ASP.NET_SessionId=vm4l0w033cdng5mevz5bkzzq', 'inventory.data.xyz.com');
http.CookieManager := cookie;
sResponse := http.Get('http://inventory.data.xyz.com/provide_data.aspx?ID=41100&Mixed=no?fc=true&lang=en');
ShowMessage(sResponse); // returns text of disclaimer
except
end;
end;
Since you have not provided a real URL, I can only speculate, but chances are that either the cookie value you are providing to TIdCookieManager is wrong or outdated by the time TIdHTTP.Get() tries to use it, or more likely TIdCookieManager.AddCookie() is rejecting the cookie outright (if the TIdCookieManager.OnNewCookie event is not triggered, then the cookie was not accepted).
Trying to inspect the header information being posted from my app but fiddler doesn't seem to pick anything up.
I am also using a web service in my app and when I invoke some of the APIs I can see these requests in Fiddler, however, when doing custom requests using Indy 10 nothing seems to be picked up.
Does Indy use WinInet? If not, that's the problem. Fiddler2 inserts itself as a proxy in your internet settings, but programs like SoapUI that use their own communication stack don't use WinInet, and therefore don't (auto-magically) pass through Fiddler2. So you may need to mess with proxy settings.
I use a construction to handle the requests in Fiddler:
try
// lHTTP.IOHandler := lIOHandler; - even without this line works
lHTTP.ProxyParams.ProxyServer := '127.0.0.1';
lHTTP.ProxyParams.ProxyPort := 8888;
sResponse := lHTTP.Post('<URL>', slRequest);
Memo1.Lines.Text := sResponse;
finally
// lIOHandler.Free;
end;
This is an interesting problem that I’ve not been able to solve yet.
I am writing a client that communicates across the Internet to a server. I am using the TIdTcpClient Internet Direct (Indy) component in Indy 10 using RAD Studio 2007 native personality.
To get data from the server, I issue an HTTP request using SSL over port 443 where my request details are contained in the HTTP message body. So far, so good. The code works like charm, with one exception.
There is one request that I am submitting that should produce a response of about 336 KB from the server (the HTTP response header contains Content-Length: 344795). The problem is that I am getting only 320KB back. The response, which is in XML, is clearly truncated in the middle of an XML element.
For what it’s worth, the XML is simple text. There are no special characters that can account for the truncation. My TIdTcpClient component is simply reporting that, after receiving the partial response, that the server closed the connection gracefully (which is how every response is expected to be completed, even those that are not truncated, so this is not a problem).
I can make nearly identical calls to the same server where the response is also more than a few K bytes, and all of these work just fine. One request I make returns about 850 KB, another returns about 300 KB, and so on.
In short, I encounter this problem only with the one specific request. All other requests, of which there are many, receive a complete response.
I have talked to the creator of the service, and have supplied examples of my request. He reported that the request is correct. He also told me that when he issues my same request to his server that he gets a complete response.
I’m at a loss. Either the creator of the service is mistaken, and there is actually a problem with the response on that end, or there is something peculiar about my request.
Is there a solution here that I'm missing? Note that I’ve also used a number of other read mechanisms (ReadString, ReadStrings, ReadBytes, etc) and all produce the same result, a truncation of this one specific response at the 320KB mark.
The code is probably not relevant, but I’ll include it anyway. Sorry, but I cannot include the XML request, as it includes proprietary information. (ReadTimeout is set to 20 seconds, but the request returns in about 1 second, so it's not a timeout issue.)
function TClient.GetResponse(PayloadCDS: TClientDataSet): String;
var
s: String;
begin
try
try
s := GetBody(PayloadCDS);
IdTcpClient1.Host := Host;
IdTcpClient1.Port := StrToInt(Port);
IdTcpClient1.ReadTimeout := ReadTimeout;
IdTcpClient1.Connect;
IdTcpClient1.IOHandler.LargeStream := True;
//IdTcpClient1.IOHandler.RecvBufferSize := 2000000;
IdTcpClient1.IOHandler.Write(s);
Result := IdTcpClient1.IOHandler.AllData;
except
on E: EIdConnClosedGracefully do
begin
//eat the exception
end;
on e: Exception do
begin
raise;
end;
end;
finally
if IdTcpClient1.Connected then
IdTcpClient1.Disconnect;
end;
end;
Since you are sending an HTTP request, you should be using the TIdHTTP component instead of the TIdTCPClient component directly. There are a lot of details about the HTTP protocol that TIdHTTP manages for you that you would have to handle manually if you continue using TIdTCPClient directly.
If you are going to continue using TIdTCPClient directly, then you should at least stop using the TIdIOHandler.AllData() method. Extract the 'Content-Length' reply header (you can Capture() the headers into a TStringList or TIdHeaderList and then use its Values[] property) and then pass the actual reported byte count to either TIdIOHandler.ReadString() or TIdIOHandler.ReadStream(). That will help ensure that the reading I/O does not stop reading prematurely because of the server's disconnect at the end of the reply.
As I mentioned in my original question, this connection uses SSL. This requires that you use an IdSSLIOHandlerSocketOpenSSL component, which you assign to the IOHandler property of the IdTcpClient component. All of this was in my original design, and as I mentioned, many of my requests were being responded to correctly.
Over the weekend I discovered another request which was returning an incomplete response. I captured that original HTTP request, and using a tool called SOAP UI, I submitted that request to the server. Using SOAP UI, the server returned a complete answer. Clearly, something was wrong with my client, and not the server.
I finally found the solution by just fiddling around with various properties. The property that finally corrrected the problem was in the SSLOptions property of the IdSSLIOHandlerSocketOpenSSL class. The default value of SSLOptions.Method is sslvSSLv2. When I changed Method to sslvSSLv23, all responses returned complete.
Why I was able to retrieve some responses in full and not all before I made this change is still a mystery to me. Nonetheless, setting IdSSLIOHandlerSocketOpenSSL.SSLOptions.Method to sslvSSLv23 solved my problem.
Thank you Remy Lebeau, for your suggestions.