This question already has an answer here:
Delphi Indy https request with custom DNS resolving
(1 answer)
Closed 3 years ago.
I'm trying to change Host header before sending get request to a website. I do that using this code:
IdHTTP1.Request.HOST := 'example.com';
memo1.Text := IdHTTP1.Get('http://stackoverflow.com');
showmessage(IdHTTP1.Request.Host); // Expected to be example.com but it's stackoverflow.com
I've got a big problem here. Even though I change Host header before getting URL, Host header will change to stackoverflow.com again. What am I doing wrong? I want to change request header to example.com.
Thanks
Unfortunately, there is no option to specify a custom Host header that specifies a different hostname than the one specified in the URL. The URL has priority. Any hostname you specify in the Request.Host or even the Request.CustomHeaders is overwritten by the hostname in the URL.
If the hostname in the URL is not registered with DNS, you will not be able to reach it with any web browser, or most HTTP libraries including TIdHTTP. While the HTTP protocol itself defines how the Host header works, current web browser technology uses the hostname from the URL, and so does TIdHTTP. So it does not make sense to have a website that uses a hostname that is not registered with DNS in the first place, as most modern client systems would not be able to retrieve it. DNS is required to convert the URL's hostname into an IP address, and then the same hostname is put into the Host header.
As Remy Lebeau said, it seems there is no way to specify a custom Host header. So I decided to modify IdHTTP codes just a little bit.
First I copied IdHTTP.pas (C:\Program Files (x86)\Embarcadero\Studio\XX.0\source\Indy10\Protocols\IdHTTP.pas) to my project directory and added it to my project. Then in TIdCustomHTTP.PrepareRequestmethod (Line 1792) I changed the code like this:
if (TextIsSame(FURI.Protocol, 'http') and (FURI.Port = IntToStr(IdPORT_HTTP))) or {do not localize}
(TextIsSame(FURI.Protocol, 'https') and (FURI.Port = IntToStr(IdPORT_https))) then {do not localize}
begin
if FURI.Host = 'stackoverflow.com' then
ARequest.Host := 'example.com'
else
ARequest.Host := FURI.Host;
end else begin
if FURI.Host = 'stackoverflow.com' then
ARequest.Host := 'example.com' + ':' + FURI.Port {do not localize}
else
ARequest.Host := FURI.Host + ':' + FURI.Port; {do not localize}
end;
I know it's not the best way and modifying libraries is not a good idea but it worked for me.
So if I change the examples above, stackoverflow.com to myblog.wordpress.com and example.com to anotherblog.wordpress.com, By IdHTTP1.Get('http://myblog.wordpress.com') we will get anotherblog.wordpress.com content.
Related
example url:
https://api.binance.com/api/v1/time
Using TIdDNSResolver and cloudflare dns I can get the host IP.
direct request in the form of
https://205.251.219.20/api/v1/time
doesn't work cause as I understand the server expects to see "api.binance.com" in the url. (it doesnt work even in browser)
Using synapce and the following patch to replace request's host with resolved IP I make it working
function THTTPSend.InternalDoConnect(needssl: Boolean): Boolean;
begin
Result := False;
FSock.CloseSocket;
FSock.Bind(FIPInterface, cAnyPort);
if FSock.LastError <> 0 then
Exit;
If ResolvedIP.IsEmpty
then FSock.Connect(FTargetHost, FTargetPort)
else FSock.Connect(ResolvedIP, FTargetPort); // !!
Is there any way to do the same using Indy?
By default, TIdHTTP uses the Host:Port that is specified/inferred by the URL that is being requested. To change that behavior, you would have to alter TIdHTTP's source code, such as in the TIdCustomHTTP.SetHostAndPort() or TIdCustomHTTP.CheckAndConnect() method (which are both not virtual), and then recompile Indy.
Alternatively, you could use TIdHTTP's OnSocketAllocated or On(Before|After)Bind event (which are not promoted to published in TIdHTTP, so you would have to access them manually at runtime) to change the TIdHTTP.IOHandler.Host property to whatever IP address you want to connect to. This will not have any affect on the HTTP headers that TIdHTTP sends, such as Host, which will still be taken from the URL.
Alternatively, if you want all of Indy's Host-to-IP DNS queries to go through Cloudflare, you could derive a custom class from TIdStack (or whatever platform-specific descendant you are interested in, such as TIdStackWindows), override its virtual HostByName() method to perform whatever DNS query you want, and then pass your class type to the SetStackClass() function in the IdStack unit at program startup before any Indy classes are instantiated.
TWebRequest has two metnod for retrive the client IP Address:
RemoteIP
RemoteAddr
From docs of RemoteIP:
Specifies the IP of the remote target machine associated with the HTTP
request message. Read the RemoteIP property to obtain the IP address
of the remote target machine associated with the HTTP request message.
From docs of RemoteAddr:
Indicates the remote IP address of the client associated with the HTTP
request message. Read RemoteAddr to obtain the IP address of the
source of the Web client request.
They seem very close but RemoteIP return an empty string.
I want retrieve the client ip address, what is the right method?
If we look into source code we will find the following in the declaration of TWebRequest
property RemoteIP: string read GetRemoteIP;
property RemoteAddr: string index 21 read GetStringVariable;
Implementation:
function TWebRequest.GetRemoteIP: string;
begin
Result := EmptyStr;
end;
and GetStringVariable is a virtual method.
Let's look into one of child classes - TISAPIRequest, we will find the following:
LResult := GetFieldByNameA(ServerVariables[Index]);
ServerVariables[21] is 'REMOTE_ADDR' header, which shows IP of client or proxy server.
But there is no implementation for GetRemoteIP.
It looks like RemoteIP isn't used in requests, because TCGIRequest, TApacheRequest and TWinCGIRequest also don't implement GetRemoteIP.
I'm new to this delphi datasnap stuff and there are barely any useful documentations about it.
I've setup a simple rest server by following this guide. Now I want to define my own authentication protocol.
As far as I know. All requests came from remote clients will be processed through WebModule methods before invoking server class methods. So far, I was able to capture the client IP adress and the requesting URL path like this:
procedure TWebModule1.WebModuleBeforeDispatch(Sender: TObject;
Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
begin
ShowMessage(request.RemoteAddr); // adress where request came from
ShowMessage(request.PathInfo); // url path
Response.SetCustomHeader('access-control-allow-origin','*');
if FServerFunctionInvokerAction <> nil then
FServerFunctionInvokerAction.Enabled := AllowServerFunctionInvoker;
end;
How am I able to:
Reject requests depend on specific IP addresses and return 401 status code to clients before they reach the class methods?
Redirect client to another methods? For example: Redirect this resquest:
request = new XMLHttpRequest();
request.open('GET', 'http://localhost:8080/datasnap/rest/TServerMethods1/ReverseString/blah', true);
To another method instead of ReverseString.
Complex redirection scenarios can be handled with an Apache HTTP server as reverse proxy with the mod_rewrite module.
For example, this answer shows that client IP addresses can be used to trigger a redirection.
https://stackoverflow.com/a/1073877/80901
RewriteCond %{REMOTE_ADDR} !^10\.0\.1\.1$
RewriteRule ^ /maintenance.html
this would send all requests with some exception to a maintenance page.
Using Apache (or a different HTTP proxy server) also allows to keep the DataSnap server running continuously, while Apache configuration changes are applied. This will remove your server down time virtually to zero.
the question is extended from the question which I asked before.
Currently I can let my proxy server with TIdHTTPProxyServer forward the request to another proxy server which also using TIdHTTPProxyServer. But when I tried to let it forward the request to other proxy servers from the free proxy servers list on web. Most servers which can be used through IE proxy setting return 403 error message to my proxy server. In fact I have tried about 20 valid proxy servers, but only two can accept the requests from my proxy server. I can not figure out why it happens.
Below is the code I use in HTTPBeforeCommand of TIdHTTPProxyServer.
TIdIOHandlerStack* tempIO = new TIdIOHandlerStack(AContext->OutboundClient);
TIdConnectThroughHttpProxy* tempProxy = new TIdConnectThroughHttpProxy(AContext->OutboundClient);
tempProxy->Enabled = true;
tempProxy->Host = ProxyServerAddress;
tempProxy->Port = ProxyServerPort ;
tempIO->TransparentProxy = tempProxy;
AContext->OutboundClient->IOHandler = tempIO;
After monitoring the behaviors of TIdHTTPProxyServer using wireshark, I found TIdHTTPProxyServer always send a CONNECT request to other proxy servers at first berfore foward the real requests(the browser does not do that).
And then receive 403 response for most proxy servers. But still do not know how to make it works.
Updated on 2012/08/07
Hi, I am not really familiar with those HTTP stuffs, so I just record what I saw in wireshark here. It seems IE uses GET/POST commands for HTTP requests and CONNECT command for HTTPS requests. And most proxy servers block Connect commands when they are not HTTPS request (For example, CONNECT www.google.com.tw:80 HTTP/1.0). That is why TIdConnectThroughHttpProxy always does not work.
Below is my workaround, I made a little bit changes in IdHTTPProxyServer.pas. Hope it is useful for someone else who meets the same problems.
For CONNECT commands, still use TIdConnectThroughHttpProxy
Within TIdHTTPProxyServer.CommandCONNECT
if UseProxy = True then
begin
tempIO := TIdIOHandlerStack.Create(LContext.FOutboundClient);
tempProxy := TIdConnectThroughHttpProxy.Create(LContext.FOutboundClient);
tempProxy.Enabled := True;
tempProxy.Host := UseProxyAddr;
tempProxy.Port := UseProxyPort ;
tempIO.TransparentProxy := tempProxy;
LContext.FOutboundClient.IOHandler := tempIO;
end;
For GET/POST commands, I should directly send GET www.google.com.tw:80 HTTP/1.0 to other proxy servers instead of sending CONNECT www.google.com.tw:80 HTTP/1.0 at first.
Within TIdHTTPProxyServer.CommandPassThrough
if UseProxy = True then
begin
TIdTCPClient(LContext.FOutboundClient).Host := UseProxyAddr;
TIdTCPClient(LContext.FOutboundClient).Port := UseProxyPort;
end else
begin
TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host;
TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
end;
Also within TIdHTTPProxyServer.CommandPassThrough, I should let header Proxy-Connection = close
LContext.Connection.IOHandler.Capture(LContext.Headers, '', False);
LContext.Headers.Values['Proxy-Connection'] := 'close';
Finally within TIdHTTPProxyServer.TransferData
if AContext.TransferSource = tsClient then
begin
if UseProxy = True then
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Target + ' HTTP/1.0');
end else
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Document + ' HTTP/1.0');
end;
end;
It is my first time to implement delphi, and hope there are better solutions.
I believe you should not use pin-holing - http://www.indyproject.org/docsite/html/TIdConnectThroughHttpProxy.html
CONNECT command is not how WWW works. No browser use it. It is how non-WWW programs try to break-through the firewalls and open direct access to all areas of internet beyond WWW.
Don't use "transparent proxy" classes.
Use regular HTTP proxy, like in How to download a file over HTTPS using Indy 10 and OpenSSL?
BTW, there is no such event handler as u name "HTTPBeforeCommand"
http://www.indyproject.org/docsite/html/!!MEMBEROVERVIEW_TIdHTTPProxyServer.html
Only a GET request to a HTTP url works. If i try to request a HTTPS url then it doesn't return anything. I pretty much tried everything. Appreciate any help.
Here's my code:
SynHttp.Sock.CreateWithSSL(TSSLOpenSSL);
SynHttp.Sock.SSLDoConnect;
SynHttp.HTTPMethod('GET', 'https://www.google.com/');
Resp.LoadFromStream(SynHttp.Document);
HtmlResponse := Resp.Text;
SynHTTP is a THTTPSend object.
Make sure:
your exe application can access ssleay32.dll and libeay32.dll - the easiest way is to copy them into the directory of your exe.
you added ssl_openssl.pas and ssl_openssl_lib.pas to your project.
Then it should work instantly.
Use HTTPSend and headers, works with SSL in HTTPMethod.
If you use in URL 'https:' instead only 'http:', then your request is made by SSL/TLS connection:
aURL:='https://api.metadefender.com/v4/file/bzIwMDYxNi1TSW42NDBPVlprTWw3YjRBMQ';
with THTTPSend.create() do begin
Headers.Add('apikey: 6b337c92c792174a54acd715ab1aae64');
writeln(botostr(HTTPMethod('GET',aURl)));
writeln('synapse get: '+StreamtoString3(document))
writeln(itoa(ResultCode)+' '+ResultString);
Clear;
Free;
end;