Only a GET request to a HTTP url works. If i try to request a HTTPS url then it doesn't return anything. I pretty much tried everything. Appreciate any help.
Here's my code:
SynHttp.Sock.CreateWithSSL(TSSLOpenSSL);
SynHttp.Sock.SSLDoConnect;
SynHttp.HTTPMethod('GET', 'https://www.google.com/');
Resp.LoadFromStream(SynHttp.Document);
HtmlResponse := Resp.Text;
SynHTTP is a THTTPSend object.
Make sure:
your exe application can access ssleay32.dll and libeay32.dll - the easiest way is to copy them into the directory of your exe.
you added ssl_openssl.pas and ssl_openssl_lib.pas to your project.
Then it should work instantly.
Use HTTPSend and headers, works with SSL in HTTPMethod.
If you use in URL 'https:' instead only 'http:', then your request is made by SSL/TLS connection:
aURL:='https://api.metadefender.com/v4/file/bzIwMDYxNi1TSW42NDBPVlprTWw3YjRBMQ';
with THTTPSend.create() do begin
Headers.Add('apikey: 6b337c92c792174a54acd715ab1aae64');
writeln(botostr(HTTPMethod('GET',aURl)));
writeln('synapse get: '+StreamtoString3(document))
writeln(itoa(ResultCode)+' '+ResultString);
Clear;
Free;
end;
Related
This question already has an answer here:
Delphi Indy https request with custom DNS resolving
(1 answer)
Closed 3 years ago.
I'm trying to change Host header before sending get request to a website. I do that using this code:
IdHTTP1.Request.HOST := 'example.com';
memo1.Text := IdHTTP1.Get('http://stackoverflow.com');
showmessage(IdHTTP1.Request.Host); // Expected to be example.com but it's stackoverflow.com
I've got a big problem here. Even though I change Host header before getting URL, Host header will change to stackoverflow.com again. What am I doing wrong? I want to change request header to example.com.
Thanks
Unfortunately, there is no option to specify a custom Host header that specifies a different hostname than the one specified in the URL. The URL has priority. Any hostname you specify in the Request.Host or even the Request.CustomHeaders is overwritten by the hostname in the URL.
If the hostname in the URL is not registered with DNS, you will not be able to reach it with any web browser, or most HTTP libraries including TIdHTTP. While the HTTP protocol itself defines how the Host header works, current web browser technology uses the hostname from the URL, and so does TIdHTTP. So it does not make sense to have a website that uses a hostname that is not registered with DNS in the first place, as most modern client systems would not be able to retrieve it. DNS is required to convert the URL's hostname into an IP address, and then the same hostname is put into the Host header.
As Remy Lebeau said, it seems there is no way to specify a custom Host header. So I decided to modify IdHTTP codes just a little bit.
First I copied IdHTTP.pas (C:\Program Files (x86)\Embarcadero\Studio\XX.0\source\Indy10\Protocols\IdHTTP.pas) to my project directory and added it to my project. Then in TIdCustomHTTP.PrepareRequestmethod (Line 1792) I changed the code like this:
if (TextIsSame(FURI.Protocol, 'http') and (FURI.Port = IntToStr(IdPORT_HTTP))) or {do not localize}
(TextIsSame(FURI.Protocol, 'https') and (FURI.Port = IntToStr(IdPORT_https))) then {do not localize}
begin
if FURI.Host = 'stackoverflow.com' then
ARequest.Host := 'example.com'
else
ARequest.Host := FURI.Host;
end else begin
if FURI.Host = 'stackoverflow.com' then
ARequest.Host := 'example.com' + ':' + FURI.Port {do not localize}
else
ARequest.Host := FURI.Host + ':' + FURI.Port; {do not localize}
end;
I know it's not the best way and modifying libraries is not a good idea but it worked for me.
So if I change the examples above, stackoverflow.com to myblog.wordpress.com and example.com to anotherblog.wordpress.com, By IdHTTP1.Get('http://myblog.wordpress.com') we will get anotherblog.wordpress.com content.
I'm developing a simple application to "talk" to the Amazon MWS API. Because a lot of existing code is at play here, I need to get this done in Delphi 2010 with Indy 10 (10.5.5) components, which I have used successfully to integrate with many other APIs in the past. However, the Amazon API seems to be incredibly sensitive to the smallest of details, to the point that all my calls are being denied with the already infamous "SignatureDoesNotMatch" error message.
Here's what I have accomplished so far:
1) My app will assemble a request, sign it with HMAC-SHA256 (using the OpenSSL libraries) and send it to the Amazon server endpoint.
2) The HMAC signature alone proved to be a challenge in itself, but it's now working correctly 100% of the time (as verified against requests generated by the Amazon Scrachpad).
However, as I pointed out earlier, my requests are always rejected by the MWS server with the SignatureDoesNotMatch error, even though they are verifiably correct. The only thing I can think of that could be causing problems is the way Indy may be handling the POST requests, specifically the text encoding process.
Has anyone been successful in connecting a Delphi/Indy client to MWS? If so, what kind of TIdHTTP settings were used? Here's what I have:
procedure TAmazon.TestGetOrder(OrderID:String);
const AwsAccessKey = 'MyAccessKey';
AwsSecretKey = 'MySecretKey';
MerchantID = 'MyMerchantID';
MarketplaceID = 'MyMarketplaceID';
ApiVersion = '2013-09-01';
CallUri = '/Orders/2013-09-01';
var HTTP:TIdHTTP;
SSL:TIdSSLIOHandlerSocketOpenSSL;
SS:TStringStream;
Params:TStringList;
S,Timestamp,QueryString,Key,Value:String;
i:Integer;
begin
HTTP:=TIdHTTP.Create(nil);
SSL:=TIdSSLIOHandlerSocketOpenSSL.Create(nil);
Params:=TStringList.Create;
try
Params.Delimiter:='&';
Params.StrictDelimiter:=True;
// HTTP Client Options
HTTP.HTTPOptions:=HTTP.HTTPOptions+[hoKeepOrigProtocol]-[hoForceEncodeParams];
HTTP.ConnectTimeout:=5000;
HTTP.ReadTimeout:=20000;
HTTP.ProtocolVersion:=pv1_1;
HTTP.IOHandler:=SSL;
HTTP.HandleRedirects:=True;
HTTP.Request.Accept:='text/plain, */*';
HTTP.Request.AcceptLanguage:='en-US';
HTTP.Request.ContentType:='application/x-www-form-urlencoded';
HTTP.Request.CharSet:='utf-8';
HTTP.Request.UserAgent:='MyApp/1.0 (Language=Delphi)';
HTTP.Request.CustomHeaders.AddValue('x-amazon-user-agent',HTTP.Request.UserAgent);
// generate the timestamp per Amazon specs
Timestamp:=TIso8601.UtcDateTimeToIso8601(TIso8601.ToUtc(Now));
// we can change the timestamp to match a value from the Scratchpad as a way to validate the signature:
//Timestamp:='2014-05-09T20:32:28Z';
// add required parameters from API function GetOrder
Params.Add('Action=GetOrder');
Params.Add('SellerId='+MerchantID);
Params.Add('AWSAccessKeyId='+AwsAccessKey);
Params.Add('Timestamp='+Timestamp);
Params.Add('Version='+ApiVersion);
Params.Add('SignatureVersion=2');
Params.Add('SignatureMethod=HmacSHA256');
Params.Add('AmazonOrderId.Id.1='+OrderID);
// generate the signature using the parameters above
Params.Add('Signature='+GetSignature(Params.Text,CallUri));
// after generating the signature, make sure all values are properly URL-Encoded
for i:=0 to Params.Count-1 do begin
Key:=Params.Names[i];
Value:=ParamEnc(Params.ValueFromIndex[i]);
QueryString:=QueryString+Key+'='+Value+'&';
end;
Delete(QueryString,Length(QueryString),1);
// there are two ways to make the call...
// #1: according to the documentation, all parameters are supposed to be in
// the URL, and the body stream is supposed to be empty
SS:=TStringStream.Create;
try
try
Log('POST '+CallUri+'?'+QueryString);
S:=HTTP.Post('https://mws.amazonservices.com'+CallUri+'?'+QueryString,SS);
except
on E1:EIdHTTPProtocolException do begin
Log('RawHeaders='+#$D#$A+HTTP.Request.RawHeaders.Text);
Log('Protocol Exception:'+#$D#$A+StringReplace(E1.ErrorMessage,#10,#$D#$A,[rfReplaceAll]));
end;
on E2:Exception do
Log('Unknown Exception: '+E2.Message);
end;
Log('ResponseText='+S);
finally
SS.Free;
end;
// #2: both the Scratchpad and the CSharp client sample provided by Amazon
// do things in a different way, though... they POST the parameters in the
// body of the call, not in the query string
SS:=TStringStream.Create(QueryString,TEncoding.UTF8);
try
try
SS.Seek(0,0);
Log('POST '+CallUri+' (parameters in body/stream)');
S:=HTTP.Post('https://mws.amazonservices.com'+CallUri,SS);
except
on E1:EIdHTTPProtocolException do begin
Log('RawHeaders='+#$D#$A+HTTP.Request.RawHeaders.Text);
Log('Protocol Exception:'+#$D#$A+StringReplace(E1.ErrorMessage,#10,#$D#$A,[rfReplaceAll]));
end;
on E2:Exception do
Log('Unknown Exception: '+E2.Message);
end;
Log('ResponseText='+S);
finally
SS.Free;
end;
finally
Params.Free;
SSL.Free;
HTTP.Free;
end;
end;
If I assemble a GetOrder call in Scratchpad, then paste the timestamp of that call into the code above, I get EXACTLY the same query string here, with the same signature and size, etc. But my Indy request must be encoding things differently, because the MWS server doesn't like the call.
I know MWS is at least "reading" the query string, because if I change the timestamp to an old date, it returns a "request expired" error instead.
Amazon's tech support is clueless, posting a message every day with basic stuff like "Make sure the secret key is correct" (as if getting a signature with HMAC-SHA256 and MD5 would work without a valid key!!!!).
One more thing: if I use Wireshark to "watch" the raw request from both the code above and the C-Sharp Amazon sample code, I can't tell a difference either. However, I'm not sure Wireshark makes a distinction between UTF-8 and ASCII or whatever encoding the text being shown has. I still think it has to do with bad UTC-8 encoding or something like that.
Ideas and suggestions on how to properly encode the API call to please the Amazon gods are welcome and appreciated.
Found the problem: Indy (and Synapse too) adds the port number to the "Host" header line, and I had not realized that extra bit until I watched the headers more closely with Fiddler (thanks, #Graymatter!!!!).
When I change the endpoint to be mws.amazonservices.com:443 (instead of just mws.amazonservices), then my signature is calculated the same way as the AWS server's, and everything works perfectly.
Is it possible to change the HTTP Response Code in my DataSnap (Delphi XE3) server when user authorization fails? Currently it's returning HTTP/1.1 500 Internal Server Error, which can happen in many other scenarios, and doesn't help describe the problem.
Along the same idea, is it possible to change the default Response Body? Currently, it's returning {"error":"USER is not authorized to perform the requested action."}, which is fine, but I'd like to return a custom JSON object if possible.
Thanks for your time!
I ended up executing code that looked like this in TDSAuthenticationManager.OnUserAuthorize
if valid = false then
begin
GetInvocationMetadata.ResponseCode := 403;
GetInvocationMetadata.ResponseMessage := JSONResponseObject.ToString;
GetInvocationMetadata.ResponseContent := JSONResponseOjbect.ToString;
end
It's possible to get the THTTPDSResponse inside a serverclass? Ex:
TServerClass1.EchoString(val: string);
begin
GetDSResponse.Header.Add('someheader');
end;
the question is extended from the question which I asked before.
Currently I can let my proxy server with TIdHTTPProxyServer forward the request to another proxy server which also using TIdHTTPProxyServer. But when I tried to let it forward the request to other proxy servers from the free proxy servers list on web. Most servers which can be used through IE proxy setting return 403 error message to my proxy server. In fact I have tried about 20 valid proxy servers, but only two can accept the requests from my proxy server. I can not figure out why it happens.
Below is the code I use in HTTPBeforeCommand of TIdHTTPProxyServer.
TIdIOHandlerStack* tempIO = new TIdIOHandlerStack(AContext->OutboundClient);
TIdConnectThroughHttpProxy* tempProxy = new TIdConnectThroughHttpProxy(AContext->OutboundClient);
tempProxy->Enabled = true;
tempProxy->Host = ProxyServerAddress;
tempProxy->Port = ProxyServerPort ;
tempIO->TransparentProxy = tempProxy;
AContext->OutboundClient->IOHandler = tempIO;
After monitoring the behaviors of TIdHTTPProxyServer using wireshark, I found TIdHTTPProxyServer always send a CONNECT request to other proxy servers at first berfore foward the real requests(the browser does not do that).
And then receive 403 response for most proxy servers. But still do not know how to make it works.
Updated on 2012/08/07
Hi, I am not really familiar with those HTTP stuffs, so I just record what I saw in wireshark here. It seems IE uses GET/POST commands for HTTP requests and CONNECT command for HTTPS requests. And most proxy servers block Connect commands when they are not HTTPS request (For example, CONNECT www.google.com.tw:80 HTTP/1.0). That is why TIdConnectThroughHttpProxy always does not work.
Below is my workaround, I made a little bit changes in IdHTTPProxyServer.pas. Hope it is useful for someone else who meets the same problems.
For CONNECT commands, still use TIdConnectThroughHttpProxy
Within TIdHTTPProxyServer.CommandCONNECT
if UseProxy = True then
begin
tempIO := TIdIOHandlerStack.Create(LContext.FOutboundClient);
tempProxy := TIdConnectThroughHttpProxy.Create(LContext.FOutboundClient);
tempProxy.Enabled := True;
tempProxy.Host := UseProxyAddr;
tempProxy.Port := UseProxyPort ;
tempIO.TransparentProxy := tempProxy;
LContext.FOutboundClient.IOHandler := tempIO;
end;
For GET/POST commands, I should directly send GET www.google.com.tw:80 HTTP/1.0 to other proxy servers instead of sending CONNECT www.google.com.tw:80 HTTP/1.0 at first.
Within TIdHTTPProxyServer.CommandPassThrough
if UseProxy = True then
begin
TIdTCPClient(LContext.FOutboundClient).Host := UseProxyAddr;
TIdTCPClient(LContext.FOutboundClient).Port := UseProxyPort;
end else
begin
TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host;
TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
end;
Also within TIdHTTPProxyServer.CommandPassThrough, I should let header Proxy-Connection = close
LContext.Connection.IOHandler.Capture(LContext.Headers, '', False);
LContext.Headers.Values['Proxy-Connection'] := 'close';
Finally within TIdHTTPProxyServer.TransferData
if AContext.TransferSource = tsClient then
begin
if UseProxy = True then
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Target + ' HTTP/1.0');
end else
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Document + ' HTTP/1.0');
end;
end;
It is my first time to implement delphi, and hope there are better solutions.
I believe you should not use pin-holing - http://www.indyproject.org/docsite/html/TIdConnectThroughHttpProxy.html
CONNECT command is not how WWW works. No browser use it. It is how non-WWW programs try to break-through the firewalls and open direct access to all areas of internet beyond WWW.
Don't use "transparent proxy" classes.
Use regular HTTP proxy, like in How to download a file over HTTPS using Indy 10 and OpenSSL?
BTW, there is no such event handler as u name "HTTPBeforeCommand"
http://www.indyproject.org/docsite/html/!!MEMBEROVERVIEW_TIdHTTPProxyServer.html
What is the best method to send a large (<50MB) file from a PHP server (written in RADPHP) to a Delphi Datasnap server (Delphi XE). Because of the connectivity issues I would prefer to use HTTP(S) but this is new territory for me.
The PHP server accepts the file upload from the user's browser OK and can encode it (base_64).
But
a) that puts it into a string which can't be the best way to handle it
b) the DataSnap server crashes with 'Max Line Length Exceeded' on receiving the string
The Datasnap server has 4 components - TDSServer, TDSServerClass, TDSHTTPService, and TDSAuthenticationManager.
The RADPHP server uses a DSRestConnection component.
I'm not very familiar with Datasnap, but couldn't you do something like:
Client uploads file to RadPHP server
now you want to send that to a Datasnap server, what you can do, is send a command with a link to download it something like:
RadPHP: hey, new file for you, here's the link: "http://www.mydomain.com/files/filename.extension"
Datasnap: sends a response, i.e. "OK", and starts downloading, on the server side you can use TIdHTTP for example.
and the implementation could be something like:
procedure DatasnapServerClass.NewFile(const ALink: string);
var
LIDHTTP: TIdHTTP;
begin
// create instance of TIdHTTP, and call the link to
// download the file to your desired local folder
// using ALink as the URL
end;