Using the following Delphi XE2 (update 4) code:
var
ConInfo: TAmazonConnectionInfo;
RespInfo: TCloudResponseInfo;
Service: TAmazonStorageService;
Content: TBytes;
Headers: TStringList;
begin
ConInfo:=TAmazonConnectionInfo.Create(self);
ConInfo.AccountName:='YOUR ACCOUNT NAME';
ConInfo.AccountKey:='YOUR ACCOUNT KEY';
ConInfo.Protocol:='http';
Service:=TAmazonStorageService.Create(ConInfo);
RespInfo:=TCloudResponseInfo.Create;
SetLength(Content, 128);
FillMemory(#Content[0], 128, Byte('x'));
Headers:=TStringList.Create;
Headers.Values['Content-type']:='text/plain';
if not Service.UploadObject('YOUR BUCKET', 'test.txt', Content, TRUE, nil, Headers, amzbaPrivate, RespInfo) then
ShowMessage('Failed:' + RespInfo.StatusMessage);
I always get an error on the call to UploadObject:
Failed:HTTP/1.1 403 Forbidden - The request signature we calculated
does not match the signature you provided. Check your key and signing
method. (SignatureDoesNotMatch)
This only happens when the Content-type is set to 'text/plain', 'text/html', or text anything. Using exactly the same code, if you just change the content-type to any other content-type, e.g. 'video/3gpp', then it works as expected and without error. The actual content of the object being uploaded isn't relevant and has no bearing on getting the error or not.
I've traced through the Indy code in Delphi, but I'm stumped as to why the text content type always gives this error.
Any ideas?
If you append "; charset=ISO-8859-1" to the Content-Type string, then it works:
Headers.Values['Content-type']:='text/plain; charset=ISO-8859-1';
Stepping through the code I see the Content-Type is being changed in TIdEntityHeaderInfo.SetHeaders (IdHTTPHeaderInfo.pas) which is called from TIdHTTPProtocol.BuildAndSendRequest (IdHTTP.pas).
Ultimately, it looks like the problem is that TIdEntityHeaderInfo.SetContentType (IdHTTPHeaderInfo.pas) is appending a character set to the content type if it is 'text' and it doesn't already have one. It shouldn't be changing the content type in these situations because the content type is part of the string to be signed, so changing it after the signing makes the signature invalid.
I had the very same problem. I also used application/octet-stream as content type, but still had some trouble. Later on, I discovered that bucket names have to be in lowercase (In US Standard Region, Amazon allows to define buckets with Uppercase or mixed case names; however, those buckets are not accessible through the HTTP API (including TAmazonStorageService). Instead of a not found message, I still got the 403 error (unauthenticated user).
However, one I changed the name to all lowercase, it worked fine.
Hope it helps
Related
I'm calling an API URL with some parameters according to the documentation provided.
The way the API is set up, the response should be an auto-download of a .pdf file.
The parameters are:
number - which stands for a 13 digit number that represents the unique file indicator in their system (example: 5277110610029)
username and user_pass respectively - which stand for the login credentials to access the system
client_id - which stands for a unique client ID associated with my account in their system
language - where I indicate the language the file contents should be in (English or other of the available languages)
Documentation indicates it should be made as a Post request.
I have the following code:
var
FileName : string;
URL: String;
Params: TStringList;
memStream: TMemoryStream;
...
begin
FileName:= MainModule.PDFfileName + '.pdf';
begin
URL := 'https://someURL/view_integrated_pdf.php?';
Params := TStringList.Create;
memStream := TMemoryStream.Create;
try
Params.Add('number='+MainModule.number+'');
Params.Add('username='+MainModule.User+'');
Params.Add('client_id='+MainModule.clientID+'');
Params.Add('user_pass='+MainModule.Pass+'');
Params.Add('language=en');
MainModule.IdHTTP.Post(URL, Params, memStream);
finally
memStream.SaveToFile(ServerModule.FilesFolderPath+'\pdfs\'+FileName);
Params.Free;
memStream.Free;
end;
end;
pdfForm.ShowModal();
end;
If I try the resulting URL and parameters in the browser - it auto-downloads the pdf file the API gives me with the name numberParameter.pdf
If I do it in Delphi using the provided code, 8 out of 10 times it saves a pdf file with a 1 KB Size (normally it is between 32 and 100 for the successful file) and when I try to open it in the program using my pdfForm and subsequent viewer, the viewer throws an error "Invalid PDF structure"
What am I doing wrong? / How do you properly save a Post requests that returns a .pdf file to download from the API?
UPDATE
As per the comments, opening up the 1kb resulting PDF in Notepad++ displays the contents as simply Error username.
This is puzzling since I checked and the username being passed is accurate + if I paste the exact same URL with parameter values filed, in the browser, it works perfectly and gives me the correct PDF.
Is my code not the correct approach to Post and save the file being sent?
If I try the resulting URL and parameters in the browser - it auto-downloads the pdf file the API gives me with the name numberParameter.pdf
The only way to do that in a browser is to put the parameters in the URL's query component, eg:
https://someURL/view_integrated_pdf.php?number=...&username=...&client_id=...&user_pass=...&language=en
... and not in a POST body, as your code is doing. Also, a browser would send a GET request, not a POST request. Unless you are filling in an HTML webform (ie <form action="<URL>" method="POST" ...>) and submitting it. Is that what the API documentation says to do?
Since you did not provide any details from the documentation about what this server is actually expecting, we can't really tell you whether your code is correct or wrong. But it does seem that your manual tests are contradicting what you claim the documentation says the server is expecting.
If I do it in Delphi using the provided code, 8 out of 10 times it saves a pdf file with a 1 KB Size (normally it is between 32 and 100 for the successful file) and when I try to open it in the program using my pdfForm and subsequent viewer, the viewer throws an error "Invalid PDF structure"
From comments, you say your file is receiving a textual error message. TIdHTTP would save such text to your TMemoryStream ONLY IF either:
the HTTP server is NOT reporting an error at the HTTP level, ie it is sending the text message using an HTTP 2xx success response. I suspect this is what is happening in your case. By default, if the server uses a proper HTTP error code, TIdHTTP will raise an EIdHTTPProtocolException containing the text message and NOT save the text to your TMemoryStream at all.
the HTTP server IS reporting an error at the HTTP level, but you are using the hoNoProtocolErrorException and hoWantProtocolErrorContent flags together in the TIdHTTP.HTTPOptions property. In which case, TIdHTTP would not raise EIdHTTPProtocolException and would instead save whatever data the server sends as-is to your TMemoryStream.
Since there is clearly no HTTP exception being raised, you will have to validate the server's response before you can use the downloaded data in your pdfForm, ie by looking at the TIdHTTP.Response.ContentType and/or TIdHTTP.Response.ContentDisposition property to know whether the server actually sent a PDF file or not.
This is puzzling since I checked and the username being passed is accurate + if I paste the exact same URL with parameter values filed, in the browser, it works perfectly and gives me the correct PDF.
Well, for one thing, you have a typo in your code: the numberr field needs to be number instead.
Beyond that, putting the URL in a browser is NOT the same operation that your code is doing, so try changing your code to mimic what you are doing manually, eg:
uses
..., IdGlobalProtocols, IdHTTP, IdURI;
...
var
URL : string;
memStream: TMemoryStream;
begin
// All parameters into the URI for a HTTP GET request
URL := 'https://someURL/view_integrated_pdf.php'
+ '?number=' + TIdURI.ParamsEncode(MainModule.number)
+ '&username=' + TIdURI.ParamsEncode(MainModule.User)
+ '&client_id=' + TIdURI.ParamsEncode(MainModule.clientID)
+ '&user_pass=' + TIdURI.ParamsEncode(MainModule.Pass)
+ '&language=en';
memStream := TMemoryStream.Create;
try
MainModule.IdHTTP.Get(URL, memStream);
// Is it really PDF? Other formats such as plaintext is not wanted.
if not IsHeaderMediaType(MainModule.IdHTTP.ContentType, 'application/pdf') then Exit;
memStream.SaveToFile(ServerModule.FilesFolderPath + '\pdfs\' + MainModule.PDFfileName + '.pdf');
finally
memStream.Free;
end;
pdfForm.ShowModal;
end;
If that does not work, then please update your question to provide the actual documentation.
please note that i had asked this question a while back but i was unable to attend to it because of some personal issue, so it got deleted, meta user suggested i reask the question and so here it is
Recently i encountered a website which doesn't send any dynamic information when you try to login, not even Username and Password !, what i mean is that after i use Fiddler and Http Analyzer i can see that 4 event happens :
All of these happens in the web browser :
1- A simple Get for the login main page
2- A Post which sends some data (These data do not include my Username and Password and they seem static! and even though there is a __USERCONTROLPATH which can be extracted from the step 1 response, it doesn't change, meaning even if i try to login multiple times using different browser at different times everything including the __USERCONTROLPATH is the same)
3- A Post which is of JSON type and the request is empty and the response only contains a single line as you can see at the image below:
4- A Get happens which is the login main page but if you look at HTML you can see that the user is logged in!
I tried doing these steps one by one, but i am unable to successfully log in to the website and i am unable to understand how my Username and Password is sent!
Here you can see some of the code i have written:
//for the sake of cookies and ... i do a Get on the main page
idHttp.get('MainLogPage');
//i send the first post which contains some information which never changes, they seems static and do not include the username and password
idhttp.post('someURL', requestList);
jsonRawRequest := TJSONObject.Create;
jsonRequest := TStringStream.Create(jsonRawRequest.ToString, TEncoding.UTF8);
/here i try to do the post that you saw in the image, no value is sent so i send an empty json request.
idhttp.post('someOtherURL', jsonRequest)
In the end my question is how can i login to such a site ?, what am i missing ?, currently i get error on Step 3, when i post the JSON, here is a picture of the error:
Update 01 :
I meant to update the whole post and explain every step in more detail but as i was messing with fiddler and checking the posts again i noticed something, when the Browser sends the POST "https://www.somewebsite.com//Index.aspx/LoginUser" the type is application/json and in the "HTTP analyzer" i can see that it send and receive a JSON but in the Fiddler the Webforms is empty! so i check the TextView thats where i saw it !, i again went and check HTTP analyzer's Raw stream, and in there i also saw a user and password being sent, i don't know how the post contains this string that you can see below, but in fiddler you can see it in "TextView" and in http analyzer you can see it in raw stream, now that i know this, can you tell me how can i imitate such a post ?, by the way the user and password are encrypted which i presume is a matter of its own, and i probably have to contact the site's admin for the algorithm ?, anyway, first things first, how can i send such a POST ?
In Fiddler :
HTTP Analyzer :
Update 02:
I tried what Remy Lebeau said,
But the JSON in the site was not standard as you can see in:
The input is not a valid Base-64 string Error, My JSON uses double quotation marks instead of single quotations
After that i tried a hard coded solution and i read the JSON from a file as you can see below:
loader := TStringList.Create;
loader.LoadFromFile('jsonWithCustomFormat.txt');
jsonRequest := TStringStream.Create(loader[0], TEncoding.UTF8);
After that i set the refer to the login page:
idHttp.Request.Referer := 'myLoginPageURL';
idHttp.Request.ContentType := 'application/json'
And Finally i send the POST:
idhttp.Post(URL, jsonRequest, ms);
The result of what i did was an alert error as you can in the image below:
I am very confused as to what the problem is, i have to solve this so any hint or help is much appreciated.
I'm using MSXML2_TLB.pas generated from the Microsoft XML type library to call a pretty simple web-service, so I don't need the full XML wrapping and just do this:
var
r:XMLHTTP;
begin
r:=CoXMLHTTP.Create;
r.open('POST',WebServiceURL,false,'','');
r.setRequestHeader('Content-Type','application/xml');
r.send('<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" '+
'xmlns:ns="http://somebigcorporation.com/api/ws1/">'+
'<soapenv:Body>'+
//snip: irrelevant to the question
'</soapenv:Body></soapenv:Envelope>');
if r.status<>200 then raise Exception.Create(r.statusText);
Result:=(r.responseXML as DOMDocument).documentElement.firstChild.firstChild.selectSingleNode('importantData');
end;
The webservice responds nicely with status 200 and Content-Type: text/xml; charset=iso-8859-1.
In some cases, documentElement is nil (and the above code throws an access violation). Apparently responseXML exists, but only to offer a parseError object (it says so in the docs), and parseError.reason in these cases is An invalid character was found in text content..
Why is the parser not taking the 'charset' value from the response header? Is there a way to tell the XMLHTTP instance to use the charset value from the response header?
Update: I don't know if it's important, but the response also doesn't start with <?xml and even if it did I have no way to request/modify it so it would have an encoding="iso-8859-1" attribute.
I think you need to set the charset in the request so the result is correct.
r.setRequestHeader('Content-Type', 'application/xml; charset=ISO-8859-1');
I'm developing a simple application to "talk" to the Amazon MWS API. Because a lot of existing code is at play here, I need to get this done in Delphi 2010 with Indy 10 (10.5.5) components, which I have used successfully to integrate with many other APIs in the past. However, the Amazon API seems to be incredibly sensitive to the smallest of details, to the point that all my calls are being denied with the already infamous "SignatureDoesNotMatch" error message.
Here's what I have accomplished so far:
1) My app will assemble a request, sign it with HMAC-SHA256 (using the OpenSSL libraries) and send it to the Amazon server endpoint.
2) The HMAC signature alone proved to be a challenge in itself, but it's now working correctly 100% of the time (as verified against requests generated by the Amazon Scrachpad).
However, as I pointed out earlier, my requests are always rejected by the MWS server with the SignatureDoesNotMatch error, even though they are verifiably correct. The only thing I can think of that could be causing problems is the way Indy may be handling the POST requests, specifically the text encoding process.
Has anyone been successful in connecting a Delphi/Indy client to MWS? If so, what kind of TIdHTTP settings were used? Here's what I have:
procedure TAmazon.TestGetOrder(OrderID:String);
const AwsAccessKey = 'MyAccessKey';
AwsSecretKey = 'MySecretKey';
MerchantID = 'MyMerchantID';
MarketplaceID = 'MyMarketplaceID';
ApiVersion = '2013-09-01';
CallUri = '/Orders/2013-09-01';
var HTTP:TIdHTTP;
SSL:TIdSSLIOHandlerSocketOpenSSL;
SS:TStringStream;
Params:TStringList;
S,Timestamp,QueryString,Key,Value:String;
i:Integer;
begin
HTTP:=TIdHTTP.Create(nil);
SSL:=TIdSSLIOHandlerSocketOpenSSL.Create(nil);
Params:=TStringList.Create;
try
Params.Delimiter:='&';
Params.StrictDelimiter:=True;
// HTTP Client Options
HTTP.HTTPOptions:=HTTP.HTTPOptions+[hoKeepOrigProtocol]-[hoForceEncodeParams];
HTTP.ConnectTimeout:=5000;
HTTP.ReadTimeout:=20000;
HTTP.ProtocolVersion:=pv1_1;
HTTP.IOHandler:=SSL;
HTTP.HandleRedirects:=True;
HTTP.Request.Accept:='text/plain, */*';
HTTP.Request.AcceptLanguage:='en-US';
HTTP.Request.ContentType:='application/x-www-form-urlencoded';
HTTP.Request.CharSet:='utf-8';
HTTP.Request.UserAgent:='MyApp/1.0 (Language=Delphi)';
HTTP.Request.CustomHeaders.AddValue('x-amazon-user-agent',HTTP.Request.UserAgent);
// generate the timestamp per Amazon specs
Timestamp:=TIso8601.UtcDateTimeToIso8601(TIso8601.ToUtc(Now));
// we can change the timestamp to match a value from the Scratchpad as a way to validate the signature:
//Timestamp:='2014-05-09T20:32:28Z';
// add required parameters from API function GetOrder
Params.Add('Action=GetOrder');
Params.Add('SellerId='+MerchantID);
Params.Add('AWSAccessKeyId='+AwsAccessKey);
Params.Add('Timestamp='+Timestamp);
Params.Add('Version='+ApiVersion);
Params.Add('SignatureVersion=2');
Params.Add('SignatureMethod=HmacSHA256');
Params.Add('AmazonOrderId.Id.1='+OrderID);
// generate the signature using the parameters above
Params.Add('Signature='+GetSignature(Params.Text,CallUri));
// after generating the signature, make sure all values are properly URL-Encoded
for i:=0 to Params.Count-1 do begin
Key:=Params.Names[i];
Value:=ParamEnc(Params.ValueFromIndex[i]);
QueryString:=QueryString+Key+'='+Value+'&';
end;
Delete(QueryString,Length(QueryString),1);
// there are two ways to make the call...
// #1: according to the documentation, all parameters are supposed to be in
// the URL, and the body stream is supposed to be empty
SS:=TStringStream.Create;
try
try
Log('POST '+CallUri+'?'+QueryString);
S:=HTTP.Post('https://mws.amazonservices.com'+CallUri+'?'+QueryString,SS);
except
on E1:EIdHTTPProtocolException do begin
Log('RawHeaders='+#$D#$A+HTTP.Request.RawHeaders.Text);
Log('Protocol Exception:'+#$D#$A+StringReplace(E1.ErrorMessage,#10,#$D#$A,[rfReplaceAll]));
end;
on E2:Exception do
Log('Unknown Exception: '+E2.Message);
end;
Log('ResponseText='+S);
finally
SS.Free;
end;
// #2: both the Scratchpad and the CSharp client sample provided by Amazon
// do things in a different way, though... they POST the parameters in the
// body of the call, not in the query string
SS:=TStringStream.Create(QueryString,TEncoding.UTF8);
try
try
SS.Seek(0,0);
Log('POST '+CallUri+' (parameters in body/stream)');
S:=HTTP.Post('https://mws.amazonservices.com'+CallUri,SS);
except
on E1:EIdHTTPProtocolException do begin
Log('RawHeaders='+#$D#$A+HTTP.Request.RawHeaders.Text);
Log('Protocol Exception:'+#$D#$A+StringReplace(E1.ErrorMessage,#10,#$D#$A,[rfReplaceAll]));
end;
on E2:Exception do
Log('Unknown Exception: '+E2.Message);
end;
Log('ResponseText='+S);
finally
SS.Free;
end;
finally
Params.Free;
SSL.Free;
HTTP.Free;
end;
end;
If I assemble a GetOrder call in Scratchpad, then paste the timestamp of that call into the code above, I get EXACTLY the same query string here, with the same signature and size, etc. But my Indy request must be encoding things differently, because the MWS server doesn't like the call.
I know MWS is at least "reading" the query string, because if I change the timestamp to an old date, it returns a "request expired" error instead.
Amazon's tech support is clueless, posting a message every day with basic stuff like "Make sure the secret key is correct" (as if getting a signature with HMAC-SHA256 and MD5 would work without a valid key!!!!).
One more thing: if I use Wireshark to "watch" the raw request from both the code above and the C-Sharp Amazon sample code, I can't tell a difference either. However, I'm not sure Wireshark makes a distinction between UTF-8 and ASCII or whatever encoding the text being shown has. I still think it has to do with bad UTC-8 encoding or something like that.
Ideas and suggestions on how to properly encode the API call to please the Amazon gods are welcome and appreciated.
Found the problem: Indy (and Synapse too) adds the port number to the "Host" header line, and I had not realized that extra bit until I watched the headers more closely with Fiddler (thanks, #Graymatter!!!!).
When I change the endpoint to be mws.amazonservices.com:443 (instead of just mws.amazonservices), then my signature is calculated the same way as the AWS server's, and everything works perfectly.
How i can determine the size in bytes of a remote file hosted in the web, before download it, using Delphi?
Thanks In advance.
You could use Indy.
First include IdHTTP.
You can retrieve the size this way:
procedure TFormMain.Button1Click(Sender: TObject);
var
Http: TIdHTTP;
begin
Http := TIdHTTP.Create(nil);
try
Http.Head('http://live.sysinternals.com/ADExplorer.exe');
ShowMessage(IntToStr(Http.Response.ContentLength));
finally
Http.Free;
end;
end;
Short answer: use the HTTP HEAD command, available in the TIdHttp component of Indy Delphi.
Details:
HTTP protocol defines a HEAD method.
9.4 HEAD
The HEAD method is identical to GET
except that the server MUST NOT return
a message-body in the response. The
metainformation contained in the HTTP
headers in response to a HEAD request
SHOULD be identical to the information
sent in response to a GET request.
This method can be used for obtaining
metainformation about the entity
implied by the request without
transferring the entity-body itself.
This method is often used for testing
hypertext links for validity,
accessibility, and recent
modification.
The response to a HEAD request MAY be
cacheable in the sense that the
information contained in the response
MAY be used to update a previously
cached entity from that resource. If
the new field values indicate that the
cached entity differs from the current
entity (as would be indicated by a
change in Content-Length, Content-MD5,
ETag or Last-Modified), then the cache
MUST treat the cache entry as stale.
HEAD Asks for the response identical to the one that would correspond to a GET request, but without the response body, retrieving the complete response headers, without the entire content.
The HTTP response headers retrieved are documented in List of HTTP headers on Wikipedia.
http://en.wikipedia.org/wiki/List_of_HTTP_headers
HTTP Headers form the core of an HTTP request,
and are very important in an HTTP response.
They define various characteristics of the data
that is requested or the data that has been provided.
The headers are separated from the request or
response body by a blank line. HTTP headers
can be near-arbitrary strings, but only some
are commonly understood.
One of the headers that is always present for a valid URL to retrieve a content is
the Content-Length header.
14.13 Content-Length
The Content-Length entity-header field indicates the
size of the entity-body, in decimal number of OCTETs,
sent to the recipient or, in the case of the HEAD method,
the size of the entity-body that would have been sent
had the request been a GET.
Content-Length = "Content-Length" ":" 1*DIGIT
An example is
Content-Length: 3495
Applications SHOULD use this field to indicate the
transfer-length of the message-body, unless this is
prohibited by the rules in section 4.4.
Any Content-Length greater than or equal to zero is a
valid value. Section 4.4 describes how to determine
the length of a message-body if a Content-Length is not given.
Note that the meaning of this field is significantly
different from the corresponding definition in MIME,
where it is an optional field used within the
"message/external-body" content-type. In HTTP,
it SHOULD be sent whenever the message's length
can be determined prior to being transferred,
unless this is prohibited by the rules in section 4.4.
From Delphi, drop a TIdHttp component to your form. And paste the following code in one of your delphi event process methods.
var
url: string; // must contain a fully qualified url
contentLength: integer;
begin
....
contentLength:=0;
try
Idhttp1.Head(url);
contentLength:=idhttp1.response.ContentLength;
except end;
....
Be aware that not ALL servers will return a valid content size for a head request. If the content length = 0, then you will ONLY know if you issue a GET request. For example the HEAD request against the Google logo returns a 0 content-length, however a GET returns the proper length, but also retrieves the image. Some servers will return content-length as the length of the packet following the header.
You can use Synapse to get at this information also. Note that the data is transfered, but the buffer is thrown away. This is a much more reliable method, but at the cost of additional bandwidth.
var
HTTP : tHTTPSend;
begin
HTTP := THTTPSend.Create;
try
HTTP.HTTPMethod('GET',url);
DownloadSize := HTTP.DownloadSize;
finally
HTTP.Free;
end;
end;