I have an AWS S3 account and got SecretAccessKey, SessionToken, Expiration, AccessKeyId items. I would like to upload some files to the cloud, in the simplest way.
Have read a some docs regarding authorization headers (http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html) but still do not understand how to build them))
also, saw another example with Indy, so, please help to build the authorization header with these items that I have: SecretAccessKey, SessionToken, Expiration, AccessKeyId. It's ok to be with a "Transferring Payload in a Single Chunk" mode + "Signed payload option".
FS := TFileStream.Create('c:\myfile.txt', fmOpenRead or fmShareDenyWrite);
try
IdHTTP1.Request.CustomHeaders.Values['Authorization'] := ...; // please help
IdHTTP1.Request.BasicAuthentication := False;
IdHTTP1.Request.Date := ...; //what should I enter here?
IdHTTP1.Request.Expect := '100-continue';
IdHTTP1.Request.ProtocolVersion := pv1_1;
...
IdHTTP1.Put('http://'+BucketName+'.s3.amazonaws.com/myfile.txt', FS);
finally
FS.Free;
end;
Thank you!
Here's my routine to upload files to Amazon using the Cloud Components:
function UploadFile(File: TBytes; FileName: string; Bucket: string): boolean;
var Service: TAmazonStorageService;
ConAmazon: TAmazonConnectionInfo;
begin
try
ConAmazon := TAmazonConnectionInfo.Create(nil);
ConAmazon.AccountKey := 'Dih71bG09****************';
ConAmazon.AccountName := 'AKIA***********';
ConAmazon.QueueEndpoint := 'queue.amazonaws.com';
ConAmazon.StorageEndpoint := 's3-eu-west-1.amazonaws.com';
ConAmazon.TableEndpoint := 'sdb.amazonaws.com';
ConAmazon.UseDefaultEndpoints := False;
Service := TAmazonStorageService.Create(ConAmazon);
Result := Service.UploadObject(Bucket, FileName, File, TRUE, nil, nil, amzbaPrivate, nil);
finally
ConAmazon.Free;
Service.Free;
end;
end;
Ok, so, finally we've solved the issue:
1.Data.Cloud from Delphi10.1 Berlin should be used. It supports Amazon AWS4 security standard.
2.TAmazonStorageService.InitHeaders should be patched by adding the following code to enable temporary session tokens usage in the header:
...
Result.Values['x-amz-security-token'] := //your session_token string;
...
tested it from many sides, works fine now:)
Related
I have to write a program (Delphi XE5, Indy 10: TIdHTTP & TIdSSLIOHandlerSocketOpenSSL) which can connect to a web service with client authentication. With several days of working, finally it has become success. I can connect using the authentication, setting the TIdSSLIOHandlerSocketOpenSSL’s SSLOptions.CertFile and SSLOptions.KeyFile properties. It’s fine. (I've got a pfx file from my partner, I exported it to a certificate and a private key file with OpenSSL so I use these 2 files in the program.)
I have one TButton, TMemo and TIdHTTP component on the form.
Source code (Button's click event - the IdHTTP1.Request.ContentType := '.......' line is necessary just for the communication because of the server settings):
procedure TForm1.Button1Click(Sender: TObject);
var
URL: string;
XML: TStrings;
S: string;
Req: TStream;
SL: TStringList;
SSL1 : TIdSSLIOHandlerSocketOpenSSL;
begin
XML := TStringList.Create;
XML.Add('<soap:Envelope xmlns:ns="http://docs.oasis-open.org/ws-sx/ws-trust/200512" ' +
'xmlns:soap="http://www.w3.org/2003/05/soap-envelope">');
…
XML.Add(' <soap:Body>');
…
XML.Add(' </soap:Body>');
XML.Add('</soap:Envelope>');
URL := 'https://…………………….';
end
Req := TStringStream.Create(XML.Text, TEncoding.UTF8);
try
SSL1 := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
SSL1.SSLOptions.CertFile := 'd:\certificate.pem';
SSL1.SSLOptions.KeyFile := 'd:\private.pem';
SSL1.SSLOptions.Mode := sslmClient;
try
SSL1.SSLOptions.Method := sslvSSLv23;
IdHTTP1.IOHandler := SSL1;
IdHTTP1.Request.ContentType := 'application/soap+xml;charset=UTF-8;action="http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue"';
S := IdHTTP1.Post(URL, Req);
finally
SSl1.Free;
end;
finally
Req.Free;
end;
ResultMemo.Lines.Add(Format('Response Code: %d', [IdHTTP1.ResponseCode]));
ResultMemo.Lines.Add(Format('Response Text: %s', [IdHTTP1.ResponseText]));
SL := TStringList.Create;
try
SL.Text := S;
ResultMemo.Lines.AddStrings(SL);
finally
SL.Free;
end;
end;
The problem is: my partner said this case is not the best if the file I use is not password-protected. They told me how to create a password-protected (and encrypted) file for the KeyFile with OpenSSL. When I set this password-protected file to the SSLOptions.KeyFile I get the following error message: „Could not load key, check password. error:0906A068:PEM routines:PEM_do_header:bad password read.”
I tried to set the password in the idHTTP1.Request.Password property, but the result is the same.
Question: how and where do I have to set the password for the KeyFile if I have to use a password-protected keyfile? Because I have to publish the certification files, too, the best solution would be to set the password in the program and use the password-protected KeyFile, instead of using not the password-protected KeyFile.
Thanks a lot.
Regards,
Attila
Use the IdSSLIOHandlerSocketOpenSSL.OnGetPassword event and set it here.
procedure TForm1.IdSSLIOHandlerSocketOpenSSL1GetPassword(var Password: string);
begin
Password := 'thepassword';
end;
I subscribe to a secure https web page containing a button that downloads some data as csv. I am trying to automate the download without the 'save as' dialog appearing but always seem to get an empty file downloaded. I suspect it has something to do with file type I'm using with IdHttp as most of my code works correctly.
Please can anyone help with my use of IdHttp or see where else I am going wrong?
The download button on the site calls some javascript to perform the download as follows
<a class="dlCSV" href="javascript:void(0);" onclick="dl_module.DownloadCsv();return false;">Download in CSV format…</a>
In Delphi I use a TWeb browser to log on securely and navigate to the page.
Clicking the download button in the TwebBrowser by hand shows the 'save as' dialog and then correctly downloads the csv data, defaulting to the filename 'data.csv'.
Automating clicking the button using execScript (below) also works, again showing the 'save as' dialog and correctly downloading the data with the same default filename.
procedure TForm1.BtnClickDownloadbuttonClick(Sender: TObject);
var TheDocument : IHTMLDocument2; // current HTML document
HTMLWindow: IHTMLWindow2; // parent window of current HTML document
begin
TheDocument := WebBrowser1.Document as IHTMLDocument2; // Get reference to current document
if not Assigned(TheDocument) then
Exit;
HTMLWindow := TheDocument.parentWindow; // Get parent window of current document
if Assigned(HTMLWindow) then
try
HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript'); // execute JS function to do download
except
on E : Exception do
begin
showmessage ('Exception class name = '+E.ClassName+ slinebreak
+ 'Exception message = '+E.Message);
end //on E
end;
end;
Then I added TLama's code from here How do I keep an embedded browser from prompting where to save a downloaded file? to use IDownloadManager to intercept the download and prevent the 'save as' dialog. This is where it seems to go wrong as I then get an empty file downloaded, and not with the name data.csv.
My code for function TWebBrowser.Download, TWebBrowser.InvokeEvent, function TWebBrowser.QueryService and TForm1.FormCreate are identical to that provided by TLama in the link above.
My procedure TForm1.Button1Click is the same except that I changed the download function being called to the one on my page by changing the line
HTMLWindow.execScript('SRT_stocFund.Export()', 'JavaScript');
to
HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript');
and my procedure TForm1.BeforeFileDownload is identical except that because I'm on a secure site I added the variable
var
LHandler: TIdSSLIOHandlerSocketOpenSSL; //<< on a https site
and after creating the Filestream I added the lines
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IdHTTP.IOHandler := LHandler;
The issue seems to be in procedure TForm1.BeforeFileDownload where I note that the value of FileSource is
https://www.the_web_site_name/Ashx/GenericCSV.ashx.
There is a short delay while IdHTTP.Get(FileSource, FileStream); executes and then a file is created on my hard disc but called 'GenericCSV.ashx' (not data.csv) and the file is zero bytes long and completely empty.
Any ideas why its not downloading the file called data.csv (Do I somehow have to execute GenericCSV.ashx as well? if so how?)
For info here is my version of procedure TForm1.BeforeFileDownload
procedure TForm1.BeforeFileDownload(Sender: TObject; const FileSource: WideString; var Allowed: Boolean);
var
IdHTTP: TIdHTTP;
FileTarget: string;
FileStream: TMemoryStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL; // added as its a https site
begin
FileSourceEdit.Text := FileSource;
Allowed := ShowDialogCheckBox.Checked;
if not Allowed then
try
IdHTTP := TIdHTTP.Create(nil);
try
FileStream := TMemoryStream.Create;
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
IdHTTP.IOHandler := LHandler; //<<< added as its a https site
try
IdHTTP.HandleRedirects := True;
IdHTTP.Get(FileSource, FileStream);
FileTarget := IdHTTP.URL.Document;
if FileTarget = '' then
FileTarget := 'File';
FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
FileStream.SaveToFile(FileTarget);
finally
FileStream.Free;
end;
finally
IdHTTP.Free;
end;
ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
FileTarget);
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
After you login, you can use this code to retrieve cookies from TWebBrowser
procedure GetHttpOnlyCookie(const AUrl: string; var ACookies: string);
const
INTERNET_COOKIE_HTTPONLY = 8192;
var
i: Integer;
hModule: THandle;
InternetGetCookieEx: function(lpszUrl, lpszCookieName, lpszCookieData
: PAnsiChar; var lpdwSize: DWORD; dwFlags: DWORD; lpReserved: pointer)
: BOOL; stdCall;
CookieSize: DWORD;
CookieData: PAnsiChar;
begin
LoadLibrary('wininet.dll');
hModule := GetModuleHandle('wininet.dll');
if (hModule <> 0) then
begin
#InternetGetCookieEx := GetProcAddress(hModule, 'InternetGetCookieExA');
if (#InternetGetCookieEx <> nil) then
begin
CookieSize := 1024;
Cookiedata := AllocMem(CookieSize);
try
if InternetGetCookieEx(PAnsiChar(AUrl), nil, Cookiedata, CookieSize, INTERNET_COOKIE_HTTPONLY, nil) then
begin
ACookies:=CookieData;
end;
finally
FreeMem(Cookiedata);
end;
end;
end;
end;
Then you just parse your cookies and add them (you have to create CookieManager in IdHTTP first)
IdHTTP1.CookieManager.AddServerCookie();
Then you start your download and it should work if you passed all parameters correctly (unfortunately, it is not possible to find out what your site requires).
Thank you smooty86 but I think its time I gave up trying to doing it this way and simply parse the page I can see.
I don't mind trying to understand code and adapting it to my needs but its so much harder trying to follow hints and suggestions when I'm working in the dark and especially don't know what parameters are needed everywhere. (I'm not daft, I've been programming for nearly 30 years and have spent over 4 years developing this particular data processing application but rarely touch web stuff)
However, the progress so far is...
Running your GetHttpOnlyCookie code after a successful login using automated filling in of the fields and clicking the login button returned an empty string so I used this code instead that at least seemed to return something that looked a little similar to your cookie string, ie seveveral strings separated by semicolons, most being name=value. (IdCookieManager1 is connected to IdHttp)
CookieList := Tstringlist.Create ;
try
CookieList.Delimiter := ';' ;
document := WebBrowser1.Document as IHTMLDocument2;
CookieList.DelimitedText := document.cookie;
for i := 0 to CookieList.Count-1 do
IdCookieManager1.AddCookie(CookieList[i],LOGIN_URL)
finally
CookieList.Free;
end;
Then in my original procedure BeforeFileDownload I try to log IdHttp into the site as well using code I adapted from here Log in to website from Delphi and the the cookies held in the cookie manager.
Displaying the string returned showed lots of HTML that appeared to represent the oringinal log in page and not the page you see after log in
procedure TFrmInportGrades.BeforeFileDownload(Sender: TObject; const FileSource: WideString; var Allowed: Boolean);
var
FileTarget: string;
FileStream: TMemoryStream;
request : Tstringlist;
s : string;
begin
FileSourceEdit.Text := FileSource;
Allowed := ShowDialogCheckBox.Checked;
if not Allowed then
begin
try
FileStream := TMemoryStream.Create;
IdHTTP.CookieManager := IdCookieManager1;
s := LogInIdHttp; //<<<< log in the IdHttp
showmessage(s); //<<<< debug
IdHTTP.Get(FileSource, FileStream);
FileTarget := IdHTTP.URL.Document;
if FileTarget = '' then
FileTarget := 'File';
FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
FileStream.SaveToFile(FileTarget);
finally
FileStream.Free;
end;
ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
FileTarget);
end;
end;
The login code I used is below but I don't really know what I am doing here or what needs to be put into the Request.Add() parameters. I used 'Inspect element' from firefox to get the name of the user and password boxes and put the correct users name and password after the '=' sign in lines {3} and {4}. In lines {2},{6} and {7} I put the url of the log in site. I've no idea what lines {1}, {2}, {5} do or even if they are correct or necessary
function TFrmInportGrades.LogInIdHttp: string;
var
Request: TStringList;
Response: TMemoryStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL; // added as its a https site
begin
Result := '';
try
Response := TMemoryStream.Create;
try
Request := TStringList.Create;
try
{1} Request.Add('op=login');
{2} Request.Add('redirect=https://www.thewebsite.com/Login.aspx' );
{3} Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl00$loginEmailInput=usernme');
{4} Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl01$passwordInput=password'});
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
IdHTTP.IOHandler := LHandler; //<<< added as its a https site
IdHTTP.AllowCookies := True;
IdHTTP.HandleRedirects := True;
{5} IdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
{6} IdHTTP.Post('https://www.thewebsite.com/Login.aspx', Request, Response);
{7} Result := IdHTTP.Get('https://www.thewebsite.com/Login.aspx');
finally
Request.Free;
end;
finally
Response.Free;
end;
except
on E: Exception do
ShowMessage(E.Message);
end;
end;
The net result of all this is that I don't get a file created at all now, not even a zero byte one. This all seems very overcomplicated simply to avoid or automate the 'Save As' dialog and is requiring lots of code that I won't be able to maintan afterwards. Unless somebody has a simpler solution I'll just parse what I can see (BTW I tried TEmbeddedWebBrowser but there is so little documentation for it I couldn't see how to make it download correctly. Might try again later.) Thank you for trying to help!
I know there's alot of Indy threads but I can't get one to match my case.
I have been given a URL with a username and password form. this then actions to a URL/reports.php on which there are multiple hyperlinks.
Each of these links will direct to a page with URL variables e.g. reports.php?report=variablename where a download will immediately start.
My thinking so far:
procedure TForm1.PostData(Sender: TObject);
var
paramList:TStringList;
url,text:string;
// IdHTTP1: TIdHTTP;
IdSSLIOHandlerSocket1: TIdSSLIOHandlerSocket;
idLogFile1 : TidLogFile;
begin
idLogFile1 := TidLogFile.Create(nil);
with idLogFile1 do
begin
idLogFile1.Filename := 'C:\HTTPSlogfile.txt';
idLogFile1.active := True;
end;
IdHTTP1 := TIdHTTP.Create(nil);
IdSSLIOHandlerSocket1 := TIdSSLIOHandlerSocket.Create(nil);
IdSSLIOHandlerSocket1.SSLOptions.Method := sslvSSLv23;
IdHTTP1.IOHandler := IdSSLIOHandlerSocket1;
IdHTTP1.HandleRedirects := true;
IdHTTP1.ReadTimeout := 5000;
IdHTTP1.Intercept := idLogFile1;
paramList:=TStringList.create;
paramList.Clear;
paramList.Add('loguser=testuser');
paramList.Add('logpass=duke7aunt');
paramList.Add('logclub=8005');
url := 'https://www.dfcdata.co.uk/integration/reports.php?report=live';
try
IdHTTP1.Post(url,paramList);
except
on E:Exception do
begin
showMessage('failed to post to: '+url);
ShowMessage('Exception message = '+E.Message);
end;
end;
end;
procedure TForm1.Button1Click(Sender: TObject);
var
reportType : String;
begin
PostData(Self);
reportType := 'live';
GetUrlToFile('',reportType+'.csv');
end;
procedure TForm1.GetUrlToFile(AURL, AFile : String);
var
Output : TMemoryStream;
success : Boolean;
begin
success := True;
Output := TMemoryStream.Create;
try
try
IdHTTP1.Get(AURL, Output);
IdHTTP1.Disconnect;
except
on E : Exception do
begin
ShowMessage('Get failed to GET from '+IdHTTP1.GetNamePath +'. Exception message = '+E.Message);
success := False;
end;
end;
if success = True then
begin
showMessage('Filed saved');
Output.SaveToFile(AFile);
end;
finally
Output.Free;
end;
end;
On each try I get "IOHandler is not valid" error. Obviously I'm not posting correctly to the initial page but can anyone advise me on what I'm missing? Also can I simply then hit the download URL after login or will I have to use cookies?
Thanks
There are several bugs in your code:
1) PostData() is requesting an HTTPS URL, but it is not assigning an SSL-enabled IOHandler to the TIdHTTP.IOHandler property. You need to do so.
2) Button1Click() is passing a URL to GetUrlToFile() that does not specify any protocol, so TIdHTTP will end up treating that URL as relative to its existing URL, and thus try to download from https://www.testurl.com/test/testurl.com/test/reports.phpinstead of https://testurl.com/test/reports.php. If you want to request a relative URL, don't include the hostname (or even the path in this case, since you are sending multiple requests to the same path, just different documents).
3) you are leaking the TIdHTTP object.
Issue 1) has now been resolved in another post:
Delphi 5 Indy/ics SSL workaround?
However I would greatly appreciate help on the rest, as follows.
Would I need to make a GET call with the same IdHTTP object and additional URL variable? or should I create a new IdHTTP object?
Would I need to record the session using cookies or can all of this be done with the same call?
Is the GET call above actually what I need to save a csv to file? I may also choose to handle it directly as the data will need importing anyway.
Currently the code gets the error: EIdHTTPProtocolException
This is the continue of my previous question:
Delphi, WebBrowser, Google Login, FusionTable
But the test with WinHTTP also failed as TWebBrowser based test...
And this is one question as you wish... :-)
I have one table what is NOW PUBLIC, but when we will buy non-free account it will be changed to private kind.
I created a simple WinHTTP test, but this also failed.
I can login, I got the "Auth" tag, but the next "private" request returns 401 error.
procedure TForm1.BitBtn1Click(Sender: TObject);
var
WinHttpReq, temp : variant;
URL, s : String;
params : TStringList;
authtoken, query, posts : string;
begin
URL := 'https://www.google.com/accounts/ClientLogin';
WinHttpReq := CreateOleObject('WinHttp.WinHttpRequest.5.1');
params := TStringList.Create;
try
params.Values['accountType'] := 'GOOGLE';
params.Values['Email'] := csEmail;
params.Values['Passwd'] := csPwd;
params.Values['service'] := 'fusiontables';
params.Values['source'] := csSource;
posts := EncodeParamsToURL(params);
finally
params.Free;
end;
URL := URL + '?' + posts;
WinHttpReq.Open('POST', URL, false);
WinHttpReq.Send();
s := WinHttpReq.ResponseText;
Memo1.Lines.Text := s;
params := TStringList.Create;
try
params.Text := s;
authtoken := params.Values['Auth'];
Edit1.Text := authtoken;
finally
params.Free;
end;
//query := URLEncode('SHOW TABLES');
query := URLEncode('select * from 1236965');
url := 'http://www.google.com/fusiontables/api/query?sql=' + query;
WinHttpReq.Open('POST', URL, false);
WinHttpReq.setRequestHeader('Authorization', 'GoogleLogin auth="' + authToken + '"');
WinHttpReq.Send();
s := WinHttpReq.ResponseText;
Memo1.Lines.Text := s;
end;
When I made "select", I got the rows.
But when I want to see the tablenames, I get 401 error...
I'm not sure what cause this error.
a. The free account don't have enough rights to access it privately
b. I set the header wrong
c. I set the csSource wrong (I set it "MyCompanyName-Test-1.0")
d. Other thing I don't know what...
Can anybody help me how to login and access the data successfully?
Change your method to GET:
WinHttpReq.Open('GET', URL, false);
And remove the " around your auth token:
WinHttpReq.setRequestHeader('Authorization', 'GoogleLogin auth=' + authToken);
The first change is to comply with the documentation (although POST also works). The second change fixes the error.
I have the following task: download a file using HTTPS and authentication. Indy seems the way to go but for some reason it doesn't work so far. I have the following in place:
a TIdHTTP component which I use for downloading
a TIdURI component used to create the URL
a TIdSSLIOHandlerSocketOpenSSL component which should provide the secure connection. The required DLLs are in the binary folder.
The site also requires authentication and I included the user/pass in the URL as in the example below. In short this is the code:
URI := TIdURI.Create('https://test.example.com/');
URI.Username := ParamUserName;
URI.Password := ParamPassword;
HTTP := TIdHTTP.Create(nil);
if URI.Protocol = 'https' then
begin
IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IOHandler.SSLOptions.Method := sslvSSLv3;
HTTP.IOHandler := IOHandler;
end;
HTTP.Get(URI.GetFullURI([ofAuthInfo]), FileStream);
Using this code I get a "Read Timeout" EIdReadTimeout exception very fast. Testing the URL in a browser works without problem. Any ideas on what's missing or what I did wrong?
I finally abandoned Indy and OpenSSL and used WinInet for downloading. This is the code that worked for me:
function Download(URL, User, Pass, FileName: string): Boolean;
const
BufferSize = 1024;
var
hSession, hURL: HInternet;
Buffer: array[1..BufferSize] of Byte;
BufferLen: DWORD;
F: File;
begin
Result := False;
hSession := InternetOpen('', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0) ;
// Establish the secure connection
InternetConnect (
hSession,
PChar(FullURL),
INTERNET_DEFAULT_HTTPS_PORT,
PChar(User),
PChar(Pass),
INTERNET_SERVICE_HTTP,
0,
0
);
try
hURL := InternetOpenURL(hSession, PChar(URL), nil, 0, 0, 0) ;
try
AssignFile(f, FileName);
Rewrite(f,1);
try
repeat
InternetReadFile(hURL, #Buffer, SizeOf(Buffer), BufferLen) ;
BlockWrite(f, Buffer, BufferLen)
until BufferLen = 0;
finally
CloseFile(f) ;
Result := True;
end;
finally
InternetCloseHandle(hURL)
end
finally
InternetCloseHandle(hSession)
end;
end;
I have seen the same thing. Setting the TIdHTTP.ReadTimeout to zero fixes the problem for me.
...
HTTP.IOHandler := IOHandler;
HTTP.ReadTimeout := 0;
All SSL related functions for secure connections will fail unless some additional libraries are installed properly.
1.) Download libraries
2.) unzip and copy both DLLs to your project folder (or somewhere in the PATH of your system)
with that your code from the question works fine for me.
--reinhard