I am trying to use IdHTTP to equivalence the following curl operation:
curl -X POST -F "message=#C:\Users\santon\Desktop\ESM_download\token.txt" "https://esm-db.eu/esmws/eventdata/1/query?eventid=IT-1997-0004&station=CLF&format=ascii" -o RecordFileName.zip
The curl command is used to download a file from the server that is then saved on the hard drive as DownloadedFileName.zip. An authorization is required through a token file on the hard drive called token.txt. The path of the token file is specified as a parameter of curl.
The best I could do is the following code:
procedure TMainForm.HTTPGetFile;
var
IdHTTP: TIdHTTP;
Params: TIdMultipartFormDataStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL;
begin
try
Params := TIdMultipartFormDataStream.Create;
Params.AddFormField('message', '#"C:\Users\santon\Desktop\ESM_download\token.txt"');
IdHTTP := TIdHTTP.Create(nil);
LHandler:= TIdSSLIOHandlerSocketOpenSSL.Create(self);
LHandler.SSLOptions.Method := sslvTLSv1;
try
IdHTTP.IOHandler := LHandler;
IdHTTP.Post('https://esm-db.eu/esmws/eventdata/1/query?eventid=IT-1997-0004&station=CLF&format=ascii',Params);
finally
IdHTTP.Free;
LHandler.Free;
Params.Free;
end;
except
on E: Exception do
ShowMessage('Error: '+E.ToString);
end;
end;
But I keep on getting a HTTP/1.1 403 Forbidden error.
Any ideas of what I am doing wrong?
Thanks in advance
You are not loading the token file into TIdMultiPartFormDataStream correctly.
Per the curl documentation:
https://curl.se/docs/manpage.html#-F
-F, --form <name=content>
(HTTP SMTP IMAP) For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to POST data using the Content-Type multipart/form-data according to RFC 2388.
...
This enables uploading of binary files etc. To force the 'content' part to be a file, prefix the file name with an # sign. To just get the content part from a file, prefix the file name with the symbol <. The difference between # and < is then that # makes a file get attached in the post as a file upload, while the < makes a text field and just get the contents for that text field from a file.
...
Example: send an image to an HTTP server, where 'profile' is the name of the form-field to which the file portrait.jpg will be the input:
curl -F profile=#portrait.jpg https://example.com/upload.cgi
...
In your code, you are creating a text field whose content is the filename itself. You are not creating a file upload field whose content is the data from the file.
Try this instead:
procedure TMainForm.HTTPGetFile;
var
IdHTTP: TIdHTTP;
Params: TIdMultipartFormDataStream;
LHandler: TIdSSLIOHandlerSocketOpenSSL;
LOutFile: TFileStream;
begin
try
Params := TIdMultipartFormDataStream.Create;
try
Params.AddFile('message', 'C:\Users\santon\Desktop\ESM_download\token.txt');
IdHTTP := TIdHTTP.Create(nil);
try
LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP);
LHandler.SSLOptions.Method := sslvTLSv1;
IdHTTP.IOHandler := LHandler;
LOutFile := TFileStream.Create('<path>\RecordFileName.zip', fmCreate);
try
IdHTTP.Post('https://esm-db.eu/esmws/eventdata/1/query?eventid=IT-1997-0004&station=CLF&format=ascii', Params, LOutFile);
finally
LOutFile.Free;
end;
finally
IdHTTP.Free;
end;
finally
Params.Free;
end;
except
on E: Exception do
ShowMessage('Error: ' + E.ToString);
end;
end;
Related
I was making an application to send backup files to the Drive, however, when I try to send the files in chunks, I am getting a 400 error in the first request.
procedure TMainForm.btnTesteClick(Sender: TObject);
const
cURL = 'URL from requset';
cFilePath = 'path of File';
var
zIdHTTP: TIdHTTP;
zParams: TIdMultiPartFormDataStream;
zIOHandler: TIdSSLIOHandlerSocketOpenSSL;
zMemFile: TFileStream;
begin
zIdHTTP:= TIdHTTP.Create();
zParams:= TIdMultiPartFormDataStream.Create();
zIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create();
try
zIOHandler.SSLOptions.SSLVersions := [sslvTLSv1_2];
zIOHandler.SSLOptions.Mode := sslmUnassigned;
zIOHandler.SSLOptions.VerifyMode:= [];
zIOHandler.SSLOptions.VerifyDepth:=0;
zIdHTTP.IOHandler:= zIOHandler;
zIdHTTP.HandleRedirects:= True;
try
zMemFile:= TFileStream.Create(cFilePath, fmOpenReadWrite);
zIdHTTP.Request.CustomHeaders.AddValue('Content-Length', '8388608');
zIdHTTP.Request.CustomHeaders.AddValue('Content-Range', 'bytes 0-8388607/178498359');
zParams.AddFormField('data', 'application/octet-stream', '', zMemFile);
ShowMessage(zIdHTTP.Put(cURL, zParams));
except
on E: EIdHTTPProtocolException do ShowMessage(IntToStr(zIdHTTP.ResponseCode));
on E:Exception do ShowMessage('Error: Code: '+IntToStr(zIdHTTP.ResponseCode)+' - Message: '+E.Message);
end;
finally
FreeAndNil(zMemFile);
FreeAndNil(zIdHTTP);
end;
end;
the error message that appears is as follows: "Connection reset by peer"
The request is being made with the TIDHTTP component, but I also tested it with other components and the error continues to happen.
Does anyone know how to fix this error ?
I've tried it with other components like REST, and I've also tested it with insomnia.
I am struggling to get a response from Google Securetoken using Delphi 10.2.2 and Indy 10.
I've gotten my RefreshToken before from https://www.googleapis.com/identitytoolkit/v3/relyingparty/verifyPassword?key=XXX. This worked fine.
Now I try to refresh the token with https://securetoken.googleapis.com/v1/token, which doesn't work. I always get a 403 Forbidden error.
procedure TForm1.Button2Click(Sender: TObject);
var
l_Response: string;
l_PostData: TIdMultiPartFormDataStream;
l_IRESAccessToken: TIRESAccessToken;
begin
IdHTTP2.Request.ContentType := 'application/json';
IdHTTP2.Request.CharSet := 'utf-8';
l_PostData := TIdMultiPartFormDataStream.Create;
try
l_PostData.AddFormField('grant_type', 'refresh_token');
l_PostData.AddFormField('refresh_token', m_IRESAuth.RefreshToken);
l_PostData.AddFormField('key', 'XXX');
try
l_Response := IdHTTP2.Post('https://securetoken.googleapis.com/v1/token', l_PostData);
l_IRESAccessToken := TJson.JsonToObject<TIRESAccessToken>(l_Response);
except
on E: Exception do
ShowMessage('Error on request: '#13#10 + e.Message);
end;
finally
l_PostData.Free;
end;
end;
I tried an IOHandler that specifies TLS 1.2, and tried to send the post with a JSON object. Also, I set the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions. Nothing worked so far, I always get 403 Forbidden.
I tried it with another program, no problems there.
Am I missing something?
Per Google's documentation:
Token Service REST API Reference:
HTTP request
POST https://securetoken.googleapis.com/v1/token
Request body
The request body contains data with the following structure:
URL-encoded representation
grant_type=string&code=string&refresh_token=string
The TStrings overload of TIdHTTP.Post() sends data in that format. That is the overload you need to use, not the TIdMultipartFormDataStream overload.
Also, as you can see above, this URL does not accept your key as input. It only accepts grant_type, code (which is ignored when grant_type is not 'authorization_code') and refresh_token.
Try this instead:
procedure TForm1.Button2Click(Sender: TObject);
var
l_Response: string;
l_PostData: TStringList;
l_IRESAccessToken: TIRESAccessToken;
begin
IdHTTP2.Request.ContentType := 'application/x-www-webform-urlencoded';
l_PostData := TStringList.Create;
try
l_PostData.Add('grant_type=refresh_token');
l_PostData.Add('refresh_token=' + m_IRESAuth.RefreshToken);
try
l_Response := IdHTTP2.Post('https://securetoken.googleapis.com/v1/token', l_PostData);
l_IRESAccessToken := TJson.JsonToObject<TIRESAccessToken>(l_Response);
except
on E: Exception do
ShowMessage('Error on request: '#13#10 + e.Message);
end;
finally
l_PostData.Free;
end;
end;
That being said, after reading the above documentation, I don't believe you can send an idToken from https://www.googleapis.com/identitytoolkit/v3/relyingparty/verifyPassword to https://securetoken.googleapis.com/v1/token as a refresh_token. I think you need to send it as an authorization_code instead, which then gives you a refresh_token. So try adding that extra step, if you have not already done so.
Sorry for the late response and thank you for the answer.
After consultation with the Main-API-Developer, he told me that they are using Firebase in the background. So the API request für Google is different.
https://cloud.google.com/identity-platform/docs/use-rest-api#section-sign-in-email-password
https://cloud.google.com/identity-platform/docs/use-rest-api#section-refresh-token
With this missing information the request works fine
procedure TForm1.Button2Click(Sender: TObject);
var
l_Response: string;
l_IRESAccessToken: TIRESAccessToken;
l_Json: string;
l_JsonToSend: TStringStream;
begin
l_Json := '{"grant_type": "refresh_token","refresh_token": "'+ m_IRESAuth.RefreshToken+ '"}';
l_JsonToSend := TStringStream.Create(l_Json, TEncoding.UTF8);
try
try
l_Response := IdHTTP2.Post('https://securetoken.googleapis.com/v1/token?key=XXX, l_JsonToSend);
l_IRESAccessToken := TJson.JsonToObject<TIRESAccessToken>(l_Response);
except
on E: Exception do
ShowMessage('Error on request: '#13#10 + e.Message);
end;
finally
l_JsonToSend.Free;
end;
end;
I am trying to call Apache-TIKA via their REST API. I have successfully been able to upload a PDF document and return the document's text via CURL
curl -X PUT --data-binary #<filename>.pdf http://localhost:9998/tika --header "Content-type: application/pdf"
That translated to INDY like so:
function GetPDFText(const FileName: String): String;
var
IdHTTP: TIdHTTP;
Params: TIdMultiPartFormDataStream;
begin
IdHTTP := TIdTTP.Create;
try
Params := TIdMultiPartFormDataStream.Create;
try
Params.Add('file', FileName, 'application/pdf')
Result := IdHTTP.PUT('http://localhost:9998/tika', Params);
finally
Params.Free;
end;
finally
IdHTTP.Free;
end;
end;
Now I want to upload a word document (.docx)
I assumed that all I would need to do is change the content Type when I add my file to Params, but that doesn't seem to produce any results, although I get no error reported back. I was able to get the following CURL command to work correctly
CURL -T <myDOCXfile>.docx http://localhost:9998/tika --header "Content-type: application/vnd.openxmlformats-officedocument.wordprocessingml.document"
How do I modify my HTTP call from CURL -X PUT to CURL -T?
There are at least two issues in your implementation:
Your translation from CURL -X PUT to TIdHTTP is wrong.
You don't specify Accept HTTP header to retrieve the extracted text in specific format.
How to translate curl -X PUT to Indy?
At first, lets make it clear that curl -X PUT --data-binary #<filename> <url> is the same as curl -T <filename> <url> when:
<url>'s scheme is HTTP or HTTPS
<url> does not end with /
Therefore using one or the other shouldn't matter in your case. See also curl documentation.
Secondly, TIdMultiPartFormDataStream is designed for use with POST verb, however nothing can stop you from passing it to TIdHTTP.Put, because it is indirectly derived from TStream. There even is a dedicated invariant of TIdHTTP.Post method that accepts TIdMultiPartFormDataStream:
function Post(AURL: string; ASource: TIdMultiPartFormDataStream): string; overload;
To upload file to the service just use TIdHTTP.Put method with TFileStream as an argument while providing proper content type of the file being uploaded in HTTP header.
And finally you're trying to extract plain text from the document, but you didn't specify content type that the service should return. This is done via Accept HTTP header. Default instance of TIdHTTP has property IdHTTP.Request.Accept initialized to 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' (this may vary depending on Indy version). Therefore by default Tika will return HTML formatted text. To get the plain text you should change it to 'text/plain; charset=utf-8'.
Fixed implementation:
uses IdGlobal, IdHTTP;
function GetDocumentText(const FileName, ContentType: string): string;
var
IdHTTP: TIdHTTP;
Stream: TIdReadFileExclusiveStream;
begin
IdHTTP := TIdHTTP.Create;
try
IdHTTP.Request.Accept := 'text/plain; charset=utf-8';
IdHTTP.Request.ContentType := ContentType;
Stream := TIdReadFileExclusiveStream.Create(FileName);
try
Result := IdHTTP.Put('http://localhost:9998/tika', Stream);
finally
Stream.Free;
end;
finally
IdHTTP.Free;
end;
end;
function GetPDFText(const FileName: string): string;
const
PDFContentType = 'application/pdf';
begin
Result := GetDocumentText(FileName, PDFContentType);
end;
function GetDOCXText(const FileName: string): string;
const
DOCXContentType = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
begin
Result := GetDocumentText(FileName, DOCXContentType);
end;
According to the Tika's documentation it also supports posting multipart form data. If you insist on using this approach, then you should change the target resource to /tika/form and switch to Post method in your implementation:
function GetDocumentText(const FileName, ContentType: string): string;
var
IdHTTP: TIdHTTP;
FormData: TIdMultiPartFormDataStream;
begin
IdHTTP := TIdHTTP.Create;
try
IdHTTP.Request.Accept := 'text/plain; charset=utf-8';
FormData := TIdMultiPartFormDataStream.Create;
try
FormData.AddFile('file', FileName, ContentType); { older Indy versions: FormData.Add(...) }
Result := IdHTTP.Post('http://localhost:9998/tika/form', FormData);
finally
FormData.Free;
end;
finally
IdHTTP.Free;
end;
end;
Why does the original implementation in question work with PDF files?
When you Post multipart form data via TIdHTTP, Indy automatically sets content type of the request to 'multipart/form-data; boundary=...whatever...'. This is not the case when you Put (unless you set it manually before performing the request) data and therefore TIdHttp.Request.ContentType remains blank. Now I can only guess that when Tika sees empty content type it falls back to some default type which could be PDF and it's still somehow able to read the document from multipart request.
I have to write a program (Delphi XE5, Indy 10: TIdHTTP & TIdSSLIOHandlerSocketOpenSSL) which can connect to a web service with client authentication. With several days of working, finally it has become success. I can connect using the authentication, setting the TIdSSLIOHandlerSocketOpenSSL’s SSLOptions.CertFile and SSLOptions.KeyFile properties. It’s fine. (I've got a pfx file from my partner, I exported it to a certificate and a private key file with OpenSSL so I use these 2 files in the program.)
I have one TButton, TMemo and TIdHTTP component on the form.
Source code (Button's click event - the IdHTTP1.Request.ContentType := '.......' line is necessary just for the communication because of the server settings):
procedure TForm1.Button1Click(Sender: TObject);
var
URL: string;
XML: TStrings;
S: string;
Req: TStream;
SL: TStringList;
SSL1 : TIdSSLIOHandlerSocketOpenSSL;
begin
XML := TStringList.Create;
XML.Add('<soap:Envelope xmlns:ns="http://docs.oasis-open.org/ws-sx/ws-trust/200512" ' +
'xmlns:soap="http://www.w3.org/2003/05/soap-envelope">');
…
XML.Add(' <soap:Body>');
…
XML.Add(' </soap:Body>');
XML.Add('</soap:Envelope>');
URL := 'https://…………………….';
end
Req := TStringStream.Create(XML.Text, TEncoding.UTF8);
try
SSL1 := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
SSL1.SSLOptions.CertFile := 'd:\certificate.pem';
SSL1.SSLOptions.KeyFile := 'd:\private.pem';
SSL1.SSLOptions.Mode := sslmClient;
try
SSL1.SSLOptions.Method := sslvSSLv23;
IdHTTP1.IOHandler := SSL1;
IdHTTP1.Request.ContentType := 'application/soap+xml;charset=UTF-8;action="http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue"';
S := IdHTTP1.Post(URL, Req);
finally
SSl1.Free;
end;
finally
Req.Free;
end;
ResultMemo.Lines.Add(Format('Response Code: %d', [IdHTTP1.ResponseCode]));
ResultMemo.Lines.Add(Format('Response Text: %s', [IdHTTP1.ResponseText]));
SL := TStringList.Create;
try
SL.Text := S;
ResultMemo.Lines.AddStrings(SL);
finally
SL.Free;
end;
end;
The problem is: my partner said this case is not the best if the file I use is not password-protected. They told me how to create a password-protected (and encrypted) file for the KeyFile with OpenSSL. When I set this password-protected file to the SSLOptions.KeyFile I get the following error message: „Could not load key, check password. error:0906A068:PEM routines:PEM_do_header:bad password read.”
I tried to set the password in the idHTTP1.Request.Password property, but the result is the same.
Question: how and where do I have to set the password for the KeyFile if I have to use a password-protected keyfile? Because I have to publish the certification files, too, the best solution would be to set the password in the program and use the password-protected KeyFile, instead of using not the password-protected KeyFile.
Thanks a lot.
Regards,
Attila
Use the IdSSLIOHandlerSocketOpenSSL.OnGetPassword event and set it here.
procedure TForm1.IdSSLIOHandlerSocketOpenSSL1GetPassword(var Password: string);
begin
Password := 'thepassword';
end;
I try to connect me to my stripe payment test account with delphi.
The connect API is here:
Stripe connect API
Curl example:
curl https://api.stripe.com/v1/charges \
-u sk_test_CpkBxhx9gcmNYYQTZIXU43Bv:
I tried using Indy TIdHTTP component with TIdSSLIOHandlerSocketOpenSSL
and calling post with Tstringlist or TIdMultipartFormDataStream as parameter
but I receive always response: 401 - Unauthorized
Here my code:
var
Data: TIdMultipartFormDataStream;
https: TIdHTTP;
ssl: TIdSSLIOHandlerSocketOpenSSL;
begin
https := TIdHTTP.Create(self);
ssl := TIdSSLIOHandlerSocketOpenSSL.Create(Self);
https.IOHandler := ssl;
https.Request.BasicAuthentication := True;
Data := TIdMultipartFormDataStream.Create;
//Data.AddFormField('api_key', 'sk_test_CpkBxhx9gcmNYYQTZIXU43Bv');
Data.AddFormField('apikey', 'sk_test_CpkBxhx9gcmNYYQTZIXU43Bv');
https.Post('https://api.stripe.com/v1/charges', Data);
Memo1.lines.Add( https.ResponseText );
Data.Free;
end;
Any help or suggestion would bee very appreciated.
Thanks,
Peter
You must not use a form field to transfer the API key. Instead, set the Request.Username property. The password is empty, so Request.Passwort is unused. From the API docs on your linked page:
Authentication to the API occurs via HTTP Basic Auth. Provide your API
key as the basic auth username. You do not need to provide a password.
This example works with Indy 10.6.2 and OpenSSL libraries in the program folder:
program Project31229779;
{$APPTYPE CONSOLE}
uses
IdHTTP, SysUtils;
var
HTTP: TIdHTTP;
begin
HTTP := TIdHTTP.Create;
try
HTTP.Request.BasicAuthentication := True;
HTTP.Request.Username := 'sk_test_CpkBxhx9gcmNYYQTZIXU43Bv';
try
WriteLn(HTTP.Get('https://api.stripe.com/v1/charges'));
except
on E: EIdHTTPProtocolException do
begin
WriteLn(E.Message);
WriteLn(E.ErrorMessage);
end;
on E: Exception do
begin
WriteLn(E.Message);
end;
end;
finally
HTTP.Free;
end;
ReadLn;
end.
Note: you may also put the user name / password in the URL:
HTTP.Request.BasicAuthentication := True;
try
WriteLn(HTTP.Get('https://sk_test_CpkBxhx9gcmNYYQTZIXU43Bv:#api.stripe.com/v1/charges'));