Can TIndyHTTP's POST method be called multiple times very quickly? - post

I'm using Indy ver 10.5498 to post some multipart/form data including an attached file to an api. My code is adapted from that supplied to my by Remy in this post, with protocol error handling taken from here.
The code I now have works well and I get a response back from the server with data about 2 seconds after making the post.
However on occasion I might need to do a post multiple times very quickly, for example by looping through the dataset returned from a database and doing a post for each record.
Is there anything I need to know or any special code I need to write in order to deal with the situation where I might be making a second POST before the first POST has completed sending (or at least before the server's response has been received? Or is the POST a blocking call that does not return control until the response is received?
At the moment the TIdHTTP component is placed on the form, not dynamically created. I could create a new TIdHTTP object for every post and destroy it afterwards if that's necessary.
The code I use to do the post at the moment is below
function TForm1.Upload(url: string; params, filenames: TStrings): string;
var
FormData : TIdMultiPartFormDataStream;
ResponseText : string;
i : integer;
begin
FormData := TIdMultiPartFormDataStream.Create;
try
for i := 0 to params.Count - 1 do
FormData.AddFormField(params.Names[i], params.ValueFromIndex[i]);
for i := 0 to filenames.Count - 1 do
FormData.AddFile('attachment', filenames[i]);
//add authorisation header
IdHTTP1.Request.CustomHeaders.Add('Authorization:Basic ' + U_generalStuff.base64encodeStr(ATHORISATION_STR));
//way to use just one try except yet get the full text server response whether its a 400 error or a success response
//see https://stackoverflow.com/questions/54475319/accessing-json-data-after-indy-post
// Make sure it uses HTTP 1.1, not 1.0, and disable EIdHTTPProtocolException on errors
IdHTTP1.ProtocolVersion := pv1_1;
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoKeepOrigProtocol, hoNoProtocolErrorException, hoWantProtocolErrorContent];
try
ResponseText := IdHTTP1.Post(url, FormData); //post to the api
except
on E: Exception do
begin
ResponseText := E.ClassName + ': ' + E.message;
raise;
end;
end; //try-except
finally
result := ResponseText;
end; //try finally
end;
I've since seen this post that talks about threading and using the parallel library. Is that something I should be investigating to answer my question?

Like most things in Indy, TIdHTTP.Post() is a synchronous operation. It does not return back to the caller until the response has been received in full (or until an error occurs). So, if you call TIdHTTP.Post() in a simple loop, you CANT send a new POST request until the previous request has fully completed. If you need to do that, you will have to call TIdHTTP.Post() asynchronously by calling it in multiple worker threads (and give each thread its own TIdHTTP object, as you can't reuse a TIdHTTP object across multiple simultaneous requests).

Related

Delphi TWebModule (IIS-ISAPI): get the current request

in a TWebModule procedure/function how get the current request?
I have tried:
procedure TWebModule1.DoSomething;
var
aRequest : TWebRequest;
begin
aRequest := Request;
end;
but it seems the first request produced on TWebModule creation.
I know i'm able to pass the request to subsequent Procedures/Functions from each TWebActionItem, but i want avoid to pass the request every where. Any tips?
Update
After digging into the code, i found WebContext and it seems the solution, eg.:
uses Web.WebCntxt;
procedure TWebModule1.DoSomething;
var
aRequest : TWebRequest;
begin
if WebContext <> nil then
aRequest := WebContext.Request;
end;
is it the right way? WebContext seems always nil.
I'm on Delphi Berlin update 2.
Every Request goes through a TWebActionItem defined in the TWebModule.Actions. The TWebActionItem has an event OnAction. There you will get the TWebRequest Object of the current Request.
Then you are able to pass it to subsequent Procedures/Functions.

Delphi REST Server, Android Client using compression

We're building a Delphi REST server that serves up rather large chunks of data (1.5MB per request, of which there are many) to a native Android application. All works fine, except the data sizes in this case will be problematic, causing long transfer times in our environment (limited mobile data rates). I've tried adding the ZLibCompression filter on the DSHTTPWebDispatcher, but the response only comes back again as uncompressed text/html.
Is there any way to force the server to use the filter added as an event before the dispatch?
The server is built using Delphi XE3.
I've managed to figure out where to add the compression and relevant header changes in the DataSnap project.
The key here is the TWebModule class. If one uses the wizard to create a new project, a default implementation of the TWebModule class is constructed, with event properties for BeforeDispatch, AfterDispatch etc.
The naming here refers to the action of dispatching the incoming request to where it will be handled. So, BeforeDispatch happens when the request arrives, some processing happens on the server and AfterDispatch triggers just before the response is sent back to the caller.
AfterDispatch is therefore the correct event to use if one wants to modify the constructed response after the fact. This can include changes to both the content and headers.
On the AfterDispatch event:
procedure TWebModule1.WebModuleAfterDispatch(
Sender: TObject;
Request: TWebRequest;
Response: TWebResponse;
var Handled: Boolean);
var
srcbuf, destbuf : TBytes;
str : string;
begin
str := Response.Content;
//prepare byte array
srcbuf := BytesOf(str);
//compress to buff (System.ZLib)
ZCompress(srcbuf, destbuf, zcMax);
//prepare responsestream and set content encoding and type
Response.Content := '';
Response.ContentStream := TMemoryStream.Create;
Response.ContentEncoding := 'deflate';
Response.ContentType := 'application/json';
//current browser implementations incorrectly handles the first 2 bytes
//of a ZLib compressed stream, remove them
Response.ContentStream.Write(#(destbuf[2]),length(destbuf)-2);
Response.ContentLength := (length(destbuf))-2;
end;
Not very fancy, one could enable/disable compression depending on the content sent back, but for our implementation we kept it simple.
This works 100% with Fiddler and browsers that can handle deflate.

A Google Chrome POST breaks this SSL_read. Anyone have code that works?

I need a minimal SSL server and came up with the following:
confirm(WSAStartup(MakeWord(1,1), WData) = 0);
SSL_library_init;
SSL_load_error_strings;
ctx := SSL_CTX_new(SSLv23_server_method);
confirm(ctx <> nil);
confirm(SSL_CTX_use_certificate_chain_file(ctx, 'cert.pem') > 0);
confirm(SSL_CTX_use_PrivateKey_file(ctx, 'key.pem', SSL_FILETYPE_PEM) > 0);
confirm(SSL_CTX_check_private_key(ctx));
SSL_CTX_set_mode(ctx, SSL_MODE_AUTO_RETRY);
listen_socket := socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
confirm(listen_socket <> 0);
sa_serv.sin_family := AF_INET;
sa_serv.sin_port := htons(DEFAULTPORT);
sa_serv.sin_addr.s_addr := INADDR_ANY;
confirm(bind(listen_socket, sa_serv, SizeOf(sa_serv)) = 0);
while TRUE do
begin
if listen(listen_socket, 100) <> 0 then continue;
client_len := SizeOf(sa_cli);
sock := accept(listen_socket, #sa_cli, #client_len);
if sock = INVALID_SOCKET then continue;
ssl := SSL_new(ctx);
if ssl = nil then continue;
SSL_set_fd(ssl, sock);
if SSL_accept(ssl) = 1 then
begin
bytesin := SSL_read(ssl, buffer, sizeof(buffer)-1);
if bytesin > 0 then
begin
buffer[bytesin] := #0;
response := getresponse(buffer);
SSL_write(ssl, pchar(response)^, length(response));
end;
end;
SSL_set_shutdown(ssl, SSL_SENT_SHUTDOWN or SSL_RECEIVED_SHUTDOWN);
CloseSocket(sock);
SSL_free(ssl);
end;
The single SSL_read will grab an entire GET or POST request
from Firefox, and everything works great. On the other hand, a Chrome GET will cause the
first few SSL_read calls to return zero bytes, but eventually
a SSL_read will grab the entire GET request and the code still works.
But when Chrome sends a POST, the first few SSL_read calls
fetch zero bytes, and the next SSL_read will grab ONLY THE HEADERS.
The getresponse() routine can't make sense of the POST because
one more SSL_read is necessary to grab the POST content.
SSL_MODE_AUTO_RETRY was set, hoping SSL_read would then not
return until an entire request was done,
but that doesn't work. SSL_pending always returns zero, before or after
every SSL_read, so that's no help either.
As this question's answer says, non-blocking SSL appears to involve lots of torture and heartburn. I've played with doing SSL_reads in a separate thread and killing the thread after timing out on a hung read, but that seems dangerous since it's unknown what state SSL is in (or how to reset it) as the thread is killed.
Does anyone have code for a minimal loop similar to the above,
but that won't hang on a Chrome POST or SSL_read, that's simple and vanilla enough
to easily convert to Delphi 6?
You need to remember, that TCP is stream-based, and single call to Read can return whatever piece of data it can, and this can be for example half of the header, or a header + a bit of the post data or anything between. You can't expect one read to return the complete data.
Consequently your only option is the following algorithm:
read something.
check if you've got a complete header inside.
If you got a header, check if it contains Content-Length (some posts might not contain it)
If you have Content-length, keep reading from the socket up to Content-length, then process the complete request.
If you don't have Content-length, then you have to deal with chunked encoding. This is a complex thing to handle which is far beyond the scope of SO question.
If I were you, I would take some existing HTTP/HTTPS server implementation. Obvious options are Indy + SSL layer or HTTPBlackbox server package of our SecureBlackbox.
I modified OpenSSL s_server.c, which now does the trick, and posted it as the answer to Question 7080958.

Need a sample/demo of using TIdTelnet to interact with telnet server

I tried to employ Indy 10.5.5 (shipped with Delphi 2010) for:
connecting to telnet server
performing username/password authentication (gaining access to the command shell)
executing a command with returning resulting data back to application
and had no success, additionally i'm completely lost in spaghetti logic of Indy's internals and now have no idea why it didnt work or how i supposed to send strings to the server and grab the results. Need some sample code to study.
Formal form of the question: Where can i get 3-rd party contributed demo covering TIdTelnet component? (indyproject.org demos webpage do not have one)
The main problem with Telnet is that it DOES NOT utilize a command/response model like most other Internet protocols do. Either party can send data at any time, and each direction of data is independant from the other direction. This is reflected in TIdTelnet by the fact that it runs an internal reading thread to receive data. Because of this, you cannot simply connect, send a command, and wait for a response in a single block of code like you can with other Indy components. You have to write the command, then wait for the OnDataAvailable event to fire, and then parse the data to determine what it actually is (and be prepared to handle situations where partial data may be received, since that is just how TCP/IP works).
If you are connecting to a server that actually implements a command/response model, then you are better off using TIdTCPClient directly instead of TIdTelnet (and then implement any Telnet sequence decoding manually if the server really is using Telnet, which is rare nowadays but not impossible). For Indy 11, we might refactor TIdTelnet's logic to support a non-threaded version, but that is undecided yet.
done with indy.
no comments.. just som old code :-)
telnet don't like the send string kommand.. use sendch.
telnetdude.Host := 1.1.1.1;
try
telnetdude.connect;
except
on E: Exception do begin
E.CleanupInstance;
end; {except}
if telnetdude.Connected then begin
for i := 1 to length(StringToSend) do telnetdude.sendch(StringToSend[i]);
telnetdude.sendch(#13);
end;
end; {while}
end; {if}
if telnetdude.Connected then telnetdude.Disconnect;
end;
I hope this helps anyone looking for answers to a similar question.
Firstly, It would seem the typical command/response model (as mentioned above, does indeed NOT apply).
So I just got it working for some very simple application (rebooting my router).
Specific additions to above code from Johnny Lanewood (and perhaps some clarification)
a) You have to send #13 to confirm the command
b) I got "hangs" on every command I sent / response I requested UNTIL I enabled ThreadedEvent. (this was my big issue)
c) the OnDataAvailable event tells you when new data is available from the Telnet Server - however there are no guarantees as to what this data is - i.e. it's pretty what you get in the command line / what ever is appended to the previous responses. But is is NOT a specific response line to your command - it's whatever the telnet server returns (could be welcome info, ASCII drawings etc etc.)
Given (c) above, one would rather check the OnDataAvailable event and parse the data (knowing what you'd expect). When the output stops (i.e. you need build a mechanism for this), you can parse the data and determine whether the server is ready for something new from the client. For the purpose of my code below, I set a read timemout and I just used Sleep(2000) - ignorantly expecting no errors and that the server would be ready after the sleep for the next command.
My biggest stumbling block was ThreadedEvent := True (see above in b)
Thus, my working solution (for specific application, and possibly horrible to some).
lIDTelnet := TIdTelnet.Create(nil);
try
lIdTelnet.ReadTimeout := 30000;
lIDTelnet.OnDataAvailable := TDummy.Response;
lIDTelnet.OnStatus := TDummy.Status;
lIdTelnet.ThreadedEvent := True;
try
lIDTelnet.Connect('192.168.0.1', 23);
if not lIDTelnet.Connected then
Raise Exception.Create('192.168.0.1 TELNET Connection Failed');
Sleep(2000);
lIdtelnet.SendString(cst_user + #13);
Sleep(2000);
lIdtelnet.SendString(cst_pass + #13);
Sleep(2000);
lIdtelnet.SendString(cst_reboot + #13);
Sleep(2000);
if lIDTelnet.Connected then
lIDTelnet.Disconnect;
except
//Do some handling
end;
finally
FreeAndNil(lIdTelnet);
end;
and then
class procedure TDummy.Response(Sender: TIdTelnet; const Buffer: TIdBytes);
begin
Write(TDummy.ByteToString(Buffer));
end;
class function TDummy.ByteToString(
const aBytes: TIdBytes): String;
var
i : integer;
begin
result := '';
for i := 0 to Length(aBytes) -1 do
begin
result := result + Char(aBytes[i]);
end;
end;

Download, pause and resume an download using Indy components

Actually i'm using the TIdHTTP component for download a file from internet. i'm wondering if is possible pause and resume the download using this component o maybe another indy component.
this is my current code, this works ok for download a file (without resume), but . now i want pause the download close my app ,and when my app restart then resume the download from the last position saved.
var
Http: TIdHTTP;
MS : TMemoryStream;
begin
Result:= True;
Http := TIdHTTP.Create(nil);
MS := TMemoryStream.Create;
try
try
Http.OnWork:= HttpWork;//this event give me the actual progress of the download process
Http.Head(Url);
FSize := Http.Response.ContentLength;
AddLog('Downloading File '+GetURLFilename(Url)+' - '+FormatFloat('#,',FSize)+' Bytes');
Http.Get(Url, MS);
MS.SaveToFile(LocalFile);
except
on E : Exception do
Begin
Result:=False;
AddLog(E.Message);
end;
end;
finally
Http.Free;
MS.Free;
end;
end;
the following code worked to me. It downloads the file by chunks:
procedure Download(Url,LocalFile:String;
WorkBegin:TWorkBeginEvent;Work:TWorkEvent;WorkEnd:TWorkEndEvent);
var
Http: TIdHTTP;
quit:Boolean;
FLength,aRangeEnd:Integer;
begin
Http := TIdHTTP.Create(nil);
fFileStream:=nil;
try
try
Http.OnWork:= Work;
Http.OnWorkEnd := WorkEnd;
Http.Head(Url);
FLength := Http.Response.ContentLength;
quit:=false;
repeat
if not FileExists(LocalFile) then begin
fFileStream := TFileStream.Create(LocalFile, fmCreate);
end
else begin
fFileStream := TFileStream.Create(LocalFile, fmOpenReadWrite);
quit:= fFileStream.Size >= FLength;
if not quit then
fFileStream.Seek(Max(0, fFileStream.Size-4096), soFromBeginning);
end;
try
aRangeEnd:=fFileStream.Size + 50000;
if aRangeEnd < fLength then begin
Http.Request.Range := IntToStr(fFileStream.Position) + '-'+ IntToStr(aRangeEnd);
end
else begin
Http.Request.Range := IntToStr(fFileStream.Position) + '-';
quit:=true;
end;
Http.Get(Url, fFileStream);
finally
fFileStream.Free;
end;
until quit;
Http.Disconnect;
except
on E : Exception do
Begin
//Result:=False;
//AddLog(E.Message);
end;
end;
finally
Http.Free;
end;
end;
Maybe the HTTP RANGE header can help you here. Have a look at archive.org's copy of http://www.west-wind.com/Weblog/posts/244.aspx for more info on resuming HTTP downloads:
(2004-02-07) A couple of days ago somebody on the Message Board asked an interesting question about how to provide resumable HTTP downloads. My first response to this question was that this isn't possible since HTTP is a stateless protocol that has no concept of file pointers and thus can't resume an HTTP download.
However it turns out HTTP 1.1 does have the ability to specify ranges in downloads by using the Range: header in the Http header sent form the client. You can do things like:
Range: 0-10000
Range: 100000-
Range: -100000
which download the first 100000 bytes, everything over 100000 bytes or the last 100000 bytes. There are more combinations but the first two are the ones that are of interest for a resumable download.
To demonstrate this feature I used wwHTTP (in Web Connection/VFP) to download a first 400k chunk of a file into a file with HTTPGetEx which is meant to simulate an aborted download. Next I do a second request to pick up the existing file and download the remainder:
#INCLUDE wconnect.h
CLEAR
CLOSE DATA
DO WCONNECT
LOCAL o as wwHTTP
lcDownloadedFile = "d:\temp\wwipstuff.zip"
*** Simulate partial output
lcOutput = ""
Text=""
tnSize = 0
o = CREATEOBJECT("wwHTTP")
o.HttpConnect("www.west-wind.com")
? o.httpgetex("/files/wwipstuff.zip",#Text,#tnSize,"Range: bytes=0-400000"+CRLF,lcDownloadedFile)
o.Httpclose()
lcOutput = Text
? LEN(lcOutput)
*** Figure out how much we downloaded
lnOpenAt = FILESIZE(lcDownloadedFile)
*** Do a partial download starting at this byte count
Text=""
tnSize =0
o = CREATEOBJECT("wwHTTP")
o.HttpConnect("www.west-wind.com")
? o.httpgetex("/files/wwipstuff.zip",#Text,#tnSize,"Range: bytes=" + TRANSFORM(lnOpenAt) + "-" + CRLF)
o.Httpclose()
? LEN(Text)
*** Read the existing partial download and append current download
lcOutput = FILETOSTR(lcDownloadedFile) + TEXT
? LEN(lcOutput)
STRTOFILE(lcOutput,lcDownloadedFile)
RETURN
Note that this approach uses a file on disk, so you have to use HTTPGetEx (with Web Connection). The second download can also be done to disk if you choose, but things will get tricky if you have multiple aborts and you need to piece them together. In that case you might want to try to keep track of each file and add a number to it, then combine the result at the very end.
If you download to memory using WinInet (which is what wwHTTP uses behind the scenes) you can also try to peel out the file from the Temporary Internet Files cache. Although this works I suspect this process will become very convoluted quickly so if you plan on providing the ability to resume I would highly recommend that you write your output to file yourself using the approach above.
Some additional information on WinInet and some of the requirements for this approach to work with it are described here: http://www.clevercomponents.com/articles/article015/resuming.asp.
The same can be done with wwHTTP for .Net by adding the Range header to the wwHTTP:WebRequest.Headers object.
(Randy Pearson) Say you don't know what the file size is at the server. Is there a way to find this out, so you can know how many chunks to request, for example? Would you send a HEAD request first, or does the header of the GET response tell you the total size also?
(Rick Strahl) You have to read the Content-Length: header to get the size of the file downloaded. If you're resuming this shouldn't matter - you just use Range: (existingsize)- to get the rest. For chunky downloads you can read the content length and only download the first x bytes. This gets tricky with wwHTTP - you have to make individual calls with HTTPGetEx and set the tnBufferSize parameter to the chunk size to retrieve to have it stop after the size is reached.
(Randy Pearson) Follow-up: It looks like a compliant server would send you enough to know the size. If it provides chunks it should reply with something like:
Content-Range: 0-10000/85432
so you could (if desired) extract that and use it in a loop to continue with intelligent chunk requests.
Also look here https://forums.embarcadero.com/message.jspa?messageID=219481 for TIdHTTP related discussion on the same topic:
(at least partly as per tfilestream.seek and offset confusion)
if FileExists(dstFile) then
begin
Fs := TFileStream.Create(dstFile, fmOpenReadWrite);
try
Fs.Seek(Max(0, Fs.Size-1024), soFromBeginning);
// alternatively:
// Fs.Seek(-1024, soFromEnd);
Http.Request.Range := IntToStr(Fs.Position) + '-';
Http.Get(Url, Fs);
finally
Fs.Free;
end;
end;

Resources