I have recently started using DataSnap in Delphi to produce what will be a RESTful web service. After following guides by the man himself Marco Cantu and several others on the internet I have successfully got the whole 'chain' working.
But there is a small issue of speed; the client can now send a stream (along with it's size) to the server (which, because of the bug here DataSnap XE2 and TStream method parameters, is read upto the sent size), and the server will reassemble it into a file and save it on disk.
But foir a 3.66MiB file, this takes over 50 seconds!
Should this be the case? On the server I have:
try
F := TFileStream.Create('written.dat', fmCreate);
F.Position := 0;
F.CopyFrom(Data, DataSize);
finally
F.Free;
And on the client end:
var
Server: TServerMethods1Client;
DBStream: TFileStream;
begin
Server := TServerMethods1Client.Create(SQLConnection1.DBXConnection);
try
DBStream := TFileStream.Create('DataSnapServer.exe', fmOpenRead);
DBStream.Position := 0;
Showmessage(IntToStr(Server.SendData(DBStream, DBStream.Size)));
finally
Server.Free;
Any help appreciated!
Cheers,
Adrian
On server side, try adjusting BufferKBSize property on TDSHTTPWebDispatcher component. Same property can be found on TsqlConnection component on client.
Related
I'm using Delphi 7 with Indy 10.6.2.5459 to make a POST request to a server from time to time:
sParams := '?ultimaAlteracao='+FormatDateTime('YYYY-MM-DD',now())+'T00:00:00.000';
FidHTTP := TIdHTTP.Create;
try
FidHTTP.Request.Clear;
FidHTTP.Request.Accept := 'application/json;charset=UTF-8';
FidHTTP.Request.CharSet := 'UTF-8';
FidHTTP.Response.ContentType := 'application/json;charset=UTF-8';
FidHTTP.Response.ContentEncoding := 'UTF-8';
FidHTTP.Request.CustomHeaders.AddValue('Authorization','Basic '+EncodeBase64('xx:xx#123456'));
JsonStream := TStringStream.Create('');
JsonStream.Position := 0;
try
FidHTTP.Get('https://'+server+'/LinxImportacaoArquivo/GetSelecaoDadosMDMNCM'+sParams, JsonStream);
DataString := JsonStream.DataString;
finally
FreeAndNil(JsonStream);
end;
finally
FreeAndNil(FidHTTP);
end;
After sometime, it begins to fail and returns:
Error creating SSL context. error:140A90F1:SSL
routines:SSL_CTX_new:unable to load ssl2 md5 routines
If I restart the application, it works for sometime again.
I found some people who had the same issue: here and here
From what I learned, it can be some other process which uses Indy to make a request that is interfering, and I quote
It appears to be an issue with some type of static member
initialization inside the openssl library. I have 2 libraries, both of
them use the openssl library, let's call them A and B. When the
application starts up both A & B are able to successfully create a
security context. Later, when library B tries to create another
security context it fails. Both library A and B are module plugins to
our application so they both will load but if one is not needed it is
unloaded.
Indeed, my application has a lot of others process executing, making http requests, through indy10 or WinInet.
So, my question is: Is there some procedure I can call on Indy10 to make it initialize something that had it freed on opensll library?
I found out that at unit IdSSLOpenSSL there's a procedure named UnLoadOpenSSLLibrary.
If I always call that procedure before my code, the error does not occurr.
I am using datasnap technology where I got a client a server application.
The Server gets the data from database and puts into stream which is then fetched at client side.
I keep getting a "Missing Data Provider or Data Packet" error on the client side code.
Server side code:
function GetFiles(ClientID: integer): TStream;
var
CDS: TClientDataSet;
begin
try
Result := TMemoryStream.Create;
CDS := TClientDataSet.Create(nil);
with adsFiles do // ads is adodataset and dspFiles is the dataset provider which has its dataset property set to adsFiles
begin
Close;
Parameters.ParamByName('ClientID').Value := ClientID;
Open;
CDS.Data := dspFiles.Data;
CDS.Open;
CDS.SaveToStream(Result);
Result.Position := 0;
end;
finally
CDS.Free;
end;
Client Side Code:
procedure ExportData;
var
StreamData: TStream;
begin
StreamData := SvrMethodClass.GetFiles(AClientID);
StreamData.Seek(0,soFromBeginning);
StreamData.Position:= 0;
cdsClientFiles.LoadFromStream(StreamData); // getting the error message "Missing Data Provider or Data Packet"
cdsClientFiles.Open;
end;
Client side I have dropped a clientdataset component and trying to load the data into this dataset where the issue is raised any help pointing me where I am going wrong would be really appreciated. Thanks
similar question: Streaming TClientDataSet using Datasnap in Delphi XE6
checked the solution but it did not work out
I spent a lot of time investigating the problems with returning data as a stream from a datasnap server - see e.g. Can't retrieve TStreams bigger than around 260.000 bytes from a Datasnap Server, but was unable to come up with a satisfactory solution for stream data.
However, experiments revealed that a similar problem does not occur with string data (I tested strings up to several hundred megabytes). So, I would suggest you try sending your binary/blob data from the Datasnap server as a Base64-encoded string.
Of course, if a string representation of your ADO or CDS data is adequate for what you want, one way to achieve that would be to do an adsFile.SaveToFile(..., pfXML) or CDS.SaveToFile(... dfXML) in the server method, then return the resulting file as a string.
I am trying to accept file uploads in a Delphi 7 Webbroker CGI.
I'm using Shiv Kumar's TMsMultipartParser, but I have a problem with Chrome. I can't access the parsed data (surprisingly, Explorer works fine).
This is my code:
with TMsMultipartFormParser.Create do
begin
Parse(Request);
lsExternalID:=ContentFields.Values['external_id'];
if (lsExternalID='') then
raise Exception.Create('No external ID');
for i := 0 to Files.Count -1 do
begin
lsFileName:=files[i].FileName;
//Rename file using external ID (not included for simplicity)
Files[i].SaveToFile(lsFilename);
end;
Response.Content := 'OK';
free;
end;
As suggested here, I tried to use http://www.mrsoft.org/Delphi/MultipartParser.pas but I can't compile it. It uses a unit called UniversalUtils that I can't find anywhere.
I know this is a very obsolete technology. Almost all references to it have already disappeared from the web (believe me, I have searched). Buy any help would be deeply appreciated.
Thanks.
I finally solved my problem, thanks to #mrabat.
This project started in Delphi 5. It was later upgraded to Delphi 7 (it can't be upgraded further, because many parts can't support Unicode strings, we use ANSI).
We were using Shiv's TMsMultipartParser because Delphi 5 didn't have any parser included.
Delphi 7 has TMultipartContentParser in unit ReqMulti.pas, and it works perfectly.
For anyone that need an example, I'll post my working code:
with TMultipartContentParser.Create(Request) do
begin
lsExternalID:=ContentFields.Values['external_id'];
if (lsExternalID='') then
raise Exception.Create('No external ID');
for i := 0 to Request.Files.Count -1 do
begin
lsFileName:=Request.Files[i].FileName;
//Rename file using external ID (not included for simplicity)
TMemoryStream(Request.Files[i].Stream).SaveToFile(lsFilename);
end;
Response.Content := 'OK';
Free;
end;
I wrote something similar once here:
https://github.com/stijnsanders/xxm/blob/master/Delphi/common/xxmParams.pas#L159
but that may be tightly coupled with SplitHeaderValue that parses the header lines, and TStreamNozzle that throttles incoming data. (and TXxmReqPar... objects, and IXxmContext...)
(Of course you're warmly welcomed to accept file uploads with xxm...)
I have a small piece of code that checks if a computer is alive by pinging it. We use to have a room with 40 computer and I wanna check remotely through my program which on is alive.
Therefore I wrote a little ping function using indy
function TMainForm.Ping(const AHost : string) : Boolean;
var
MyIdIcmpClient : TIdIcmpClient;
begin
Result := True;
MyIdIcmpClient := TIdIcmpClient.Create(nil);
MyIdIcmpClient.ReceiveTimeout := 200;
MyIdIcmpClient.Host := AHost;
try
MyIdIcmpClient.Ping;
Application.ProcessMessages;
except
Result := False;
MyIdIcmpClient.Free;
Exit;
end;
if MyIdIcmpClient.ReplyStatus.ReplyStatusType <> rsEcho Then result := False;
MyIdIcmpClient.Free;
end;
So I've developped that at home on my wifi network and everthing just work fine.
When I get back to work I tested and I get an error saying
Socket Errod # 10040 Message too long
At work we have fixed IPs and all the computer and I are in the same subnet.
I tried to disconnect from the fixed IP and connect to the wifi which of course is DHCP and not in the same subnet, and it is just working fine.
I have tried searching the internet for this error and how to solve it but didn't find much info.
Of course I have tried to change the default buffer size to a larger value but it didn't change anything I still get the error on the fixed IP within same subnet.
Moreover, I don't know if this can help finding a solution, but my code treats exceptions, but in that case it takes about 3-4 seconds to raise the error whereas the Timeout is set to 200 milliseconds. And I cannot wait that long over each ping.
By the way I use delphi 2010 and I think it is indy 10. I also have tested on XE2 but same error.
Any idea
----- EDIT -----
This question is answered, now I try to have this running in multithread and I have asked another question for that
Delphi (XE2) Indy (10) Multithread Ping
Set the PacketSize property to 24:
function TMainForm.Ping(const AHost : string) : Boolean;
var
MyIdIcmpClient : TIdIcmpClient;
begin
Result := True;
MyIdIcmpClient := TIdIcmpClient.Create(self);
MyIdIcmpClient.ReceiveTimeout := 200;
MyIdIcmpClient.Host := AHost;
MyIdIcmpClient.PacketSize := 24;
MyIdIcmpClient.Protocol := 1;
MyIdIcmpClient.IPVersion := Id_IPv4;
try
MyIdIcmpClient.Ping;
// Application.ProcessMessages; // There's no need to call this!
except
Result := False;
Exit;
end;
if MyIdIcmpClient.ReplyStatus.ReplyStatusType <> rsEcho Then result := False;
MyIdIcmpClient.Free;
end;
For XE5 and Indy10 this is still a problem, even with different Packet Size.
To answer the more cryptical fix:
ABuffer := MyIdIcmpClient1.Host + StringOfChar(' ', 255);
This is a "magic" fix to get around the fact that there is a bug in the Indy10 component (if I have understood Remy Lebeau right).
My speculation is that this has some connection with the size of the receive buffer. To test my theory I can use any character and don't need to include the host address at all. Only use as many character you need for the receive buffer. I use this small code (C++ Builder XE5) to do a Ping with great success (all other values at their defaults):
AnsiString Proxy = StringOfChar('X',IcmpClient->PacketSize);
IcmpClient->Host = Host_Edit->Text;
IcmpClient->Ping(Proxy);
As you can see I create a string of the same length as the PacketSize property. What you fill it with is insignificant.
Maybe this can be of help to #RemyLebeau when he work on the fix.
use this code
ABuffer := MyIdIcmpClient1.Host + StringOfChar(' ', 255);
MyIdIcmpClient.Ping(ABuffer);
Actually i'm using the TIdHTTP component for download a file from internet. i'm wondering if is possible pause and resume the download using this component o maybe another indy component.
this is my current code, this works ok for download a file (without resume), but . now i want pause the download close my app ,and when my app restart then resume the download from the last position saved.
var
Http: TIdHTTP;
MS : TMemoryStream;
begin
Result:= True;
Http := TIdHTTP.Create(nil);
MS := TMemoryStream.Create;
try
try
Http.OnWork:= HttpWork;//this event give me the actual progress of the download process
Http.Head(Url);
FSize := Http.Response.ContentLength;
AddLog('Downloading File '+GetURLFilename(Url)+' - '+FormatFloat('#,',FSize)+' Bytes');
Http.Get(Url, MS);
MS.SaveToFile(LocalFile);
except
on E : Exception do
Begin
Result:=False;
AddLog(E.Message);
end;
end;
finally
Http.Free;
MS.Free;
end;
end;
the following code worked to me. It downloads the file by chunks:
procedure Download(Url,LocalFile:String;
WorkBegin:TWorkBeginEvent;Work:TWorkEvent;WorkEnd:TWorkEndEvent);
var
Http: TIdHTTP;
quit:Boolean;
FLength,aRangeEnd:Integer;
begin
Http := TIdHTTP.Create(nil);
fFileStream:=nil;
try
try
Http.OnWork:= Work;
Http.OnWorkEnd := WorkEnd;
Http.Head(Url);
FLength := Http.Response.ContentLength;
quit:=false;
repeat
if not FileExists(LocalFile) then begin
fFileStream := TFileStream.Create(LocalFile, fmCreate);
end
else begin
fFileStream := TFileStream.Create(LocalFile, fmOpenReadWrite);
quit:= fFileStream.Size >= FLength;
if not quit then
fFileStream.Seek(Max(0, fFileStream.Size-4096), soFromBeginning);
end;
try
aRangeEnd:=fFileStream.Size + 50000;
if aRangeEnd < fLength then begin
Http.Request.Range := IntToStr(fFileStream.Position) + '-'+ IntToStr(aRangeEnd);
end
else begin
Http.Request.Range := IntToStr(fFileStream.Position) + '-';
quit:=true;
end;
Http.Get(Url, fFileStream);
finally
fFileStream.Free;
end;
until quit;
Http.Disconnect;
except
on E : Exception do
Begin
//Result:=False;
//AddLog(E.Message);
end;
end;
finally
Http.Free;
end;
end;
Maybe the HTTP RANGE header can help you here. Have a look at archive.org's copy of http://www.west-wind.com/Weblog/posts/244.aspx for more info on resuming HTTP downloads:
(2004-02-07) A couple of days ago somebody on the Message Board asked an interesting question about how to provide resumable HTTP downloads. My first response to this question was that this isn't possible since HTTP is a stateless protocol that has no concept of file pointers and thus can't resume an HTTP download.
However it turns out HTTP 1.1 does have the ability to specify ranges in downloads by using the Range: header in the Http header sent form the client. You can do things like:
Range: 0-10000
Range: 100000-
Range: -100000
which download the first 100000 bytes, everything over 100000 bytes or the last 100000 bytes. There are more combinations but the first two are the ones that are of interest for a resumable download.
To demonstrate this feature I used wwHTTP (in Web Connection/VFP) to download a first 400k chunk of a file into a file with HTTPGetEx which is meant to simulate an aborted download. Next I do a second request to pick up the existing file and download the remainder:
#INCLUDE wconnect.h
CLEAR
CLOSE DATA
DO WCONNECT
LOCAL o as wwHTTP
lcDownloadedFile = "d:\temp\wwipstuff.zip"
*** Simulate partial output
lcOutput = ""
Text=""
tnSize = 0
o = CREATEOBJECT("wwHTTP")
o.HttpConnect("www.west-wind.com")
? o.httpgetex("/files/wwipstuff.zip",#Text,#tnSize,"Range: bytes=0-400000"+CRLF,lcDownloadedFile)
o.Httpclose()
lcOutput = Text
? LEN(lcOutput)
*** Figure out how much we downloaded
lnOpenAt = FILESIZE(lcDownloadedFile)
*** Do a partial download starting at this byte count
Text=""
tnSize =0
o = CREATEOBJECT("wwHTTP")
o.HttpConnect("www.west-wind.com")
? o.httpgetex("/files/wwipstuff.zip",#Text,#tnSize,"Range: bytes=" + TRANSFORM(lnOpenAt) + "-" + CRLF)
o.Httpclose()
? LEN(Text)
*** Read the existing partial download and append current download
lcOutput = FILETOSTR(lcDownloadedFile) + TEXT
? LEN(lcOutput)
STRTOFILE(lcOutput,lcDownloadedFile)
RETURN
Note that this approach uses a file on disk, so you have to use HTTPGetEx (with Web Connection). The second download can also be done to disk if you choose, but things will get tricky if you have multiple aborts and you need to piece them together. In that case you might want to try to keep track of each file and add a number to it, then combine the result at the very end.
If you download to memory using WinInet (which is what wwHTTP uses behind the scenes) you can also try to peel out the file from the Temporary Internet Files cache. Although this works I suspect this process will become very convoluted quickly so if you plan on providing the ability to resume I would highly recommend that you write your output to file yourself using the approach above.
Some additional information on WinInet and some of the requirements for this approach to work with it are described here: http://www.clevercomponents.com/articles/article015/resuming.asp.
The same can be done with wwHTTP for .Net by adding the Range header to the wwHTTP:WebRequest.Headers object.
(Randy Pearson) Say you don't know what the file size is at the server. Is there a way to find this out, so you can know how many chunks to request, for example? Would you send a HEAD request first, or does the header of the GET response tell you the total size also?
(Rick Strahl) You have to read the Content-Length: header to get the size of the file downloaded. If you're resuming this shouldn't matter - you just use Range: (existingsize)- to get the rest. For chunky downloads you can read the content length and only download the first x bytes. This gets tricky with wwHTTP - you have to make individual calls with HTTPGetEx and set the tnBufferSize parameter to the chunk size to retrieve to have it stop after the size is reached.
(Randy Pearson) Follow-up: It looks like a compliant server would send you enough to know the size. If it provides chunks it should reply with something like:
Content-Range: 0-10000/85432
so you could (if desired) extract that and use it in a loop to continue with intelligent chunk requests.
Also look here https://forums.embarcadero.com/message.jspa?messageID=219481 for TIdHTTP related discussion on the same topic:
(at least partly as per tfilestream.seek and offset confusion)
if FileExists(dstFile) then
begin
Fs := TFileStream.Create(dstFile, fmOpenReadWrite);
try
Fs.Seek(Max(0, Fs.Size-1024), soFromBeginning);
// alternatively:
// Fs.Seek(-1024, soFromEnd);
Http.Request.Range := IntToStr(Fs.Position) + '-';
Http.Get(Url, Fs);
finally
Fs.Free;
end;
end;