I'm trying to transfer some big streams (~1Mb) between DataSnap server/client but to no avail. I'm trying to understand the code of Jim Tierney (http://blogs.embarcadero.com/jimtierney/2009/04/06/31461) with no luck and i can't even compile the code because of a missing library, anyway ...
The max size of a stream i`m able to receive is 64k, so any tips/ideas/code samples you can provide for a weekend programmer like me will be very welcomed. Thank you!
my server code:
function TsrvMethods.getStream(iCount: integer): TStream;
begin
Result := dummyStream('0123456789', iCount);
end;
function dummyStream(sCnt: string; iCount: integer): TStream;
begin
Result := TMemoryStream.Create;
while iCount > 1 do begin
Result.Write(Pointer(sCnt)^, Length(sCnt));
Dec(iCount);
end;
Result.Seek(0, TSeekOrigin.soBeginning);
end;
my client calling code:
procedure TfrmMain.butStreamClick(Sender: TObject);
var
sStr : TStream;
begin
cycleConnection; //make sure we have an active connection
with TsrvMethodsClient.Create( SQLConn.DBXConnection, False ) do begin
sStr := getStream( Integer(SpinCount.Value) );
Free;
end;
FreeAndNil(sStr);
end;
Actually, i think i`ve got it. I'm posting this as an answer maybe somebody else need this.
procedure TfrmMain.butStreamClick(Sender: TObject);
const
iBufSize = 128;
var
sStr : TStream;
sMem : TMemoryStream;
buf: PByte;
iRead: integer;
begin
cycleConnection;
with TsrvMethodsClient.Create( SQLConn.DBXConnection, False ) do begin
sStr := getStream( 500000 ); //500k stream
GetMem(buf, iBufSize);
sMem := TMemoryStream.Create;
try
repeat
iRead := sStr.Read( Pointer(buf)^, iBufSize);
if iRead > 0 then sMem.WriteBuffer( Pointer(buf)^, iRead);
if iRead < iBufSize then break;
until iRead < iBufSize;
finally
FreeMem(buf, iBufSize);
end;
Free;
end;
FreeAndNil(sStr);
FreeAndNil(sMem);
end;
P.S.
Searching through DataSnap code samples i`ve found that one (speed related) improvement would be to have iBufSize set to 61440 (or equivalent hex value $F000) which seems to be the biggest size can be received in one go. If receiving stream is bigger then reported size will be -1 and the code above is needed to read the entire stream.
Related
I am in need of passing a couple things using Data Snap in Delphi XE6 (TStream & TClientdataSet). Lets start with a TStream - maybe what I learn here, I can figure out the TClientDataSet.
Here is my attempt, but it throws an error:
Remote error: Access violaion at address 0040801C in module DSServer.exe
Client demo (DSClient.exe)
//RunReportObj is the real object I will be passing to the server method
Procedure TForm8.Button1Click(Sender: TObject);
var
RunReportObj: TRunReportObject;
S: TStream;
FS: TFileStream;
begin
RunReportObj:= TRunReportObject.Create;
RunReportObj.ID:= '10101';
RunReportObj.ReportName:= 'Test';
RunReportObj.ExportType:= 'PDF';
S:= TStream.Create;
try
S:= ClientModule1.ServerMethods1Client.GetReport(RunReportObj);
S.Seek(0,soFromBeginning);
FS:= TFileStream.Create(RunReportObj.ReportName + '.' + RunReportObj.ExportType, fmOpenWrite);;
try
FS.CopyFrom(S, S.Size);
finally
FS.Free;
end;
finally
S.Free;
end;
end;
ClientClassesUnit1.pas
function TServerMethods1Client.GetReport(RunReportObj: TRunReportObject): TStream;
begin
if FGetReportCommand = nil then
begin
FGetReportCommand := FDBXConnection.CreateCommand;
FGetReportCommand.CommandType := TDBXCommandTypes.DSServerMethod;
FGetReportCommand.Text := 'TServerMethods1.GetReport';
FGetReportCommand.Prepare;
end;
if not Assigned(RunReportObj) then
FGetReportCommand.Parameters[0].Value.SetNull
else
begin
FMarshal := TDBXClientCommand(FGetReportCommand.Parameters[0].ConnectionHandler).GetJSONMarshaler;
try
FGetReportCommand.Parameters[0].Value.SetJSONValue(FMarshal.Marshal(RunReportObj), True);
if FInstanceOwner then
RunReportObj.Free
finally
FreeAndNil(FMarshal)
end
end;
FGetReportCommand.ExecuteUpdate;
Result := FGetReportCommand.Parameters[1].Value.GetStream(FInstanceOwner);
end;
Server demo (DSServer.exe)
//not really doing anything with the RunReportObj yet,
// just trying to test whether or not I can pass a TStream back first
function TServerMethods1.GetReport(RunReportObj: TRunReportObject): TStream;
var
Stream: TMemoryStream;
Writer: TBinaryWriter;
Bytes: TBytes;
begin
result := TMemoryStream.Create;
try
Writer := TBinaryWriter.Create(result);
try
Writer.Write(TEncoding.UTF8.GetBytes('Hello World' + sLineBreak));
finally
Writer.Free;
end;
finally
Stream.Free;
end;
end;
I'm sure I did something foolish :)
You have to take care who is responsible for freeing objects sent with DataSnap. TServerMethods1.GetReport() should not free the Result, as it has to be sent to the client first. On the other side, the client should not free the TStream it gets from TServerMethods1Client.GetReport(), as long as FInstanceOwner is true (which it is by default).
The first condition is fulfilled more by accident, although as David pointed out, you are freeing the uninitialized local variable Stream.
Without being able to actually test this in the moment, the correct code for the client should look like:
Procedure TForm8.Button1Click(Sender: TObject);
var
RunReportObj: TRunReportObject;
S: TStream;
FS: TFileStream;
begin
RunReportObj:= TRunReportObject.Create;
RunReportObj.ID:= '10101';
RunReportObj.ReportName:= 'Test';
RunReportObj.ExportType:= 'PDF';
S:= ClientModule1.ServerMethods1Client.GetReport(RunReportObj);
S.Seek(0,soFromBeginning);
FS:= TFileStream.Create(RunReportObj.ReportName + '.' + RunReportObj.ExportType, fmOpenWrite);;
try
FS.CopyFrom(S, S.Size);
finally
FS.Free;
end;
end;
And for the server side:
function TServerMethods1.GetReport(RunReportObj: TRunReportObject): TStream;
var
Writer: TBinaryWriter;
Bytes: TBytes;
begin
result := TMemoryStream.Create;
Writer := TBinaryWriter.Create(result);
try
Writer.Write(TEncoding.UTF8.GetBytes('Hello World' + sLineBreak));
finally
Writer.Free;
end;
end;
I have searched high and low for a working solution for sending an image (e.g. tpngimage) to the server using a datasnap method - but I cannot make it to work.
When I load the image and save it to a memorystream, I'm able to read the image back from the stream - locally in the client, which is not really a surprise. But when the server method gets called, the stream is nil and contains nothing, the other parameters are fine (simple datatype and an object).
Did I miss something obvious here? I was under the impression that TStream is a valid datatype for datasnap methods, but maybe I'm wrong?
The from client side it looks like this.
function TPerson.Update: Boolean;
var
AStream : TMemoryStream;
APicture : TPngImage;
ASize : Integer;
begin
if (FId > 0) then // if Id below zero we have a problem
begin
ClientModule1.ServerMethods1Client.UpdatePerson(Self);
APicture := TPngImage.Create;
AStream := TMemoryStream.Create;
try
// Temp just use a file
AStream.LoadFromFile('.\images\075.png');
ASize := AStream.Size;
AStream.Position := 0; // wind back if needed
// just for testing, we can read back the image from the stream
APicture.LoadFromStream(AStream);
ClientModule1.ServerMethods1Client.UpdatePersonPicture(self, ASize, AStream);
finally
FreeAndNil(AStream);
FreeAndNil(APicture);
end;
end;
FModified := False;
end;
And the proxy method looks like this
procedure TServerMethods1Client.UpdatePersonPicture(APerson: TPerson; ASize: Integer; APictureStream: TMemoryStream);
begin
if FUpdatePersonPictureCommand = nil then
begin
FUpdatePersonPictureCommand := FDBXConnection.CreateCommand;
FUpdatePersonPictureCommand.CommandType := TDBXCommandTypes.DSServerMethod;
FUpdatePersonPictureCommand.Text := 'TServerMethods1.UpdatePersonPicture';
FUpdatePersonPictureCommand.Prepare;
end;
if not Assigned(APerson) then
FUpdatePersonPictureCommand.Parameters[0].Value.SetNull
else
begin
FMarshal := TDBXClientCommand(FUpdatePersonPictureCommand.Parameters[0].ConnectionHandler).GetJSONMarshaler;
try
FUpdatePersonPictureCommand.Parameters[0].Value.SetJSONValue(FMarshal.Marshal(APerson), True);
if FInstanceOwner then
APerson.Free
finally
FreeAndNil(FMarshal)
end
end;
FUpdatePersonPictureCommand.Parameters[1].Value.SetInt32(ASize);
FUpdatePersonPictureCommand.Parameters[2].Value.SetStream(APictureStream, FInstanceOwner);
FUpdatePersonPictureCommand.ExecuteUpdate;
end;
The Server method looks like this - it fails due to the APictureStream is nil.
procedure TServerMethods1.UpdatePersonPicture(APerson: TPerson; ASize: integer;
APictureStream: TMemoryStream);
var
APicture : TPngImage;
begin
fdqPersons.Close;
fdqPersons.SQL.Clear;
fdqPersons.Connection.StartTransaction;
try
fdqPersons.SQL.Add('update Persons set Picture=:Picture ');
fdqPersons.SQL.Add('where Id=:Id');
fdqPersons.ParamByName('Id').Value := APerson.Id;
APicture := TPngImage.Create;
try
// APicture for testing - but APictureStream is nil!
APicture.LoadFromStream(APictureStream);
fdqPersons.ParamByName('Picture').Assign(APicture);
fdqPersons.ExecSQL;
finally
FreeAndNil(APicture);
end;
fdqPersons.Close;
fdqPersons.Connection.Commit;
LogEvent(format('Person picture updated ID: %d',[APerson.id]));
except
on e:exception do
begin
fdqPersons.Connection.Rollback;
LogEvent(format('Error updating person picture %s',[e.Message]));
raise;
end;
end;
end;
When you call APicture.LoadFromStream(AStream); the stream's position goes to the end, and therefore when passing it into ClientModule1 it does not have any data left to be read. Either get rid of the unnecessary portion where you write the stream to a TPngImage or reset the stream's position back to 0 just after that part.
I am trying to use the system.filesize function to get the size of a file in delphi, it works ok for files < 4GB but fails for files > 4GB.
so i implemented my own that opens the required file as a filestream and gets the streamsize which works perfectly.
Here is a Snippet
function GiveMeSize(PathtoFile : string): int64;
var
stream : TFileStream;
size : int64;
begin
try
stream := TFileStream.Create(PathtoFile, fmOpenReadWrite or fmShareDenyNone);
size := stream.size;
except
showmessage('Unable to get FileSize');
end
finally
stream.free;
end;
but the problem with my above function is that it opens the file which incurs some overhead when processing a large number of files.
is there any function that can get filesize of files > 4GB without opening the file first?
I have tried some functions online but they tend to report wrong file size for files greater than 4GB.
Delphi Version : XE5
Thanks.
System.FileSize is a Pascal I/O function that operates on Pascal File variables. If you want to get the size of a file specified by path, then System.FileSize is simply wrong function to use.
What's more, you quite likely don't want to open the file just to obtain its size. I obtain the file size like this:
function FileSize(const FileName: string): Int64;
var
AttributeData: TWin32FileAttributeData;
begin
if GetFileAttributesEx(PChar(FileName), GetFileExInfoStandard, #AttributeData) then
begin
Int64Rec(Result).Lo := AttributeData.nFileSizeLow;
Int64Rec(Result).Hi := AttributeData.nFileSizeHigh;
end
else
Result := -1;
end;
Googling for the keywords "delphi get file size int64" gives you plenty of examples
I use this:
function GetSizeOfFile(const Filename: string): Int64;
var
sr : TSearchRec;
begin
if FindFirst(fileName, faAnyFile, sr ) <> 0 then
Exit(-1);
try
result := Int64(sr.FindData.nFileSizeHigh) shl Int64(32) + Int64(sr.FindData.nFileSizeLow);
finally
System.SysUtils.FindClose(sr) ;
end;
end;
You can avoid bit-shifting by assigning into a variant record which I think makes the code below more efficient.
function GetSizeOfFile(const Filename: string): Int64;
type
TSizeType = (stDWORD, stInt64);
var
sizerec: packed record
case TSizeType of
stDWORD: (SizeLow: LongWord; SizeHigh: LongWord);
stInt64: (Size: Int64);
end;
sr : TSearchRec;
begin
if FindFirst(fileName, faAnyFile, sr ) <> 0 then
begin
Result := -1;
Exit;
end;
try
sizerec.SizeLow := sr.FindData.nFileSizeLow;
sizerec.SizeHigh := sr.FindData.nFileSizeHigh;
Result := sizerec.Size;
finally
SysUtils.FindClose(sr) ;
end;
end;
I could've just used "case Boolean of" but like to use the power of Pascal to make the code more descriptive.
I need to compare if two TStream descendant have the same content.
The only interesting result for me is the boolean Yes / No.
I'm going to code a simple loop checking byte after byte the streams content's.
But I'm curious to know if there is an already existing function. I haven't found any inside DelphiXE or JCL/JVCL libs.
Of course, the two streams have the same size !
Exactly, as Nickolay O. said you should read your stream in blocks and use CompareMem. Here is an example (including size test) ...
function IsIdenticalStreams(Source, Destination: TStream): boolean;
const Block_Size = 4096;
var Buffer_1: array[0..Block_Size-1] of byte;
Buffer_2: array[0..Block_Size-1] of byte;
Buffer_Length: integer;
begin
Result := False;
if Source.Size <> Destination.Size then
Exit;
while Source.Position < Source.Size do
begin
Buffer_Length := Source.Read(Buffer_1, Block_Size);
Destination.Read(Buffer_2, Block_Size);
if not CompareMem(#Buffer_1, #Buffer_2, Buffer_Length) then
Exit;
end;
Result := True;
end;
The IsIdenticalStreams function posted by daemon_x is excellent - but needs one adjustment to work properly. (Uwe Raabe caught the issue already.) It is critical that you reset the stream positions before starting the loop - or this procedure will probably return an incorrect TRUE if the two streams were already accessed outside this function.
This is the final solution that works every time. I just renamed the function to suit my naming conventions. Thank you daemon_x for the elegant solution.
function StreamsAreIdentical(Stream1, Stream2: TStream): boolean;
const
Block_Size = 4096;
var
Buffer_1: array[0..Block_Size-1] of byte;
Buffer_2: array[0..Block_Size-1] of byte;
Buffer_Length: integer;
begin
Result := False;
if Stream1.Size <> Stream2.Size then exit;
// These two added lines are critical for proper operation
Stream1.Position := 0;
Stream2.Position := 0;
while Stream1.Position < Stream1.Size do
begin
Buffer_Length := Stream1.Read(Buffer_1, Block_Size);
Stream2.Read(Buffer_2, Block_Size);
if not CompareMem(#Buffer_1, #Buffer_2, Buffer_Length) then exit;
end;
Result := True;
end;
There is no such built-in function. Only one thing I can recommend - read not byte-to-byte, but using blocks of 16-64kbytes, that would be much faster.
Answers from user532231 and Mike are working in 99% cases, but there are additional checks to be made!
Descendants of TStream can be almost anything, so it's not guaranteed that Stream.Read will return same amount of data, even if streams are of the same length (stream descendant can also download data, so may return readed=0 bytes, while waiting for next chunk). Streams can be also on completelly different media and stream read error could occur on just one.
For 100% working code all these checks should be made. I modified the function from Mike.
If this function is used for example to rewrite stream 2 if not identical to Stream1, all errors should be checked. When function result is True, everthing is ok, but if it is False, it would be very smart to check if Streams are actually different or just some error occured.
Edited: Added some additional checks, FilesAreIdentical function based on StreamsAreIdentical and usage example.
// Usage example
var lError: Integer;
...
if FilesAreIdentical(lError, 'file1.ext', 'file2.ext')
then Memo1.Lines.Append('Files are identical.')
else case lError of
0: Memo1.Lines.Append('Files are NOT identical!');
1: Memo1.Lines.Append('Files opened, stream read exception raised!');
2: Memo1.Lines.Append('File does not exist!');
3: Memo1.Lines.Append('File open exception raised!');
end; // case
...
// StreamAreIdentical
function StreamsAreIdentical(var aError: Integer;
const aStream1, aStream2: TStream;
const aBlockSize: Integer = 4096): Boolean;
var
lBuffer1: array of byte;
lBuffer2: array of byte;
lBuffer1Readed,
lBuffer2Readed,
lBlockSize: integer;
begin
Result:=False;
aError:=0;
try
if aStream1.Size <> aStream2.Size
then Exit;
aStream1.Position:=0;
aStream2.Position:=0;
if aBlockSize>0
then lBlockSize:=aBlockSize
else lBlockSize:=4096;
SetLength(lBuffer1, lBlockSize);
SetLength(lBuffer2, lBlockSize);
lBuffer1Readed:=1; // just for entering while
while (lBuffer1Readed > 0) and (aStream1.Position < aStream1.Size) do
begin
lBuffer1Readed := aStream1.Read(lBuffer1[0], lBlockSize);
lBuffer2Readed := aStream2.Read(lBuffer2[0], lBlockSize);
if (lBuffer1Readed <> lBuffer2Readed) or ((lBuffer1Readed <> lBlockSize) and (aStream1.Position < aStream1.Size))
then Exit;
if not CompareMem(#lBuffer1[0], #lBuffer2[0], lBuffer1Readed)
then Exit;
end; // while
Result:=True;
except
aError:=1; // stream read exception
end;
end;
// FilesAreIdentical using function StreamsAreIdentical
function FilesAreIdentical(var aError: Integer;
const aFileName1, aFileName2: String;
const aBlockSize: Integer = 4096): Boolean;
var lFileStream1,
lFilestream2: TFileStream;
begin
Result:=False;
try
if not (FileExists(aFileName1) and FileExists(aFileName2))
then begin
aError:=2; // file not found
Exit;
end;
lFileStream1:=nil;
lFileStream2:=nil;
try
lFileStream1:=TfileStream.Create(aFileName1, fmOpenRead or fmShareDenyNone);
lFileStream2:=TFileStream.Create(aFileName2, fmOpenRead or fmShareDenyNone);
result:=StreamsAreIdentical(aError, lFileStream1, lFileStream2, aBlockSize);
finally
if lFileStream2<>nil
then lFileStream2.Free;
if lFileStream1<>nil
then lFileStream1.Free;
end; // finally
except
aError:=3; // file open exception
end; // except
end;
I am trying to stream an XML File from Server To Client using DataSnap, with the help of ldsandon, i was able to download the sample from embarcadero, but my problem is I cannot follow it.
a pseudo of the program should work this way.
client will request from the server for selected xml file in the combobox.
the server will load the client selected xml file back to client.
i am just am trying to figure it out using delphi DataSnap, if not I will either use synapse or indy for tranferring the file, but I found Datasnap to be interesting.
could anyone help me please, a working if possible?
thanks a lot.
Please Help me, I need your help very badly.. thanks and thanks
I found this link, but I could not figure out how to convert it to TFileStream
// server side
function TServerMethods1.GetCDSXML(SQL: String; var FileSize: Integer): TStream;
begin
QryMisc.Close;
QryMisc.SQL.Text := SQL;
CDSMisc.Open;
Result := TMemoryStream.Create;
try
CDSMisc.SaveToStream(Result, dfXML);
FileSize := Result.Size; // not CDSMisc.DataSize;
Result.Position := 0; // Seek not implemented in abstract class
finally
CDSMisc.Close;
end;
end;
// client side
procedure TClientModule1.PopMiscCDS(SQL: String);
const
BufSize = $8000;
var
RetStream: TStream;
Buffer: PByte;
MemStream: TMemoryStream;
BytesRead: Integer;
FileSize: Integer;
begin
try
MemStream := TMemoryStream.Create;
GetMem(Buffer, BufSize);
try
//---------------------------------------------------------
RetStream := ServerMethods1Client.GetCDSXML(SQL, FileSize);
//---------------------------------------------------------
repeat
BytesRead := RetStream.Read(Pointer(Buffer)^, BufSize);
if BytesRead > 0 then
MemStream.WriteBuffer(Pointer(Buffer)^, BytesRead);
until BytesRead < BufSize;
if FileSize <> MemStream.Size then
raise Exception.Create('Error downloading xml');
MemStream.Seek(0, TSeekOrigin.soBeginning);
CDSMisc.Close;
CDSMisc.LoadFromStream(MemStream);
finally
FreeMem(Buffer, BufSize);
MemStream.Free;
end;
except
on E: Exception do
begin
ShowMessage(E.Message);
end;
end;
end;