Copy database to tethered client - delphi

I have created a tethered app. The server needs to copy a Sqlite db
and stream it to the client.
I get the db with this code:
procedure TfmxServer.actStreamTheDbExecute(Sender: TObject);
var
ms: TMemoryStream;
begin
ms := tmemorystream.Create;
ms := dmplanner.GetDbAsStream; // get it from the datamodule
ms.Position := 0;
thrprofServer.SendStream(thrmanServer.RemoteProfiles.First,
'Stream_TheDB', ms); // send it to the client
end;
function TdmPlanner.GetDbAsStream: TMemoryStream; // datamodule
var
fs: TFilestream;
ms: TMemoryStream;
begin
fs := tfilestream.Create(consqlite.Params.Values['Database'] , fmOpenRead);
ms := tmemorystream.Create;
try
ms.loadfromstream(fs); // ms.size = 315392, file size = (315,392 bytes
result := ms; // so I am getting the full db3 file.
result.Position := 0;
finally
freeandnil(fs);
freeandnil(ms); // does this kill the result?
end;
end;
I catch the stream and to write the db with this code:
procedure TfrmMobile_Client_Main.DoStreamTheDb(
const Aresource: TremoteResource);
var
fs: TFilestream;
ms: TMemoryStream;
begin
fs := tfilestream.Create
(dmplannerclient.consqlite.Params.Values['Database'] ,
fmopenreadwrite or fmCreate);
try
ms := TMemoryStream.Create;
ms := TMemoryStream(AResource.Value.AsStream);
ms.Position := 0; // ms.size = 315392, so I got the whole file.
ms.SaveToStream(fs);
dmPlannerClient.FillLbx(lbxRecipeNames);
// now fill a listbox, but when I open a query, I get
// [FireDAC][Phys][SQLite] ERROR: unable to open database file.
finally
freeandnil(fs);
freeandnil(ms);
end;
end;
So my question is, How do I copy the db to the client
and then use it on the client?
Better yet, How do I an in-memory db instead of an on-disk db?
I have tried setting the FDConnection filename to :memory: but that
did not work.
Delphi CE Rio 10.3.2
Thanks...Dan'l' +

I don't think there is a way to copy a Sqlite database in its entirety to a tethered
client short of copying the entire database file to the client, because it may contain
numerous tables and other resources like views, stored procs, etc.
However, copying the entire database as a file is actually quite
simple to do. In the client, you can open a table in it using a local FDConnection
and FDQuery.
Server code:
procedure TApp1Form.SendDBAsStream;
var
StreamToSend : TMemoryStream;
const
DBName = 'D:\Delphi\Code\Sqlite\DB1.Sqlite';
begin
StreamToSend := TMemoryStream.Create;
try
StreamToSend.LoadFromFile(DBName);
StreamToSend.Position := 0;
TetheringAppProfile1.Resources.FindByName('SqliteDB').Value := StreamToSend;
finally
// Don't free StreamToSend ?
end;
end;
Client code
procedure TApp2Form.TetheringAppProfile1Resources0ResourceReceived(const Sender:
TObject; const AResource: TRemoteResource);
var
ReceivedStream : TStream;
FileStream : TFileStream;
begin
FileName := ExtractFilePath(Application.ExeName) + 'Temp.Sqlite';
AResource.Value.AsStream.Position := 0;
FileStream := TFileStream.Create(FileName, fmCreate);
ReceivedStream := AResource.Value.AsStream;
try
ReceivedStream.Position := 0;
FileStream.CopyFrom(ReceivedStream, ReceivedStream.Size);
finally
FileStream.Free;
// ReceivedStream.Free; No! The tethering framework frees the stream
end;
OpenTable;
end;
procedure TApp2Form.OpenTable;
begin
if FDConnection1.Connected then
FDConnection1.Connected := False;
FDConnection1.Params.Clear;
FDConnection1.Params.Add('Database=' + FileName);
FDConnection1.DriverName := 'Sqlite';
try
FDConnection1.Connected := True;
FDQuery1.Open('select * from mytable');
except
ShowMessage(Exception(ExceptObject).Message + ' ' + FileName);
end;
end;
I tested the above in Delphi 10.2.3 on Win10 64-bit and it works fine for me.
If you wanted to copy only a few tables to the client, what I would do is
In the server, open one of the tables in an FDQuery, then assign its data to an
FDMemtable by FDMemTable1.Data := FDQuery1.Data
Call SaveToStream on FDMemTable1 and send the stream as a stream resource to the client
On the client, call FDMemTable.LoadFromStream to load the received stream. I think,
because I haven't tried it that the client would need to contain a TFDPhysSQLiteDriverLink
to support loading from the stream.

Related

TIniFile.WriteBinaryStream creates exception

In Delphi 10.4, I try to save a valid TPicture base64-encoded to an INI file:
procedure TForm1.SavePictureToIniFile(const APicture: TPicture);
var
LInput: TMemoryStream;
LOutput: TMemoryStream;
MyIni: TIniFile;
ThisFile: string;
begin
if FileSaveDialog1.Execute then
ThisFile := FileSaveDialog1.FileName
else EXIT;
LInput := TMemoryStream.Create;
LOutput := TMemoryStream.Create;
try
APicture.SaveToStream(LInput);
LInput.Position := 0;
TNetEncoding.Base64.Encode(LInput, LOutput);
LOutput.Position := 0;
MyIni := TIniFile.Create(ThisFile);
try
MyIni.WriteBinaryStream('Custom', 'IMG', LOutput); // Exception# 234
finally
MyIni.Free;
end;
finally
LInput.Free;
LOutput.Free;
end;
end;
WriteBinaryStream creates an exception:
ERROR_MORE_DATA 234 (0xEA) More data is available.
Why? What does this mean? How can this problem be solved?
EDIT: Taking into consideration what #Uwe Raabe and #Andreas Rejbrand said, this code (which does not use base64-encoding) now works:
procedure TForm1.SavePictureToIniFile(const APicture: TPicture);
var
LInput: TMemoryStream;
MyIni: System.IniFiles.TMemIniFile;
ThisFile: string;
begin
if FileSaveDialog1.Execute then
ThisFile := FileSaveDialog1.FileName
else EXIT;
LInput := TMemoryStream.Create;
try
APicture.SaveToStream(LInput);
LInput.Position := 0;
MyIni := TMemIniFile.Create(ThisFile);
try
MyIni.WriteBinaryStream('Custom', 'IMG', LInput);
MyIni.UpdateFile;
finally
MyIni.Free;
end;
finally
LInput.Free;
end;
end;
I believe this is a limitation in the operating system's functions for handling INI files; the string is too long for it.
If you instead use the Delphi INI file implementation, TMemIniFile, it works just fine. Just don't forget to call MyIni.UpdateFile at the end.
Yes, this is indeed a limitation in the Windows API, as demonstrated by the following minimal example:
var
wini: TIniFile;
dini: TMemIniFile;
begin
wini := TIniFile.Create('C:\Users\Andreas Rejbrand\Desktop\winini.ini');
try
wini.WriteString('General', 'Text', StringOfChar('W', 10*1024*1024));
finally
wini.Free;
end;
dini := TMemIniFile.Create('C:\Users\Andreas Rejbrand\Desktop\pasini.ini');
try
dini.WriteString('General', 'Text', StringOfChar('D', 10*1024*1024));
dini.UpdateFile;
finally
dini.Free;
end;
(Recall that INI files were initially used to store small amounts of configuration data in the 16-bit Windows era.)
Also, Uwe Raabe is right: you should save the Base64 string as text.

Delphi and Indy TIdFTP: Copy all files from one folder on the server to another

I'm using TIdFTP (Indy 10.6) for a client application and I need to be able to copy all files from one folder on the server to another. Can this be done?
I know how to rename or move a file, we can use TIdFTP.Rename(Src, Dst).
How about the copy? Would I need to use Get() and Put() with a new path / name, knowing that the number of files in the server can exceed 500,000 files.
In our company, we have some files whose size exceeds 1.5 GB. By using my code, it consumes a lot of memory and the file is not copied from one directory to another: in less code, the source directory is named "Fichiers" and the destination directory is named "Sauvegardes".
Here is my code:
var
S , directory : String;
I: Integer;
FichierFTP : TMemoryStream;
begin
IdFTP1.Passive := True;
idftp1.ChangeDir('/Fichiers/');
IdFTP1.List();
if IdFTP1.DirectoryListing.Count > 0 then begin
IdFTP1.List();
for I := 0 to IdFTP1.DirectoryListing.Count-1 do begin
with IdFTP1.DirectoryListing.Items[I] do begin
if ItemType = ditFile then begin
FichierFTP := TMemoryStream.Create;
S := FileName;
idftp1.Get( FileName , FichierFTP , false );
Application.ProcessMessages
idftp1.ChangeDir('/Sauvegardes/' );
idftp1.Put(FichierFTP , S );
Application.ProcessMessages;
FichierFTP.Free;
end;
end;
end;
IdFTP1.Disconnect;
end;
Does anyone have any experience with this? How can I change my code to resolve this problem?
There are no provisions in the FTP protocol, and thus no methods in TIdFTP, to copy/move multiple files at a time. Only to copy/move individual files one at a time.
Moving a file from one FTP folder to another is easy, that can be done with the TIdFTP.Rename() method. However, copying a file typically requires issuing separate commands to download the file locally first and then re-upload it to the new path.
Some FTP servers support custom commands for copying files, so that you do not need to download/upload them locally. For example, ProFTPD's mod_copy module implements SITE CPFR/CPTO commands for this purpose. If your FTP server supports such commands, you can use the TIdFTP.Site() method, eg:
Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
try
IdFTP1.Site('CPFR ' + Item.FileName);
IdFTP1.Site('CPTO /Sauvegardes/' + Item.FileName);
except
// fallback to another transfer option, see further below...
end;
end;
If that does not work, another possibility to avoid having to copy each file locally is to use a site-to-site transfer between 2 separate TIdFTP connections to the same FTP server. If the server allows this, you can use the TIdFTP.SiteToSiteUpload() and TIdFTP.SiteToSiteDownload() methods to make the server transfer files to itself, eg:
IdFTP2.Connect;
...
Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
try
IdFTP1.SiteToSiteUpload(IdFTP2, Item.FileName, '/Sauvegardes/' + Item.FileName);
except
try
IdFTP2.SiteToSiteDownload(IdFTP1, Item.FileName, '/Sauvegardes/' + Item.FileName);
except
// fallback to another transfer option, see further below...
end;
end;
end;
...
IdFTP2.Disconnect;
But, if using such commands is simply not an option, then you will have to resort to downloading each file locally and then re-uploading it. When copying a large file in this manner, you should use TFileStream (or similar) instead of TMemoryStream. Do not store large files in memory. Not only do you risk a memory error if the memory manager can't allocate enough memory to hold the entire file, but once that memory has been allocated and freed, the memory manager will hold on to it for later reuse, it does not get returned back to the OS. This is why you end up with such high memory usage when you transfer large files, even after all transfers are finished.
If you really want to use a TMemoryStream, use it for smaller files only. You can check each file's size on the server (either via TIdFTPListItem.Size if available, otherwise via TIdFTP.Size()) before downloading the file, and then choose an appropriate TStream-derived class to use for that transfer, eg:
const
MaxMemoryFileSize: Int64 = ...; // for you to choose...
var
...
FichierFTP : TStream;
LocalFileName: string;
RemoteFileSize: Int64;
Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
LocalFileName := '';
if Item.SizeAvail then
RemoteFileSize := Item.Size
else
RemoteFileSize := IdFTP1.Size(Item.FileName);
if (RemoteFileSize >= 0) and (RemoteFileSize <= MaxMemoryFileSize) then
begin
FichierFTP := TMemoryStream.Create;
end else
begin
LocalFileName := MakeTempFilename;
FichierFTP := TFileStream.Create(LocalFileName, fmCreate);
end;
try
IdFTP1.Get(Item.FileName, FichierFTP, false);
IdFTP1.Put(FichierFTP, '/Sauvegardes/' + Item.FileName, False, 0);
finally
FichierFTP.Free;
if LocalFileName <> '' then
DeleteFile(LocalFileName);
end;
end;
There are other optimizations you can make to this, for instance creating a single TMemoryStream with a pre-sized Capacity and then reuse it for multiple transfers that will not exceed that Capacity.
So, putting this all together, you could end up with something like the following:
var
I: Integer;
Item: TIdFTPListItem;
SourceFile, DestFile: string;
IdFTP2: TIdFTP;
CanAttemptRemoteCopy: Boolean;
CanAttemptSiteToSite: Boolean;
function CopyFileRemotely: Boolean;
begin
Result := False;
if CanAttemptRemoteCopy then
begin
try
IdFTP1.Site('CPFR ' + SourceFile);
IdFTP1.Site('CPTO ' + DestFile);
except
CanAttemptRemoteCopy := False;
Exit;
end;
Result := True;
end;
end;
function CopyFileSiteToSite: Boolean;
begin
Result := False;
if CanAttemptSiteToSite then
begin
try
if IdFTP2 = nil then
begin
IdFTP2 := TIdFTP.Create(nil);
IdFTP.Host := IdFTP1.Host;
IdFTP.Port := IdFTP1.Port;
IdFTP.UserName := IdFTP1.UserName;
IdFTP.Password := IdFTP1.Password;
// copy other properties as needed...
IdFTP2.Connect;
end;
try
IdFTP1.SiteToSiteUpload(IdFTP2, SourceFile, DestFile);
except
IdFTP2.SiteToSiteDownload(IdFTP1, SourceFile, DestFile);
end;
except
CanAttemptSiteToSite := False;
Exit;
end;
Result := True;
end;
end;
function CopyFileManually: Boolean;
const
MaxMemoryFileSize: Int64 = ...;
var
FichierFTP: TStream;
LocalFileName: String;
RemoteFileSize: Int64;
begin
Result := False;
try
if Item.SizeAvail then
RemoteFileSize := Item.Size
else
RemoteFileSize := IdFTP1.Size(SourceFile);
if (RemoteFileSize >= 0) and (RemoteFileSize <= MaxMemoryFileSize) then
begin
LocalFileName := '';
FichierFTP := TMemoryStream.Create;
end else
begin
LocalFileName := MakeTempFilename;
FichierFTP := TFileStream.Create(LocalFileName, fmCreate);
end;
try
IdFTP1.Get(SourceFile, FichierFTP, false);
IdFTP1.Put(FichierFTP, DestFile, False, 0);
finally
FichierFTP.Free;
if LocalFileName <> '' then
DeleteFile(LocalFileName);
end;
except
Exit;
end;
Result := True;
end;
begin
CanAttemptRemoteCopy := True;
CanAttemptSiteToSite := True;
IdFTP2 := nil;
try
IdFTP1.Passive := True;
IdFTP1.ChangeDir('/Fichiers/');
IdFTP1.List;
for I := 0 to IdFTP1.DirectoryListing.Count-1 do
begin
Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
SourceFile := Item.FileName;
DestFile := '/Sauvegardes/' + Item.FileName;
if CopyFileRemotely then
Continue;
if CopyFileSiteToSite then
Continue;
if CopyFileManually then
Continue;
// failed to copy file! Do something...
end;
end;
finally
IdFTP2.Free;
end;
IdFTP1.Disconnect;
end;

Delphi - Indy Client/Server sending buffer

I'm trying to send buffer from client to the server...Buffer revived but i get error message 'data error' while converting the buffer into steam on the server side.
Also i tried to send that buffer as a Stream but i get error message on the server side Out of memory
Client:
procedure TAudio.Buffer(Sender: TObject; Data: Pointer; Size: Integer);
var
Stream: TMemoryStream;
Buff:string;
begin
Move(Data^, ACMC.BufferIn^, Size);
if AConn.Client.Connected then begin
Stream := TMemoryStream.Create;
Stream.WriteBuffer(ACMC.BufferOut^, ACMC.Convert);
Stream.Position := 0;
Buff := ZCompressStreamToString(Stream);
AConn.Client.IOHandler.WriteLn(Buff);
Stream.Free;
Writeln('sent');
end;
end;
Server Thread:
try
List := MainForm.idtcpsrvrMain.Contexts.LockList;
try
if List.IndexOf(Ctx) <> -1 then
begin
TMainContext(Ctx).Queue.Add(EncryptStr('AUDIO|2|'+BYTES));
Stream:
Buffer := TMainContext(Ctx).Connection.IOHandler.ReadLn;
mStream := TMemoryStream.Create;
try
ZDecompressStringToStream(Buffer,mStream);
mStream.Position := 0;
SetLength(Buffer,mStream.Size);
mStream.ReadBuffer(pointer(Buffer)^,mStream.Size);
SendMessage(hLstbox,LB_ADDSTRING,0,lparam(Buffer));
iList := SendMessage(hLstbox,LB_GETCOUNT,0,0);
SendMessage(hLstbox,LB_SETTOPINDEX,iList-1,0);
ACMO.Play(Pointer(Buffer)^,Length(Buffer));
finally
mStream.Free;
end;
if NodesList.Items[index].TerminateAudioThreads then
begin
..
..
Terminate;
end
else goto Stream;
Note:
both ZCompressStreamToString & ZDecompressStringToStream functions are tested on the client side and its worked.

How can I download a file from MongoDB using delphi mongo-delphi-driver.GridFS.pas?

I'm experimenting with MongoDB and Delphi (mongo-delphi-driver)
While I was able to upload a file using the code at bottom, I'm struggling to download it back from MongoDB to my filesystem.
Does someone has already a code snippet that can show to me?
Thank you all in advance
uses
...
MongoDB, MongoBson, GridFS;
...
procedure UploadFile();
const
LOCAL_FILE_NAME = 'C:\local_file_name';
REMOTE_FILE_NAME = 'remote_file_name';
var
GridFSStoreFileSuccess: Boolean;
myGridFS: TGridFS;
begin
myGridFS := TGridFS.Create(mongo, db);
try
GridFSStoreFileSuccess := myGridFS.storeFile(LOCAL_FILE_NAME, REMOTE_FILE_NAME);
if GridFSStoreFileSuccess then
ShowMessage('File upload was successful!')
else
ShowMessage('File upload was *NOT* successful!');
finally
myGridFS.Free;
end;
end;
Currently I also try to use mongodb with delphi. This is a snippet for your needs
var
GridFSTest : TGridFS;
p : Pointer;
stream : TMemoryStream;
filename : string;
f : TGridfile;
begin
// Create and link TGridFS to 'test' db
GridFSTest := TGridFS.Create(Mongo,'test');
stream := TMemoryStream.Create;
// Find the image file...
f := GridFSTest.find('topo gigio.jpg');
p := AllocMem(f.getLength);
f.read(p,f.getLength);
stream.Write(p^,f.getLength);
stream.Position := 0;
ClientDataSet1.Close;
ClientDataSet1.CreateDataSet;
ClientDataSet1.Insert;
TBlobField(ClientDataSet1.FieldByName('immagine')).LoadFromStream(stream);
ClientDataSet1.Post;
// stream.SaveToFile('c:\a.jpg');
stream.Free;
FreeMem(p);
GridFS.Free;
end;

delphi DataSnap 2010 Stream File working Sample

I am trying to stream an XML File from Server To Client using DataSnap, with the help of ldsandon, i was able to download the sample from embarcadero, but my problem is I cannot follow it.
a pseudo of the program should work this way.
client will request from the server for selected xml file in the combobox.
the server will load the client selected xml file back to client.
i am just am trying to figure it out using delphi DataSnap, if not I will either use synapse or indy for tranferring the file, but I found Datasnap to be interesting.
could anyone help me please, a working if possible?
thanks a lot.
Please Help me, I need your help very badly.. thanks and thanks
I found this link, but I could not figure out how to convert it to TFileStream
// server side
function TServerMethods1.GetCDSXML(SQL: String; var FileSize: Integer): TStream;
begin
QryMisc.Close;
QryMisc.SQL.Text := SQL;
CDSMisc.Open;
Result := TMemoryStream.Create;
try
CDSMisc.SaveToStream(Result, dfXML);
FileSize := Result.Size; // not CDSMisc.DataSize;
Result.Position := 0; // Seek not implemented in abstract class
finally
CDSMisc.Close;
end;
end;
// client side
procedure TClientModule1.PopMiscCDS(SQL: String);
const
BufSize = $8000;
var
RetStream: TStream;
Buffer: PByte;
MemStream: TMemoryStream;
BytesRead: Integer;
FileSize: Integer;
begin
try
MemStream := TMemoryStream.Create;
GetMem(Buffer, BufSize);
try
//---------------------------------------------------------
RetStream := ServerMethods1Client.GetCDSXML(SQL, FileSize);
//---------------------------------------------------------
repeat
BytesRead := RetStream.Read(Pointer(Buffer)^, BufSize);
if BytesRead > 0 then
MemStream.WriteBuffer(Pointer(Buffer)^, BytesRead);
until BytesRead < BufSize;
if FileSize <> MemStream.Size then
raise Exception.Create('Error downloading xml');
MemStream.Seek(0, TSeekOrigin.soBeginning);
CDSMisc.Close;
CDSMisc.LoadFromStream(MemStream);
finally
FreeMem(Buffer, BufSize);
MemStream.Free;
end;
except
on E: Exception do
begin
ShowMessage(E.Message);
end;
end;
end;

Resources