In trying to update 10000 records by using batch update method over remote mysql connection. my server has 200+ms latency and by using this method it would take forever to do this since its sending queries one by one! any workaround?
query.params.arraysize := 10000;
query.sql.text := 'update table set field=:f1 where id=:f2;'
for i := 0 to query.params.arraysize-1 do
begin
query.params[0].asstrings[i] := 'VERY LONG STRING > 10KB';
query.params[1].asintegers[i] := id;
end;
query.execute(10000);
Try this:
Define the parameterized query.
Configure query params for performance.
e.g:
FDQuery1.Params[0].DataType := ftString;
Set the array size.
Supply the parameters values.
Execute the query in a transaction.
FDConnection.StartTransaction;
try
FDQuery1.Execute(FDQuery1.Params.ArraySize);
FDConnection.Commit;
except
FDConnection.Rollback;
raise;
end;
Related
I'm trying to use the FireDAC DBMS Identifiers for generating a database specific query.
I'm connecting to a MySQL-Server currently (DriverID = MySQL). I want to query either from mysql or mssql with a limit / top query.
My current statement looks like this:
SELECT
{IF MSSQL} TOP(1) {fi} `tr`.`TaxRate_Primkey`
FROM
`tbl_taxrates` AS `tr`
WHERE
`tr`.`TaxRate_TaxCodeId` = `tc`.`TaxCode_Primkey`
AND
`tr`.`TaxRate_ValidSince` <= :DATE
ORDER BY `tr`.`TaxRate_ValidSince` DESC
{IF MySQL} LIMIT 1 {fi}
Of course I'm aware, that the escaping will not be correct for mssql, but that's a different story.
When I inspect the FireDAC Monitor the preprocessed query looks like this:
SELECT
TOP(1) `tr`.`TaxRate_Primkey`
FROM
`tbl_taxrates` AS `tr`
WHERE
`tr`.`TaxRate_TaxCodeId` = `tc`.`TaxCode_Primkey`
AND
`tr`.`TaxRate_ValidSince` <= ?
ORDER BY
`tr`.`TaxRate_ValidSince` DESC
LIMIT 1
Of course this will result in error
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '.`TaxRate_Primkey` FROM `tbl_taxrates` AS `tr` WHERE `tr`.`TaxRate_TaxCodeId` ' at line 1 [errno=1064, sqlstate="42000"]
since it should only add the LIMIT 1 and omit the TOP(1).
Since I only have the Delphi Community Edition 10.4, I don't have access to the MSSQL-Driver. I thought, that could cause this error and I tried with others. But also with {if FIREBIRD} and {if ADS} it was the same.
The constructor looks like this:
constructor TFireDACTenantRepository.Create(
ADBName: string;
ADBServer: string;
APort: Integer;
AUserName: string;
APassword: string;
ALogger: TLogger
);
var
oParams: TStrings;
begin
inherited Create;
FLogger := ALogger;
Self.MonitorLink := nil;
Self.MonitorBy := mbRemote;
Self.Tracing := True;
FConnection := TFDConnection.Create(nil);
oParams := TStringList.Create;
try
oParams.Add('Server=' + ADBServer);
oParams.Add('Port=' + IntToStr(APort));
oParams.Add('Database=' + ADBName);
oParams.Add('User_Name=' + AUserName);
oParams.Add('Password=' + APassword);
oParams.Add('OSAuthent=No');
FDManager.AddConnectionDef('MySQLConnectionTenant', 'MySQL', oParams);
FConnection.Params.MonitorBy := Self.MonitorBy;
FConnection.ConnectionDefName := 'MySQLConnectionTenant';
FConnection.ResourceOptions.ParamCreate := True;
FConnection.ResourceOptions.MacroCreate := True;
FConnection.ResourceOptions.ParamExpand := True;
FConnection.ResourceOptions.MacroExpand := True;
FConnection.ResourceOptions.PreprocessCmdText := True;
FConnection.ResourceOptions.EscapeExpand := True;
finally
oParams.Free;
end;
FConnection.AfterConnect := DoAfterConnect;
FConnection.AfterDisconnect := DoAfterDisconnect;
end;
My question looks a bit related to this question here
How do I use FireDAC DBMS identifiers to conditionally change SQL text
Thanks
I found the solution. If you want to use the conditional escape sequence, you also have to use the FireDAC.Phys.XXX unit for all involved databases.
Since I want to use MSSQL, I have to add FireDAC.Phys.MSSQL to the use-clause.
The units are listed here:
https://docwiki.embarcadero.com/RADStudio/Sydney/en/Databases_(FireDAC)
Maybe this answer helps others.
I use Delphi Berlin.
I am trying to make a function/procedure to copy data from a FireDac query (connected to a database) to another FireDac query (connected to another database) using ArrayDML. first database is firebird and the other is MSSQL in the first care, but in another case both databases are Firebird.
So far so good and almost all datatype are working correct except ftBlob.
Here is the body of the function:
while not querySource.Eof do begin
paramPosition := -1;
Inc(mIndex);
for i := 0 to querySource.FieldCount - 1 do begin
Inc(paramPosition);
// daca exista o valoare
if querySource.FieldByName(querySource.Fields[i].FieldName).AsVariant <> Null then begin
case querySource.Fields[i].DataType of
ftDateTime, ftDate, ftTime, ftTimeStamp : queryInsert.Params[paramPosition].AsDateTimes[mIndex] := querySource.FieldByName(querySource.Fields[i].FieldName).AsDateTime;
ftFloat, ftCurrency, ftBCD, ftFMTBcd : queryInsert.Params[paramPosition].AsFloats[mIndex] := querySource.FieldByName(querySource.Fields[i].FieldName).AsFloat;
ftSmallint, ftInteger, ftLargeint : queryInsert.Params[paramPosition].AsIntegers[mIndex] := querySource.FieldByName(querySource.Fields[i].FieldName).AsInteger;
ftString : queryInsert.Params[paramPosition].AsStrings[mIndex] := querySource.FieldByName(querySource.Fields[i].FieldName).AsString;
ftBlob, ftMemo, ftGraphic : queryInsert.Params[paramPosition].AsBlobs[mIndex] := querySource.FieldByName(querySource.Fields[i].FieldName).AsVariant;
end;
end;
end;
the blob value is not copy the correct value from the source.
how to use the arrayDML in this case? any workaround?
I won't answer your question but suggest you to use the TFDBatchMove component because you are reinventing wheel here. TFDBatchMove is just for what you want to do, for moving data from one database to another (and not only that).
You simply setup the TFDBatchMoveSQLReader as Reader and Writer, write SQL queries for both and the component will automatically map the fields by matching names. If the queries won't have matching field names, you can fine tune this by the Mappings property. Then you just call Execute.
I have the following problem:
1) I use Delphi XE7 to develop a 3-layer system.
2) The server layer, created with datasnap using REST.
3) I use Firebird as database and the access is performed with FireDAC.
4) I have a sequence with value 01.
5) I created the following query in the server layer:
Select GEN_ID (gen_my_sequence, 1) from rdb $ database
6) On the server returns the sequence value in the query is: 02.
7) But the client layer returns 03.
I do not understand why the query is executed twice.
Can anyone help me?
This is the nature of generators (sequences) in firebird. Their value is increased every time you request it and the value of the generator is updated from that request and remains updated. Also generators live outside of transaction control.
See this firebirdsql generatorguide-basics. It doesn't matter where you request it from.
I use technical standards that the Embarcadero indicates.
What I realized was this:
1) The unit Data.FireDACJSONReflect in TFDJSONInterceptor.ItemListToJSONObject routine has this block of code:
if not LActive then
LDataSet.Active := True;
try
LJSONDataSet := DataSetToJSONValue(LDataSet);
// Use AddPair overload that will accept blank key
AJSONObject.AddPair(TJSONPair.Create(LPair.Key, LJSONDataSet))
finally
if not LActive then
LDataSet.Active := False;
end;
See he activates the query once, causing the sequence to be incremented.
But in DataSetToJSONValue (LDataSet) routine; This code block is:
if (LMemTable = nil) then
begin
LMemTable := TFDMemTable.Create(nil);
LAdapter := TFDTableAdapter.Create(nil);
LMemTable.Adapter := LAdapter;
LAdapter.SelectCommand := ADataSet.Command;
LMemTable.Active := True;
end;
See he again activates the query, where the sequence is again incremented.
Now I do not know if I made a mistake or if it is a bug, but I created a new class inherited from TFDMemTable and thought there was some mistake in this class, but did a test with TFDMemTable component, standard component of FireDAC, and even then the activation of any query is performed twice, because the code does not consider any of these two classes, as a TFDCustomMemTable, even though they were inherited directly from this class.
I commented the code of DataSetToString routine (const ADataSet: TFDAdaptedDataSet) that looked like this:
LMemTable := nil;
LAdapter := nil;
try
//if (ADataSet is TFDCustomMemTable) then
LMemTable := TFDCustomMemTable(ADataSet);
{if (LMemTable = nil) then
begin
LMemTable := TFDMemTable.Create(nil);
LAdapter := TFDTableAdapter.Create(nil);
LMemTable.Adapter := LAdapter;
LAdapter.SelectCommand := ADataSet.Command;
LMemTable.Active := True;
end;}
In this way the problem was solved, and the performance of the application seemed to have improved.
I have one IBDatabase in DataModule linked with my IBTransaction.
In one module of project I need to control the persistence in two database.
For this, I am adding the second IBDatabase this way:
constructor TConnections.Create(AIBDatabase: TIBDatabase);
begin
if AIBDatabase = nil then
raise Exception.Create('The base connection is needed!');
inherited Create;
FIBDatabase := TIBDatabase.Create(nil);
FIBDatabase.LoginPrompt := false;
FIBDatabase.Params.Clear;
FIBDatabase.Params.Text := AIBDatabase.Params.Text;
FIBDatabase.DatabaseName := AIBDatabase.DatabaseName.Replace('DB.GDB', 'DB2.GDB');
end;
procedure TConnections.SetTransaction(AIBTransaction: TIBTransaction);
begin
if AIBTransaction = nil then
raise Exception.Create('Then Transaction is needed!');
AIBTransaction.AddDatabase(FIBDatabase);
FIBDatabase.DefaultTransaction := AIBTransaction;
FIBDatabase.Open;
end;
Any select commands are work fine, but in insert command the error occurs.
Well, I have this:
connections := TConnections.Create(Dm.Database);
try
connection.SetTransaction(Dm.Transaction);
qry := TIBQuery.Create(nil);
qry.Database := Dm.Database;
try
// here are commands with Dm.Transaction
// ...
qry.ExecSql;
finally
qry.Free;
end;
otherQry := TIBQuery.Create(nil);
otherQry.Database := connection.OtherDatabase;
try
// here are commands with connection.OtherDatabase but same Transaction
// ...
otherQry.ExecSql; // The error occurs here.
finally
otherQry.Free;
end;
Dm.Transaction.Commit;
finally
connection.Free;
end;
'invalid transaction handle (expecting explicit transaction start)'
These block is envolved in try except.
So, if I try again, after the error, the process runs smoothly.
What's wrong in my configuration?
This may occur if you started transaction explicitly. Every explicit transactions must be finished explicitly. So, if your connection is open explicitly, you should close it explicitly.
You may use :
//Commit(Stop) the transaction before open an other connection
if Dm.Transaction.InTransaction then
dm.Transaction.Commit;
Note: In applications that connect an InterBaseExpress dataset to a client dataset, every query must be in its own transaction. You must use one transaction component for each query component.
http://docwiki.embarcadero.com/Libraries/XE8/en/IBX.IBDatabase.TIBTransaction
I want to store images into a database using SQL commands, I know other ways using TBlobField.LoadFromFile etc, but we make our own sql commands to update the database that's why I need to do this.
How should I go about doing this?
I've never tried this (and away from desk at the moment), but would parameters work? eg:
Query.Sql.Clear;
Query.Sql.Add('update myTable set myField = :blobVal where myId = :idVal');
Query.ParamByName('idVal').AsInteger := SomeId;
Query.ParamByName('blobVal').LoadFromFile(....
//or
Query.ParamByName('blobVal').LoadFromStream(....
Query.ExecSql;
This allows you to use SQL (rather than the .Edit, etc methods) and still insert blob data
If I understand correctly, you have some sort of SQL-generation system instead of using datasets, and you want to know how to generate the SQL to do an INSERT or UPDATE to a Blob field correctly.
What you'll need is a way to serialize your image to a string. For example:
function ImageToString(image: TImage): string;
var
stream: TStringStream;
begin
stream := TStringStream.Create;
try
image.SaveToStream(stream);
result := image.DataString;
finally
stream.Free;
end;
end;
Once that's ready, put a token in your SQL, something like set IMAGE_FIELD = $IMAGE$, and then use StringReplace to replace the $IMAGE$ token with the string representation of your image.
Or you could use parameters, like Graza suggested, if the system you're working with supports them.
Try this:
Query.Sql.Clear;
Query.Sql.Add('update myTable set myField = :blobVal where myId = :idVal');
Query.ParamByName('idVal').AsInteger := SomeId;
Query.Params.CreateParam(ftBlob,'blobVal',ptInput).LoadFromFile('c:\path.png',ftGraphic);
Query.ExecSql;
jpg := TJPEGImage.Create;
jpg.Assign(Image1.Picture.Graphic);
strm := TMemoryStream.Create;
strm.Position:= 0;
jpg.SaveToStream(strm);
IBSQL1.Close;
IBSQL1.SQL.Clear;
IBSQL1.SQL.Add('INSERT INTO ENTRY(FORMNUM, JPG) VALUES(');
IBSQL1.SQL.Add( quotedstr(edtFormNum.Text));
IBSQL1.SQL.Add(', :JPG');
IBSQL1.SQL.Add(')');
IBSQL1.Params.ByName('JPG').LoadFromStream(strm);
IBSQL1.ExecQuery;
strm.Free;
jpg.Free;