I have a query that is dynamically set up with a parameter.
I query for lines with a varchar field that contains values that can begin with a '!'.
But I get no match of those.
I use SQLServer as the database server.
If I take the sqlcode and run it directly in the database manager it works but not with TFDQuery.
Se the code example below:
myParameter := '!Tommy';
with qryExec do
begin
SQL.Clear ;
SQL.Add('SELECT * FROM myTable T WHERE T.Name='+quotedStr(myParameter));
active := true ;
first;
if Not Eof then
begin
Result := True;
end;
end; //with
I have no idea what's wrong here, so I would be happy if anyone could come with an explanation.
I would suggest using actual parameters which also avoids the possibility of SQL injection. There are also overloaded versions of Open that reduce the housekeeping lines.
FDQuery1.Open('SELECT * FROM myTable T WHERE T.Name= :NAME',['!Tommy'],[ftWideString]);
Related
I'am using a firebird 2.5 server to write in a Database file(BD.fbd). My delphi XE8 project has a Data module(DMDados) with:
SQLConnection (conexao)
TSQLQUery1 (QueryBDPortico_Inicial) + TDataSetProvider1 (DSP_BDPortico_Inicial) + TClientDataSet1 (cdsBDPortico_Inicial)
TSQLQUery2 (QueryConsulta) (just for use SQL strings)
My database file has this table:
PORTICO_INICIAL
The table has these fields (all integer):
NPORTICO
ELEMENTO
ID
None of those fields are primary keys because I will have repeated values in some cases. The connection with the file is ok. The client data set is open when I run the code. The TSQLQUery2 (QueryConsulta) is open when needed.
My code, when triggered by a button, has to delete all tables' records (if exist) then full the table with integer numbers created by a LOOP.
In the first try the code just work fine, but when I press the button the second time i get the error 'Unable to find record. No key specified' then when I check the records the table is empty.
I tried to change the ProviderFlags of my query but this make no difference. I checked the field names, the table name or some SQL text error but find nothing.
My suspect is that when my code delete the records the old values stay in memory then when try apply updates with the new values the database use the old values to find the new record's place therefore causing this error.
procedure monta_portico ();
var
I,K,L,M, : integer;
begin
with DMDados do
begin
QUeryCOnsulta.SQL.Text := 'DELETE FROM PORTICO_INICIAL;';
QueryConsulta.ExecSQL();
K := 1;
for I := 1 to 10 do
begin
L := I*100;
for M := 1 to 3 do
begin
cdsBDPortico_Inicial.Insert;
cdsBDPortico_Inicial.FieldbyName('NPORTICO').AsInteger :=
M+L;
cdsBDPortico_Inicial.FieldbyName('ELEMENTO').AsInteger := M;
cdsBDPortico_Inicial.ApplyUpdates(0);
K := K +1;
end;
end;
end;
end;
I want that every time I use the code above it first delete all records in the table then fill it again with the loop.
When I use the code for the first time it do what I want but in the second time it just delete the records and can not fill the table with the values.
Update I've added some example code below. Also, when I wrote the original version of this answer, I'd forgotten that one of the TDataSetProvider Options is
poAllowMultiRecordUpdates, but I'm not sure that's involved in your problem.
The error message Unable to find record. No key specified is generated by the DataSetProvider, so isn't directly connected to your
QUeryCOnsulta.SQL.Text := 'DELETE FROM PORTICO_INICIAL;'
because that bypasses the DataSetProvider. The error is coming from an failed attempt to ApplyUpdates on the CDS. Try changing your call to it to
Assert(cdsBDPortico_Inicial.ApplyUpdates(0) = 0);
That will show you when the error occurs because the return result of ApplyUpdates gives the number of errors that occurred when calling it.
You say
will have repeated values in some cases
If that's true when the problem occurs, it's because you are hitting a fundamental limitation in the way a DataSetProvider works. To apply the updates on the source dataset, it has to generate SQL to send back to the source dataset (TSqlQuery1) which uniquely identifies the row to update in the source data, which is impossible if the source dataset contains duplicated rows.
Basically, you need to re-think your code so that the source dataset rows are all unique. Once you've done that, setting the DSP's UpdateMode to upWhereAll should avoid the problem. It would be best for the source dataset to have a primary key, of course.
A quick work-around would be to use CDS.Locate in the loop where you insert the records, to see if it can locate an already-existing record with the values you're about to add.
Btw, sorry for raising the point about the ProviderFlags. It's irrelevant if there are duplicated rows, because whatever they are set to, the DSP will still fail to update a single record.
In case it helps, here is some code which might help populating your table
in a way which avoids getting duplicates. It only populates the first two
columns, as in the code you show in your q.
function RowExists(ADataset : TDataSet; FieldNames : String; Values : Variant) : Boolean;
begin
Result := ADataSet.Locate(FieldNames, Values, []);
end;
procedure TForm1.PopulateTable;
var
Int1,
Int2,
Int3 : Integer;
i : Integer;
RowData : Variant;
begin
CDS1.IndexFieldNames := 'Int1;Int2';
for i := 1 to 100 do begin
Int1 := Round(Random(100));
Int2 := Round(Random(100));
RowData := VarArrayOf([Int1, Int2]);
if not RowExists(CDS1, 'Int1;Int2', RowData) then
CDS1.InsertRecord([Int1, Int2]);
end;
CDS1.First;
Assert(CDS1.ApplyUpdates(0) = 0);
end;
Splite the problem into small parties using functions and procedures
create an instance of TSqlQuery Execute the SQL statment's and destroy the instance when you finish with it...
procedure DeleteAll;
var
Qry: TSqlQuery;
begin
Qry := TSqlQuery.Create(nil);
try
Qry.SqlConnection := DMDados.conexao;
Qry.Sql.Text := 'DELETE FROM PORTICO_INICIAL;';
Qry.ExecSql;
finally
Qry.Free;
end;
end;
your can even execute directly from TSQlConnection with one line...
DMDados.conexao.ExecuteDirect('DELETE FROM PORTICO_INICIAL;')
procedure monta_portico ();
var
I,K,L,M, : integer;
begin
with DMDados do
begin
DeleteAll;
K := 1;
for I := 1 to 10 do
begin
L := I*100;
for M := 1 to 3 do
begin
cdsBDPortico_Inicial.Insert;
cdsBDPortico_Inicial.FieldbyName('NPORTICO').AsInteger :=
M+L;
cdsBDPortico_Inicial.FieldbyName('ELEMENTO').AsInteger := M;
cdsBDPortico_Inicial.ApplyUpdates(0);
K := K +1;
end;
end;
end;
end;
Just few obvervations, cause the primary answers were given, but they not deal with the secondary problems.
cdsBDPortico_Inicial.FieldbyName('NPORTICO').AsInteger :=
FieldByName is slow function - it is linear search over objects array with uppercased string comparison over each one. You better only call it once for every field, not do it again in again in the loop.
cdsBDPortico_Inicial.ApplyUpdates(0);
Again, applying updates is relatively slow - it requires roundtrip to the server all through internal guts of DataSnap library, why so often?
BTW, you delete rows from SQL table - but where do you delete rows from cdsBDPortico_Inicial ??? I do not see that code.
Was I in your shows I would write something like that (granted I am not big fan of Datasnap and CDS):
procedure monta_portico ();
var
Qry: TSqlQuery;
_p_EL, _p_NP: TParam;
Tra: TDBXTransaction;
var
I,K,L,M, : integer;
begin
Tra := nil;
Qry := TSqlQuery.Create(DMDados.conexao); // this way the query would have owner
try // thus even if I screw and forget to free it - someone eventually would
Qry.SqlConnection := DMDados.conexao;
Tra := Qry.SqlConnection.BeginTransaction;
// think about making a special function that would create query
// and set some its properties - like connection, transaction, preparation, etc
// so you would not repeat yourself again and again, risking mistyping
Qry.Sql.Text := 'DELETE FROM PORTICO_INICIAL'; // you do not need ';' for one statement, it is not script, not a PSQL block here
Qry.ExecSql;
Qry.Sql.Text := 'INSERT INTO PORTICO_INICIAL(NPORTICO,ELEMENTO) '
+ 'VALUES (:NP,:EL)';
Qry.Prepared := True;
_p_EL := Qry.ParamByName('EL'); // cache objects, do not repeat linear searches
_p_NP := Qry.ParamByName('NP'); // for simple queries you can even do ... := Qry.Params[0]
K := 1;
for I := 1 to 10 do
begin
L := I*100;
for M := 1 to 3 do
begin
_p_NP.AsInteger := M+L;
_p_EL.AsInteger := M;
Qry.ExecSQL;
Inc(K); // why? you seem to never use it
end;
end;
Qry.SqlConnection.CommitFreeAndNil(tra);
finally
if nil <> tra then Qry.SqlConnection.RollbackFreeAndNil(tra);
Qry.Destroy;
end;
end;
This procedure does not populate cdsBDPortico_Inicial - but do you really need it?
If you do - maybe you can re-read it from the database: there could be other programs that added rows into the table too.
Or you can insert many rows and then apply them all in one command, before committing the transaction (often abreviated tx), but even then, do not call FieldByName more than once.
Also, think about logical blocks of your program work in advance, those very transactions, temporary TSQLQuery objects etc.
However boring and tedious it is now, you would bring yourself many more spaghetti trouble if you don't. Adding this logic retroactively after you have many small functions calling one another in unpredictable order is very hard.
Also, if you make Firebird server auto-assigning the ID field (and your program does not need any special values in ID and will be ok with Firebird-made values) then the following command might server yet better for you: INSERT INTO PORTICO_INICIAL(NPORTICO,ELEMENTO) VALUES (:NP,:EL) RETURNING ID
I'm currently using AdoQuery's and append post commands. But for data security I want to change my code with insert into and update table name...
But I have a lot of forms and tables...
Because of that I think maybe someone has already developed code for generating insert statements.
Actually I have found a way but I'm stuck.
I have query1. it contains the fieldlist.
I'm creating a parameter list in another query from this fieldlist.
I'm updating the parameters field by field.
This is not very convenient
Can someone give me a easy ways to do this.
Note: I prefer coding this job with only standard components. I don't want to install additional components.
Maybe not the reply you want. I think you need to raise the abstraction level. You need to skip SQL. An ORM framework can do this for you. It maybe feels like a big step for you but I promise it is also a relief to just use code like:
Person.name := 'Bob';
Invoice.customer.address.street := 'Abbey road';
Edit1.text := Invoice.customer.name;
To actually update database you need to call an update method that differ depending on framework. For a list of frameworks see here. I am also aware of TMS Aurelius. I use Bold on daily use. Bold also have features like OCL, derived attributes and links in the model, some boldaware components (it updates whenever db changes). But it has one big disadvantage. It is only available for D2006/D2007. I am working for a solution on this because I think it is the best and most mature ORM framework for Delphi. See also my blog on Bold for Delphi. Ask if you have questions!
You take the fieldlist from your query.
Create a new query with parameters.
And fill in the values.
Something like this:
const
TableNameEscapeStart = '['; //SQL server, use '`' for MySQL
TableNameEscapeEnd = ']'; //SQL server, use '`' for MySQL
FieldNameEscapeStart = '[';
FieldNameEscapeEnd = ']';
function CreateInsertStatementFromTable1ToTable2(Table1, Table2: TTable): String;
var
i: integer;
comma: string;
begin
i:= 0;
Result:= 'INSERT INTO '+TableNameEscapeStart + Table2.TableName + TableNameEscapeEnd + ' (';
comma:= ' , '
while i < Table1.FieldCount do begin
if (i = Table1.FieldCount -1) then begin comma:= ' '; end;
Result:= Result + FieldNameEscapeStart + Table1.Fields.Field[i].Name + FieldNameEscapeEnd + comma;
end;
Result:= Result +' ) VALUES ( ';
i:= 0;
comma:= ' , '
while i < Table1.FieldCount do begin
if (i = Table1.FieldCount -1) then begin comma:= ' '; end;
Result:= Result +':' + IntToStr(i+1) + comma;
end; {while}
Result:= Result + ' ); ';
end;
There are three avenues for SQL injection here.
1. The field values
2. The table name
3. The field names
The first is covered by the use of parameters.
The second and third are covered, because you're using the table and field names of the table directly.
If you don't have a trusted source of table and fields names, then you need to compare these against the table and fieldnames obtained directly from the table.
See: Delphi - prevent against SQL injection
You insert the data using ParamByName (slowly) or more efficiently using Param[i] where i starts at 0.
In MySQL it's even easier:
If table1 and table2 have the same fields, the following SQL will insert all data in table2 into table1:
INSERT INTO table1 SELECT * FROM table2;
I have a problem when trying to apply a filter to a ADO dataset in Delphi XE2 ...
Filter := ' [Name] like ''%john'' ';
It raises an exception:
Project Test.exe raised exception class EOleException with message
'Arguments are of the wrong type, are out of acceptable range,
or are in conflict with one another'. Process stopped.
but when using:
Filter := ' [Name] like ''john%'' '
it works fine!
why?
The Operator can only be one of the following:
= < > <= >= <> LIKE
If you use the LIKE operator, you can also use the * or % wildcards as the last character in the string or as the first and last character in the string.
http://www.devguru.com/technologies/ado/quickref/recordset_filter.html
You can catch the filter on the event OnFilterRecord:
procedure TForm1.ADODataSet1FilterRecord(DataSet: TDataSet; var Accept: Boolean);
var
iPos: Integer;
begin
iPos:= pos('john',ADODataSet1name.AsString);
if (iPos>0) and
(iPos = length(ADODataSet1name.AsString)-3) then
begin
Accept:= True;
end
else
begin
Accept:= False;
end;
end;
or
function TForm1.IsLastCriteria(AText: String): Boolean;
var
iPos: Integer;
begin
iPos:= pos(AText,ADODataSet1name.AsString);
Result:= (iPos>0) and
(iPos = length(ADODataSet1name.AsString)-length(AText)-1);
end;
procedure TForm1.ADODataSet1FilterRecord(DataSet: TDataSet; var Accept: Boolean);
begin
Accept:= IsLastCriteria('john');
end;
AS. I still ask you to read http://www.catb.org/esr/faqs/smart-questions.html#beprecise and describe your environment accordingly.
What is database server ?
What version are MDAC/ADO components?
What the query is ?
What is Name column type in SQL ?
You are "upping" the comments, that si nice. But you don't answer the questions. And that is not nice. We are not ESPers, we canot read you mind.
I put you few suggestions in comments above. Did you tried them ? Did they worked ? To quote them:
Maybe you can update ADO/MDAC to 2.8sp1 version. Referenced MSDN KB articles are told to apply to MDAC up to 2.7 version. Maybe 2.8sp1 no more has that limitation.
Maybe you can use some data-server specific tricks like copying last 4 letters of Name into a separate column.
Maybe you can move that condition into SQL SELECT WHERE clause and re-open the query.
Maybe there is event-handler like TBDEDataSet.OnFilterRecord
There are 4 workarounds and you either did not tried them or did not reported the results. Not nice.
Idea #1 is self-explanatory
Idea #3 was detailed by #SertacAkyuz as far as he could do it, giving lack of information about your program.
Idea #4 was detailed by #Ravaut123
Idea #2 is outlined below
Assuming your "ADO database" is backed by Microsoft Access, and according to http://www.databasedev.co.uk/access-sql-string-functions.html ...
Or using MS SQL and according to http://msdn.microsoft.com/en-us/library/ms177532.aspx ...
Or...
Query.SQL.Text := 'select Right(Name, 4) as name_tail, * from table where ...'
See that extra column added to query! 4 is length for "john" there.
With query like that the following filtering conditions are to be equivalent:
Filter := ' [Name] like ''%john'' ';
Filter := ' [name_tail] = ''john'' ';
However, if on some rows Name column is shorter than 4 letters, i don't know what RIGHT function would do. Maybe it will truncate the result, or maybe throw an error and abort the query. The latter maybe - depending on the real data - maybe can be alleviated by padding with spaces to the 4 length like
LTrim(Right(' ' || Name, 4)) as name_tail
That is for you to test, since only you know the details of your environment.
The simplest answer to this question was mentioned in comments.
If you used asterisk instead of percent symbol, everything would work great
Filter := ' [Name] like ''*john'' ';
how can i load millions of records from tadotable into a stringlist very fast?
procedure TForm1.SlowLoadingIntoStringList(StringList: TStringList);
begin
StringList.Clear;
with SourceTable do
begin
Open;
DisableControls;
try
while not EOF do
begin
StringList.Add(FieldByName('OriginalData').AsString);
Next;
end;
finally
EnableControls;
Close;
end;
end;
in your loop you get the field.
Search the field out of the loop
procedure TForm1.SlowLoadingIntoStringList(StringList: TStringList);
var
oField: TField;
begin
StringList.Clear;
with SourceTable do
begin
Open;
DisableControls;
try
oField:= FieldByName('OriginalData');
if oField<>Nil then
begin
while not EOF do
begin
StringList.Add(oField.AsString);
Next;
end;
end;
finally
EnableControls;
Close;
end;
end;
end;
Unfortunately, you can't do this quickly. It is an inherently slow operation that involves large amounts of CPU time and memory bandwidth to achieve. You could throw more hardware at it, but I suspect you should be re-thinking your task instead.
With 'millions of records' you may consider :
1/ Change your Query from
SELECT * FROM MYTABLE;
in
SELECT OriginalData FROM MYTABLE;
You'll use less memory and be more efficient.
2/ Look another component than TStringList depending on your needs.
3/ Look all good previous advices, mainly :
don't use FieldByName
direct link to the OleDB provider
Is it sorted?
// Turn off the sort for now
StringList.Sorted := False;
// Preallocate the space
StringList.Capacity := recordCount;
// Now add the data with Append()
...
// Now turn the sort back on
StringList.Sorted := True;
Seriously? Millions of records in a stringlist?
Ok, let's assume you really do need to take this approach...
There are some good suggestions already posted.
If you want to experiment with a different approach you could consider concatenating the individual records server side (via a stored procedure) and then returning the concatenated data as a blob (or possibly nvarchar(max)), which is basically the list of concatenated strings delimited by say a carriage return (assuming this is a reasonable delimiter for your needs).
You can then simply assign the returned value to the Text property of the TStringList.
Even if you cannot do all of the strings in a single hit, you could do them in groups of say 1000 at a time.
This should save you a ton of time looping around each record client side.
Expanding on #Ravaut123's answer I would suggest the following code:
Make sure your Query is not connected to any visual other component, and does not have any events set that fire on rowchanges because this will cause it to to updates on every change in the active record, slowing things way down.
You can disable the visual controls using disablecontrols, but not the events and non-visual controls.
...
SQLatable:= 'SELECT SingleField FROM atable ORDER BY indexedfield ASC';
AQuery:= TAdoQuery.Create(Form1);
AQuery.Connection:= ....
AQuery.SQL.Text:= SQLatable;
Using a Query makes sure you only select 1 field, in the order that you want, this reduces network traffic. A table fetches all fields, causing much more overhead.
function TForm1.LoadingAllIntoStringList(AQuery: TAdoQuery): TStringList;
var
Field1: TField;
begin
Result:= nil;
try
if not(AQuery.Active) then begin
AQuery.Open;
end else begin
AQuery.First;
end;
AQuery.DisableControls;
AQuery.Filtered:= false; //Filter in the SQL `where` clause
AQuery.FetchAll; //Preload all data into memory
Result:= TStringlist.Create;
except
{ignore error, will return nil}
end;
try
Result.Sorted:= false; //Make sure you don't enable sorting
Result.Capacity:= AQuery.RecordCount; //Preallocate the needed space
Field1:= AQuery.FieldByName('SingleField'); //Never use `fieldbyname` in a loop!
while not AQuery.EOF do begin
Result.Add(Field1.AsString);
AQuery.Next;
end; {while}
AQuery.EnableControls;
except
FreeAndNil(Result);
end;
If you want to load the data into the stringlist to do some processing, consider doing that in the SQL statement instead. The DB can use indexes and other optimizations that the stringlist cannot use.
If you want to save that data into a CSV file, consider using a build-in DB function for that.
e.g. MySQL has:
SELECT X FROM table1 INTO OUTFILE 'c:/filename_of_csv_file.txt'
Which will create a CSV file for you.
Many DB's have simular functions.
I am having a problem getting a list of fields from a query defined at run time by the users of my program. I let my users enter a SQL query into a memo control and then I want to let them go through the fields that will return and do such things as format the output, sum column values and so forth. So, I have to get the column names so they have a place to enter the additional information.
I would do fine if there were no parameters, but I also have to let them define filter parameters for the query. So, if I want to set the parameters to null, I have to know what the parameter's datatype is.
I am using Delphi 2006. I connect to a Firebird 2.1 database using the DBExpress component TSQLConnection and TSQLQuery. Previously, I was successful using:
for i := 0 to Qry.Params.Count - 1 do Qry.Params[i].value := varNull;
I discovered I had a problem when I tried to use a date parameter. It was just a coincidence that all my parameters up until then had been integers (record IDs). It turns out that varNull is just an enumerated constant with a value of 1 so I was getting acceptable results (no records) was working okay.
I only need a list of the fields. Maybe I should just parse the SELECT clause of the SQL statement. I thought setting Qry.Prepared to True would get me a list of the fields but no such luck. It wants values for the parameters.
If you have an idea, I would sure like to hear it. Thanks for any help.
Replied again 'coz I'm interested. My methods works (with my queries) because they have been pre-defined with the params' datatypes preset to the correct type:)
I'm not sure how you are expecting the query to know or derive the datatype of the param given that you are not even selecting the field that it operates against.
So I think your query setup and user input method will need more attention. I've just looked up how I did this a while ago. I do not use a parameterised query - I just get the "parameter values" from the user and put them directly into the SQL. So your sql would then read:
SELECT s.hEmployee, e.sLastName
FROM PR_Paystub s
INNER JOIN PR_Employee e ON e.hKey = s.hEmployee
WHERE s.dtPaydate > '01/01/2008'
therefore no parameter type knowledge is necessary. Does not stop your users entering garbage but that goes back to input control :)
Although a slightly different dataset type this is what I use with TClientDataset simple and effective :)
for i := 0 to FilterDataSet.Params.Count -1 do
begin
Case FilterDataSet.Params.Items[i].Datatype of
ftString:
ftSmallint, ftInteger, ftWord:
ftFloat, ftCurrency, ftBCD:
ftDate:
ftTime:
ftDateTime:
.
.
.
end;
end;
can you not do something similar with the query?
You guys are making this way too hard:
for i := 0 to Qry.Params.Count - 1 do begin
Qry.Params[i].Clear;
Qry.Params[i].Bound := True;
end;
I'm not sure what version of Delphi you are using. In the Delphi 2006 help under Variant Types, it says:
Special conversion rules apply to the
Borland.Delphi.System.TDateTime type
declared in the System unit. When a
Borland.Delphi.System.TDateTime is
converted to any other type, it
treated as a normal Double. When an
integer, real, or Boolean is converted
to a Borland.Delphi.System.TDateTime,
it is first converted to a Double,
then read as a date-time value. When a
string is converted to a
Borland.Delphi.System.TDateTime, it is
interpreted as a date-time value using
the regional settings. When an
Unassigned value is converted to
Borland.Delphi.System.TDateTime, it is
treated like the real or integer value
0. Converting a Null value to Borland.Delphi.System.TDateTime raises
an exception.
The last sentence seems important to me. I would read that as varNull cannot be converted to a TDateTime to put into the field, and hence you get the exception that you're experiencing.
It also implies that this is the only special case.
Couldn't you do something like:
for i := 0 to Qry.Params.Count - 1 do
begin
if VarType(Qry.Params[i].value) and varTypeMask = varDate then
begin
Qry.Params[i].value := Now; //or whatever you choose as your default
end
else
begin
Qry.Params[i].value := varNull;
end;
end;
What I ended up doing was this:
sNull := 'NULL';
Qry.SQL.Add(sSQL);
for i := 0 to Qry.Params.Count - 1 do begin
sParamName := Qry.Params[i].Name;
sSQL := SearchAndReplace (sSQL, ':' + sParamName, sNull, DELIMITERS);
end;
I had to write SearchAndReplace but that was easy. Delimiters are just the characters that signal the end of a word.
TmpQuery.ParamByName('MyDateTimeParam').DataType := ftDate;
TmpQuery.ParamByName('MyDateTimeParam').Clear;
TmpQuery.ParamByName('MyDateTimeParam').Bound := True;