A json is assigned to result which is greater than 50MB in a DB2 stored procedure. But the size of the result which is inserted or returned is only approx. 1MB. Should I have to define the size of the CLOB?
CREATE OR REPLACE PROCEDURE CALCULATE_CLOB
(
OUT result CLOB
)
LANGUAGE SQL
SPECIFIC SQL2433453455
BEGIN
DECLARE result CLOB;
SET result = ... json
INSERT INTO DATA
VALUES(result);
return result;
The default length of a CLOB in Db2 is 1M (1 megabyte) for z/os, luw, and i-series. So CLOB is the same as CLOB(1M).
When you want a larger clob, you must specify the size when you declare a variable, or when you specify a parameter to a routine. For example, CLOB(50M), or CLOB(1G) etc. The maximum length is 2G.
This is per the Db2 documentation, see the create table statement for your Db2-server platform and version.
Always specify your Db2-server platform (z/os, i-series, linux/unix/windows/cloud) and Db2-server version when asking for help, because the answer can depend on these facts.
As you appear to be learning how to code stored procedures, you may benefit from studying the examples provided by IBM for SQL PL, which are available in various places. For example, online in the Db2 Knowledge Centre for your version of db2-server, and on github, and also in the samples subdirectory tree of your Db2-LUW server installation (if installed). Spending time on such examples will help you because you will be able to answer simple questions without waiting for answers on stackoverflow.
Related
Is there limit to either the number of params or to the overall size of a params in a TStoredProc ExecProc call?
Currently running a system that is still using the BDE to connect to Oracle and a recent change to the number of parameters to a package procedure as started producing access violations. The params count is now up to 291 and the AV is being created in the ExecProc call of TStoredProc.
If we remove a single param from the list (any param, does not have to be a specific param), the ExecProc call works fine.
I have debugged through the code and the access violation is being thrown with the TStoredProc.BindParams procedure within DBTables.pas. I have several watches set up, one of which is SizeOf(FRecordBuffer) and as I step through this procedure, the value is 65535. This is MaxWord (Windows.pas). I don't see is any specified limits within the DBTables code.
The callstack is TStoredProd.ExecProc -> TStoredProc.CreateCursor -> TStoredProc.GetCursor -> TStoredProc.BindParams and the access violation is thrown in the for-loop that iterates through the FParams.
Thanks in advance, we need to find something we can pinpoint so we can steer clear.
I'm not at all versed in Oracle SQL, but since you're maintaining the thing, I would see if I could change the call with all that parameters to a single insert into a new dedicated table (with that many columns plus an autonumber primary key), and change the stored procedure to take this key as input and call the values from this new record to do its job. This may just deliver a bit faster than finding out what's the maximum number of parameters and try to find a fix there. (Though it's a bit of a strange number, as in not a power of 2, it may well be 291...)
I am working with multiple databases within the same application. I am using drivers from two different companies. Both companies have tTable and tQuery Descendants that work well.
I need a way to have generic access to the data, regardless of which driver/tQuery component I am using to return a set of data. This data would NOT tie to components, just to my logic.
For example...(pseudocode) Let's create a function which can run against either tQuery component
function ListAllTables(NameOfDatabase : String) :ReturnSet??
begin
If NameOfDataBase = 'X' then use tQuery(Vendor 1)
else use tQuery(Vendor 2)
RunQuery;
Return Answer...
end;
When NORMALLY running a query, I do
Query.Open;
While not Query.EOF do
begin
Read my rows..
next;
end;
If I am CALLING ListAllTables, what is my return type so that I can iterate through the rows? Each tQuery Vendor is different, so I can't use that (can I, and if so, would I want to?) I could build a Memory Table, and pass that back, but that seems like extra work for ListAllRows to build a memory table, and then to pass it back to the calling routine so that it can "un-build", i.e. iterate through the rows...
What are your ideas and suggestions?
Thanks
GS
Almost all Delphi datasets descend from TDataset, and most useful behavior is defined on TDataset.
Therefore, if you assign each table or query to a variable of type TDataset, you should be able to perform your logic on that dataset in a vendor neutral fashion.
I would also isolate the production of the datasets into a set of factory functions that only create the vendor-specific dataset and return it as a TDataset. Each factory function goes in it's own unit. Then, only those small units need have any knowledge of the vendor specific components.
You can use IProviderSupport to generalize the query execution. I am expecting, that used query's are supporting IProviderSupport. This interface allows to set command text, parameters, execute commands, etc.
The common denominator for used query's is TDataSet. So, you will need pass TDataSet reference.
For example:
var
oDS: TDataSet;
...
if NameOfDataBase = 'X' then
oDS := T1Query.Create(nil)
else
oDS := T2Query.Create(nil);
(oDS as IProviderSupport).PSSetCommandText('select * from mytab');
oDS.Open;
while not oDS.Eof do begin
//
oDS.Next;
end;
Perhaps you could consider a universal data access component such as UniDAC or AnyDAC. This allows you to use only one component set to access different databases in a consistent way.
You might also be interested in DataAbstract from RemObjects. A powerful data abstraction, multi-tier, remoting solution with a lot of features. Not inexpensive, but excellent value for the money.
Relevant links:
http://www.da-soft.com/anydac/
http://www.devart.com/unidac/
http://www.remobjects.com/da/
A simple approach would be to have ListAllTables return (or populate) a stringlist.
What do you people use for generating unique account numbers?
Some use Autoinc field, others something else...
What would be the proper way i.e to get an account number before I run the insert query?
If you are using a SQL database, use a Generator. If you want to use an independent mechanism you could consider using a GUID.
You haven't told us what database system you are using, but from the sound of it, you're talking about the Paradox tables in Delphi. If so, an autoInc column can work, although if I recall correctly, you have to be very careful when moving your data around with Paradox autoInc columns because they re-generate from zero when moved.
As has been mentioned, you can use GUIDs - sysutils.function CreateGUID(out Guid: TGUID): HResult; - they will always be unique, but the downside in GUIDS is that ordering by these keys will not be intuitive and probably be meaningless, so you'll need a timestamp column of some sort to maintain the order of your inserts, which can be important. Also, a GUID is a rather long character string and will not be very efficient for use as an account#, which assumedly will be a primary or foreign key in many tables.
So I'd stick to autoInc if you want something automatic, but if you have to move data around and you need to maintain your original keys, load your original autoincs as integer columns in their new location or you could end up corrupting your entire database. (I believe there are other scenarios that also cause autoIncs to reset in Paradox tables - research this if it's relevant - been a long time since I've used Pdox, and it may not be a problem with other flat file databases)
If you are indeed using a database server - SQLServer, Oracle, Interbase, etc, they all have autoInc/indentity or generator functionality, sometimes in conjuction with a trigger - that is your best option.
Dorin's answer is also an excellent solution if you want to handle this yourself from within your Delphi code. Create a global, thread safe function to implement it - that will ensure a very high level of safety.
HTH
Depending on how long you want the number, you can go with Jamies MD5 conversion or:
var
LDateTime: TDateTime;
LBytes: array[0..7] of Byte absolute LDateTime;
LAccNo: string;
Index: Integer;
begin
LDateTime := Now;
LAccNo := EmptyStr;
for Index := 0 to 7 do
LAccNo := LAccNo + IntToHex( LBytes[ Index ], 2 );
// now you have a code in LAccNo, use it wisely (:
end;
I use this PHP snippet to generate a decent account number:
$account_number = str_replace(array("0","O"),"D",strtoupper(substr(md5(time()),0,7)));
This will create a 7 digit varchar string that doesn't contain 0's or o's (to avoid errors on the phone or transcribing them in e-mails, etc.) You get something like EDB6DA6 or 76337D5 or DB2E624.
I want to read the entire contents of a table into memory, as quickly as possible. I am using Nexus database, but there might be some techniques I could use that are applicable to all database types in Delphi.
The table I am looking at has 60,000 records with 20 columns. So not a huge data set.
From my profiling, I have found the following so far:
Accessing tables directly using TnxTable is no faster or slower than using a SQL query and 'SELECT * FROM TableName'
The simple act of looping through the rows, without actually reading or copying any data, takes the majority of the time.
The performance I am getting is
Looping through all records takes 3.5 seconds
Looping through all the records, reading the values and storing them, takes 3.7 seconds (i.e. only 0.2 seconds more)
A sample of my code
var query:TnxQuery;
begin
query.SQL.Text:='SELECT * FROM TableName';
query.Active:=True;
while not query.Eof do
query.Next;
This takes 3.5 seconds on a 60,000 row table.
Does this performance sound reasonable? Are there other approaches I can take that would let me read the data faster?
I am currently reading data from a server on the same computer, but eventually this may be from another server on a LAN.
You should be using BlockRead mode with a TnxTable for optimal read speed:
nxTable.BlockReadOptions := [gboBlobs, gboBookmarks];
//leave out gboBlobs if you want to access blobs only as needed
//leave out gboBookmarks if no bookmark support is required
nxTable.BlockReadSize := 1024*1024; //1MB
// setting block read size performs an implicit First
// while block read mode is active only calls to Next and First are allowed for navigation
try
while not nxTable.Eof do begin
// do something....
nxTable.Next;
end;
finally
nxTable.BlockReadSize := 0;
end;
Also, if you don't need to set a range on a specifc index, make sure to use the sequential access index for fastest possible access.
I am looking the fastest way to insert many records at once (+1000) to an table using ADO.
Options:
using insert commands and parameters
ADODataSet1.CommandText:='INSERT INTO .....';
ADODataSet1.Parameters.CreateParameter('myparam',ftString,pdInput,12,'');
ADODataSet1.Open;
using TAdoTable
AdoTable1.Insert;
AdoTable1.FieldByName('myfield').Value:=myvale;
//..
//..
//..
AdoTable1.FieldByName('myfieldN').value:=myvalueN;
AdoTable1.Post;
I am using delphi 7, ADO, and ORACLE.
Probably your fastest way would be option 2. Insert all the records and tell the dataset to send it off to the DB. But FieldByName is slow, and you probably shouldn't use it in a big loop like this. If you already have the fields (because they're defined at design time), reference the fields in code their actual names. If not, call FieldByName once for each field and store them in local variables, and reference the fields by these when you're inserting.
Using ADO I think you may be out of luck. Not all back-ends support bulk insert operations and so ADO implements an abstraction to allow consistent coding of apparent bulk operations (batches) irrespective of the back-end support which "under the hood" is merely inserting the "batch" as a huge bunch of parameterised, individual inserts.
The downside of this is that even those back-ends which do support bulk inserts do not always code this into their ADO/OLEDB provider(s) - why bother? (I've seen it mentioned that the Oracle OLEDB provider supports bulk operations and that it is ADO which denies access to it, so it's even possible that the ADO framework simply does not allow a provider to support this functionality more directly in ADO itself - I'm not sure).
But, you mention Oracle, and this back-end definitely does support bulk insert operations via it's native API's.
There is a commercial Delphi component library - ODAC (Oracle Direct Access Components) for, um, direct access to Oracle (it does not even require the Oracle client software to be installed).
This also directly supports the bulk insert capabilities provided by Oracle and is additionally a highly efficient means for accessing your Oracle data stores.
What you are trying to do is called bulk insert. Oracle provides .NET assembly Oracle.DataAccess.dll that you can use for this purpose. There is no hand-made solution that you can think of that would beat the performance of this custom vendor library for the Oracle DBMS.
http://download.oracle.com/docs/html/E10927_01/OracleBulkCopyClass.htm#CHDGJBBJ
http://dotnetslackers.com/articles/ado_net/BulkOperationsUsingOracleDataProviderForNETODPNET.aspx
The most common idea is to use arrays of values for each column and apply them to a template SQL. In the example below employeeIds, firstNames, lastNames and dobs are arrays of the same length with the values to insert.
The Array Binding feature in ODP.NET
allows you to insert multiple records
in one database call. To use Array
Binding, you simply set
OracleCommand.ArrayBindCount to the
number of records to be inserted, and
pass arrays of values as parameters
instead of single values:
> 01. string sql =
> 02. "insert into bulk_test (employee_id, first_name, last_name,
> dob) "
> 03.
> + "values (:employee_id, :first_name, :last_name, :dob)";
> 04.
>
> 05. OracleConnection cnn = new OracleConnection(connectString);
> 06. cnn.Open();
> 07. OracleCommand cmd = cnn.CreateCommand();
> 08. cmd.CommandText = sql;
> 09. cmd.CommandType = CommandType.Text;
> 10. cmd.BindByName = true;
> 11.
>
> 12. // To use ArrayBinding, we need to set ArrayBindCount
> 13. cmd.ArrayBindCount = numRecords;
> 14.
>
> 15. // Instead of single values, we pass arrays of values as parameters
> 16. cmd.Parameters.Add(":employee_id", OracleDbType.Int32,
> 17. employeeIds, ParameterDirection.Input);
> 18. cmd.Parameters.Add(":first_name", OracleDbType.Varchar2,
> 19. firstNames, ParameterDirection.Input);
> 20. cmd.Parameters.Add(":last_name", OracleDbType.Varchar2,
> 21. lastNames, ParameterDirection.Input);
> 22. cmd.Parameters.Add(":dob", OracleDbType.Date,
> 23. dobs, ParameterDirection.Input);
> 24. cmd.ExecuteNonQuery();
> 25. cnn.Close();
As you can see, the code does not look that much different
from doing a regular single-record
insert. However, the performance
improvement is quite drastic,
depending on the number of records
involved. The more records you have to
insert, the bigger the performance
gain. On my development PC, inserting
1,000 records using Array Binding is
90 times faster than inserting the
records one at a time. Yes, you read
that right: 90 times faster! Your
results will vary, depending on the
record size and network
speed/bandwidth to the database
server.
A bit of investigative work reveals
that the SQL is considered to be
"executed" multiple times on the
server side. The evidence comes from
V$SQL (look at the EXECUTIONS column).
However, from the .NET point of view,
everything was done in one call.
You can really improve the insert performance by using the TADOConnection object directly.
dbConn := TADOConnection......
dbConn.BeginTrans;
try
dbConn.Execute(command, cmdText, [eoExecuteNoRecords]);
dbConn.CommitTrans;
except
on E:Exception do
begin
dbConn.RollbackTrans;
Raise e;
end;
end;
Also, the speed can be improved further by inserting more than one records at once.
You could also try the BatchOptmistic mode of the TADODataset. I don't have Oracle so no idea whether it is supported for Oracle, but I have used similar for MS SQL Server.
ADODataSet1.CommandText:='select * from .....';
ADODataSet1.LockType:=ltBatchOptimistic;
ADODataSet1.Open;
ADODataSet1.Insert;
ADODataSet1.FieldByName('myfield').Value:=myvalue1;
//..
ADODataSet1.FieldByName('myfieldN').value:=myvalueN1;
ADODataSet1.Post;
ADODataSet1.Insert;
ADODataSet1.FieldByName('myfield').Value:=myvalue2;
//..
ADODataSet1.FieldByName('myfieldN').value:=myvalueN2;
ADODataSet1.Post;
ADODataSet1.Insert;
ADODataSet1.FieldByName('myfield').Value:=myvalue3;
//..
ADODataSet1.FieldByName('myfieldN').value:=myvalueN3;
ADODataSet1.Post;
// Finally update Oracle with entire dataset in one batch
ADODataSet1.UpdateBatch(arAll);
1000 rows is probably not the point where this approach becomes economic but consider writing the inserts to a flat file & then running the SQL*Loader command line utility. That is seriously the fastest way to bulk upload data into Oracle.
http://www.oracleutilities.com/OSUtil/sqlldr.html
I've seen developers spend literally weeks writing (Delphi) loading routines that performed several orders of magnitude slower than SQL*Loader controlled by a control file that took around an hour to write.
Remenber to disable posible control that are linked to the Dataset/Table/Query/...
...
ADOTable.Disablecontrols;
try
...
finally
ADOTable.enablecontrols;
end;
...
You might try Append instead of Insert:
AdoTable1.Append;
AdoTable1.FieldByName('myfield').Value:=myvale;
//..
//..
//..
AdoTable1.FieldByName('myfieldN').value:=myvalueN;
AdoTable1.Post;
With Append, you will save some effort on the client dataset, as the records will get added to the end rather than inserting records and pushing the remainder down.
Also, you can unhook any data aware controls that might be bound to the dataset or lock them using BeginUpdate.
I get pretty decent performance out of the append method, but if you're expecting bulk speeds, you might want to look at inserting multiple rows in a single query by executing the query itself like this:
AdoQuery1.SQL.Text = 'INSERT INTO myTable (myField1, myField2) VALUES (1, 2), (3, 4)';
AdoQuery1.ExecSQL;
You should get some benefits from the database engine when inserting multiple records at once.