I'm testing a stored procedure on SQL Server 2008 R2 Express. Eventually it will be run in production on a full version of SQL Server 2008 R2.
Testing the stored procedure in Management Studio I notice that the first time I run it after a restore of the database or a restart of the SQL Server service the procedure takes about 35 seconds to run. The second and subsequent time it completes in a blink of an eye.
From this I assume that the query plan takes a while to be created. The symptoms seem to match with what causes the query plan to be recreated.
My problem is when I execute this stored procedure from my C# application if it goes over 30 seconds it times out. I think I have figured out how to increase the timeout but from all the threads I have read people say that in all but exceptional circumstances 30 seconds should be heaps of time. "Fix the problem not increase the timeout".
As this stored procedure will only be run once per month the chances are pretty good that it will create a new query plan every time it is used.
None of my other stored procedures have this problem.
Any ideas on the best way to diagnose the real issue?
The procedure isn't hugely complicated, it reads from a couple of joined tables and inserts about 4000 rows into another table based on a few conditions.
I'm not the best SQLer in the world so maybe I have done some silly things.
I can't really add indexes to the database or anything as it belongs to a business application written by an external company. So I'm very cautious making any changes as it may cause other unforseen problems.
Let me know if you think it is worth posting my sql code here.
Thanks
David
Looks like I answered my own question.
I clicked on the little "Display Estimated Query Plan" button in Management Studio and took a look at the query plan. In nice big letters it was recommending that I ad an index to a table and provided me the exact code short of an index name that I needed to use to create it. I took a look at that table and it has over 3 million rows in it. Not a lot in todays terms but obviously a problem if you have to keep reading it end to end which it appears to do while trying to create the query plan.
Anyway I created the index and run my application again and I can't reproduce the problem. It runs like a bullet every time.
I guess I just need to think about the consequences of me adding an index.
First will be to give it a name that the external company is never likely to duplicate. :)
Thanks to all who took the time to read my post and I hope this answer helps someone else.
David
Related
We have a Firebird 2.5 stored procedure that carries out multiple inserts & updates, and is called as a result of website actions, so can be called simultaneously.
We need to stop it running more than once at the same time, and I'm wondering whether there's a solution like there is with SQL server - something that can be called/set that effectively can 'lock' it on the way in, and 'unlocked' on the way out.
I appreciate that this is not the best solution - queuing being more satisfactory - but we need to get this sorted out quickly & don't have time to implement anything else at the moment. I think I recall seeing one somewhere, but have been unable to find any reference to it over the past few days.
I have two stored procedures, the first one (SP1) creates a table, and the second one (SP2) uses that table.
I cannot run both SP sequentially in one statement.
CALL SP1();
CALL SP2();
I get an error saying " Could not find table/view..."
when SP2 was created, the table existed due to testing SP1, but now I got rid of all the tables that are created temporarily during runtime SP2 doesn't run.
I realize I can just run SP1 and then SP2 and that will work, but there are 39 SPs and they take about 2 hours when running one after another, so waiting would make the process take even longer.
The only solution I can think of is to just leave the tables that are needed and truncate them rather than drop them. If nothing else can be done I will have to do that, but I rather drop them. The person that created this SPs was evil enough to create over 500 of these tables so I don't want them permanently in my schema.
As a side note, these tables are used for debugging mainly so they can be dropped after a few days of execution, once the processes validated.
I wonder if there is any way to ignore the error or something like or any other workaround.
Thanks in advance for your time and expertise.
Regards,
Leonardo
I found a way to run it regardless of the missing tables.
I just had to put each SP in a different DO BEGIN.... END; block, that seems to be only validating one block at a time and since the previous block creates the table that use going to be used on block number 2, everything seems to be working fine.
DO
BEGIN
CALL SP1();
END;
DO
BEGIN
CALL SP2();
END;
and so on.
The blocks have a bit more things in them, I just put the SP call to simplify them.
I am just leaving this here in case someone else has the same problem as I have.
I would still appreciate it if someone has a better solution.
This sounds a lot like that those tables are in fact temporary tables used to "transfer" data between the different stored procedures.
With HANA it is not necessary nor recommended to dynamically create tables for that use.
Instead, use table variables and "hand over" the data via table typed parameters.
That way, no tables have to be created/dropped at runtime at all.
Also, since table variables are effectively handled as internal, session local temporary tables, a lot of overhead is removed (i.e. those tables don't need to take care of logging or concurrent access) which makes them a lot more lightweight and better performing than dynamically created tables.
Another negative aspect of dynamically creating objects is that this always comes with locking of the database catalog, checking for and invalidating potentially dependent objects, and numerous internal caches.
If possible, one really should avoid this if performance is important.
I'm developing a software that displays information in a DBGrid via a TSimpleDataSet (dbExpress components)
The software in question is used on 2 different computers by 2 different people.
They both view and edit the same information at different times.
I'm trying to figure out a way to automatically update the DBGrid (or rather, the DataSet, right?) on Computer B once Computer A makes a change to a row (edits something/whatever) and vice-versa.
Currently I've set up a TButton named Refresh that once clicked executes the following code:
procedure TForm2.actRefreshDataExecute(Sender: TObject);
begin
dbmodule.somenameDataSet.MergeChangeLog;
dbmodule.somenameDataSet.ApplyUpdates(-1);
dbmodule.somenameDataSet.Refresh;
dbmodule.somename1DataSet.MergeChangeLog;
dbmodule.somename1DataSet.ApplyUpdates(-1);
dbmodule.somename1DataSet.Refresh;
dbmodule.somename2DataSet.MergeChangeLog;
dbmodule.somename2DataSet.ApplyUpdates(-1);
dbmodule.somename2DataSet.Refresh;
dbmodule.somename3DataSet.MergeChangeLog;
dbmodule.somename3DataSet.ApplyUpdates(-1);
dbmodule.somename3DataSet.Refresh;
end;
This is fine and works as intended, once clicked.
I'd like an auto update feature for this, for example when Computer A edits information in a row, Computer B's DBGrid should update it's display accordingly, without the need to click the refresh button.
I figured I would use a TTimer and set it at a specific interval, on both software on both PC's.
My actual question is:
Is there a better way than a TTimer for this? If so, please elaborate.
Also, if the TTimer route is the way to go any further info you might find useful to state would be appreciated (pro's and con's and so on)
I'm using Rad Studio 10 Seattle and dbExpress components, the datasets connect to a MySQL database on my hosting where my website is.
Thanks!
Well, Ken White and Sertac Akyuz are certainly correct that using a server-originated notification to determine when to refresh your local dataset is preferable to continually re-reading all the data you are using from the server.
The problem AFAIK is that there is no Emba-supplied notification system which works with MySql. See this list of databases supported by FireDAC's Database Alerts:
http://docwiki.embarcadero.com/RADStudio/XE8/en/Database_Alerts_(FireDAC)
and note that it does not list MySql.
Luckily, I think there is a work-around which should be viable for a v. small system like yours currently is. As I understand it, you and your colleague's PCs are on a LAN and the MySql Server is outside your LAN and on the internet. In that situation, it doesn't need a round trip to the server for one of you to get a notification that the other has changed something in the database. Using an analogy akin to Ken's, you can, as it were, lean over the desk and say to your colleague "Hey, I've changed something, so you need to refresh your data."
A very low-tech way of implementing that would be to have somewhere on your LAN a resource that both of you can easily get at, which you can update when you make a change to the DB that means that the other of you should update your data from the server. One way to do that is to have a small, shared datafile with a number of records in it, one per server db table, which has some sort of timestamp or version-ID number which gets updated when you update the corresponding server table. Then, you can periodically check (poll) this datafile to see whether a given table has changed since you last checked; obviously, if it has, you then re-read the data you want from it from the server and update your local record of the info you read from the shared file.
You can update the shared file using handlers for the events of your Delphi client-side datasets.
There are a number of variations on this theme that I'm sure will be apparent to you; the implementational details really don't matter.
To update the shared file I'm talking about, you will need to lock it while writing to it. This answer:
How do I get the handle for locking a file in Delphi?
will show you how to do that.
Of course, the shared local resource doesn't have to be a data file. One alternative would be to use a Microsoft Message Queue service, which is sometimes used for this kind of thing, but has a steeper learning curve than a shared data file.
By the way, this kind of thing is far easier to do (at least on a small scale like you have) if you use 3-tier database access (e.g. using datasnap).
In a three tier system, only the middle tier (a Delphi datasnap server which you write, but it's not that hard) talks to the server, and the clients only talk to the middle tier. This makes it easy for the middle tier server to notify the other client(s) when one of them changes the db data.
The three-tier arrangement also helps minimise the security problems with accessing a database server via the internet, because you only need one secure connection to the server, not one per client. But that's straying a bit far from your immediate problem.
I hope all this is clear, if not, ask.
Just use a timer and make it refresh the dataset every 5 min. No big deal.
If the usage is not frequent then you can set it to fire every 10 or 15 min.
There is nothing wrong with the timer if it set on longer intervals.
Today's broadband connection's can easily handle the traffic so can Access.
If the table is not huge of course.
I have a multiuser delphi program which has Firebird database behind it.
And I want 2 user can insert 2 records same time but with given automated number for a field.
On the other hand I am not sure Firebird is eligible for this without one use commit and close the table. And the other one refreshing it...
I heard bad things about commitretaining and I don't now what to do now. Like:
Which transaction setting is best for me?
Wait or No-wait if I have to use commitretaining how can I do that safely?
Use GENERATORS. With GENERATORS you get always unique numbers. It doesn't matter how many transactions are active, they live outside the transaction control.
See Firebird Generator Guide
I’m using SSIS to synchronize data between two databases. I’ve used SSIS and DTS in the past, but I generally write an application for things of this nature (I’m coder and it just comes easier to me).
In my package I use a SQL Task that returns about 15,000 rows. I’ve hooked that up to a Foreach Container, and within that I assign the resultset column values to variables, and then map those variables to parameters that are fed to another SQL Task.
The problem I’m having is with debugging, and not just more complicated debugging like breakpoints and evaluating values at runtime. I simply mean that if I run this with debugging rather than without, it takes hours to complete.
I ended up rewriting the process in Delphi, and the following is what I came up with:
Full Push of Data:
This pulls 15,000 rows, updates a destination table for each row, then pulls 11,000 rows and updates a destination table for each row.
Debugging:
Delphi App: 139s
SSIS: 4 hours, 46 minutes
Not Debugging:
Delphi App: 132s
SSIS: 384s
Update of Data:
This pulls 3,000 rows, but no updates are needed or made to the destination table. It then pulls 11,000 rows but, again, no updates are needed or made to the destination table.
Debugging:
Delphi App: 42s
SSIS: 1 hours, 10 minutes
Not Debugging:
Delphi App: 34s
SSIS: 205s
The odd thing is, I get the feeling that most of this time spent debugging is just updating UI elements in Visual Studio. If I watch the progress tab, a node is added to a tree for each iteration (thousands total), and this gets slower and slower as the process goes on. Trying to stop debugging usually doesn’t work, as Visual Studio seems caught in a loop updating the UI. If I check the profiler for SQL Server no actual work is being done. I'm not sure if the machine matters, but it should be more than up to the job (quad core, 4 gig of ram, 512 mb video card).
Is this sort of behavior normal? As I’ve said I’m a coder by trade, so I have no problem writing an app for this sort of thing (in fact it takes much less time for me to code an application than “draw” it in SSIS, but I figure that margin will shrink with more work done in SSIS), but I’m trying to figure out where something like SSIS and DTS would fit into my toolbox. So far nothing about it has really impressed me. Maybe I’m misusing or abusing SSIS in some way?
Any help would be greatly appreciated, thanks in advance!
SSIS control flow and loops are not very high performance, and not designed for processing these amounts of data. Especially during the debugging - before and after each task execution, debugger sends notifications to designer process, which updates colors of the shapes and this could be slow.
You could get much better performance using data flow. Data flow does not operate with single rows, it works with buffers of rows - much faster, and the debugger is only notified about beginning/end of the buffers - so its impact is less noticeable.
SSIS is not designed to do a foreach like that. If you are doing something for each row coming in, you probably want to read those into a dataflow and then using a lookup or merge join, determine whether to do an INSERT (these happen in bulk) or a database command object for multiple SQL UPDATE commands (a better performing option is to batch these into staging table and do a single UPDATE).
In another typical sync situation, you read all the data into a staging table, and do a SQL Server UPDATE on the existing rows (INNER JOIN) and INSERT on the new rows (LEFT JOIN, rhs IS NULL). There is also the possibility of using linked servers, but joins over that can be slow, since all (or a lot of) the data may have to come across the network.
I have SSIS packages that regular import 24 million rows, including handling data conversion and validation and slowly changing dimensions using the TableDifference component, and it performs relatively quickly for that large amount of data versus a separate client program.
I have noticed this is the behavior, I had an SSIS package for moves, that did somewhere in the neighborhood of 3 million entries, it was not possible to debug as it would run for about 3-4 days.
SSIS is still the way I did it, I just don't "debug" with SSIS, I run them when working with the full datasets. If I must debug, I use very small datasets.