I'm trying to track down a performance issue by looking at the "Active Queries" tab in the Advantage Management Utility.
The documentation for this tab says:
Active: True if the query is being actively processed by the server. A query must be active to be cancelled.
Is a query Active until it completes? Or can it become inactive for another reason, like waiting on a resource (disk IO or a lock)?
I ask because I only 1-2 queries in an "active" state at a given time, but I also have 20+ worker threads running. Which makes little sense to me.
Active means the server is actively looking for rows to populate the cursor with for the request. It will remain active until enough rows have been made to satisfy the request. If the query needs to wait on a lock or disk I\O it will remain active. One caveat to this is live cursors. Live cursors are treated as tables by the client rather than SQL statements. Are the SQL statements that are open, but not active live cursors?
You might try calling the stored procedure sp_mgGetWorkerThreadActivity to see what commands the other threads are doing.
Related
My idea is to use ETS as a temporary cache for my GenServer state.
For example when I restart my application, the GenServer state should be transported to ETS and when the application starts again, the GenServer should be able to get the state from there.
I would like to keep it simple, so all the state (a Map) from the GenServer should take a single entry. Is there a limit for entry-sizes?
Another approach would be, to simply create a file, and load it again, when needed. Maybe this is even better/simpler, but I am not sure :)
In case of an ETS table, the App could start on a completely other host, and connect to the Cache Node (ETS).
This can most certainly be done in a wide variety of ways. You could have a separate store, like Mnesia(which is ETS under the hood), Redis or a plain database. In the latter two cases, you would need to cast your Genserver state to a string and back doing: :erlang.term_to_binary and :erlang.binary_to_term respectively.
In the case that you are dealing with multiple GenServer processes that need to be cached in this way, e.g. every GenServer represents a unique customer cart for instance, then that unique identifier can be utilized as the key under which to store the state which can then later on be retrieved. This is particularly useful when you are running your shopping application on multiple nodes behind a load balancer, and every new request on part of a customer can get 'round robin'-ned around to any random node.
When the request comes in:
fetch the unique identifier belonging to that customer in one way or the other,
fetch the stored contents from wherever that may be(Mnesia/Redis/...),
spawn up a new GenServer process initialized with that stored contents,
do the various operations required for that request,
store the latest modified GenServer shopping cart into Redis/Mnesia/wherever,
tear down the GenServer and
respond to the request with whatever data is required.
Based on the Benchmarks I have done of ETS vs Redis on my local, it is no surprise that ETS is the more performant way to go, but ElastiCache is an awesome alternative if you are not in the mood to bother spinning up a dedicated Mnesia store.
In the case it pertains to a specific GenServer that needs to run, then you are most likely looking at failover as opposed to managing individual user requests.
In such a case, you could consider using something like: https://hexdocs.pm/elixir/GenServer.html#c:terminate/2 to have the state first persisted to some store and in your init make the GenServer first look in that store and reuse the cache accordingly.
The complicated matter here is in the scenario where you have multiple applications running, which key will you utilize in order to have the crashed application reinitialize the GenServer with the correct state?
There are several open ended questions over here that revolve around your exact use case, but what has been presented so far should give you a fair idea as to when it makes sense to utilize this caching solution and how to start implementing it.
I have a very strange problem with transactions in Interbase 7.5 which seem to be stuck.
I can track the problem with IBConsole -> right click DB -> Performance Monitor -> Transactions
Usually this list should show only a few active transaction. But I get several hundred active transactions when I start my application (a web module for an apache webserver using Delphi 7 Interbase components, e.g. IBQuery, IBTransaction, ...)
Transaction type is always listed as snapshot, if this is of relevance.
I have already triple checked all sql statements and cannot find anything that should produce such problems...
Is there any way get the sql statements of a specific transaction?
Any other suggestion how to find such a problem would be very welcome.
Is there any way get the sql statements of a specific transaction?
Yes, you can SELECT from TMP$STATEMENTS WHERE TRANSACTION_ID = .... That's from memory, but should get you started.
In IB Performance Monitor, you can locate the transaction from the statements tab, using the button on the toolbar. Can't remember if you can go the other way in that app. It's been a long time since I wrote it!
Active IBX data-sets require an active transaction all the time. If you don't have active data-sets just don't forget to commit all the active transactions.
If you have active data-sets, you can configure all your components to use the same TIbTransaction object, and you can also configure the unique TIbTransaction to commit or rollback after a idle time-out period via the IdleTimer and DefaultAction properties.
Terminating the transaction (by manually or automatically committing or rolling back) will close all the linked datasets (TIBQuery, TIBTable and the like).
You may be tempted to use the CommitRetaining or RollbackRetaining methods to terminate the transaction without closing the related data-sets, but this may affect the performance of the server, and my advise is to always avoid using it.
If you want to improve your application, you should consider changing your database connection layer or introducing a in-memory capable dataset over IBX, for example, Delphi's TClientDataSet, which allows you to retrieve data and retain it in memory while closing all the underlying datasets (and transactions), while allowing you to use the traditional Insert/Append/Edit/Delete methods to modify the data and then apply that changes to the database in a new short-time transaction.
My question's pretty simple, but I'm somewhat new ruby on rails (and web programming at that) and can't find a good answer. How do I ensure that two (or more) database transaction occur atomically on Heroku?
Can I code the application with mutexes, as I would any multi-threaded application, and expect it to work properly in a distributed server environment like Heroku? If not, what's the best way to ensure the atomicity of two or more database transactions?
I know Heroku uses PostgreSQL which lets me acquire an array of database locks, but these locks only persist for one transaction. Furthermore, I'm familiar with mutexes (and pthreads) to synchronize multi-threaded applications, but I'm not sure if these conventions will work as expected on Heroku, especially if my actual application code is running on multiple servers.
EDIT
I definitely should explain what I'm trying to do a little more. I need to make two calls to the database. The first call is a read of a boolean value. If that boolean value is true, I need to do some action, and if it is false I need to do something else. If I read a TRUE, then I need to do some other stuff (interface with external APIs etc...), and if that succeeds then I need to set the boolean to false. The problem is that I don't know how to avoid the situation where multiple clients read true from the database, when, in actuality, one client is en route to write false but hasn't yet completed the other stuff I need to do before writing false.
Here's an alternative that might work:
Modify your boolean field to allow storing a tri-state variable. (Date/time might be nice too, I'll explain in a bit.)
Then code the operations like this:
begin transaction
read flag
if true
set flag to "processing" state
if false
perform other activity
commit
If the flag was true, then it will now be set to processing; call the external API, do whatever, and when the results are known:
begin transaction
read flag
if "processing"
set flag to false
if true
warn loudly
if false
warn loudly
commit
Of course, if the process handling the external API dies or blocks for too long, you'll want to know, and be able to recover. That's why it would be nice to store a date/time into the flag -- if it has been in the state processing for six minutes, then kill it and restart.
There would be something nice about using posix semaphores or similar tools, but I would expect your Rails backend processes might execute on different machines from minute to minute.
Althoug both Jet/OLE DB parameters are relativly well documented I fail to understand the difference between these two connection parameters:
The first one:
Jet OLEDB:Transaction Commit Mode
(DBPROP_JETOLEDB_TXNCOMMITMODE)
Indicates whether Jet writes data to
disk synchronously or asynchronously
when a transaction is committed.
The second one:
Jet OLEDB:User Commit Sync
(DBPROP_JETOLEDB_USERCOMMITSYNC)
Indicates whether changes that were
made in transactions are written in
synchronous or asynchronous mode.
What's the difference? When to use which?
This is very long, so here's the short answer:
Don't set either of these. The default settings for these two options are likely to be correct. The first, Transaction Commit Mode, controls Jet's implicit transactions, and applies outside of explicit transactions, and is set to YES (asynchronous). The second controls how Jet interacts with its temporary database during an explicit transaction and is set to NO (synchronous). I can't think of a situation where you'd want to override the defaults here. However, you might want to set them explicitly just in case you're running in an environment where the Jet database engine settings have been altered from their defaults.
Now, the long explanation:
I have waded through a lot of Jet-related resources to see if I can find out what the situation here is. The two OLEDB constants seem to map onto these two members of the SetOptionEnum of the top-level DAO DBEngine object (details here for those who don't have the Access help file available):
dbImplicitCommitSync
dbUserCommitSync
These options are there for overriding the default registry settings for the Jet database engine at runtime for any particular connection, or for permanently altering the stored settings for it in the registry. If you look in the Registry for HLKM\Software\Microsoft\Jet\X.X\ you'll find that under the key there for the Jet version you're using there are keys, of which two are these:
ImplicitCommitSync
UserCommitSync
The Jet 3.5 Database Engine Programmer's Guide defines these:
ImplicitCommitSync: A value of Yes indicates that Microsoft Jet will wait for commits to finish. A value other than Yes means that Microsoft Jet will perform commits asynchronously.
UserCommitSync: When the setting has a value of Yes, Microwsoft Jet will wait for commits to finish. Any other value means that Microsoft Jet will perform commits asynchronously.
Now, this is just a restatement of what you'd already said. The frustrating thing is that the first has a default value of NO while the second defaults to YES. If they really were controlling the same thing, you'd expect them to have the same value, or that conflicting values would be a problem.
But the key actually turns out to be in the name, and it reflects the history of Jet in regard to how data writes are committed within and outside of transactions. Before Jet 3.0, Jet defaulted to synchronous updates outside of explicit transactions, but starting with Jet 3.0, IMPLICIT transactions were introduced, and were used by default (with caveats in Jet 3.5 -- see below). So, one of these two options applies to commits OUTSIDE of transactions (dbImplicitCommitSync) and the other for commits INSIDE of transactions (dbUserCommitSync). I finally located a verbose explanation of these in the Jet Database Engine Programmer's Guide (p. 607-8):
UserCommitSynch
The UserCommitSynch setting determines
whether changes made as part of an
explicit transaction...are written to
the database in synchronous mode or
asynchronous mode. The default value...is Yes, which specifies
asynchronous mode. It is not
recommended that you change this value
because in synchronous mode, there is
no guarantee that information has been
written to disk before your code
proceeds to the next command.
ImplicitCommitSync
By default, when
performing operations that add,
delete, or update records outside of
explicit transactions, Microsoft Jet
automatically performs internal
transactions called implicit
transactions that temporarily save
data in its memory cache, and then
later write the data as a chunk to the
disk. The ImplicitCommitSync setting
determines whether changes made by
using implicit transactions are
written to the database in synchronus
mode or asynchronous mode. The default
value...is No, which specifies that
these changes are written to the
database in asynchronous mode; this
provides the best performance. If you
want implicit transactions to be
written to the database in synchronous
mode, change the value...to Yes. If
you change the value...you get
behavior similar to Microsoft Jet
versions 2.x and earlier when you
weren't using explicit transactions.
However, doing so can also impair
performance considerably, so it is not
recommended that you change the value
of this setting.
Note: There is no longer a need to use
explicit transactions to improve the
performance of Microsoft Jet. A
database application using Microsoft
Jet 3.5 should use explicit
transactions only in situations where
there may be a need to roll back
changes. Micosoft Jet can now
automatically perform implicit
transactions to improve performance
whenever it adds, deletes or changes
records. However, implicit
transactions for SQL DML statements
were removed in Microsoft Jet
3.5...see "Removal of Implicit Transactions for SQL DML Statements"
later in this chapter.
That section:
Removal of Implicit Transactions for SQL DML Statements
Even with all the work in Microsoft
Jet 3.0 to eliminate transactions in
order to obtain better performance,
SQL DML statements were still placed
in an implicit transaction. In
Microsoft Jet 3.5, SQL DML statements
are not placed in an implicit
transaction. This substantially
improves performance when running SQL
DML statements that affect many
records of data.
Although this change provides a
substantial performance improvement,
it also introduces a change to the
behavior of SQL DML statements. When
using Microsoft Jet 3.0 and previous
versions that use implicit
transactions for SQL DML statements,
an SQL DML statement rolls back if any
part of the statement is not
completed. When using Microsoft Jet
3.5, it is possible to have some of the records committed by SQL DML
statement while others are not. An
example of this would be when the
Microsoft Jet cache is exceeded. The
data in the cache is written to disk
and the next set of records is
modified and placed in the cache.
Therefore, if the connection is
terminated, it is possible that some
of the records were saved to disk, but
others were not. This is the same
behavior as using DAO looping routines
to update data withoug an explicit
transaction in Microsoft Jet 3.0. If
you want to avoid this behavior, you
need to add explicit transactions
around the SQL DML statement to define
a set of work and you must sacrifice
the performance gains.
Confused yet? I certainly am.
The key point to me seems to me to be that dbUserCommitSync seems to control the way Jet writes to the TEMPORARY database it uses for staging EXPLICIT transactions, while dbImplicitCommitSync relates to where Jet uses its implicit transactions OUTSIDE of an explicit transaction. In other words, dbUserCommitSync controls the behavior of the engine while inside a BeginTrans/CommitTrans loop, while dbImplicitCommitSync controls how Jet behaves in regard to asynch/synch outside of explicit transactions.
Now, as to the "Removal of Implicit Transactions" section: my reading is that implicit transactions apply to updates when you're looping through a recordset outside of a transaction, but no longer apply to a SQL UPDATE statement outside a transaction. It stands to reason that an optimization that improves the performance of row-by-row updates would be good and wouldn't actually help so much with a SQL batch update, which is already going to be pretty darned fast (relatively speaking).
Also note that the fact that it is possible to do it both ways is what enables DoCmd.RunSQL to make incomplete updates. That is, a SQL command that would fail with CurrentDB.Execute strSQL, dbFailOnError, can run to completion if executed with DoCmd.RunSQL. If you turn off DoCmd.SetWarnings, you don't get a report of an error, and you don't get the chance to roll back to the initial state (or, if you are informed of the errors and decide to commit, anyway).
So, what I think is going on is that SQL executed through the Access UI is wrapped in a transaction by default (that's how you get a confirmation prompt), but if you turn off the prompts and there's an error, you get the incomplete updates applied. This has nothing to do with the DBEngine settings -- it's a matter of the way the Access UI executes SQL (and there's an option to turn it off/on).
This contrasts to updates in DAO, which were all wrapped in the implicit transactions starting with Jet 3.0, but starting with Jet 3.5, only sequential updates were wrapped in the implicit transactions -- batch SQL commands (INSERT/UPDATE/DELETE) are not.
At least, that's my reading.
So, in regard to the issue in your actual question, in setting up your OLEDB connection, you'd set the options for the Jet DBEngine for that connection according to what you were doing. It seems to me that the default Jet DBEngine settings are correct and shouldn't be altered -- you want to use implicit transactions for edits where you're walking through a recordset and updating one row at a time (outside of an explicit transaction). On the other hand, you can wrap the whole thing in a transaction and get the same result, so really, this only applies to cases where you're walking a recordset and updating and have not used an explicit transaction, and the default setting seems quite correct to me.
The other setting, UserCommitSync, seems to me to be something you'd definitely want to leave alone as well, as it seems to me to apply to the way Jet interacts with its temp database during an explicit transaction. Setting it to asynchronous would seem to me to be quite dangerous as you'd basically not know the state of the operation at the point that you committed the data.
You'd think that USERCOMMITSYNC=YES would be the option to commit synchronously. And that is the cause of the confusion.
I spent ages googling on this topic because I found that the behavior I was getting with old vb6 applications was not the same as I get in .net oledb/jet4
Now I really should back up what I'm going to say with a link to the actual page(s) I read but I can't find those pages now.
Anyway, I was browsing MSDN website and found a page that described a 'by design' error in Jet3 which transposed the functionality of USERCOMMITSYNC meaning a value of NO gets synchronous commit.
Therefore MS set the default to NO and we get synchronous commit by default. Exactly as described above by David Fenton. A behavior we've all come to accept.
But, the document then went on to explain that the behavior in oledb/Jet4 has been changed. Basically MS fixed their bug and now a setting of USERCOMMITSYNC=YES does what it says.
But did they change the default? I think not because now my explicit transactions are NOT committing synchronously in .Net applications using oledb/jet4.
I have a website where the user can upload an excel spreadsheet to load data in a table. There can be a few 100k rows in the excel spreadsheet. When he uploads the file the website needs to insert an equal amount of rows in a database table.
What strategy should i take to do this? I was thinking of displaying a "Please wait page" until the operation is completed but i want him to be able to continue browsing the website. Also, since the database at that time will be kind of busy - wouldn't that stop people from working on the website?
My data access layer is in NHibernate.
Thanks,
Y
Displaying a please wait page would be pretty unfriendly as your user could be wating quite a while and would block threads on your web server.
I would upload the file, store it and create an entry in a queue (you'll need anouther table for this) to indicate that there is a file waiting to be processed. You can then have another process (which could even run on it's own server) which picks up tasks from this queue table and processes the xls file in it's own time.
I would create an upload queue that would submit this request to. Then the user could just check in on the queue every once in a while. You could store the progress of the batch operation in the queue as the rows are processed.
Also, database servers are robust, powerful, multi-tasking systems. Unless you have observed a problem with the website while the inserts are happening don't assume it will stop people from working on the website.
However, as far as insert or concurrent read/write performance goes there are mechanisms to deal with this. You could use the "INSERT LOW PRIORITY" syntax in MySQL or have your application throttle the inserts by sleeping a millisecond between each insert. Also, how you craft your insert statements, wether you use bound parameters or not, and wether you use multi-valued inserts can affect the insert performance and how it affects clients to a large degree.
On Submit you could pass the DB Operation to a asynchronous RequestHandler and set a Session Value when its done.
While the asynch process is in progress you can check the Session Value on each request and if it is set (operation = completed) display a message, eg in a modal or whatever message mechanism you have.