OLE DB has a sneaky feature where if your current connection is busy, it will silently open more database connections.
these happen without your knowledge
and they are not taken from, or returned to, the connection pool
With the SQL Server 2005 "native client", Microsoft introduced a feature where one connection could support Multiple Active Record Sets; so that you can officially have multiple recordsets active over a single connection. But they note it's a tricky thing to enabled, which is why it's been opt-in since beta 2 of the feature.
Secret connections? Really?
Microsoft notes this behavior:
Using ADO with SQL Server Native Client
In prior versions of the OLE DB provider, this code would cause an implicit connection to be created on the second execution because only one active set of results could be opened per a single connection. Because the implicit connection was not pooled in the OLE DB connection pool this would cause additional overhead. With the MARS feature exposed by the SQL Server Native Client OLE DB provider, you get multiple active results on the one connection.
It's also noted by John C. Gordon [MSFT] in the Microsoft forums: (archive)
In OLE DB, there is something called an "implicit connection" spawned when a query is performed while an active resultset is still pending. These are not apparent to the user, as the original poster noticed. I know that SQLNCLI, SQLNCLI10 implement these, but it I do not remember if SQLOLEDB does. What then happens is the server has 2 connections and each one has one pending resultset. This is unpleasant when the licensing model you choose is by connection.
It's also noted in a MARS announcement blog entry: (archive)
Using MARS with SQL Native Client [Chris Lee]
The second result set is using a new connection each time it is opened. This clearly has some overhead (and it turns out that the additional connections aren’t pooled, so the overhead is for a full server connection network protocol exchange each time). This is the default behavior for SQLOLEDB and SQL Native Client (OLE DB) – a new implicit connection is spawned when the main connection is busy with a default result set.
It's a feature deep inside OLE DB, designed to make data access easier.
Bonus Chatter
OLE DB is a low level, complicated, API
ADO is simplified wrapper around OLE DB
How to turn it off?
This automatic creation of secondary connections is obviously not good. It's so not good that the SQL Server ODBC driver does not do it (because it's a feature deep inside OLE DB, and comes to SQL Server's OLE DB drivers for free. It's not a feature inside ODBC).
I want to make sure i'm not doing it. To this end i want the driver to throw an error at the point in my code where i accidentally try to do this.
How can i turn off OLE DB implicit additional connections?
The solution is to disable to feature where OLEDB will open multiple connections behind your back.
You can set the Multiple Connections property of your ADO Connection to false to prevent it from implicitly opening 2nd connections when the main connection is busy with a results set.
//Set Connection.Properties["Multiple Connections"] = false
for int i = 0 to Connection.Properties.Count-1
{
Property prop = Connection.Properties.Item[i];
if (prop.Name == "Multiple Connections")
{
prop.Value = false;
break;
}
}
Note that turning off the ability to open a 2nd connection will then mean that code that used to appear to work will suddenly start to fail with the error "An object was open":
OleDbErr.h
//
// MessageId: DB_E_OBJECTOPEN
//
// MessageText:
//
// An object was open
//
#define DB_E_OBJECTOPEN ((HRESULT)0x80040E05L)
Warning
The Multiple Connections property doesn't apply to the ODBC OLEDB driver (MSDASQL), or the Active Directory driver. (ADsDSOObject)
In fact, they very act of querying for the "Multiple Connections" property of
the ADsDSOObject (active directory) provider causes it to throw an
"Item could not be found in the collection"
exception later on when running a SELECT query.
Even commenting out the code that sets:
prop.Value = false;
still causes the error.
It's the very act of calling asking for a property that doesn't exist that causes it to fail. That's why we iterate properties, and only set it if we find it: bugs in OLEDB drivers.
Related
I'm in process of upgrading our client-server ERP app to multi-tier. We want to offer our customers the possibility having their databases in cloud(hosted in our server).So, clients are written in Delphi, server is a http IOCP server written also in Delphi (from mORMot framework), for dbs we use Firebird embedded.
Our customers(let's say 200), can have 25-30 Firebird databases (total 5000-6000 databases), accessed by 4-5 user per each customer. This is not happening all at once. One user can work in one db, other 2 users can work in another db, but all the dbs should be available and online. So, I can have 800-1000 users working at 700-900 dbs. Databases are not big, typically 20-30 MB each but can go to 200 MB.
This is not data sharding so please don't suggest to merge all databases together, I really need them individually with possibility to backup/restore/replace each one of them.
So, I need multiple pools of connections - for every database I need a pool of let's say 2 connections. I read about Firedac connection pooling. It seems that TFDManager should be perfect for me. I define multiple "ConnectionDef"s with "Pooled=true" and it can maintain multiple pools of connections (each connection lasting until some minutes of inactivity).
Questions:
I have to create all "ConnectionDef"s before server starts serving requests?
Can TFDManager "handle" requests (and time-out connections on inactivity), while in other thread I need to create a new database , so automatically I need to create a new pool of connections and start serving requests from newly created database. Practically can I call FDManager.AddConnectionDef(..) while other pools are in use?
AFAIK Firebird embedded does not have any "connection". As its name states, it is embedded within the same process, so there is no connection pool needed. Connection pools are needed when several clients connect/disconnect to the same DB over the network, whereas here all is embedded, and you would have direct access to the Firebird engine.
As such, you may:
Define one "connnection" per Firebird embedded database;
Protect your SOA code via a mutex (aka critical section) per DB. In fact, mORMot's HTTP IOCP server would run the incoming requests from a thread pool, so ensure that all DB access is safely made.
Ensure you use at least Firebird 2.5 since the embedded version is told to be threadsafe only since revision 2.5 (see the release notes).
Instead of FireDAC, consider using ZDBC/Zeos (in latest 7.2/7.3 branch), which has nice features, together with the native mORMot SynDB libraries.
Looking at Firedac sources, seems that all about adding connection definitions and acquiring connections in pooled mode is thread-safe.
Adding a connection definition or matching one is guarded by a TMultiReadExclusiveWriteSynchronizer and acquiring a connection from the pool is guarded by a TCriticalSection.
So, answers:
I don't have to create all "ConnectionDef"s before server starts serving requests.
Yes, I can call safely FDManager.AddConnectionDef(..) while other pools are in use.
Using Firedac, acquiring a connection for any of those databases will be guarded by one TCriticalSection. The solution proposed by #Arnaud Bouchez presents a more grained access by creating one TCriticalSection per database and I think will scale better, but you should be aware of a bug when using multiple TCriticalSection, especially that all will be initiated at once:
https://www.delphitools.info/2011/11/30/fixing-tcriticalsection/
In that article present a very simple fix for this bug.
I built an Web API with Delphi using FireDac and an HTTPServer component: The application is using a dbms powered by firebird.
Everything was working fine until I the moment I started to simulate multiple requests to the same API endpoint. This is causing internal server exceptions reporting that a second trasaction is being opened when there are already a transaction opened.
I know that all connections are being closed after being used and objects are being destroyed in order to prevent memory leaks but I couldn't understand why do the application triggers the exception.
Any help or toughs that might drive me to a solution?
Multiple requests will be processed concurrently by the HTTP server.
So if two clients try to access the same resource (URL) at the same time, the server will need two sets of database connections and data access components.
If your application uses distinct objects - one set per client - and does this in a thread safe way, both connection should work fine.
However if you use only one datamodule to serve all incoming HTTP requests, proper serialization is required. It does not help to close connections after use, the connections must be used only from one thread at a time.
So to understand the potential reason of the error, more information about the actual design of your server is needed.
Which use of connection management is better while developing a windows based application which uses a Database as its data store? What about web-based applications?
when user loads the first form of an application, the global
connection opens and on closing the last form of the application
the connection closes and disposes.
for each form within the application, there is a local connection
(form scope) and when user wants to perform an operation like
insert, update, delete, search, ... the application uses the
connection and by unloading the form the connection also closes and
disposes.
for every operation within a form of an application, there is a
local connection (procedure scope) and when user wants to perform
an operation like insert, update, delete, search, ... the
application uses procedure connection and at the end of every
procedure within the form, the connection also closes and disposes.
Go with #3
You should try to only ever keep connections open for just as long as is required.
Also have a look at
Understanding Connection Pooling
SQL Server Connection Pooling
(ADO.NET)
Connecting to a database server
typically consists of several
time-consuming steps. A physical
channel such as a socket or a named
pipe must be established, the initial
handshake with the server must occur,
the connection string information must
be parsed, the connection must be
authenticated by the server, checks
must be run for enlisting in the
current transaction, and so on.
In practice, most applications use
only one or a few different
configurations for connections. This
means that during application
execution, many identical connections
will be repeatedly opened and closed.
To minimize the cost of opening
connections, ADO.NET uses an
optimization technique called
connection pooling.
Connection pooling reduces the number
of times that new connections must be
opened. The pooler maintains ownership
of the physical connection. It manages
connections by keeping alive a set of
active connections for each given
connection configuration. Whenever a
user calls Open on a connection, the
pooler looks for an available
connection in the pool. If a pooled
connection is available, it returns it
to the caller instead of opening a new
connection. When the application calls
Close on the connection, the pooler
returns it to the pooled set of active
connections instead of closing it.
Once the connection is returned to the
pool, it is ready to be reused on the
next Open call.
This is quite a broad question. But usually, for any database server and application environment, opening and keeping a new connection is an expensive operation. That's why you definitely don't want to open multiple connections from a single client, and should stick to process-scope for connections.
In a desktop application using a database server, strategy for handling it's single connection depends a lot on the DB usage pattern. Say, if the app reads or writes something a lot within 5 minutes, and then just does nothing with the DB for hours, it makes no sense to keep the connection open all the time (assuming there are many other clients). You may introduce some kind of time-out for closing a connection.
The Web server situation depends a lot on the used technology. Say, in PHP every request is a "fresh start" WRT database connection. You open and close a connection for each mouse click. While popular Java application servers have DB connections pool, reusing the same connection instances for many HTTP request handling threads.
We have a Delphi 7 application that runs as an ISAPI extension in IIS6. The code use ADO to connect to a MS SQL 2000 database and performs many reads on the database (no writes). If I watch the audit login and logout events in SQL profiler I can see that numerous requests to the app result in only 1 audit login event. However, if I run that same code from outside IIS (i.e. a test app calling the same method in the dll) I see many login and logout events. My guess is that IIS is performing some automatic connection pooling without my doing anything. I'd like to see the same behavior when I run the dll from outside of IIS for performance reasons - the app is almost 100% slower in this situation. How can I get ADO connection pooling when the dll runs outside of IIS?
EDIT - I'm actually using the SQL ole provider. The connection string looks like this:
Provider=SQLOLEDB.1;Initial Catalog=%s;Data Source=%s;Password=%s;User ID=%s;Pooling=True;Min Pool Size=5;Max Pool Size=50;Connection Lifetime=120
I tried adding the Pooling=True attribute but this doesn't change things. Also, I learned that audit login and logout events don't necessarily change for connection pooling so I started using the Logins/sec, Logouts/sec and User Connections performance counters (SQLServer:GeneralStatistics) to determine if connection pooling occurs. From inside IIS I see many logins/sec and no logouts/sec. Outside of IIS I see many logins and logouts per second and user connections fluctuates (it holds steady in IIS).
It's hard to say based on the info given, but connection pooling is definitely based off of the connection string - if the connection string is exactly the same then the connection can be pooled... it sounds like your outside application may be altering the connection string?
IIS isn't pooling ADO connections. It is likely caching the ISAPI dll though. Are you starting/stopping your outside application continuously? Or is it a single run causing multiple login events?
We notice something strange in our struts web application which was hosted on sun app server enterprise edition 8.1.
The NumConnUsed for Monitoring of JDBC resources stays at 100 over connections even though there was relatively very low user activities.
I try to do some research and found the following links
http://j2ee-performance.blogspot.com/
http://www.ibm.com/developerworks/websphere/library/techarticles/0506_johnsen/0506_johnsen.html
"When the application closes a shareable connection, the connection is not truly closed, nor is it returned to the free pool. Rather, it remains in the Shared connection pool, ready for another request within the same LTC for a connection to the same resource."
Base on the above comments, it is true that if my web.xml resource ref scope is set to shareable, when application side close conneciton, it remains in the shared conneciton pool thus the numconnused is always so high?
If I interpret the links in my own special way (;)), the shared vs. unshared connections is based on different connections in the same page.
java.sql.Connection connectionOne = DriverManager.getConnection(...);
...
java.sql.Connection connectionTwo = DriverManager.getConnection(...);
These two, at a glance, seem to be individual - but if your AS is set to shareable connections, the second one will be created with a pointer to the first connection instead of returning a new connection. When the page finishes the connection should be sent back to the pool.
The AS is probably keeping the pool filled with connections to enhance performance.
This is not fact, only my own iterpretation of the links.