when using a DBaaS (database as a service) such as Xeround with a Rails app hosted on EC2 instances, how is it possible to limit the number of concurrent connections to the database (according to the DB service plan limits) ? is it necessary to do so at all ?
I know that ActiveRecord connections pool is per process and is thread safe, but what if there are several processes (also across several different machines) ?
Unfortunately there is no way to correctly limit the number of connections across multiple clients (applications). The only way, which is pretty much static and empirical, is to divide the number of max allowed connection by the number of apps and set the result as the connections limit per application.
Use a Connection pool base class for managing Active Record database connections.
Related
I have a stateless Rest API, that is currently deployed manually on several backend servers for scalability and redundancy. This service is talking to a central database by using a connection from its application pool of connection.
The settings of this application pool are carefully computed and configured by hand depending on the capacity of the database ( ie: RAM, CPU, Disk, and max connections allowed: that we adjust also manually from month to month ).
In the future we plan to make the deployment less painless, more frequent, and also have dynamic scalability for this API. So for instance under heavy load, we will have a lot of new dockers running in parallel.
So I am wondering what will be the way to configure the connection pool to the database, so we would not consume more than the database can handle for example? But also at the same time have the rest API that could handle the load during a spike of calls for example with a reasonable minimum and maximum connection in its pool?
I'm in process of upgrading our client-server ERP app to multi-tier. We want to offer our customers the possibility having their databases in cloud(hosted in our server).So, clients are written in Delphi, server is a http IOCP server written also in Delphi (from mORMot framework), for dbs we use Firebird embedded.
Our customers(let's say 200), can have 25-30 Firebird databases (total 5000-6000 databases), accessed by 4-5 user per each customer. This is not happening all at once. One user can work in one db, other 2 users can work in another db, but all the dbs should be available and online. So, I can have 800-1000 users working at 700-900 dbs. Databases are not big, typically 20-30 MB each but can go to 200 MB.
This is not data sharding so please don't suggest to merge all databases together, I really need them individually with possibility to backup/restore/replace each one of them.
So, I need multiple pools of connections - for every database I need a pool of let's say 2 connections. I read about Firedac connection pooling. It seems that TFDManager should be perfect for me. I define multiple "ConnectionDef"s with "Pooled=true" and it can maintain multiple pools of connections (each connection lasting until some minutes of inactivity).
Questions:
I have to create all "ConnectionDef"s before server starts serving requests?
Can TFDManager "handle" requests (and time-out connections on inactivity), while in other thread I need to create a new database , so automatically I need to create a new pool of connections and start serving requests from newly created database. Practically can I call FDManager.AddConnectionDef(..) while other pools are in use?
AFAIK Firebird embedded does not have any "connection". As its name states, it is embedded within the same process, so there is no connection pool needed. Connection pools are needed when several clients connect/disconnect to the same DB over the network, whereas here all is embedded, and you would have direct access to the Firebird engine.
As such, you may:
Define one "connnection" per Firebird embedded database;
Protect your SOA code via a mutex (aka critical section) per DB. In fact, mORMot's HTTP IOCP server would run the incoming requests from a thread pool, so ensure that all DB access is safely made.
Ensure you use at least Firebird 2.5 since the embedded version is told to be threadsafe only since revision 2.5 (see the release notes).
Instead of FireDAC, consider using ZDBC/Zeos (in latest 7.2/7.3 branch), which has nice features, together with the native mORMot SynDB libraries.
Looking at Firedac sources, seems that all about adding connection definitions and acquiring connections in pooled mode is thread-safe.
Adding a connection definition or matching one is guarded by a TMultiReadExclusiveWriteSynchronizer and acquiring a connection from the pool is guarded by a TCriticalSection.
So, answers:
I don't have to create all "ConnectionDef"s before server starts serving requests.
Yes, I can call safely FDManager.AddConnectionDef(..) while other pools are in use.
Using Firedac, acquiring a connection for any of those databases will be guarded by one TCriticalSection. The solution proposed by #Arnaud Bouchez presents a more grained access by creating one TCriticalSection per database and I think will scale better, but you should be aware of a bug when using multiple TCriticalSection, especially that all will be initiated at once:
https://www.delphitools.info/2011/11/30/fixing-tcriticalsection/
In that article present a very simple fix for this bug.
My use case is a multi-tenant Rails app. Each app uses a different database server, and the connections are managed through the AR connection pool. Currently it's doing well with 10 connections, but I'm curious if there is an upper limit to this resource.
The upper limit is your database upper limit, not Rails or ActiveRecord limits.
So I have a connection pool setup. Which is great and all since I have an application that really needs it. However what I would like to know is if it is possible to share this connection pool with other J2SE apps? Would this even be worth it, as opposed to creating a connection pool based on each apps needs? If it would be prudent, how can I accomplish this?
It is not hard having connection pools in a single JVM doing multiple things - that is what applications servers do everyday (using JNDI to throw objects across classloaders)
The interesting part is when you have the connection pool in a separate JVM from the client code needing it, as this does not immediately allow simply asking for and getting a connection from the pool and returning it afterwards.
Basically you have two options:
Doing remote requests for all your JDBC commands over the network. This will most likely mean that the data will travel over the network twice, from the database to the connection pool, and then from the connection pool to your application. If the database connections are very expensive objects then this might be a viable solution.
Use RMI to get the connection object from the connection pool JVM to your own machine. This is a very expensive operation, but can as far as I know include the actual driver classes, allowing your connection pool to provide connections to databases not known to your application JVM. To me this would only make sense if the database connections were ridiculoulusly expensive or it was a requirement to be able to support additional databases after deployment without changing the original deployments.
Note that the primary reason for having connection pools at all is because connections are expensive to create, use shortly and then discard. Some databases more than others, e.g. MySQl is (or was when I tried) very cheap so it might be the simplest just to do that.
So. First of all: Measure what your connection pool buys you in time, and then consider if it is worth your while to centralize this further.
I'm building a web application that uses Oracle Database 10g as the database backend. I realize the Express edition has limitations, but I just wanted to make sure that number of connections wasn't one of them.
Does Oracle Express Edition (XE) limit the amount of concurrent connections (for example, the number of users viewing the site)?
According to:
http://forums.oracle.com/forums/thread.jspa?messageID=1673347
(among others)
There's no hardcoded limit, however there is an effective default limit of ~20 concurrent connections, however you can extend that with something like:
ALTER SYSTEM SET processes=200 scope=spfile
(and restart the DB)
In practice you've probably either hit other XE limits well before 200 connections, or should have been using a simpler DB to begin with.
There are some benchmarks for XE 10g here:
http://ora-00001.blogspot.com/2011/03/stress-testing-oracle-xe-10g.html
Number of connections depends on RAM and other dependant things.