I've been recently applying threads for making queries to a MYSQL database, I use MyDAC for connection to DB, 'cause TMyConnection doesnot let making simultaneously queries per a connection, I create a new connection and a new query object per every thread executing a query, so in certain time could happens that server has several connections per a client. If we consider this scenario for several clients connecting to database, this is would be a problem, I guess. Is there a better solution for using threads in queries?
Thanks in advance
Use a second tier where you can pool some connections (you can do with datasnap or remobjetcs...) This way you can reuse connections of all of your users and mantain the number of connections in a smaller level.
Have a look Cary Jansen article called
Using Semaphores in Delphi, Part 2: The Connection Pool
He goes in to great detail about how to provide thread-safe access to a limited number of database connections
Getting is code to work with MyDac - TMyConnection is trivial.
Related
I am working on asp.net MVC+WebAPI entity framework code-first project, and using async methods. Currently, I am creating database connection on per-call basis.
Creating database connection takes small time,
so every request to server needs small time to create a database connnection,
and for thousand requests it takes more time.
Now, my question is if I use one database connection instance in whole application then what will be happened?
SQL Server and Oracle have connection pools, so the actual resources behind the database connection are typically going to be reused even though you create a connection class for each call.
If you have a singleton connection you won't be able to have concurrent database requests. You'll have a single-thread bottleneck to the database.
The connection pool will grow and shrink depending on how much concurrency you application actually has.
I have an ASP.NET MVC application which gathers data from multiple Databases.
The databases hold information for various sites and for every new site we have a new Database. The database for each site is connected at two points, from the site and then from HQ.
A web application updated data every minute from the site and the data is is served to the HQ (via another web application) every minute. Sometimes the application response is very slow and from what I have investigated, it may be because the connection pool starts filling up swiftly.
I want to ask what is the best approach to such application, where I can get the best performance out of it. Any guidance is welcome.
How to improve your web application performance regarding to database, really depends on your architecture. But there are some general rules which you should always follow:
Check about thread starvation:On the Web server, the .NET Framework
maintains a pool of threads that are used to service ASP.NET
requests. When a request arrives, a thread from the pool is
dispatched to process that request. If the request is processed
synchronously, the thread that processes the request is blocked
while the request is being processed, and that thread cannot service
another request.
This might not be a problem, because the thread
pool can be made large enough to accommodate many blocked threads.
However, the number of threads in the thread pool is limited. In
large applications that process multiple simultaneous long-running
requests, all available threads might be blocked. This condition is
known as thread starvation. When this condition is reached, the Web
server queues requests. If the request queue becomes full, the Web
server rejects requests with an HTTP 503 status (Server Too Busy).
for "thread starvation" the best approach is using "Asynchronous
Methods". refer here for more information.
Try to use using block for your datacontext, to dispose them immediately after finishing with them.
Huge data amount in transaction: you should check your code.
May be you using too much data without need to all of them. For
example you transfer all object which you may need just one
properties of object. In this case use "projection"(refer here for
an example).
Also you may use "lazy loading" or "eager loading" base on you
scenarios. But please be noted that none of these are magic tool for
every scenario. In some cases "lazy loading" improve performance and
on others "eager loading" makes things faster. It depends to your
deep understanding of these two terms and also your case of issue,
your code and your design.
Filter your data on server side or client side. Filtering data on server side helps to keep your server load and network traffic as less as possible. It also makes your application more responsive and with better performance. Use IQueryable Interface for server side filtering (check here for more information).
One side effect of using server side filtering is having better security
Check your architecture to see do you have any bottleneck. A
controller which gets called too much, a methods which handles lots
of objects with lots of data, a table in database which receives
requests continuously, all are candidates for bottle neck.
Ues cashing data when applicable for most requested data. But again
use cashing wisely and based on your situation. Wrong cashing makes
your server very slow.
If you think your speed issue is completely on your database, the best approach is using sql profiling tools to find out which point you have critical situation. Maybe redesign of your own tables could be an answer. Try to separate reading and writing tables as much as possible. Separation could be done by creating appropriate views. Also check this checklist for monitoring your database.
Rails' database.yml file has a setting, pool: 5. I understand what a database connection pool is but I'm being tripped by a few subtleties:
A connection is used then returned to its pool. The next request can then use a connection from the pool rather than creating a new connection.
How is it determined which request gets which connection?
Suppose I have a concurrent connections limit of 5 and one of my web pages needs to make 10 queries to the database:
Is each query a separate connection or all 10 queries are considered one connection?
In terms of queries, connections, or speed, what can be an example of a situation that would overwhelm that 5 concurrent connections limit?
And suppose that, in a different database, I set the database connection pool size to 5.
How are pool size and concurrent connections related, if at all?
In terms of queries, connections, or speed, what can be an example of a situation that would overwhelm this pool size?
1) ActiveRecord::Base loads a connection when required (lazily on a request or it's current one is closed/disconnected)
2) No, The same connection will be used to make multiple queries
3) No way to answer that without using diagnostic utilities which your db vendor supplied with your db
4) That is db vendor/adapter specific
5) same answer as 3.
If you are experiencing a slow down, the only way to solve them is by using diagnostic tools to inform you where your bottleneck is concurring. 90% of the time, it's not your db or the connections to it (It's usually the indexing, n+1, etc... )
If you are NOT experiencing any slow down, then keep the defaults and move on. Premature optimization will lead to an over engineered solution
Hi I am using neo4j in my application and my structure is as following:
I am using Embedded Graph API
I have several databases that I point to using a pool that I maintain in my application eg-> db1, db2, db3, ..... db100
When I want to access a particular database I point to it using new EmbeddedGraphDatabase("Path to db(n)")
The problem is that when the connection pool count increases the RAM size being consumed by the application keep increasing and breaks down the application at a point of limit.
So I am Thinking of migrating from Neo4j to some other Database.
Additionally only a small part of my database is utilizing the graph structure.
One way for migration is that I write a script for it. Is there any better option?
My another question is what is the best Database so that my structure can be maintained.
Other view-point that I am thinking about is I can keep part of my data into Neo4j and shift another part to some other database.
If anything is unclear I can clarify.
Thanks in advance.
An EmbeddedGraphDatabase instance is not the equivalent of a "connection" in SQL. It's designed to run a long time (days, months). Hence starting/stopping is costly.
What is the use case for having hundreds of separate databases in the same JVM?
Your lots of small databases will perform poorly as the graphdb is designed to hold the whole datamodel on a single host.
Do you run a single JVM per database?
You can control the amount of memory used by neo4j by providing the correct properties for memory mapping and also use the gcr cache from neo4j-enterprise and control the cache size-property variables.
I think it still makes sense to keep the graph part in Neo4j and only move the non-graphy part.
I'm using JAVA DB (derby)
I want to import a public view of my data to another database (also in java db).
I want to pass this data and save in to the other database. I'm having trouble since the general rule is one connection to one database.
Help would be much appreciated.
You need two connections, one to each database.
If you want the two operations to be a single unit of work, you should use XA JDBC drivers so you can do two-phase commit. You'll also need a JTA transaction manager.
This is easy to do with Spring.
SELECT from one connection; INSERT into the other. Just standard JDBC is what I'm thinking. You'll want to batch your INSERTs and checkpoint them if you have a lot of rows so you don't build up a huge rollback segment.
I'd wonder why you have to duplicate data this way. "Don't Repeat Yourself" would be a good argument against it. Why do you think you need it in two places like this?