For example, suppose there is the code in Rails 3.2.3
def test_action
a = User.find_by_id(params[:user_id])
# some calculations.....
b = Reporst.find_by_name(params[:report_name])
# some calculations.....
c = Places.find_by_name(params[:place_name])
end
This code does 3 requests to database and opens 3 different connections. Most likely it's going to be a quite long action.
Is there any way to open only one connection and do 3 requests within it? Or I want to control which connection to use by myself.
You would want to bracket the calls with transaction:
Transactions are protective blocks where SQL statements are only
permanent if they can all succeed as one atomic action. The classic
example is a transfer between two accounts where you can only have a
deposit if the withdrawal succeeded and vice versa. Transactions
enforce the integrity of the database and guard the data against
program errors or database break-downs. So basically you should use
transaction blocks whenever you have a number of statements that must
be executed together or not at all.
def test_action
User.transaction do
a = User.find_by_id(params[:user_id])
# some calculations.....
b = Reporst.find_by_name(params[:report_name])
# some calculations.....
c = Places.find_by_name(params[:place_name])
end
end
Even though they invoke different models the actions are encapsulated into one call to the DB. It is all or nothing though. If one fails in the middle then the entire capsule fails.
Though the transaction class method is called on some Active Record
class, the objects within the transaction block need not all be
instances of that class. This is because transactions are per-database
connection, not per-model.
You can take a look at ActiveRecord::ConnectionAdapters::ConnectionPool documentation
Also AR doesn't open a connection for each model/query it reuses the existent connection.
[7] pry(main)> [Advertiser.connection,Agent.connection,ActiveRecord::Base.connection].map(&:object_id)
=> [70224441876100, 70224441876100, 70224441876100]
Related
Let's say on some Child model method I need to do calculations based on some data stored on its Parent model. For example,
def child_method(minutes)
remaining_time = minutes % self.parent.parent_settings
if remaining_time >= 1
return minutes/ self.parent.parent_settings
else
return [minutes/self.parent.parent_settings - 1 , 0].max
end
end
In the above I've called self.parent.parent_settings 3 times. Based on how Rails works, is this efficient? Or is it a terrible idea, and I should instead set the parent_settings locally, e.g.,:
def child_method(minutes)
parent_settings = self.parent.parent_settings
remaining_time = minutes % parent_settings
if remaining_time >= 1
return minutes/ parent_settings
else
return [minutes/parent_settings - 1 , 0].max
end
end
I have more complex instances of this (e.g., where in one child method I'm accessing multiple parent attributes, and also in some instances, grandparent attributes). I realize the answer might be "it depends" on exactly what is the data, etc., but looking to see if there are general rules of thumb or convention
Like you said, it depends.
Rails will cache fetched associations as long as the object remains in memory:
puts self.parent.parent_settings.object_id
# ... Some code
puts self.parent.parent_settings.object_id # => This should be the same object ID as before
This cache is cleared automatically by the framework and can be explicitly cleared via #reload:
self.reload
Your code should be fine as long as you're not running child_method multiple times in a request/response cycle. Even if you do run child_method multiple times in the same request/response cycle, there's another database query cache that will intercept the same DB queries. The db query cache is only active when in production mode or when a special ENV var is set.
Add connection execute in model
def self.execute_sql(*sql_array)
connection.execute(send(:sanitize_sql_array, sql_arry))
end
How to close the connection by this case? Is it a good way?
def self.close_connection
connection.close
end
By the way, it's a little different from ActiveRecord::Base.connection.execute. Are they the same?
Assuming defaults, all of your models should be inheriting from ActiveRecord::Base, therefore ActiveRecord::Base.connection.execute and your connection.execute(...) methods should be exactly the same (except only if you super methods being called).
Now, your execute_sql method seems to be a bit different that it accepts *sql_array argument (which is an Array), which means that the syntax is a bit different than ActiveRecord::Base.connection.execute, because you can now supply sanitized arguments to the SQL: i.e. User.execute_sql('SELECT * FROM users WHERE created_at > ?, Time.zone.now).
Rails makes use of ConnectionPool. So each HTTP request will try to obtain from a Connection in that pool, and when finished the Connection will not be closed but instead will just be returned to that pool. Rails will not infinitely try to open connections as time goes by. Specify :pool size in database.yml according to your production needs and DB connections limits. The :pool size there is max connections, and not number of connections; Rails does not create connections immediately; only by demand.
However, if you intentionally are not gonna use anymore a Connection for the duration of the Thread / request, and you want to yield back the Connection to the pool immediately, then you can do simply do ActiveRecord::Base.connection_pool.release_connection
For demo purposes, suppose that I have a class called DemoThing with a method called do_something.
Is there a way that (in code) I can check the number of times that do_something hits the database? Is there a way that I can "spy" on active record to count the number of times that the database was called?
For instance:
class DemoThing
def do_something
retVal = []
5.times do |i|
retVal << MyActiveRecordModel.where(:id => i)
end
retVal
end
end
dt = DemoThing.new
stuff = dt.do_something # want to assert that do_something hit the database 5 times
ActiveRecord should be logging each query in STDOUT.
But for the above code, it's pretty obvious that you're making 5 calls for each iteration of i.
Queries can be made more efficient by not mixing Ruby logic with querying.
In this example, gettings the ids before querying will mean that the query isn't called for each Ruby loop.
ids = 5.times.to_a
retVal = MyActiveRecordModel.where(id: ids) # .to_a if retVal needs to be an Array
Sure is. But first you must understand Rails' Query Cache and logger. By default, Rails will attempt to optimize performance by turning on a simple query cache. It is a hash stored on the current thread (one for every active database connection - Most rails processes will have just one ). Whenever a select statement is made (like find or where etc.), the corresponding result set is stored in a hash with the SQL that was used to query them as the key. You'll notice when you run the above method your log will show Model Load statement and then a CACHE statement. Your database was only queried one time, with the other 4 being loaded via cache. Watch your server logs as you run that query.
I found a gem for queries count https://github.com/comboy/sql_queries_count
I have a simple rails app that scrapes JSON from a remote URL for each instance of a model (let's call it A). The app then creates a new data-point under an associated model of the 1st. Let's call this middle model B and the data point model C. There's also a front end that let's users browse this data graphically/visually.
Thus the hierarchy is A has many -> B which has many -> C. I scrape a URL for each A which returns a few instances of B with new Cs that have data for the respective B.
While attempting to test/scale this app I have encountered a problem where rails will stop processing, hang for a while, and finally throw a "ActiveRecord::ConnectionTimeoutError could not obtain a database connection within 5.000 seconds" Obviously the 5 is just the default.
I can't understand why this is happening when 1) there are no DB calls being made explicitly, 2) the log doesn't show any under the hood DB calls happening when it does work 3) it works sometimes and not others.
What's going on with rails 4 AR and the connection pool?!
A couple of notes:
The general algorithm is to spawn a thread for each model A, scrape the data, create in memory new instances of model C, save all the C's in one transaction at the end.
Sometimes this works, other times it doesn't, i can't figure out what causes it to fail. However, once it fails it seems to fail more and more.
I eager load all the model A's and B's to begin with.
I use a transaction at the end to insert all the newly created C instances.
I currently use resque and resque scheduler to do this work but I highly doubt they are the source of the problem as it persists even if I just do "rails runner Class.do_work"
Any suggestions and or thoughts greatly appreciated!
I believe I have found the cause of this problem. When you loop through an association via
model.association.each do |a|
#work here
end
Rails does some behind the scenes work that "uses" a DB connection. I put uses in quotes because in my case I think the result is actually returned from memory. I eager loaded the association and thus the DB is never actually hit.
Preliminary testing of wrapping my block in a
ActiveRecord::Base.connection_pool.with_connection do
#something me doing?
end
seems to have resolved the issue.
I uncovered this by adding a backtrace to my thread's error message that was printing out.
-----For those using resque----
I also had to add a bit in my resque.rake file to get this fully working as intended.
task 'resque:setup' => :environment do
Resque.after_fork do |job|
ActiveRecord::Base.establish_connection
end
end
If you are you using
ActiveRecord::Base.transaction do
... code
end
to accomplish faster transactions in a thread, note that this locks the database. I had an app that did this for a hugely expensive process, in a thread, and it would lock the DB for over 5 seconds. It is faster, though it will lock your database
Folks,
I am fairly new to transactions in activerecord in rails and I have a piece of code, where I do something like:
transaction do
specimen = Specimen.find_by_doc_id(25)
specimen.state = "checking"
specimen.save
result = Inventory.do_check(specimen)
if result
specimen.state="PASS"
else
specimen.state="FAIL"
end
specimen.save
end
My goal here for using a transaction is if I get an exception in Inventory.do_check(it is a client to external web-services and does a bunch of HTTP calls and checks) then I want the specimen.state to rollback to its previous value. I wanted to know if this will work as above? Also, it looks like on my development machine the lock is set on the entire Specimen table, when I try to query that table/model I get a BUSY exception(I am using SQLLite). I was thinking that the lock should only be set on that object/record.
Any feedback is much appreciated, as I said I am really new to this so my question may be very naive.
Implementation and locking depends on the DB. I don't use SQLLite and I won't be surprised if it locks the entire table in such case. But reading should still work, so it's probably because it doesn't allow two concurrent operations on a single connection, so is waiting for your transaction to finish before allowing any other operation. See, for example, this SO answer: https://stackoverflow.com/a/7154699/2117020.
However, my main point is you shouldn't be holding the transaction while accessing external services in any case. However it is implemented, keeping the transaction for seconds is not what you'd want. Looks like in your case all you want is to recover from an exception. Do you simply want to set the state to "FAIL" or "initial" as a result, or does do_check() modify your specimen? If do_check() doesn't modify the specimen, you should better do something like:
specimen = Specimen.find_by_doc_id(25)
specimen.state="checking"
specimen.save
# or simply specimen.update_attribute( :state, "checking" )
begin
specimen.state = Inventory.do_check(specimen) ? "PASS" : "FAIL"
rescue
specimen.state = "FAIL" # or "initial" or whatever
end
specimen.save
The locking is going to be highly dependent on your database. You could use a row lock. Something like this:
specimen = Specimen.find_by_doc_id(25)
success = true
# reloads the record and does a select for update which locks the row until the block exits (its wrapped in a transation)
specimen.with_lock do
result = Inventory.do_check(specimen)
if(result)
specimen.state="PASS"
else
specimen.state="FAIL"
end
specimen.save!
end
Checking the external site in a transaction is not ideal, but if you use with_lock and your database supports row lock, you should just be locking this single row (it will block reads, so use carefully)
Take a look at the pessimistic locking documentation in active record:
http://ruby-docs.com/docs/ruby_1.9.3-rails_3.2.2/Rails%203.2.2/classes/ActiveRecord/Locking/Pessimistic.html