I am new to Ruby on Rails and Datamapper. I have written models using Datamapper, one of my model name is Student. In one view haml file I have written the following code:
-students = Student.all
-students.each |student|
%tr
%td= student.roll_no
%td= student.type if student.type
%td= student.department.name
Here I have used newrelic -rpm for profiling my code. Here I found that in each iteration of the above block, one query of the form select prop1, prop2,... from students where id ="some value" is being generated. This is very undesired as it is taking time on each iteration of the block. I think it's due to lazy loading. I have spent nearly a week on that but found nothing to avoid this. If anyone have any idea regarding this please help me. Thank you.
It could be helpful if you could show us the schema of the Students table in app/db/schema.rb.
I suspect your problem is not that each iteration of student takes so long because of lazy evaluation of each row in students, but that it has to load the department in each step. The Student.all is, since Rails 4, lazily evaluated, but it loads the whole set at once.
In order to solve your issue, you have to write Student.includes(:department) in your first line.
You can force DataMapper to preload data before embarking on the loop.
After loading the Collection into students, add a line such as:
prefetch = students.collect { |s| s.department }
You don't have to do anything with the prefetch object; DataMapper will load all the data and associate it with the students object. You should then find that you can iterate through the loop as expected without a separate query been generated for each. (I've just tested this on a project I'm working on and it succeeds.)
Related
I am using pagy. I combined two models into one, and I used pagy on that combined model. I am getting this error:
undefined method `offset' for #<Array:0x00007f886f88b3b0>
With the last line of the code below highlighted.
My code:
#problems = Problem.me_and_friends(current_user)
#activities = Activity.me_and_friends(current_user)
#combine = (#problems + #activities).sort{|a,b| a.created_at <=> b.created_at }
#pagy, #combined = pagy_countless(#combine, items:100, link_extra: 'class="" style="color:black; margin:3px;"')
It worked fine with using pagination on #problems alone.
I'd appreciate any help.
As soon as you call the (#problems + #activities), you transform the ActiveRecord::Relation into an array (which is also not good because you are loading all the database rows into memory, sorting and then paginating them). Pagy expects an ActiveRecord::Relation to work, hence the error.
You can consider multiple solutions,
Change your UI to show problems and activities in separate UIs, then you can paginate them separately
Update your models to store both problems and activities in the same table (maybe just a reference table which points to either a Problem or an Activity)
If either of these is not feasible, you can consider rolling out a custom solution for the pagination, but it will be tricky.
Update: June 21, 2021
If you are using Rails 6, it introduces the concept of Delegated Types which fits well into this scenario. The example given in the link mentions the issue of pagination across different tables.
I have a model (News) associated with another model (Category), so in News model i have:
has_and_belong_to_many :news_categories, join:table: 'news_categories_news'
I want to take all news with own categories, so:
News.find(/*conditions*/).includes(:news_categories)
If I check in console I see the right inner join query, but when I call
#news.news_categories
(Where news is a single news in the result array) if I check in console I see another query to take the categories for the current news, how can I avoid this redundant query?
p.s: sorry for my english...
First of all, .includes can't work when chained after .find. Reason - find will not return the ActiveRecord::Relation which is necessary for relational chaining; it will rather return the matching News object or error.
You should do:
#all_news = News.includes(:news_categories).where(id: 1)
#news = #all_news.first
#news.news_categories # shouldn't invoke new query
Thank you to all but i resolved with eager_load()
It's generate just once query!
When I executing query
Mymodel.all.each do |model|
# ..do something
end
It uses allot of memory and amount of used memory increases at all the time and at the and it crashes. I found out that to fix it I need to disable identity_map but when I adding to my mongoid.yml file identity_map_enabled: false I am getting error
Invalid configuration option: identity_map_enabled.
Summary:
A invalid configuration option was provided in your mongoid.yml, or a typo is potentially present. The valid configuration options are: :include_root_in_json, :include_type_for_serialization, :preload_models, :raise_not_found_error, :scope_overwrite_exception, :duplicate_fields_exception, :use_activesupport_time_zone, :use_utc.
Resolution:
Remove the invalid option or fix the typo. If you were expecting the option to be there, please consult the following page with repect to Mongoid's configuration:
I am using Rails 4 and Mongoid 4, Mymodel.all.count => 3202400
How can I fix it or maybe some one know other way to reduce amount of memory used during executing query .all.each ..?
Thank you very much for the help!!!!
I started with something just like you by doing loop through millions of record and the memory just keep increasing.
Original code:
#portal.listings.each do |listing|
listing.do_something
end
I've gone through many forum answers and I tried them out.
1st attempt: I try to use the combination of WeakRef and GC.start but no luck, I fail.
2nd attempt: Adding listing = nil to the first attempt, and still fail.
Success Attempt:
#start_date = 10.years.ago
#end_date = 1.day.ago
while #start_date < #end_date
#portal.listings.where(created_at: #start_date..#start_date.next_month).each do |listing|
listing.do_something
end
#start_date = #start_date.next_month
end
Conclusion
All the memory allocated for the record will never be released during
the query request. Therefore, trying with small number of record every
request does the job, and memory is in good condition since it will be
released after each request.
Your problem isn't the identity map, I don't think Mongoid4 even has an identity map built in, hence the configuration error when you try to turn it off. Your problem is that you're using all. When you do this:
Mymodel.all.each
Mongoid will attempt to instantiate every single document in the db.mymodels collection as a Mymodel instance before it starts iterating. You say that you have about 3.2 million documents in the collection, that means that Mongoid will try to create 3.2 million model instances before it tries to iterate. Presumably you don't have enough memory to handle that many objects.
Your Mymodel.all.count works fine because that just sends a simple count call into the database and returns a number, it won't instantiate any models at all.
The solution is to not use all (and preferably forget that it exists). Depending on what "do something" does, you could:
Page through all the models so that you're only working with a reasonable number of them at a time.
Push the logic into the database using mapReduce or the aggregation framework.
Whenever you're working with real data (i.e. something other than a trivially small database), you should push as much work as possible into the database because databases are built to manage and manipulate big piles of data.
Outlet.rb:
def latest_reports
weekly_reports.limit(10)
end
Outlet_controller.rb:
#all_outlets = Outlet.includes(:weekly_reports)
#search = #all_outlets.search(params[:q]) # load all matching records
#outlets = #search.result.order("created_at DESC").page(params[:page])
outlet/index.slim:
- #outlets.each do |outlet|
tr
td= link_to outlet.name, outlet_path(outlet)
th
ul.reports
li class="#{'done' if outlet.monitored_today}"
th
ul.reports
- for report in outlet.latest_reports
li class="#{'done' if report.quota_met}"= report.times_monitored
I'm not sure why, but this loads it up as several different queries. I'm pretty sure it's because the include in my controller isn't correct (because I'm using a method in the model).
If anyone could help me improve this, I would be extremely grateful :).
Note: I'm developing on PostgreSQL
Update:: Posted the full controller action.
In rails 3 at least, if you use
Model1.includes :model2
then the result is one query for each model. You can access instances of the associated model from the result and no extra queries will be made.
If you really want it all in one query, you can do this:
Model1.joins(:model2).includes(model2)
This will produce a nice long JOIN query that loads all the data for both models in one go. Rails will populate the result with instances of both models already loaded.
So, you should be able to replace
#all_outlets = Outlet.includes(:weekly_reports)
with
#all_outlets = Outlet.includes(:weekly_reports).joins(:weekly_reports)
and it should combine everything into one query.
So I am trying to find all entries in my table that have a specific value in a particular column. The only way I can think off to do this is to look at each entry and see if it has the value but I was hoping there would be a more efficient solution - this gets unwieldy once you have a sizable database.
Does anyone have a better idea?
Update - I am creating an HTML table and I want to populate the table with all the entries in my model that have a certain value in a particular column. I am trying to do:
<%= render #users.where("column_name = 'value'") %>
as the answer below recommends but I get "undefined method `where' for nil:NilClass" error.
Update 2 - I am not sure why #users would be nil but I will try to figure that out later. For now, I tried
<% #user_message = User.where("column_name = 'value'") %>
<%= render #user_message %>
but it doesn't show any entries at all.
Update 3 - When I do, User.all in rails console, I get all the users so I know the data is there. However, when I do User.where("column_name = 'value'"), I get an empty array. I double checked the column name and value to make sure that the data was present.
Update 4 - Fixed! - I'm not sure why it didn't work in rails console but I got it to work in the site. I called my partial _user_message.html.erb. Apparently it still needs to be called _user.html.erb. Thanks for the help everyone!
Sounds like you want to do a where query, i.e.
#records = Model.where(:some_column => some_value)
Rails has excellent documentation, I suggest you take a look at the ActiveRecord Query guide:
http://guides.rubyonrails.org/active_record_querying.html
ian.