Using: Rails 3.1.1
I am trying to create a search engine in my application that browses a large database (apprx 100 000 items) for string matches.
I am using the following:
fp = FeededProduct.where("name LIKE '%blue%' OR description LIKE '%blue%' OR category LIKE '%blue%'")
for the search query for "blue".
When I run the equivalent search phrase in MySQL it works fine but when I try to run this in rails it shows:
NoMethodError: undefined method `fields' for nil:NilClass: SELECT `feeded_products`.* FROM `feeded_products` WHERE (name LIKE '%blue%' OR description LIKE '%blue%' OR category LIKE '%blue%')
Clues and troubleshooting:
This happens only for large search results, I have not been able to distinguish a number but it crashes when it should have returned 920 results but it does NOT crash when returning 6 results!
My conclusion of the above is either that it cannot keep all the 920 results in the memory OR that there is some type of row that makes it crash and the more results, the more likely it is that it will contain that type of row. I am leaning more towards the first conclusion.
I cannot troubleshoot it very well because it crashes (with the same error code) when I try to run:
raise fp.inspect
It also crashes for:
fp.each do |prod|
begin
puts 'Do nothing'
rescue
puts 'Something crashed'
end
but it does NOT crash for:
raise fp.count.inspect
So, am I having a memory type of problem? What can I do to trouble shoot this further and/or solve the problem?
Using: Mac OS X 10.7.2. Lion
Database: InnoDB
Adapter: Mysql2 (don't know which version)
Stack:
ActiveRecord::StatementInvalid (NoMethodError: undefined method fields' for nil:NilClass: SELECT feeded_products`.* FROM feeded_products WHERE (name LIKE '%blue%' OR description LIKE '%blue%' OR category LIKE '%blue%')): app/controllers/search_results_controller.rb:190:in `show'
Edit 2012-03-06 Additional trouble shooting:
I tried with
fp2 = FeededProduct.limit(60000)
to create a really big array of hits and it worked fine. So I guess that rules out my guess that the fp variable cannot hold a certain amount of items.
The core of the problem seems to be that if I use the:
fp = FeededProduct.where("name LIKE '%blue%' OR description LIKE '%blue%' OR category LIKE '%blue%'")
I cannot use the fp-variable for anything afterwards without the application crashing.
Can you post the code at line 190 in search_results_controller.rb, and any others that possibly refers to the "fields" attribute in your show method? Also relevant parts of your model FeededProduct from app/models/feeded_product.rb
app/controllers/search_results_controller.rb:190:in `show'
It's not clear what fields refers to from the info you posted. It could be a typo, bad code or a migration that needs to be run.
Also note that
fp.each.do |prod|
is not syntactically correct. It should be:
fp.each do |prod|
What if you moved that fp logic into your model instead?
class Item < ActiveRecord::Base
...
def self.feeded_products(query)
self.where("name LIKE '%#{query}%' OR description LIKE '%#{query}%' OR category LIKE '%#{query}%'")
end
...
end
...and then in your controller, you can just call:
Item.feeded_products(query)
Hope this helps.
I changed to mysql instead of mysql2 adapter and it solved the problem. Thanks everyone for trying! I learned a lot trouble shooting your suggestions.
For these situations in which you have large sets of objects you can consider the use of find_each and find_in_batches methods or some kind of pagination.
Related
I'm rendering a view in Rails using form_for and nested_form_fields. Here, #procedure_step is a record that has_many :procedure_step_actions, each of which belongs_to :error, which is a ProcedureError that has (among some relations to other models) an integer :code that I'm trying to access and print out to the page. Here's my template:
<%= form_for #procedure_step do |f| %>
<%= f.nested_fields_for :procedure_step_actions do |act| %>
<%= act.object.error.code %>
<% end %>
<% end %>
When I run this, I get undefined method 'code' for nil:NilClass. Okay, so my relations are messed up and I can't access act.object.error, right? Changing my template to display that instead yields #<ProcedureError:0x0000000ece02a8>, which is what one would expect of a functioning relation. Dumping its contents to the screen using debug shows all the attributes of the record, including code, but I still can't access it with the original template! Clearly act.object.error is not nil, so Rails telling me that act.object.error is nil doesn't make any sense to me.
Frustrated, I tried to work around the problem by using act.object.error.to_json. This printed the correct JSON for the record with all its attributes. Using JSON.load() on this gave me a correct Hash of all the attributes, but using [:code] to try to access the code gives me undefined method '[]' for nil:NilClass. Again, I know that object isn't nil, but Rails still refuses to allow me to access it.
Running out of ideas, I tried to use regular expressions to pull the code out of the raw JSON string. /"code":([0-9]+)/.match(act.object.error.to_json) returned #<MatchData "\"code\":69" 1:"69">, which is right. I used [1] to try to access the code number that was matched, but again I got undefined method '[]' for nil:NilClass.
Enough with ActiveRecord, I thought to myself. I decided to turn to raw SQL queries. I got the ID of the error in question using act.object.error_id, then printed that to the screen first to make sure I could access it. Luckily, I could. Then I inserted it into my SQL query with "... WHERE id = #{act.object.error_id}". I refreshed the page again and was greeted with a SQL error. It showed the final SQL query string I had generated, but it ended with WHERE id =. The ID of the error didn't get added to the string. ProcedureError.find(action.object.error_id) gave a similar error.
I'm totally out of ideas. What could possibly be preventing me from accessing one simple integer in so many different ways?
There are at least a couple of issues here. The first is that you probably want to be using fields_for, rather than nested_fields_for, if you're using 4.x.
The second is similar to what the first answer has indicated. You have a nested fields form, which allows you to nest one level in, but you are trying to nest two levels in. By addressing your law of demeter violation you should be able to make some more progress.
Debugging things like this you can get more information by throwing in a binding.pry or byebug right in your erb.
<%- binding.pry %>
Then reload the page. Your server will be stopped at that point in your code and you can play with variable values to learn more about what's going on.
One thing I can see right off the bat is you are violating Law of Demeter here
act.object.error.code
The form object obviously has to stay but you can delegate access to the subobjects by making a method on the procedure_step which can help with handling nulls, and other error cases.
Try delegating that first as I'm not sure if the scope that is created by nested_forms_for will allow the ActiveRecord::Relation object to perform properly. I'll double check locally.
A delegation might look like the following
class ProcedureStepActions
belongs_to :error
def error_code
#error.code
end
end
EDIT:
Other things that might be helpful are the version of Ruby and Rails you are using and any other additional gems or libraries.
I am new to rails and notice a very odd pattern. I thought some error messages in Django were obscenely cryptic, but building my second app in rails I notice my errors come up NoMethodError more than 90% of the time.
How do rails people tell the difference between all these errors with the same name?
What does NoMethodError mean at it's core? It seems like what you're calling in the template is misspelled, or you're accessing attributes that don't exist?
Where can this error happen? Is the only possible cause in the template/view (the html.erb file)? Or can a bad call in the controller and whatnot cause same error?
Also, what is the best debugger gem to alleviate these issues? Reading the full trace isn't too helpful for beginners at least for a while, I;d like a first hand account of what debugger someone uses instead of reading hype
Thank you kindly
NoMethodError means you are calling a method on an object, but that object doesn't provide such method.
This is a quite bad error in the sense that is reveals a poorly designed and tested application. It generally means your code is not behaving as you are expected.
The way to reduce such errors is:
Make sure that when you are writing the code you are taking care of the various edge cases that may happen, not just the correct path. In other words, you need to take care of validating what's going on and making sure that if something is not successful (e.g. the user is not supplying all the input requested) your application will handle the case gracefully.
Make sure you write automatic test cases that covers the behavior of each method in your codebase
Keep track of the errors. When an error occurs, write a test to reproduce the behavior, fix the code and check the test passes. That will reduce the risk of regression.
This is not a Rails specific error actually. I'll try to explain what's happening at its core.
Ruby is a language that functions through message passing. Objects communicate by sending messages to each other.
The message needs to be defined as a method on the object to respond to it. This can be directly defined on the object itself, the object's class, the object's class's parents/ancestors or through included modules.
class MyObject
def some_method
puts "Yay!"
end
end
> MyObject.new.some_method
Yay!
Objects can define method_missing to handle unexpected messages.
class MyObject
def method_missing(name, *args, &block)
puts name
end
end
> MyObject.new.some_undefined_method
some_undefined_method
Without the method_missing handler, the object will raise a NoMethodError
class MyObject
end
> MyObject.new.some_undefined_method
NoMethodError: undefined method 'some_undefined_method' for #<MyObject...>
So what does this look like in Rails?
$ rails generate model User name:string
Produces this
# models/user.rb
class User < ActiveRecord::Base
end
Which has the following methods implemented among others (by ActiveRecord)
def name
end
def name=(value)
end
When you do the following:
# create a new User object
> user = User.new
#<User ... >
# call the 'name=' method on the User object
> user.name = "name"
"name"
# call the 'name' method on the User object. Note that these are 2 different methods
> user.name
"name"
> user.some_undefined_method
NoMethodError: undefined method 'some_undefined_method' for #<User...>
You'll see the same results whether you're calling it in your console, your model, your controller or in the view as they're all running the same Ruby code.
ERB view templates are slightly different in that what you enter is only evaluated as Ruby code when it's between <% %> or <%= %>. Otherwise, it gets written out to the page as text.
How do rails people tell the difference between all these errors with
the same name?
We usually look at the stack trace that comes back with the response (in development mode) as well as looking in the logs. In a dev environment, I am running my server in a console where I can scroll through the request.
What does NoMethodError mean at it's core? It seems like what you're
calling in the template is misspelled, or you're accessing attributes
that don't exist?
Due to dynamic coupling nature of Ruby. When Ruby determines that an object doesn't have a method by the name of the one that was called. It looks for a method by the name of "method_missing" within that object. If it's not defined then the super one is called which has the default behaviour of raising an exception. Rails leverages this mechanism heavily in it's internal dispatching
Where can this error happen? Is the only possible cause in the
template/view (the html.erb file)? Or can a bad call in the controller
and whatnot cause same error?
This error can happen wherever you have Ruby code, It has nothing to do with rails
Also, what is the best debugger gem to alleviate these issues? Reading
the full trace isn't too helpful for beginners at least for a while,
I;d like a first hand account of what debugger someone uses instead of
reading hype
An Invaluable tool for debugging is the gem 'pry'. It has many useful plug-able tools that greatly simplify debugging
When I executing query
Mymodel.all.each do |model|
# ..do something
end
It uses allot of memory and amount of used memory increases at all the time and at the and it crashes. I found out that to fix it I need to disable identity_map but when I adding to my mongoid.yml file identity_map_enabled: false I am getting error
Invalid configuration option: identity_map_enabled.
Summary:
A invalid configuration option was provided in your mongoid.yml, or a typo is potentially present. The valid configuration options are: :include_root_in_json, :include_type_for_serialization, :preload_models, :raise_not_found_error, :scope_overwrite_exception, :duplicate_fields_exception, :use_activesupport_time_zone, :use_utc.
Resolution:
Remove the invalid option or fix the typo. If you were expecting the option to be there, please consult the following page with repect to Mongoid's configuration:
I am using Rails 4 and Mongoid 4, Mymodel.all.count => 3202400
How can I fix it or maybe some one know other way to reduce amount of memory used during executing query .all.each ..?
Thank you very much for the help!!!!
I started with something just like you by doing loop through millions of record and the memory just keep increasing.
Original code:
#portal.listings.each do |listing|
listing.do_something
end
I've gone through many forum answers and I tried them out.
1st attempt: I try to use the combination of WeakRef and GC.start but no luck, I fail.
2nd attempt: Adding listing = nil to the first attempt, and still fail.
Success Attempt:
#start_date = 10.years.ago
#end_date = 1.day.ago
while #start_date < #end_date
#portal.listings.where(created_at: #start_date..#start_date.next_month).each do |listing|
listing.do_something
end
#start_date = #start_date.next_month
end
Conclusion
All the memory allocated for the record will never be released during
the query request. Therefore, trying with small number of record every
request does the job, and memory is in good condition since it will be
released after each request.
Your problem isn't the identity map, I don't think Mongoid4 even has an identity map built in, hence the configuration error when you try to turn it off. Your problem is that you're using all. When you do this:
Mymodel.all.each
Mongoid will attempt to instantiate every single document in the db.mymodels collection as a Mymodel instance before it starts iterating. You say that you have about 3.2 million documents in the collection, that means that Mongoid will try to create 3.2 million model instances before it tries to iterate. Presumably you don't have enough memory to handle that many objects.
Your Mymodel.all.count works fine because that just sends a simple count call into the database and returns a number, it won't instantiate any models at all.
The solution is to not use all (and preferably forget that it exists). Depending on what "do something" does, you could:
Page through all the models so that you're only working with a reasonable number of them at a time.
Push the logic into the database using mapReduce or the aggregation framework.
Whenever you're working with real data (i.e. something other than a trivially small database), you should push as much work as possible into the database because databases are built to manage and manipulate big piles of data.
I've created a model for an existing table using the following generator command:
script/generate model Group
The table in question have a different name, so I've changed the model to account for it.
set_table_name 'demandegroupe'
Then I've fired up the console to look if everything was working.
>> Group.all
[#<Group login: "XXXXXX", ...>, ...]
But, to my surprise, using this model in a view throws out weird errors. I returned to the console to make sure I wasn't hallucinating and here's what happened:
>> Group.first
#<Group login: "XXXXXX", ...>
>> Group.first.login
NoMethodError: undefined method `generated_methods' for 50:Fixnum
from /usr/lib/ruby/gems/1.8/gems/activerecord-2.3.8/lib/active_record/attribute_methods.rb:247:in `method_missing'
from (irb):2
After that the first expression is not working anymore:
>> Group.first
NoMethodError: undefined method `column_names' for 50:Fixnum
All columns are either varchar or int, where's that 50:Fixnum type is coming from?
Thanks
Are you using Mysql? Group is a reserved sql keyword. It looks like Rails 2.3.5+ handles this for mysql, but it may be causing the problem you're seeing on other databases or other versions of Rails.
Finally, I made a huge mistake, or more precisely overlooked an important detail! It's a simple column name clash, I had review them for potential issues, but missed one buried in the 26 others. That malicious column was named class and once AR generated it's magic code after accessing any column it just replaced the vital class method without throwing any error.
I expected few name clashed as the column names seemed all written in French, so didn't took enough time and must have read "classe" or something like that. Beware the mighty name clash!
Thanks to ehsanul and dmajkic for making me review those column names a second time.
I'm trying to update one of my objects in my rails app and the changes just don't stick. There are no errors, and stepping through with the debugger just reveals that it thinks everything is updating.
Anyway, here is the code in question...
qm = QuestionMembership.find(:first, :conditions => ["question_id = ? AND form_id = ?", q_id, form_id])
qm.position = x
qm.save
For reference sake, QuestionMembership has question_id, form_id, and position fields. All are integers, and have no db constraints.
That is basically my join table between Forms and Questions.
Stepping through the code, qm gets a valid object, the position of the object does get changed to the value of x, and save returns 'true'.
However, after the method exits, the object in the db is unchanged.
What am I missing?
You may not be finding the object that you think you are. Some experimenting in irb might be enlightening.
Also, as a general rule when changing only one attribute, it's better to write
qm.update_attribute(:position, x)
instead of setting and saving. Rails will then update only that column instead of the entire row. And you also get the benefit of the data being scrubbed.
Is there an after_save?
Is the correct SQL being emitted?
In development log, you can actually see the sql that is generated.
For something like this:
qm = QuestionMembership.find(:first, :conditions => ["question_id = ? AND form_id = ?", q_id, form_id])
qm.position = x
qm.save
You should see something to the effect of:
SELECT * FROM question_memberships WHERE question_id=2 AND form_id=6 LIMIT 1
UPDATE question_memberships SET position = x WHERE id = 5
Can you output what sql you are actually seeing so we can compare?
Either update the attribute or call:
qm.reload
after the qm.save
What is the result of qm.save? True or false? And what about qm.errors, does that provide anything that makes sense to you? And what does the development.log say?
I have run into this problem rather frequently. (I was about to say consistently, but I cannot, as that would imply that I would know when it was about to happen.)
While I have no solution to the underlying issue, I have found that it seems to happen to me only when I am trying to update mysql text fields. My workaround has been to set the field to do something like:
qm.position = ""
qm.save
qm.position = x
qm.save
And to answer everyone else... when I run qm.save! I get no errors. I have not tried qm.save?
When I run through my code in the rails console everything works perfectly as evidenced by re-finding the object using the same query brings the expected results.
I have the same issue when using qm.update_attribute(... as well
My workaround has gotten me limping this far, but hopefully someone on this thread will be able to help.
Try changing qm.save to qm.save! and see if you get an exception message.
Edit: What happens when you watch the log on the call to .save!? Does it generate the expected SQL?
Use ./script/console and run this script.. step by step..
see if the position field for the object is update or not when you run line 2
then hit qm.save or qm.save!... to test
see what happens. Also as mentioned by Tim .. check the logs
Check your QuestionMembership class and verify that position does not have something like
attr_readonly :position
Best way to debug this is to do
tail -f log/development.log
And then open another console and do the code executing the save statement. Verify that the actual SQL Update statement is executed.
Check to make sure your database settings are correct. If you're working with multiple databases (or haven't changed the default sqlite3 database to MySQL) you may be working with the wrong database.
Run the commands in ./script/console to see if you see the same behavior.
Verify that a similar object (say a Form or Question) saves.
If the Form or Question saves, find the difference between the QuestionMembership and Form or Question object.
Turns out that it was emitting the wrong SQL. Basically it was looking for the QuestionMembeship object by the id column which doesn't exist.
I was under the impression that that column was unnecessary with has_many_through relationships, although it seems I was misguided.
To fix, I simply added the id column to the table as a primary key. Thanks for all the pointers.