I am getting this type of error in the logs :
Parameters: {"id"=>"4", "step"=>{"documents_attributes"=>{"0"=>
{"file"=>"\x89PNG\r\n\u001A\n\u0000\u0000\u0000\rIHDR\u0000\..."}}}}
def update
#step = Step.find_by(id: params[:id])
if #step.update(steps_params)
render :json => #step
else
render :json => { :responseStatus => 402,
:responseMessage => #step.errors.full_messages.first}
end
end
During update, it rollbacks without giving any error (not execute else condition)
ArgumentError (invalid byte sequence in UTF-8):
(0.2ms) ROLLBACK
How can I fix or handle this type of request?
Your question is how to handle this type of request or error. So here is my suggestion of a general strategy.
First, do your homework. You could easily find this past question, for example. If you have tried the way already but found it did not work, you should have described what you did and what did not work in your question.
Now, I am assuming you can reproduce the case or at least you can expect you will encounter the same problem in near future (or you can wait till then) so you will have a more chance to pin down the problem next time. If you know what parameters caused the error, I guess you can reproduce the case in your development environment. However, if not, it is more tricky to pin down — it heavily depends how much information about the error and input you have and what development environment you can use, and my answer does not cover the case.
The first objective should be to pin down which command (method) exactly in your code caused an error. Did it happen just inside Rails or did your DB raise an error?
In your specific case, did it occur at Step.find_by or #step.update or else? What is steps_params? It seems like a method you have defined. Are you sure steps_params is working as expected? (You may be sure, but we don't know…)
A convenient way to find it out is simply to insert logger.debug (or logger.error) etc before and after each sentence. In doing it, it is recommended to split a sentence into smaller units in some cases. For example, steps_params and update() should be separated, such as (in the simplest case),
logger.debug 'Before steps_params'
res_steps_params = steps_params
logger.debug 'Before update'
res_update = #step.update(res_steps_params)
logger.debug 'Before if'
if res_update
# ……
Obviously you can (and perhaps should) log more detailed information, such as, res_steps_params.inspect, and you may also enclose a part with a begin-rescue clause so that you can get the detailed infromation about the exception and log it. Also, I can recommend to split update into 2 parts – substitutions and save – to find out exactly what action and parameter cause a problem.
Once you have worked out which of DB or Rails or something before (like HTTP-server or Client-browser) is to blame and which parameter causes a problem, then you can proceed to the next stage. The error message suggests it is a character-encoding issue. Is the character encoding of a string invalid (as a UTF-8), or wrongly recognised by Rails (which might be not a fault of Rails but of the client), or not recognised correctly by the DB?
Wherever the problem lies, it is usually (though not always!) possible to fix or circumvent character-encoding problems with Ruby (Rails). The Ruby methods of String#encode, String#encoding, and String#force_encoding would be useful to diagnose and perhaps fix the problem.
As an added note, it can be useful, if possible in your environment, to browse the logfile of your DB (PostgreSQL?) to find out which query passed from Rails to the DB caused a problem (if a query was indeed passed to them!). Alternatively, Rails Gem SQL Query Tracker might be handy to know what queries your Rails app create (though I have never used it and so can't tell much.)
At the end of the day, when a code misbehaves mysteriously, I am afraid only the sure way to solve is to narrow down the problematic clause or parameter step by step. Good luck!
Related
When I executing query
Mymodel.all.each do |model|
# ..do something
end
It uses allot of memory and amount of used memory increases at all the time and at the and it crashes. I found out that to fix it I need to disable identity_map but when I adding to my mongoid.yml file identity_map_enabled: false I am getting error
Invalid configuration option: identity_map_enabled.
Summary:
A invalid configuration option was provided in your mongoid.yml, or a typo is potentially present. The valid configuration options are: :include_root_in_json, :include_type_for_serialization, :preload_models, :raise_not_found_error, :scope_overwrite_exception, :duplicate_fields_exception, :use_activesupport_time_zone, :use_utc.
Resolution:
Remove the invalid option or fix the typo. If you were expecting the option to be there, please consult the following page with repect to Mongoid's configuration:
I am using Rails 4 and Mongoid 4, Mymodel.all.count => 3202400
How can I fix it or maybe some one know other way to reduce amount of memory used during executing query .all.each ..?
Thank you very much for the help!!!!
I started with something just like you by doing loop through millions of record and the memory just keep increasing.
Original code:
#portal.listings.each do |listing|
listing.do_something
end
I've gone through many forum answers and I tried them out.
1st attempt: I try to use the combination of WeakRef and GC.start but no luck, I fail.
2nd attempt: Adding listing = nil to the first attempt, and still fail.
Success Attempt:
#start_date = 10.years.ago
#end_date = 1.day.ago
while #start_date < #end_date
#portal.listings.where(created_at: #start_date..#start_date.next_month).each do |listing|
listing.do_something
end
#start_date = #start_date.next_month
end
Conclusion
All the memory allocated for the record will never be released during
the query request. Therefore, trying with small number of record every
request does the job, and memory is in good condition since it will be
released after each request.
Your problem isn't the identity map, I don't think Mongoid4 even has an identity map built in, hence the configuration error when you try to turn it off. Your problem is that you're using all. When you do this:
Mymodel.all.each
Mongoid will attempt to instantiate every single document in the db.mymodels collection as a Mymodel instance before it starts iterating. You say that you have about 3.2 million documents in the collection, that means that Mongoid will try to create 3.2 million model instances before it tries to iterate. Presumably you don't have enough memory to handle that many objects.
Your Mymodel.all.count works fine because that just sends a simple count call into the database and returns a number, it won't instantiate any models at all.
The solution is to not use all (and preferably forget that it exists). Depending on what "do something" does, you could:
Page through all the models so that you're only working with a reasonable number of them at a time.
Push the logic into the database using mapReduce or the aggregation framework.
Whenever you're working with real data (i.e. something other than a trivially small database), you should push as much work as possible into the database because databases are built to manage and manipulate big piles of data.
I've inherited some code written in ruby 1.8.7 with Rails ~> 2.3.15 and it contains a test which looks something like this:
class Wibble < ActiveRecord::Base
# Wibbles have integer primary keys and string names
end
def test
create_two_wibbles
w1 = Wibble.first
sleep 2 # This sleep is necessary to
w2 = Wibble.first
w1.name = "Name for w1"
w2.name = "Name for w2"
w1.save
w3 = Wibble.first
assert(!w3.update_attributes(w2.attributes))
end
That comment next to the sleep line hasn't been cut off, it literally says This sleep is necessary to. Without that sleep, this test fails - removing it changes the behaviour somehow beyond making it run 2 seconds faster. I've been digging through the file's history in our version control system, and the messages were uninformative. I also cannot contact the original author of this test to figure out what they were trying to do.
From my understanding, we're pulling the same record out of the database twice, editing it in two different ways, saving the first, and asserting that we can't then save the second. I suspect this is a test to make sure our database locks the table correctly, but surely if this were to fail our Wibbles would be fine, and ActiveRecord would be at fault. Nobody in the office can figure out why this test may have been necessary, nor what difference the sleep statement might make. Any ideas?
This is likely caused by Active Record caching and memoization stopping the second find from actually going to the DB and get fresh data.
In fact, try printing the object_id of each wibble; they might be the exact same memoized object. If that is the case, then the test kinda makes sense.
You could also do the same test in a controller action and look at the verbose SQL logs from Rails; I'd expect the second find call to tell you it just used cached data and did not actually run any SQL query.
I have the following code in a rails model:
foo = Food.find(...)
foo.with_lock do
if bar = foo.bars.find_by_stuff(stuff)
# do something with bar
else
bar = foo.bars.create!
# do something with bar
end
end
The goal is to make sure that a Bar of the type being created is not being created twice.
Testing with_lock works at the console confirms my expectations. However, in production, it seems that in either some or all cases the lock is not working as expected, and the redundant Bar is being attempted -- so, the with_lock doesn't (always?) result in the code waiting for its turn.
What could be happening here?
update
so sorry to everyone who was saying "locking foo won't help you"!! my example initially didin't have the bar lookup. this is fixed now.
You're confused about what with_lock does. From the fine manual:
with_lock(lock = true)
Wraps the passed block in a transaction, locking the object before yielding. You pass can the SQL locking clause as argument (see lock!).
If you check what with_lock does internally, you'll see that it is little more than a thin wrapper around lock!:
lock!(lock = true)
Obtain a row lock on this record. Reloads the record to obtain the requested lock.
So with_lock is simply doing a row lock and locking foo's row.
Don't bother with all this locking nonsense. The only sane way to handle this sort of situation is to use a unique constraint in the database, no one but the database can ensure uniqueness unless you want to do absurd things like locking whole tables; then just go ahead and blindly try your INSERT or UPDATE and trap and ignore the exception that will be raised when the unique constraint is violated.
The correct way to handle this situation is actually right in the Rails docs:
http://apidock.com/rails/v4.0.2/ActiveRecord/Relation/find_or_create_by
begin
CreditAccount.find_or_create_by(user_id: user.id)
rescue ActiveRecord::RecordNotUnique
retry
end
("find_or_create_by" is not atomic, its actually a find and then a create. So replace that with your find and then create. The docs on this page describe this case exactly.)
Why don't you use a unique constraint? It's made for uniqueness
A reason why a lock wouldn't be working in a Rails app in query cache.
If you try to obtain an exclusive lock on the same row multiple times in a single request, query cached kicks in so subsequent locking queries never reach the DB itself.
The issue has been reported on Github.
I have a rails 2.3.4 app and a line that looks like:
temp = Rails.cache.fetch(:temp_id) { User.find_by_name('Temp').id }
and everything worked fine, until I decided to switch the caching layer to memcached by adding the following to my environment.rb:
config.cache_store = :mem_cache_store
Now the line which used to work fine gives me the following error:
undefined method 'length' for :temp_id:Symbol
/usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.4/lib/active_support/vendor/memcache-client-1.7.4/memcache.rb:645:in 'get_server_for_key'
I understand the error, but I would imagine this common case would have been quickly discovered by a rails test case, so I am wondering if I am doing something wrong. Otherwise, I'm sure I can monkeypatch this issue to convert the symbol to a string.
Thanks
Just use string keys if you can. All the documentation examples use string keys. Although it's not explicitly mentioned as far as I can see, other keys are not supported.
The key arguments are passed directly to the cache implementation, so the different caching flavours may disagree on whether or not they accept anything other than strings.
Because the caches are external with the exception of in-memory cache, I'm not sure that supporting symbols would be useful apart from preventing cases like yours. The key will actually be written to some output somewhere (it's not just internal to your Ruby app), so conceptually the key should be a string.
Edit in reaction to comment: yes, it is of course possible and perfectly reasonable in this case to create a monkey patch to circumvent having to change all calls. What you're suggesting is this (copied into the answer for readability):
class MemCache
def get_server_for_key_with_symbols(key, options = {})
key = key.to_s if key.is_a? Symbol
get_server_for_key_without_symbols(key, options)
end
alias_method_chain :get_server_for_key, :symbols
end
I would also consider just doing a project wide search-and-replace for \.fetch(:\w+) and replace it with \.fetch("$1") (repeat for read and write if necessary). This should probably cover 95% of all cases and a subsequent run of your test suite should catch the rest of the errors.
In general: While the documentation of Rails is pretty good these days, a lot of assumptions are unfortunately still implicit. It's generally a good idea to take a good look at the examples that are given in the documentation, and use the same style. The documented examples are always how the framework was intended to be used.
FWIW, it's canonically Rails.cache.read and Rails.cache.write.
I'm trying to update one of my objects in my rails app and the changes just don't stick. There are no errors, and stepping through with the debugger just reveals that it thinks everything is updating.
Anyway, here is the code in question...
qm = QuestionMembership.find(:first, :conditions => ["question_id = ? AND form_id = ?", q_id, form_id])
qm.position = x
qm.save
For reference sake, QuestionMembership has question_id, form_id, and position fields. All are integers, and have no db constraints.
That is basically my join table between Forms and Questions.
Stepping through the code, qm gets a valid object, the position of the object does get changed to the value of x, and save returns 'true'.
However, after the method exits, the object in the db is unchanged.
What am I missing?
You may not be finding the object that you think you are. Some experimenting in irb might be enlightening.
Also, as a general rule when changing only one attribute, it's better to write
qm.update_attribute(:position, x)
instead of setting and saving. Rails will then update only that column instead of the entire row. And you also get the benefit of the data being scrubbed.
Is there an after_save?
Is the correct SQL being emitted?
In development log, you can actually see the sql that is generated.
For something like this:
qm = QuestionMembership.find(:first, :conditions => ["question_id = ? AND form_id = ?", q_id, form_id])
qm.position = x
qm.save
You should see something to the effect of:
SELECT * FROM question_memberships WHERE question_id=2 AND form_id=6 LIMIT 1
UPDATE question_memberships SET position = x WHERE id = 5
Can you output what sql you are actually seeing so we can compare?
Either update the attribute or call:
qm.reload
after the qm.save
What is the result of qm.save? True or false? And what about qm.errors, does that provide anything that makes sense to you? And what does the development.log say?
I have run into this problem rather frequently. (I was about to say consistently, but I cannot, as that would imply that I would know when it was about to happen.)
While I have no solution to the underlying issue, I have found that it seems to happen to me only when I am trying to update mysql text fields. My workaround has been to set the field to do something like:
qm.position = ""
qm.save
qm.position = x
qm.save
And to answer everyone else... when I run qm.save! I get no errors. I have not tried qm.save?
When I run through my code in the rails console everything works perfectly as evidenced by re-finding the object using the same query brings the expected results.
I have the same issue when using qm.update_attribute(... as well
My workaround has gotten me limping this far, but hopefully someone on this thread will be able to help.
Try changing qm.save to qm.save! and see if you get an exception message.
Edit: What happens when you watch the log on the call to .save!? Does it generate the expected SQL?
Use ./script/console and run this script.. step by step..
see if the position field for the object is update or not when you run line 2
then hit qm.save or qm.save!... to test
see what happens. Also as mentioned by Tim .. check the logs
Check your QuestionMembership class and verify that position does not have something like
attr_readonly :position
Best way to debug this is to do
tail -f log/development.log
And then open another console and do the code executing the save statement. Verify that the actual SQL Update statement is executed.
Check to make sure your database settings are correct. If you're working with multiple databases (or haven't changed the default sqlite3 database to MySQL) you may be working with the wrong database.
Run the commands in ./script/console to see if you see the same behavior.
Verify that a similar object (say a Form or Question) saves.
If the Form or Question saves, find the difference between the QuestionMembership and Form or Question object.
Turns out that it was emitting the wrong SQL. Basically it was looking for the QuestionMembeship object by the id column which doesn't exist.
I was under the impression that that column was unnecessary with has_many_through relationships, although it seems I was misguided.
To fix, I simply added the id column to the table as a primary key. Thanks for all the pointers.