According to the redis-rb ruby gem docs;
redis.watch("key") do
if redis.get("key") == "some value"
redis.multi do |multi|
multi.set("key", "other value")
multi.incr("counter")
end
else
redis.unwatch
end
end
I have two questions about this;
Why is the unwatch line required? My thought is that if key == "some value", set "key" to other value and increase counter if "key" has not changed by the time EXEC is called. In my mind, the else statement never gets executed in this scenario. If you run this code without the else/unwatch, nil is returned.
If redis.watch "key" is entered without a block (redis.watch "key"); Why am I unable to unwatch a specific key? I get (wrong number of arguments (given 1, expected 0))??
Here is a link to the ruby gem https://github.com/redis/redis-rb
I am not a Ruby nor Redis expert.
Though by reading the Redis documentation about transaction here https://redis.io/topics/transactions we learn that a watched key is unwatched in two ways : either after EXEC is used. (then the transactions completes) or either by UNWATCH.
So there is indeed a reason to use UNWATCH. If the multi block is not called then EXEC is not called either. Then the key is never flushed.
Also I think the conditionnal bit containing redis.unwatch can indeed happen to be called. As redis.get("key") == "some value" seems to be purely arbitrary and basically a code design.
It could well be redis.get("key") > "some value" for example.
(Also it seem having a block passed to watched does not UNWATCH the key automatically as per documentation, as the block is optionnal)
For your second question you are correct, you cannot unwatch a specific key it seems. Though redis.unwatch seems to work fine.
This does not seem to be a Ruby design. Watching a key is only useful in order to process an atomic transaction, there is no point in leaving watched keys behind us when transaction is either successful or aborted.
I don't see any example of using UNWATCH in the Redis documentation https://redis.io/topics/transactions but it feels natural to flush every key after the transaction block.
Related
I am getting this type of error in the logs :
Parameters: {"id"=>"4", "step"=>{"documents_attributes"=>{"0"=>
{"file"=>"\x89PNG\r\n\u001A\n\u0000\u0000\u0000\rIHDR\u0000\..."}}}}
def update
#step = Step.find_by(id: params[:id])
if #step.update(steps_params)
render :json => #step
else
render :json => { :responseStatus => 402,
:responseMessage => #step.errors.full_messages.first}
end
end
During update, it rollbacks without giving any error (not execute else condition)
ArgumentError (invalid byte sequence in UTF-8):
(0.2ms) ROLLBACK
How can I fix or handle this type of request?
Your question is how to handle this type of request or error. So here is my suggestion of a general strategy.
First, do your homework. You could easily find this past question, for example. If you have tried the way already but found it did not work, you should have described what you did and what did not work in your question.
Now, I am assuming you can reproduce the case or at least you can expect you will encounter the same problem in near future (or you can wait till then) so you will have a more chance to pin down the problem next time. If you know what parameters caused the error, I guess you can reproduce the case in your development environment. However, if not, it is more tricky to pin down — it heavily depends how much information about the error and input you have and what development environment you can use, and my answer does not cover the case.
The first objective should be to pin down which command (method) exactly in your code caused an error. Did it happen just inside Rails or did your DB raise an error?
In your specific case, did it occur at Step.find_by or #step.update or else? What is steps_params? It seems like a method you have defined. Are you sure steps_params is working as expected? (You may be sure, but we don't know…)
A convenient way to find it out is simply to insert logger.debug (or logger.error) etc before and after each sentence. In doing it, it is recommended to split a sentence into smaller units in some cases. For example, steps_params and update() should be separated, such as (in the simplest case),
logger.debug 'Before steps_params'
res_steps_params = steps_params
logger.debug 'Before update'
res_update = #step.update(res_steps_params)
logger.debug 'Before if'
if res_update
# ……
Obviously you can (and perhaps should) log more detailed information, such as, res_steps_params.inspect, and you may also enclose a part with a begin-rescue clause so that you can get the detailed infromation about the exception and log it. Also, I can recommend to split update into 2 parts – substitutions and save – to find out exactly what action and parameter cause a problem.
Once you have worked out which of DB or Rails or something before (like HTTP-server or Client-browser) is to blame and which parameter causes a problem, then you can proceed to the next stage. The error message suggests it is a character-encoding issue. Is the character encoding of a string invalid (as a UTF-8), or wrongly recognised by Rails (which might be not a fault of Rails but of the client), or not recognised correctly by the DB?
Wherever the problem lies, it is usually (though not always!) possible to fix or circumvent character-encoding problems with Ruby (Rails). The Ruby methods of String#encode, String#encoding, and String#force_encoding would be useful to diagnose and perhaps fix the problem.
As an added note, it can be useful, if possible in your environment, to browse the logfile of your DB (PostgreSQL?) to find out which query passed from Rails to the DB caused a problem (if a query was indeed passed to them!). Alternatively, Rails Gem SQL Query Tracker might be handy to know what queries your Rails app create (though I have never used it and so can't tell much.)
At the end of the day, when a code misbehaves mysteriously, I am afraid only the sure way to solve is to narrow down the problematic clause or parameter step by step. Good luck!
I've just started to take on my first model spec task at work. After writing a lot of feature specs, I find it hard to get into the different perspective of writing model specs (not taking the context into consideration). I'll take a method of the Order model as an example, to explain which difficulties I am experiencing:
def update_order_prices
self.shipping_price_cents = SHIPPING_PRICE_CENTS unless shipping_price_cents
return if order_lines.empty?
self.total_price_cents = calculate_order_price
self.total_line_items_price_cents = calculate_total_order_line_price
self.total_tax_cents = calculate_tax_amount
end
EDIT TL;DR
I am totally happy with an answer that simply writes me a spec for this method. The rest of the post just shows what I tried so far but is not necessary to answer this question.
First approach:
At first I didn't know what to test for. I tried to find out when and where the method was called and to find a scenario where I would know what the attributes that are touched in this method should be equal to. Put short, I spent a lot of time trying to understand the context. Then a coworker said that I should test methods in model specs self-contained, independent from the context. I should just make sure I identify all cases. So for this method that would be:
it sets shipping price cents to default (if not done already)
it returns early if order_lines is empty
it sets values if order_line is set
Current approach:
I tried writing the tests for these points but still questions arise:
Test 1
it 'sets shipping price cents to default (if not done already)' do
order.shipping_price_cents = nil
order.update_order_prices
expect(order.shipping_price_cents).to eq(Order::SHIPPING_PRICE_CENTS)
end
I am confident I got this one right, but feel free to prove me wrong. I set shipping_price_cents to nil to trigger the code that sets it, call the tested method on the cents to be equal to the default value as defined in the model.
Test 2
it 'returns early if order_lines is empty' do
expect(order.update_order_prices).to eq(nil)
end
So here I want to test that the method returns early when there is no object in the order_lines association. I didn't have a clue how to do that so I went into the console, took an order, removed the order_lines associated with it, and called the method to see what would be returned.
2.3.1 :011 > o.order_lines
=> #<ActiveRecord::Associations::CollectionProxy []>
2.3.1 :012 > o.update_order_prices
=> nil
Then did the same for an order with associated order_line:
2.3.1 :017 > o.update_order_prices
=> 1661
So I tested for 'nil' to be returned. But it doesn't feel like I am testing the right thing.
Test 3
it 'sets (the correct?) values if order_line is set' do
order_line = create(:order_line, product: product)
order = create(:order, order_lines: [order_line])
order.update_order_prices
expect(order.total_price_cents).to eq(order.calculate_order_price)
expect(order.total_line_items_price_cents).to eq(order.calculate_order_line_price)
expect(order.total_tax_cents).to eq(order.calculate_tax_amount)
end
I simply test that the attributes equal what they are set to, without using actual values, as I shouldn't look outside. If I wanted to test for an absolute value, I would have to investigate outside of this function which then wouldn't test the method but also status of the Order object etc.?
Running the tests
Failures:
1) Order Methods: #update_order_prices sets (the correct?) values if order_line is set
Failure/Error: expect(order.total_price_cents).to eq(order.calculate_order_price)
NoMethodError:
private method `calculate_order_price' called for #<Order:0x007ff9ee643df0>
Did you mean? update_order_prices
So, the first two tests passed, the third one didn't. At this point I feel a bit lost and would love hear how some experienced developers would write this seemingly simple test.
Thanks
I guess you have to spec against the exact values you are expecting after update_order_prices.
Let's say you set up your order and order lines to have a total price of 10 euros then I'd add the following expectation
expect(order.total_price_cents).to eq(1000)
Same for the other methods. Generally I try to test against specific values. Also as you are relying on the result of a private method you only care about the result.
I've inherited some code written in ruby 1.8.7 with Rails ~> 2.3.15 and it contains a test which looks something like this:
class Wibble < ActiveRecord::Base
# Wibbles have integer primary keys and string names
end
def test
create_two_wibbles
w1 = Wibble.first
sleep 2 # This sleep is necessary to
w2 = Wibble.first
w1.name = "Name for w1"
w2.name = "Name for w2"
w1.save
w3 = Wibble.first
assert(!w3.update_attributes(w2.attributes))
end
That comment next to the sleep line hasn't been cut off, it literally says This sleep is necessary to. Without that sleep, this test fails - removing it changes the behaviour somehow beyond making it run 2 seconds faster. I've been digging through the file's history in our version control system, and the messages were uninformative. I also cannot contact the original author of this test to figure out what they were trying to do.
From my understanding, we're pulling the same record out of the database twice, editing it in two different ways, saving the first, and asserting that we can't then save the second. I suspect this is a test to make sure our database locks the table correctly, but surely if this were to fail our Wibbles would be fine, and ActiveRecord would be at fault. Nobody in the office can figure out why this test may have been necessary, nor what difference the sleep statement might make. Any ideas?
This is likely caused by Active Record caching and memoization stopping the second find from actually going to the DB and get fresh data.
In fact, try printing the object_id of each wibble; they might be the exact same memoized object. If that is the case, then the test kinda makes sense.
You could also do the same test in a controller action and look at the verbose SQL logs from Rails; I'd expect the second find call to tell you it just used cached data and did not actually run any SQL query.
What I'm doing
I'm using the twitter gem (a Ruby wrapper for the Twitter API) in my app, which is run on Heroku. I use Heroku's Scheduler to periodically run caching tasks that use the twitter gem to, for example, update the list of retweets for a particular user. I'm also using delayed_job so scheduler calls a rake task, which calls a method that is 'delayed' (see scheduler.rake below). The method loops through "authentications" (for users who have authenticated twitter through my app) to update each authorized user's retweet cache in the app.
My question
What am I doing wrong? For example, since I'm using Heroku's Scheduler, is delayed_job redundant? Also, you can see I'm not catching (rescuing) any errors. So, if Twitter is unreachable, or if a user's auth token has expired, everything chokes. This is obviously dumb and terrible because if there's an error, the entire thing chokes and ends up creating a failed delayed_job, which causes ripple effects for my app. I can see this is bad, but I'm not sure what the best solution is. How/where should I be catching errors?
I'll put all my code (from the scheduler down to the method being called) for one of my cache methods. I'm really just hoping for a bulleted list (and maybe some code or pseudo-code) berating me for poor coding practice and telling me where I can improve things.
I have seen this SO question, which helps me a little with the begin/rescue block, but I could use more guidance on catching errors, and one the higher-level "is this a good way to do this?" plane.
Code
Heroku Scheduler job:
rake update_retweet_cache
scheduler.rake (in my app)
task :update_retweet_cache => :environment do
Tweet.delay.cache_retweets_for_all_auths
end
Tweet.rb, update_retweet_cache method:
def self.cache_retweets_for_all_auths
#authentications = Authentication.find_all_by_provider("twitter")
#authentications.each do |authentication|
authentication.user.twitter.retweeted_to_me(include_entities: true, count: 200).each do |tweet|
# Actually build the cache - this is good - removing to keep this short
end
end
end
User.rb, twitter method:
def twitter
authentication = Authentication.find_by_user_id_and_provider(self.id, "twitter")
if authentication
#twitter ||= Twitter::Client.new(:oauth_token => authentication.oauth_token, :oauth_token_secret => authentication.oauth_secret)
end
end
Note: As I was posting this, I noticed that I'm finding all "twitter" authentications in the "cache_retweets_for_all_auths" method, then calling the "User.twitter" method, which specifically limits to "twitter" authentications. This is obviously redundant, and I'll fix it.
First what is the exact error you are getting, and what do you want to happen when there is an error?
Edit:
If you just want to catch the errors and log them then the following should work.
def self.cache_retweets_for_all_auths
#authentications = Authentication.find_all_by_provider("twitter")
#authentications.each do |authentication|
being
authentication.user.twitter.retweeted_to_me(include_entities: true, count: 200).each do |tweet|
# Actually build the cache - this is good - removing to keep this short
end
rescue => e
#Either create an object where the error is log, or output it to what ever log you wish.
end
end
end
This way when it fails it will keep moving on to the next user but will still making a note of the error. Most of the time with twitter its just better to do something like this then try to do with each error on its own. I have seen so many weird things out of the twitter API, and random errors, that trying to track down every error almost always turns into a wild goose chase, though it is still good to keep track just in case.
Next for when you should use what.
You should use a scheduler when you need something to happen based on time only, delayed jobs for when its based on an user action, but the 'action' you are going to delay would take to long for a normal response. Sometimes you can just put the thing plainly in the controller also.
So in other words
The scheduler will be fine as long as the time between updates X is less then the time it will take for the update to happen, time Y.
If X < Y then you might want to look at calling the logic from the controller when each indvidual entry is accessed, isntead of trying to do them all at once. The idea being you would only update it after a certain time as passed so. You could store the last time update either on the model itself in a field like twitter_udpate_time or in a redis or memecache instance at a unquie key for the user/auth.
But if the individual update itself is still too long, then thats when you should do the above, but instead of doing the actually update, call a delayed job.
You could even set it up that it only updates or calls the delayed job after a certain number of views, to further limit stuff.
Possible Fancy Pants
Or if you want to get really fancy you could still do it as a cron job, but have a point system based on views that weights which entries should be updated. The idea being certain actions would add points to certain users, and if their points are over a certain amount you update them, and then remove their points. That way you could target the ones you think are the most important, or have the most traffic or show up in the most search results etc etc.
Next off a nick picky thing.
http://api.rubyonrails.org/classes/ActiveRecord/Batches.html
You should be using
#authentications.find_each do |authentication|
instead of
#authentications.each do |authentication|
find_each pulls in only 1000 entries at a time so if you end up with a lof of Authentications you don't end up pulling a crazy amount of entries into memory.
Look at the discussion on this thread . I am not able to follow how having a block to a fetch is a better solution.
In the first patch on Rails ticket #4558:
options.fetch(:alt, File.basename(src, '.*').capitalize)
This line executes the basename and capitalize functions and then passes the result into Hash#fetch regardless of if a value for :alt already exists in the options hash.
In the updated patch:
options.fetch(:alt) { File.basename(src, '.*').capitalize }
The same basename/capitalize code is only executed when Hash#fetch needs the default value (i.e. when the :alt key does not exist in the options hash). This means the (possibly expensive) calculation of the default value can be skipped if it's not needed.
See the documentation on Hash#fetch for more details.
I don't know what duck-punching Rails has been doing to Hash#fetch, but with Plain Old Ruby Objects, I use it rather than Hash#[] because when I ask for something and it's not available, I want to know about it. This is called "Failing early" (or "Crashing early" in The Pragmatic Programmer's List of Tips)