I'm using Ruby on Rails (v3.2.13), Dalli (v2.6.4) and MemCached (v1.4.13).
I do caching like this:
result = Rails.cache.fetch("test_key", :expires_in => 1.week) do
get_data() # slow call, result of which should be cached
end
I want to update cache expiration date based on the data, since some of my data can be kept longer.
Right now the following code does the job:
if keep_longer(result)
Rails.cache.write("test_key", result, :expires_in => 6.months)
end
I know that MemCached supports "touch" command that allows to update expiration date without sending the value. And I don't see how to use it through the Dalli gem. Is there a way to update expiration date without resending the result?
UPDATE:
Rails.cache.dalli.touch('some_key', 24.hours)
This should work, but for me it doesn't. Does it work for you?
Here is small example you can try. After execution of the following code in the IRB
dc = Dalli::Client.new("localhost:11211")
dc.set("test_key", "test_value", 5.minutes)
dc.set( "key", "value", 5.minutes)
dc.touch( "key", 10.minutes)
I'm checking the expiration dates using telnet:
telnet localhost 11211
Then given the correct slab_id and using "stats cachedump" command I obtain expiration times in seconds:
stats cachedump 1 0
ITEM key [9 b; 1375733492 s]
ITEM test_key [14 b; 1375905957 s]
Note that the expiration time of the key "key" points to the past. When I expect it to be 300 seconds later than "test_key" expiration time. Also I noticed that "key" expiration time is approximately 1 second before the MemCached server has started. Which probably indicates that this key has no expiration time. And in fact "key" doesn't get deleted in the near future.
Am I doing something wrong or it is a bug of Dalli/MemCached?
Dalli does support this - there's a touch method on Dalli::Client that does exactly what it says on the tin. Rails.cache returns a cache store rather than the underlying Dalli object so you need to do
Rails.cache.dalli.touch('some_key', 24.hours)
To bump the cache entry's expiry time by 24 hours (and of course memcache may decide to drop the entry anyway)
I found that my version of MemCached (v1.4.13) has the bug: binary touch operation did not update expiration time correctly. This bug was fixed in v1.4.14 (release notes):
Fixed issue with invalid binary protocol touch command expiration time
The problem now: as of today versions v1.4.14 and later cannot be installed using apt-get.
Related
I am trying to validate the system time of client’s computer with the actual time (internet time). If for some reason the client’s time settings are not correct or the time and timezone don’t match the local time, I want to notify them to sync the time with their local time in order to use the application. If my question is not clear then this is something that I am trying to mimic, https://knowledge.autodesk.com/support/autocad/troubleshooting/caas/sfdcarticles/sfdcarticles/Incorrect-System-time-warning-when-starting-an-Autodesk-360-application.html
How can I do this time comparison/validation in dart?
The main question is IMHO what accuracy you need.
You can just query a NTC and report if there is a discrepancy. If the server is synchronized with such a time server, there shouldn't be a problem.
You can also add an API to your server that returns the server time.
Then you read the time from the local system and from the server and check the difference
bool compareTime() {
var serverTime = await getTimeFromServer(); // not yet existing method to fetch the date and time from the server
var clientTime = new DateTime.now().toUtc();
var diff = serverTime.difference(clientTime).abs();
if(diff > const Duration(seconds: 5)) {
print('Time difference too big');
} else {
print('Time is fine');
}
}
Ensure that the time returned from the server is UTC as well, otherwise you're comparing apples with pears.
If you're running server-side, you can shell out to ntpdate -d pool.ntp.org and read the output, parsing the last line. If the time offset is small enough, you're good.
For browser apps, see the StackOverflow at Getting the current GMT world time for a few options.
I am seeing some very slow cache reads in my rails app. Both redis (redis-rails) and memcached (dalli) produced the same results.
It looks like it is only the first call to Rails.cache that causes the slowness (averaging 500ms).
I am using skylight to instrument my app and see a graph like:
I have a Rails.cache.fetch call in this code, but when I benchmark it I see it average around 8ms, which matches what memcache-top shows for my average call time.
I thought this might be dalli connections opening slowly, but benchmarking that didnt show anything slow either. I'm at a loss for what else to check into.
Does anyone have any good techniques for tracking this sort of thing down in a rails app?
Edit #1
Memcache server is stored in ENV['MEMCACHE_SERVERS'], all the servers are in the us-east-1 datacenter.
Cache config looks like:
config.cache_store = :dalli_store, nil, { expires_in: 1.day, compress: true }
I ran something like:
100000.times { Rails.cache.fetch('something') }
and calculated the average timings and got something on the order of 8ms when running on one of my webservers.
Testing my theory of the first request is slow, I opened a console on my web server and ran the following as the first command.
irb(main):002:0> Benchmark.ms { Rails.cache.fetch('someth') { 1 } }
Dalli::Server#connect my-cache.begfpc.0001.use1.cache.amazonaws.com:11211
=> 12.043342
Edit #2
Ok, I split out the fetch into a read and write, and tracked them independently with statsd. It looks like the averages sit around what I would expect, but the max times on the read are very spiky and get up into the 500ms range.
http://s16.postimg.org/5xlmihs79/Screen_Shot_2014_12_19_at_6_51_16_PM.png
I noticed when PingFederate(PF) sends the date it is off by a day from my Rails app. It appears that the PF date is the one off by a day. For example in the PF SAML response I get:
<saml:Assertion ID="pEaf1kce93SpAxfIpuohOv6QP-T" IssueInstant="2014-05-03T03:15:20.020Z" Version="2.0" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
and
<saml:Conditions NotBefore="2014-05-03T03:10:20.021Z" NotOnOrAfter="2014-05-03T03:20:20.021Z">
whereas in Rails I get
Time.now = 2014-05-02 20:15:19 -0700
which makes me think that I need to set the date in PF. Note that PF is running on the same computer that the Rails app is running on.
Is there a way to set the date in PF?
The time is off as well. Is there a way in PF to set the time too?
SAML assertions are always in UTC. Which is what PingFed is using. Set your Rails application to use UTC as well.
I don't see why you think that the time is off. Your time from Rails is 2015 and -7. That means in UTC, it's tomorrow at 0315 - when it was issued. PingFed is setting an allowance of +/-5 minutes, so the SP should not accept it before 0310 or after 0320.
Your server and PingFed are correct so far...
I'm using a gem to get code results from Ideone.com. The gem submits code to Ideone and then checks for the results page. It checks timeout times and then gives up if there's no result. The problem is it might give up too early, but I also don't want it to wait too long if there's not going to be a result. Is there a way to know when one should give up hope?
This is the relevant code:
begin
sleep 3 if i > 0
res = JSON.load(
Net::HTTP.post_form(
URI.parse("http://ideone.com/ideone/Index/view/id/#{loc}/ajax/1"),
{}
).body
)
i += 1
end while res['status'] != '0' && i < timeout
if i == timeout
raise IdeoneError, "Timed out while waiting for code result."
end
Sounds like you want to adjust sleep timeout and number of attempts parameters. There is no absolute values suitable for each case, so you should pick some which are most appropriate for you application.
Unfortunatelly the gem code have both this parameters (3 seconds delay and 4 attempts) hardcoded so you don't have an elegant way to change them. So you can either fork the gem and change its code or try to monkey-patch the value of TIMEOUT constant with http://apidock.com/ruby/Module/const_set . However you won't be able to monkey-patch the delay between attempts value without rewriting method .run of the gem.
FYI. Net::HTTP has their own timeouts - how much time to wait for ideone.com connection and response. If they are exceeded Net::HTTP raises Timeout exception. The setters are
http://ruby-doc.org/stdlib-2.0/libdoc/net/http/rdoc/Net/HTTP.html#method-i-read_timeout-3D and #open_timeout=.
This question already has an answer here:
Overriding/Modifying Rails Class (ActiveResource)
(1 answer)
Closed 3 years ago.
I'm trying to contact a REST API using ActiveResource on Rails 2.3.2.
I'm attempting to use the timeout functionality so that if the resource I'm contacting is down I can fail quickly - I'm doing this with the following:
class WorkspaceResource < ActiveResource::Base
self.timeout = 5
self.site = "http://mysite.com/restAPI"
end
However, when I try to contact the service when I know it isn't available, the class only times out after the default 60 seconds. I can see from the error stack that the timeout error does indeed come from an ActiveResource class in my gem folder that has the proper functions to allow timeout settings, but my set timeout never seems to work.
Any thoughts?
So apparently the issue is not that timeout is not functioning. I can run a server locally, make it not return a response within the timeout limit, and see that timeout works.
The issue is in fact that if the server does not accept the connection, timeout does not function as I expected it to - it doesn't function at all. It appears as though timeout only works when the server accepts the connection but takes too long to respond.
To me, this seems like an issue - shouldn't timeout also work when the server I'm contacting is down? If not, there should be another mechanism to stop a bunch of requests from hanging...anyone know of a quick way to do this?
The problem
If you're running on Ruby 1.8.x then the problem is its lack of real system threads.
As you can read first hereand then here, there are systemic problems with timeouts in Ruby. An interesting discussion but for you in particular some comments suggest that the timeout is effectively ignored and defaults to 60 seconds - exactly what you are seeing.
Solutions ...
I have a similar issue with our own product when trying to send emails - if the email server is down the thread blocks. For me the solution was to spin the request off on a separate thread and therefore my main request-processing thread doesn't block.
There are non-blocking libraries out there for Ruby but perhaps you could take a look first at this System Timeout Gem.
An option open to anyone using Rails behind a proxy like nginx would be to set the upstream timeout to a lower number - that way you'll get notified if the server is taking too long. I'd only do this if I were really stuck for a solution.
Last but not least, it's possible that running Rails 2.3.2 on top of Ruby 1.9.1 will fix the issue.
Alternatively, you could try to catch these connection errors and retry once (after certain period of time) just to make sure the connection is really out.
retried = false
begin
#businesses = Business.find(:all, :params => { :shop_domain => #shop.domain })
retried = false
rescue ActiveResource::TimeoutError => ex
#raise ex
rescue ActiveResource::ConnectionError, ActiveResource::ServerError, ActiveResource::ClientError => ex
unless retried
sleep(((ex.respond_to?(:response) && ex.response['Retry-After']) || 5).to_i)
retried = true
retry
else
# raise ex
end
end
Inspired by this solution from Shopify for paginating a large number of records. https://ecommerce.shopify.com/c/shopify-apis-and-technology/t/paginate-api-results-113066