To get the twitter rate limit in v.1.1, I am doing :
$rate_limit = $connection->get('application/rate_limit_status', array('resource'=>'search'));
and $rate_limit->resources->search gives me:
stdClass Object
(
[/search/tweets] => stdClass Object
(
[limit] => 180
[remaining] => 177
[reset] => 1389642298
)
)
How to extract the remaining value from such an object? Trying to the twitter rate limit. Thanks
I have done more research and realised I should use the curly bracket
$rate_limit->resources->search->{'/search/tweets'}->limit
Thanks
Related
I'm trying to get the value of id returned from the response.body function. The only way I got it to work is to loop on elements then assign $record_id to it.
JSON.parse(response.body).each do | k , v |
if k == 'id'
$record_id = v
end
I have similar functions like above used in my code, with the result that the script started to take a long time before it finishes.
Is there is a way I could get the value of first hash without looping?
record_id = JSON.parse(response.body)["id"]
Form the ruby documentation of JSON.parse:
my_hash = JSON.parse('{"hello": "goodbye"}')
puts my_hash["hello"] => "goodbye"
How would you improve this:
time = Time.now
#time = []
#time.push(
(time-1.week).strftime("%m-%d"),
(time-6.days).strftime("%m-%d"),
(time-5.days).strftime("%m-%d"),
(time-4.days).strftime("%m-%d"),
(time-3.days).strftime("%m-%d"),
(time-2.days).strftime("%m-%d"),
(time-1.day).strftime("%m-%d"),
(time).strftime("%m-%d")
)
I'm trying out some of the suggestions below:
time = Time.now
iterations = 1000
Benchmark.bm do |bm|
bm.report do
iterations.times do
#time = 7.downto(0).map { |v| (time - v.days).strftime("%m-%d") }
end
end
bm.report do
iterations.times do
#time = []
#time.push(
(time-1.week).strftime("%m-%d"),
(time-6.days).strftime("%m-%d"),
(time-5.days).strftime("%m-%d"),
(time-4.days).strftime("%m-%d"),
(time-3.days).strftime("%m-%d"),
(time-2.days).strftime("%m-%d"),
(time-1.day).strftime("%m-%d"),
(time).strftime("%m-%d")
)
end
end
end
user system total real
0.350000 0.960000 1.310000 ( 1.310054)
0.310000 0.840000 1.150000 ( 1.156484)
downto is demonstrably slower than my method.
The next test used the method:
#time = (0..7).map { |x| (time - x.days).strftime("%m-%d") }.reverse
1000 iterations
user system total real
0.340000 0.980000 1.320000 ( 1.321518)
0.300000 0.840000 1.140000 ( 1.149759)
5000 iterations
user system total real
1.720000 4.800000 6.520000 ( 6.545335)
1.530000 4.180000 5.710000 ( 5.712035)
I'm having a hard time wrapping my head around this without looking at both downto and map in the Ruby core, but in both cases my more elongated method of writing this responds faster than the more simplified methods (I like the answers below much more from a readability standpoint). Please shed some light on my tests if I'm doing it wrong. I expected map to blow my way out of the water.
UPDATED FOR STEFAN'S ANSWER
So I see Stefan's answer below and throw it in the tester:
user system total real
0.040000 0.000000 0.040000 ( 0.035976)
1.520000 4.180000 5.700000 ( 5.704401)
Holy crap! 5000 iterations and it absolutely destroys my method.
Because he accurately points out that I'm only interested in Dates, I decide to change my own method from Time.now to Date.today and test it:
user system total real
0.090000 0.000000 0.090000 ( 0.085940)
0.390000 0.000000 0.390000 ( 0.398143)
It's somewhat weird that in the first test Stefan's method clocks in at 0.0359 and in the second clocks at 0.0859, more than double, but it's still hundredths of a second over 5000 iterations - so I think I'm deep into splitting hairs territory here.
Nevertheless - Stefan's way obliterates my own method - so I'm giving him the check mark.
You can do as below using #downto -
time = Time.now
#time = 7.downto(0).map {|v| (time - v.days).strftime("%m-%d") }
Since you're only interested in dates, you could use a Range of Date instances (dates increment in 1-day steps):
today = Date.today
(today-7..today).map { |date| date.strftime("%m-%d") }
#=> ["04-24", "04-25", "04-26", "04-27", "04-28", "04-29", "04-30", "05-01"]
You can use not pushing itself, but collection, i.e. combination of Range from zero to seven, and Array's #map method:
time = Time.now
#time = (0..7).map {|v| (time - v.days).strftime("%m-%d") }.reverse
# => ["04-24", "04-25", "04-26", "04-27", "04-28", "04-29", "04-30", "05-01"]
I would write it using the Array#map method
time = Time.now
#time = (0..7).map { |x| (time - x.days).strftime("%m-%d") }.reverse
I am currently thinking about improving the performance of Rails.cache.write when using dalli to write items to the memcachier cloud.
The stack, as it relates to caching, is currently:
heroku, memcachier heroku addon, dalli 2.6.4, rails 3.0.19
I am using newrelic for performance monitoring.
I am currently fetching "active students" for a given logged in user, represented by a BusinessUser instance, when its active_students method is called from a controller handling a request that requires a list of "active students":
class BusinessUser < ActiveRecord::Base
...
def active_students
Rails.cache.fetch("/studio/#{self.id}/students") do
customer_users.active_by_name
end
end
...
end
After looking at newrelic, I've basically narrowed down one big performance hit for the app in setting key values on memcachier. It takes an average of 225ms every time. Further, it looks like setting memcache key values blocks the main thread and eventually disrupts the request queue. Obviously this is undesirable, especially when the whole point of the caching strategy is to reduce performance bottlenecks.
In addition, I've benchmarked the cache storage with plain dalli, and Rails.cache.write for 1000 cache sets of the same value:
heroku run console -a {app-name-redacted}
irb(main):001:0> require 'dalli'
=> false
irb(main):002:0> cache = Dalli::Client.new(ENV["MEMCACHIER_SERVERS"].split(","),
irb(main):003:1* {:username => ENV["MEMCACHIER_USERNAME"],
irb(main):004:2* :password => ENV["MEMCACHIER_PASSWORD"],
irb(main):005:2* :failover => true,
irb(main):006:2* :socket_timeout => 1.5,
irb(main):007:2* :socket_failure_delay => 0.2
irb(main):008:2> })
=> #<Dalli::Client:0x00000006686ce8 #servers=["server-redacted:11211"], #options={:username=>"username-redacted", :password=>"password-redacted", :failover=>true, :socket_timeout=>1.5, :socket_failure_delay=>0.2}, #ring=nil>
irb(main):009:0> require 'benchmark'
=> false
irb(main):010:0> n = 1000
=> 1000
irb(main):011:0> Benchmark.bm do |x|
irb(main):012:1* x.report { n.times do ; cache.set("foo", "bar") ; end }
irb(main):013:1> x.report { n.times do ; Rails.cache.write("foo", "bar") ; end }
irb(main):014:1> end
user system total real
Dalli::Server#connect server-redacted:11211
Dalli/SASL authenticating as username-redacted
Dalli/SASL: username-redacted
0.090000 0.050000 0.140000 ( 2.066113)
Dalli::Server#connect server-redacted:11211
Dalli/SASL authenticating as username-redacted
Dalli/SASL: username-redacted
0.100000 0.070000 0.170000 ( 2.108364)
With plain dalli cache.set, we are using 2.066113s to write 1000 entries into the cache, for an average cache.set time of 2.06ms.
With Rails.cache.write, we are using 2.108364s to write 1000 entries into the cache, for an average Rails.cache.write time of 2.11ms.
⇒ It seems like the problem is not with memcachier, but simply with the amount of data that we are attempting to store.
According to the docs for the #fetch method, it looks like it would not be the way I want to go, if I want to throw cache sets into a separate thread or a worker, because I can't split out the write from the read - and self-evidently, I don't want to be reading asynchronously.
Is it possible to reduce the bottleneck by throwing Rails.cache.write into a worker, when setting key values? Or, more generally, is there a better pattern to do this, so that I am not blocking the main thread every time I want to perform a Rails.cache.write?
There are two factors that would contribute to overall latency under normal circumstances: client side marshalling/compression and network bandwidth.
Dalli mashalls and optionally compresses the data, which could be quite expensive. Here are some benchmarks of Marshalling and compressing a list of random characters (a kind of artificial list of user ids or something like that). In both cases the resulting value is around 200KB. Both benchmarks were run on a Heroku dyno - performance will obviously depend on the CPU and load of the machine:
irb> val = (1..50000).to_a.map! {rand(255).chr}; nil
# a list of 50000 single character strings
irb> Marshal.dump(val).size
275832
# OK, so roughly 200K. How long does it take to perform this operation
# before even starting to talk to MemCachier?
irb> Benchmark.measure { Marshal.dump(val) }
=> 0.040000 0.000000 0.040000 ( 0.044568)
# so about 45ms, and this scales roughly linearly with the length of the list.
irb> val = (1..100000).to_a; nil # a list of 100000 integers
irb> Zlib::Deflate.deflate(Marshal.dump(val)).size
177535
# OK, so roughly 200K. How long does it take to perform this operation
irb> Benchmark.measure { Zlib::Deflate.deflate(Marshal.dump(val)) }
=> 0.140000 0.000000 0.140000 ( 0.145672)
So we're basically seeing anywhere from a 40ms to 150ms performance hit just for Marshaling and/or zipping data. Marshalling a String will be much cheaper, while marshalling something like a complex object will be more expensive. Zipping depends on the size of the data, but also on the redundancy of the data. For example, zipping a 1MB string of all "a" characters takes merely about 10ms.
Network bandwidth will play some of a role here, but not a very significant one. MemCachier has a 1MB limit on values, which would take approximately 20ms to transfer to/from MemCachier:
irb(main):036:0> Benchmark.measure { 1000.times { c.set("h", val, 0, :raw => true) } }
=> 0.250000 11.620000 11.870000 ( 21.284664)
This amounts to about 400Mbps (1MB * 8MB/Mb * (1000ms/s / 20ms)), which makes sense. However, for even a relatively large, but still smaller value of 200KB, we'd expect a 5x speedup:
irb(main):039:0> val = "a" * (1024 * 200); val.size
=> 204800
irb(main):040:0> Benchmark.measure { 1000.times { c.set("h", val, 0, :raw => true) } }
=> 0.160000 2.890000 3.050000 ( 5.954258)
So, there are several things you might be able to do to get some speedup:
Use a faster marshalling mechanism. For example, using Array#pack("L*") to encode a list of 50,000 32-bit unsigned integers (like in the very first benchmark) into a string of length 200,000 (4 bytes for each integer), takes only 2ms rather than 40ms. Using compression with the same marshalling scheme, to get a similar sized value is also very fast (about 2ms as well), but the compression doesn't do anything useful on random data anymore (Ruby's Marshal produces a fairly redundant String even on a list of random integers).
Use smaller values. This would probably require deep application changes, but if you don't really need the whole list, you should be setting it. For example, the memcache protocol has append and prepend operations. If you are only ever adding new things to a long list, you could use those operations instead.
Finally, as suggested, removing the set/gets from the critical path would prevent any delays from affecting HTTP request latency. You still have to get the data to the worker, so it's important that if you're using something like a work queue, the message you send to the worker should only contain instructions on which data to construct rather than the data itself (or you're in the same hole again, just with a different system). A very lightweight (in terms of coding effort) would be to simply fork a process:
mylist = Student.where(...).all.map!(&:id)
...I need to update memcache with the new list of students...
fork do
# Have to create a new Dalli client
client = Dalli::Client.new
client.set("mylistkey", mylist)
# this will block for the same time as before, but is running in a separate process
end
I haven't benchmarked a full example, but since you're not execing, and Linux fork is copy-on-write, the overhead of the fork call itself should be minimal. On my machine, it's about 500us (that's micro-seconds not milliseconds).
Using Rails.cache.write to prefetch and store data in cache with workers (e.g. Sidekiq) is what I've seen at high volumes. Of course there is a trade off between speed and the money you want to spend. Think about:
the most used paths in your app (is active_students accessed often?);
what to store (just IDs or the entire objects or further down the chain);
if you can optimize that query (n+1?).
Also, if you really need speed, consider using a dedicated memcache service, instead of a Heroku add-on.
I have a load of data I want to store in /apps/frontend/modules/builder/config/module.yml.
I have it looking something like:
all:
series_options:
compact:
name: Compact
description: Something small.
enabled: 1
large:
name: Large
description: Bit bigger.
enabled: 0
In the actions.class if I write this:
sfConfig::get('mod_builder_series_options_compact');
I get this
Array
(
[name] => Compact
[description] => Something small.
[enabled] => 1
)
Perfect. But I want to write this:
sfConfig::get('mod_builder_series_options');
Which gives NULL.
Is there any way I can get this to return the full associative array to its full depth so I can iterate through the different options?
You can add a level with a dot before its name to force array on certain level:
all:
.options:
series_options:
compact:
name: Compact
description: Something small.
enabled: 1
large:
name: Large
description: Bit bigger.
enabled: 0
Now you should be able to access your settings with:
sfConfig::get('mod_builder_series_options');
Remember that module configuration is only accessible in the module it is defined.
Typical, as soon as I resort to posting the answer hits me in the face...
$series = sfYaml::load('../apps/frontend/modules/builder/config/module.yml');
print_r($series);die;
Returns:
Array
(
[all] => Array
(
[series_options] => Array
(
[compact] => Array
(
[name] => Compact
[description] => Something small.
[enabled] => 1
)
[large] => Array
(
[name] => Large
[description] => Bit bigger.
[enabled] => 0
)
)
)
)
Guessing that sfConfig wasnt really meant for this purpose, where sfYaml certainly looks like it was!
Hope this helps someone else!
I would like to "cap" a number in Ruby (on Rails).
For instance, I have, as a result of a function, a float but I need an int.
I have very specific instructions, here are some examples:
If I get 1.5 I want 2 but if I get 2.0 I want 2 (and not 3)
Doing number.round(0) + 1 won't work.
I could write a function to do this but I am sure one already exists.
If, nevertheless, it does not exist, where should I create my cap function?
Try ceil:
1.5.ceil => 2
2.0.ceil => 2
How about number.ceil?
This returns the smallest Integer greater than or equal to number.
Be careful if you are using this with negative numbers, make sure it does what you expect:
1.5.ceil #=> 2
2.0.ceil #=> 2
(-1.5).ceil #=> -1
(-2.0).ceil #=> -2
.ceil is good, but remember, even smallest value in float will cause this:
a = 17.00000000000002
17.0
a.ceil
18
Use Numeric#ceil:
irb(main):001:0> 1.5.ceil
=> 2
irb(main):002:0> 2.0.ceil
=> 2
irb(main):003:0> 1.ceil
=> 1
float.ceil is what you want for positive numbers. Be sure to consider the behavior for negative numbers. That is, do you want -1.5 to "cap" to -1 or -2?