Sum and Average Array of Array - ruby-on-rails

Question

I am trying to sum array of array and get average at the same time. The original data is in the form of JSON. I have to parse my data to array of array in order to render the graph. The graph does not accept array of hash.
I first convert the output to JSON using the definition below.
ActiveSupport::JSON.decode(#output.first(10).to_json)
And the result of the above action is shown below.
output =
[{"name"=>"aaa", "job"=>"a", "pay"=> 2, ... },
{"name"=>"zzz", "job"=>"a", "pay"=> 4, ... },
{"name"=>"xxx", "job"=>"a", "pay"=> 6, ... },
{"name"=>"yyy", "job"=>"a", "pay"=> 8, ... },
{"name"=>"aaa", "job"=>"b", "pay"=> 2, ... },
{"name"=>"zzz", "job"=>"b", "pay"=> 4, ... },
{"name"=>"xxx", "job"=>"b", "pay"=> 6, ... },
{"name"=>"yyy", "job"=>"b", "pay"=> 10, ... },
]
Then I retrieved the job and pay by converting to array of array.
ActiveSupport::JSON.decode(output.to_json).each { |h|
a << [h['job'], h['pay']]
}
The result of the above operation is as below.
a = [["a", 2], ["a", 4], ["a", 6], ["a", 8],
["b", 2], ["b", 4], ["b", 6], ["b", 10]]
The code below will give me the sum of each element in the form of array of array.
a.inject({}) { |h,(job, data)| h[job] ||= 0; h[job] += data; h }.to_a
And the result is as below
[["a", 20], ["b", 22]]
However, I am trying to get the average of the array. The expected output is as below.
[["a", 5], ["b", 5.5]]
I can count how many elements in an array and divide the sum array by the count array. I was wondering if there is an easier and more efficient way to get the average.

Answer 1

output = [
{"name"=>"aaa", "job"=>"a", "pay"=> 2 },
{"name"=>"zzz", "job"=>"a", "pay"=> 4 },
{"name"=>"xxx", "job"=>"a", "pay"=> 6 },
{"name"=>"yyy", "job"=>"a", "pay"=> 8 },
{"name"=>"aaa", "job"=>"b", "pay"=> 2 },
{"name"=>"zzz", "job"=>"b", "pay"=> 4 },
{"name"=>"xxx", "job"=>"b", "pay"=> 6 },
{"name"=>"yyy", "job"=>"b", "pay"=> 10 },
]
output.group_by { |obj| obj['job'] }.map do |key, list|
[key, list.map { |obj| obj['pay'] }.reduce(:+) / list.size.to_f]
end
The group_by method will transform your list into a hash with the following structure:
{"a"=>[{"name"=>"aaa", "job"=>"a", "pay"=>2}, ...], "b"=>[{"name"=>"aaa", "job"=>"b", ...]}
After that, for each pair of that hash, we want to calculate the mean of its 'pay' values, and return a pair [key, mean]. We use a map for that, returning a pair with:
They key itself ("a" or "b").
The mean of the values. Note that the values list has the form of a list of hashes. To retrieve the values, we need to extract the last element of each pair; that's what list.map { |obj| obj['pay'] } is used for. Finally, calculate the mean by suming all elements with .reduce(:+) and dividing them by the list size as a float.
Not the most efficient solution, but it's practical.
Comparing the answer with #EricDuminil's, here's a benchmark with a list of size 8.000.000:
def Wikiti(output)
output.group_by { |obj| obj['job'] }.map do |key, list|
[key, list.map { |obj| obj['pay'] }.reduce(:+) / list.size.to_f]
end
end
def EricDuminil(output)
count_and_sum = output.each_with_object(Hash.new([0, 0])) do |hash, mem|
job = hash['job']
count, sum = mem[job]
mem[job] = count + 1, sum + hash['pay']
end
result = count_and_sum.map do |job, (count, sum)|
[job, sum / count.to_f]
end
end
require 'benchmark'
Benchmark.bm do |x|
x.report('Wikiti') { Wikiti(output) }
x.report('EricDuminil') { EricDuminil(output) }
end
user system total real
Wikiti 4.100000 0.020000 4.120000 ( 4.130373)
EricDuminil 4.250000 0.000000 4.250000 ( 4.272685)

Answer 2

This method should be reasonably efficient. It creates a temporary hash with job name as key and [count, sum] as value:
output = [{ 'name' => 'aaa', 'job' => 'a', 'pay' => 2 },
{ 'name' => 'zzz', 'job' => 'a', 'pay' => 4 },
{ 'name' => 'xxx', 'job' => 'a', 'pay' => 6 },
{ 'name' => 'yyy', 'job' => 'a', 'pay' => 8 },
{ 'name' => 'aaa', 'job' => 'b', 'pay' => 2 },
{ 'name' => 'zzz', 'job' => 'b', 'pay' => 4 },
{ 'name' => 'xxx', 'job' => 'b', 'pay' => 6 },
{ 'name' => 'yyy', 'job' => 'b', 'pay' => 10 }]
count_and_sum = output.each_with_object(Hash.new([0, 0])) do |hash, mem|
job = hash['job']
count, sum = mem[job]
mem[job] = count + 1, sum + hash['pay']
end
#=> {"a"=>[4, 20], "b"=>[4, 22]}
result = count_and_sum.map do |job, (count, sum)|
[job, sum / count.to_f]
end
#=> [["a", 5.0], ["b", 5.5]]
It requires 2 passes, but the created objects aren't big. In comparison, calling group_by on a huge array of hashes isn't very efficient.

Answer 3

How about this (Single pass iterative average calculation)
accumulator = Hash.new {|h,k| h[k] = Hash.new(0)}
a.each_with_object(accumulator) do |(k,v),obj|
obj[k][:count] += 1
obj[k][:sum] += v
obj[k][:average] = (obj[k][:sum] / obj[k][:count].to_f)
end
#=> {"a"=>{:count=>4, :sum=>20, :average=>5.0},
# "b"=>{:count=>4, :sum=>22, :average=>5.5}}
Obviously average is just recalculated on every iteration but since you asked for them at the same time this is probably as close as you are going to get.
Using your "output" instead looks like
output.each_with_object(accumulator) do |h,obj|
key = h['job']
obj[key][:count] += 1
obj[key][:sum] += h['pay']
obj[key][:average] = (obj[key][:sum] / obj[key][:count].to_f)
end
#=> {"a"=>{:count=>4, :sum=>20, :average=>5.0},
# "b"=>{:count=>4, :sum=>22, :average=>5.5}}

Answer 4

as Sara Tibbetts comment suggests, my first step would be to convert it like this
new_a = a.reduce({}){ |memo, item| memo[item[0]] ||= []; memo[item[0]] << item[1]; memo}
which puts it in this format
{a: [2, 4, 6, 8], b: [2, 4, 6, 20]}
you can then use slice to filter the keys you want
new_a.slice!(key1, key2, ...)
Then do another pass through to do get the final format
new_a.reduce([]) do |memo, (k,v)|
avg = v.inject{ |sum, el| sum + el }.to_f / v.size
memo << [k,avg]
memo
end

Answer 5

I elected to use Enumerable#each_with_object with the object being an array of two hashes, the first to compute totals, the second to count the number of numbers that are totalled. Each hash is defined Hash.new(0), zero being the default value. See Hash::new for a fuller explanation, In short, if a hash defined h = Hash.new(0) does not have a key k, h[k] returns 0. (h is not modified.) h[k] += 1 expands to h[k] = h[k] + 1. If h does not have a key k, h[k] on the right of the equality returns 0.1
output =
[{"name"=>"aaa", "job"=>"a", "pay"=> 2},
{"name"=>"zzz", "job"=>"a", "pay"=> 4},
{"name"=>"xxx", "job"=>"a", "pay"=> 6},
{"name"=>"yyy", "job"=>"a", "pay"=> 8},
{"name"=>"aaa", "job"=>"b", "pay"=> 2},
{"name"=>"zzz", "job"=>"b", "pay"=> 4},
{"name"=>"xxx", "job"=>"b", "pay"=> 6},
{"name"=>"yyy", "job"=>"b", "pay"=>10}
]
htot, hnbr = output.each_with_object([Hash.new(0), Hash.new(0)]) do |f,(g,h)|
s = f["job"]
g[s] += f["pay"]
h[s] += 1
end
htot.merge(hnbr) { |k,o,n| o.to_f/n }.to_a
#=> [["a", 5.0], ["b", 5.5]]
If .to_a at the end is dropped the the hash {"a"=>5.0, "b"=>5.5} is returned. The OP might find that more useful than the array.
I've used the form of Hash#merge that uses a block to determine the values of keys that are present in both hashes being merged.
Note that htot={"a"=>20, "b"=>22} and hnbr=>{"a"=>4, "b"=>4}.
1 If the reader is wondering why h[k] on the left of = doesn't return zero as well, it's a different method: Hash#[]= versus Hash#[]