I have a performance issue in my application. I would like to gather some ideas on what I can do to improve it. The application is very easy: I need to add values inside a nested table to get the total an user wants to pay out of all the pending payments. The user chooses a number of payments and I calculate how much it is they will pay.
This is what I have:
jsonstr = "{ "name": "John",
"surname": "Doe",
"pending_payments": [
{
"month": "january",
"amount": 50,
},
{
"month": "february",
"amount": 40,
},
{
"month": "march",
"amount": 45,
},
]
}"
local lunajson = require 'lunajson'
local t = lunajson.decode(jsonstr)
local limit -- I get this from the user
local total = 0;
for i=1, limit, 1 do
total = total + t.pending_payments[i].amount;
end;
It works. At the end I get what I need. However, I notice that it takes ages to do the calculation. Each JSON has only twelve pending payments (one per month). It is taking between two to three seconds to come up with a result!. I tried in different machines and LUA 5.1, 5.2., 5.3. and the result is the same.
Can anyone please suggest how I can implement this better?
Thank you!
For this simple string, try the test code below, which extracts the amounts directly from the string, without a json parser:
jsonstr = [[{ "name": "John",
"surname": "Doe",
"pending_payments": [
{
"month": "january",
"amount": 50,
},
{
"month": "february",
"amount": 40,
},
{
"month": "march",
"amount": 45,
},
]
}]]
for limit=0,4 do
local total=0
local n=0
for a in jsonstr:gmatch('"amount":%s*(%d+),') do
n=n+1
if n>limit then break end
total=total+tonumber(a)
end
print(limit,total)
end
I found the delay had nothing to do with the calculation in LUA. It was related with a configurable delay in the retrieval of the limit variable.
I have nothing to share here related to the question asked since the problem was actually in an external element.
Thank #lfh for your replies.
Related
I'm trying to combine requests from two services in one endpoint.
The first returns the list of users, like this:
{
"data": [
{ "id": 1, "name": "first", "photo": "employee_photos/key1.png" },
{ "id": 2, "name": "second", "photo": null },
{ "id": 3, "name": "third", "photo": "employee_photos/key3.png" }
]
}
The second is supposed to receive the POST request with the JSON listing photo keys to get the required version URLs.
Request:
{
"keys": [ "employee_photos/key1.png", "employee_photos/key3.png" ],
"versions": [ "small", "large" ]
}
I created a small Lua script to process the response of the first request and collect the list of keys. It looks like this:
function post_employees(request, employeesResponse)
local resp = employeesResponse.load()
local data = resp:data():get("data")
local photo_keys = {}
for i = 0, data:len() - 1, 1 do
local rec = data:get(i)
local id = rec:get("id")
local photo = rec:get("photo")
if photo then
table.insert(photo_keys, photo)
end
end
resp:data():set("photos", table.concat(photo_keys, ","))
end
But then... I can't find a way to use this list of keys in the second request. Is it even possible?
This is my column:
[
{ id: 1, value: 1, complete: true },
{ id: 2, value: 1, complete: false },
{ id: 3, value: 1, complete: true }
]
First, is there a "correct" way to work with a jsonb scheme? should I redesign to work with a single json instead of the array of hashes?
I have about 200 entries on the database, the column status has 200 of those itens.
How would I perform a query to get the count of true/false?
How can I query for ALL complete itens? I can query for the database rows in which the json has an item complete, but I can't query for all the itens, in all rows of the database that are complete.
Appreciate the help, thank you
Aha! I found it here:
https://levelup.gitconnected.com/how-to-query-a-json-array-of-objects-as-a-recordset-in-postgresql-a81acec9fbc5
Say your dataset is like this:
[{
"productid": "3",
"name": "Virtual Keyboard",
"price": "150.00"
}, {
"productid": "1",
"name": "Dell 123 Laptop Computer",
"price": "1300.00"
},
{
"productid": "8",
"name": "LG Ultrawide Monitor",
"price": "190.00"
}]
The proper way to count it, is like this:
select items.name, count(*) as num from
purchases,jsonb_to_recordset(purchases.items_purchased) as items(name text)
group by items.name
order by num Desc
Works like a charm and is extremely fast.
To do it in Rails, you need to use Model.find_by_sql(....) and indicate your select therem. I'm sure there are probably better ways to do it.
I have a third party JSON feed which is huge - lots of data. Eg
{
"data": [{
"name": "ABC",
"price": "2.50"
},
...
]
}
I am required to strip the quotation marks from the price as the consumer of the JSON feed requires it in this way.
To do this I am performing a regex to find the prices and then iterating over the prices and doing a string replace using gsub. This is how I am doing it:
price_strings = json.scan(/(?:"price":")(.*?)(?:")/).uniq
price_strings.each do |price|
json.gsub!("\"#{price.reduce}\"", price.reduce)
end
json
The main bottle neck appears to be on the each block. Is there a better way of doing this?
If this JSON string is going to be serialised into a Hash at some point in your application or in another 3rd-party dependency of your code (i.e. to be consumed by your colleagues or modules), I suggest negotiating with them to convert the price value from String to Numeric on demand when the json is already a Hash, as this is more efficient, and allows them to...
...handle edge-case where say if "price": "" of which my code below will not work, as it would remove the "", and will be a JSON syntax error.
However, if you do not have control over this, or are doing once-off mutation for the whole json data, then can you try below?
json =
<<-eos
{
"data": [{
"name": "ABC",
"price": "2.50",
"somethingsomething": {
"data": [{
"name": "DEF",
"price": "3.25", "someprop1": "hello",
"someprop2": "world"
}]
},
"somethinggggg": {
"price": "123.45" },
"something2222": {
"price": 9.876, "heeeello": "world"
}
}]
}
eos
new_json = json.gsub /("price":.*?)"(.*?)"(.*?,|})/, '\1\2\3'
puts new_json
# =>
# {
# "data": [{
# "name": "ABC",
# "price": 2.50,
# "somethingsomething": {
# "data": [{
# "name": "DEF",
# "price": 3.25, "someprop1": "hello",
# "someprop2": "world"
# }]
# },
# "somethinggggg": {
# "price": 123.45 },
# "something2222": {
# "price": 9.876, "heeeello": "world"
# }
# }]
# }
DISCLAIMER: I am not a Regexp expert.
This is truly a fools errand.
JSON.parse('{ "price": 2.50 }')
> {price: 2.5}
As you can see from this javascript example the parser on the consuming side will truncate the float to whatever it wants.
Use a string if you want to provide a formatted number or leave formatting up to the client.
In fact using floats to represent money is widely known as a really bad idea since floats and doubles cannot accurately represent the base 10 multiples that we use for money. JSON only has a single number type that represents both floats and integers.
If the client is going to do any kind of calculations with the value you should use an integer in the lowest monetary denomation (cents for euros and dollars) or a string that's interpreted as a BigDecimal equivilent type by the consumer.
I am building a Rails 5 app and in this app I got four models.
User, Report, Trip and Expense.
User has_many Reports, trips and expenses
Reports has_many trips and expenses.
I want to get a JSON response with all the trips and expenses that a user has done this month, grouped by Report.
I can do this but I need only the Id and Title of the report (when grouping) and now I get only the object name.
I use this method (located in the User model):
def grouped_reports
trips = self.trips
expenses = self.expenses
items = trips + expenses
items.group_by(&:report)
end
The output in JSON is this:
{
"#<Report:0x007fa225163ba8>": [{
"id": 12,
"account_id": 20,
"user_id": 92,
"vehicle_id": null,
"ttype": null,
"description": "saf",
"start_latitude": 57.4874919,
"start_longitude": 12.0761927999999,
"end_latitude": 59.3293235,
"end_longitude": 18.0685808000001,
"start_address": "Chicago",
"end_address": "New york",
"distance": 490,
"time": null,
"status": "pending",
"registered_at": "2018-08-24T02:00:00.000+02:00",
"approvals_count": 0,
"rejections_count": 0,
"created_at": "2018-08-24T22:39:22.637+02:00",
"updated_at": "2018-08-24T22:39:22.637+02:00",
"report_id": 79,
"return_trip": null,
"triptype_id": 10
},
{
"id": 13,
"account_id": 20,
"user_id": 92,
"cost": 100,
"etype": null,
"description": "sdsd",
"start_address": null,
"end_address": null,
"distance": null,
"reimbursable": false,
"status": "pending",
"registered_at": "2018-08-08T00:00:00.000+02:00",
"created_at": "2018-08-24T22:39:40.343+02:00",
"updated_at": "2018-08-24T22:39:40.343+02:00",
"approvals_count": 0,
"rejections_count": 0,
"report_id": 79,
"expensetype_id": 15
}
]
}
There is two things I need to improve.
Display id and title of report not .
Get only this months reports.
Update, this works but is it the right way in terms of performance?
And not only performance, is it actually using the title of the report as a grouping factor? Which is not good since may reports can share the same title. I want to group by report_id but display report title.
def grouped_reports
trips = self.trips
expenses = self.expenses
items = trips + expenses
items.group_by{ |t| [t.report.id, t.report.title] }
end
Code would differ depending on what JSON format you want to output.
I usually use json generator such jbuilder, but this time I suggest a array and hash structure.
class User < ApplicationRecord
has_many :reports
has_many :expenses
has_many :trips
def grouped_reports
start_date = Time.current.beginning_of_month
monthly_trips = trips.where('created_at >= ?', start_date)
monthly_expenses = expenses.where('created_at >= ?', start_date)
report_ids = (monthly_trips.map(&:report_id) + monthly_expenses.map(&:report_id)).uniq
reports = Report.where(id: report_ids)
reports.map do |report|
{
id: report.id,
title: report.title,
trips: trips.select {|t| t.report_id == report.id},
expenses: expenses.select {|e| e.report_id == report.id}
}
end
end
end
You are grouping with trips and expenses joined to one array, but it is not preferable to put different types in same arrays for JSON. It would be safe to have a hash format and separate key for trip and expense.
To extract records for this month, use where to filter.
It is possible to fetch trips by using includes for reports and expenses, but from a performance point of view it is better to get the related trips at once.
If you want to further improve performance, narrow down only columns used when outputting JSON by using select method. This would be a huge improvement if a lot of records are outputted.
I'm using a Ruby script to interface with an application API and the results being returned are in a JSON format. For example:
{
"incidents": [
{
"number": 1,
"status": "open",
"key": "abc123"
}
{
"number": 2,
"status": "open",
"key": "xyz098"
}
{
"number": 3,
"status": "closed",
"key": "lmn456"
}
]
}
I'm looking to search each block for a particular "key" value (yzx098 in this example) and return the associated "number" value.
Now, I'm very new to Ruby and I'm not sure if there's already a function to help accomplish this. However, a couple days of scouring the Googles and Ruby resource books hasn't yielded anything that works.
Any suggestions?
First of all, the JSON should be as below: (note the commas)
{
"incidents": [
{
"number": 1,
"status": "open",
"key": "abc123"
},
{
"number": 2,
"status": "open",
"key": "xyz098"
},
{
"number": 3,
"status": "closed",
"key": "lmn456"
}
]
}
Strore the above json in a variable
s = '{"incidents": [{"number": 1,"status": "open","key": "abc123"},{"number": 2,"status": "open","key": "xyz098"},{"number": 3,"status": "closed","key": "lmn456"}]}'
Parse the JSON
h = JSON.parse(s)
Find the required number using map
h["incidents"].map {|h1| h1['number'] if h1['key']=='xyz098'}.compact.first
Or you could also use find as below
h["incidents"].find {|h1| h1['key']=='xyz098'}['number']
Or you could also use select as below
h["incidents"].select {|h1| h1['key']=='xyz098'}.first['number']
Do as below
# to get numbers from `'key'`.
json_hash["incidents"].map { |h| h['key'][/\d+/].to_i }
json_hash["incidents"] - will give you the value of the key "incidents", which is nothing but an array of hash.
map to iterate thorough each hash and collect the value of 'key'. Then applying Hash#[] to each inner hash of the array, to get the value of "key". Then calling str[regexp], to get only the number strings like '098' from "xyz098", finally applying to_i to get the actual integer from it.
If the given hash actually a json string, then first parse it using JSON::parse to convert it to a hash.Then do iterate as I said above.
require 'json'
json_hash = JSON.parse(json_string)
# to get values from the key `"number"`.
json_hash["incidents"].map { |h| h['number'] } # => [1, 2, 3]
# to search and get all the numbers for a particular key match and take the first
json_hash["incidents"].select { |h| h['key'] == 'abc123' }.first['number'] # => 1
# or to search and get only the first number for a particular key match
json_hash["incidents"].find { |h| h['key'] == 'abc123' }['number'] # => 1