Related
I have a json object. It has multiple fields "passthrough_fields" which is unnecessary for me and I want to remove them. Is there a way to get all those attributes filtered out?
JSON:
{
"type": "playable_item",
"id": "p06s0lq7",
"urn": "urn:bbc:radio:episode:p06s0mk3",
"network": {
"id": "bbc_radio_five_live",
"key": "5live",
"short_title": "Radio 5 live",
"logo_url": "https://sounds.files.bbci.co.uk/v2/networks/bbc_radio_five_live/{type}_{size}.{format}",
"passthrough_fields": {}
},
"titles": {
"primary": "Replay",
"secondary": "Bill Shankly",
"tertiary": null,
"passthrough_fields": {}
},
"synopses": {
"short": "Bill Shankly with Sue MacGregor in 1979 - five years after he resigned as Liverpool boss.",
"medium": null,
"long": "Bill Shankly in conversation with Sue MacGregor in 1979, five years after he resigned as Liverpool manager.",
"passthrough_fields": {}
},
"image_url": "https://ichef.bbci.co.uk/images/ic/{recipe}/p06qbz1x.jpg",
"duration": {
"value": 1774,
"label": "29 mins",
"passthrough_fields": {}
},
"progress": null,
"container": {
"type": "series",
"id": "p06qbzmj",
"urn": "urn:bbc:radio:series:p06qbzmj",
"title": "Replay",
"synopses": {
"short": "Colin Murray unearths classic sports commentaries and interviews from the BBC archives.",
"medium": "Colin Murray looks back at 90 years of sport on the BBC by unearthing classic commentaries and interviews from the BBC archives.",
"long": null,
"passthrough_fields": {}
},
"activities": [],
"passthrough_fields": {}
},
"availability": {
"from": "2018-11-16T16:18:54Z",
"to": null,
"label": "Available for over a year",
"passthrough_fields": {}
},
"guidance": {
"competition_warning": false,
"warnings": null,
"passthrough_fields": {}
},
"activities": [],
"uris": [
{
"type": "latest",
"label": "Latest",
"uri": "/v2/programmes/playable?container=p06qbzmj&sort=sequential&type=episode",
"passthrough_fields": {}
}
],
"passthrough_fields": {}
}
Is there a way I can remove all those fields and store the updated json in a new variable?
You can do this recursively to tackle nested occurances of passthrough_fields, whether they're found in an array or a sub hash. Inline comments to explain things a little as it goes:
hash = JSON.parse(input) # convert the JSON to a hash
def remove_recursively(hash, *to_remove)
hash.each do |key, val|
hash.except!(*to_remove) # the heavy lifting: remove all keys that match `to_remove`
remove_recursively(val, *to_remove) if val.is_a? Hash # if a nested hash, run this method on it
if val.is_a? Array # if a nested array, loop through this checking for hashes to run this method on
val.each { |el| remove_recursively(el, *to_remove) if el.is_a? Hash }
end
end
end
remove_recursively(hash, 'passthrough_fields')
To demonstrate, with a simplified example:
hash = {
"test" => { "passthrough_fields" => [1, 2, 3], "wow" => '123' },
"passthrough_fields" => [4, 5, 6],
"array_values" => [{ "to_stay" => "I am", "passthrough_fields" => [7, 8, 9]}]
}
remove_recursively(hash, 'passthrough_fields')
#=> {"test"=>{"wow"=>"123"}, "array_values"=>[{"to_stay"=>"I am"}]}
remove_recursively(hash, 'passthrough_fields', 'wow', 'to_stay')
#=> {"test"=>{}, "array_values"=>[{}]}
This will tackle any arrays, and will dig for nested hashes however deep it needs to go.
It takes any number of fields to remove, in this case a single 'passthrough_fields'.
Hope this helps, let me know how you get on.
I think that the easiest solution would be to:
convert JSON into hash (JSON.parse(input))
use this answer to extend the functionality of Hash (save it in config/initializers/except_nested.rb)
on the hash from 1st step, call:
without_passthrough = your_hash.except_nested('passthrough_fields')
covert hash to JSON (without_passthrough.to_json)
Please keep in mind that it will work for passthrough_fields that is nested directly in hashes. In your JSON, you have the following part:
"uris" => [
{
"type"=>"latest",
"label"=>"Latest",
"uri"=>"/v2/programmes/playable?container=p06qbzmj&sort=sequential&type=episode",
"passthrough_fields"=>{}
}
]
In this case, the passthrough_fields will not be removed. You have to find a more sophisticated solution :)
You can do something like this:
def nested_except(hash, except_key)
sanitized_hash = {}
hash.each do |key, value|
next if key == except_key
sanitized_hash[key] = value.is_a?(Hash) ? nested_except(value, except_key) : value
end
sanitized_hash
end
json = JSON.parse(json_string)
sanitized = nested_except(json, 'passthrough_fields')
See example:
json = { :a => 1, :b => 2, :c => { :a => 1, :b => { :a => 1 } } }
nested_except(json, :a)
# => {:b=>2, :c=>{:b=>{}}}
This helper can easily be converted to support multiple keys to except, simply by except_keys = Array.wrap(except_key) and next if except_keys.include?(key)
I am trying to retrieve data from 2 buckets, no error but nothing shows up (I do have documents I need in these buckets).
1st bucket: a_bucket
here is the document I am interested in (I do have 3 different docs)
author_ID document:
{
"author_ID": 1,
"profil_creation_date": "2017/01/01/01:23:05/+5",
"prefix": "Mr.",
"first_name": "Dylan",
"middle_name_s": "Alfred",
"last_name": "Kerr",
"date_of_birth": "1974/01/02",
"sex": "M",
"marital_status": "Single",
"mobile_phone": "(860) 231-3336",
"address": [
{
"address_1": {
"address_ID": 1,
"home_address": "338 Counts Lane",
"city": "West Hartford",
"province/state": "CT",
"postal_code": "06105"
}
},
{
"address_2": {
"address_ID": 2,
"work_address": "977 Copperhead Rd",
"city": "Newington",
"province/state": "CT",
"postal_code": "06111"
}
}
]
}
2nd bucket: b_bucket
here are the 2 docs I am interested in:
p_output_ID document:
{
"p_output_ID": 1,
"author_ID": 2,
"overall_score": 4.41,
"status": {
"r_status_first": "TRUE",
"r_status_second": "FALSE",
"r_status_third": "YES",
"y_status_second": "TRUE",
"y_status_third": "FALSE",
"g_status_third": "TRUE"
}
}
timing_ID document:
{
"timing_ID": 1,
"p_output_ID": 1,
"author_ID": 1,
"date_and_time": "2017-06-06/23:45:25.25/+5",
"time_in_seconds": 12525,
"incremental_time_in_seconds": "time_in_seconds",
"current_state_and_duration": {
"state": "RED",
"duration_in_seconds": 33333
}
}
my goal is to grab these informations in one query ():
prefix, first_name, middle_name_s, last_name (from author_ID document in a_bucket)
overall_score (from p_output_ID document in b_bucket)
date_and_time, state (from timing_ID document in b_bucket)
Here is my query:
select p2.current_state_and_duration.state, p1.overall_score, p2.date_and_time
from proc_data_bucket p1 USE KEYS "p_output_ID"
JOIN proc_data_bucket p2 ON KEYS "author_ID";
The syntax is OK, but I am getting no data
Please help me with that...
CREATE INDEX ix1 ON b_bucket(timing_ID);
SELECT p1.prefix, p1.first_name, p1.middle_name_s, p1.last_name,
p2.date_and_time,p2.state,
p3.overall_score
FROM b_bucket p2
JOIN a_bucket p1 ON KEYS ("author_" || TO_STRING(p2.author_ID))
JOIN b_bucket p3 ON KEYS ("p_output_" || TO_STRING(p2.p_output_ID))
WHERE p2.timing_ID BETWEEN 10 AND 50;
I am building a helpdesk application. I have a model called TicketDetail, with a table which uses datatables to get its data via JSON. This is in order to periodically recalculate the time a ticket has been open. The time taken is formatted by a simple helper so it's in the format "dd:hh:mm", but it should be sorted by the time (stored as a decimal) multiplied by a weighting. Here's the datatables definition
var table = $('#ticket_details').DataTable({
order: [[ 8, "desc" ], [ 9, "desc" ], [ 2, "asc" ]],
stateSave: true,
deferRender: true,
ajax: $('#ticket_details').data('source'),
"columns": [
{ "data": "reference_number" },
{ "data": "location" },
{ "data": "title" },
{ "data": "parent", className: "hidden-md hidden-sm hidden-xs" },
{ "data": { _:"time_display.time", sort: "time_display.decimal_time"}},
{ "data": "created_by", className: "hidden-md hidden-sm hidden-xs" }
]
} );
setInterval( function () {
table.ajax.reload( null, false ); }, 60000 );
Here's a simplified sample record, where the ticket has been open 3 days and 6 hours, with a weighting of x2 (i.e. 3.25 * 2 = 6.5:
{
data: [
{
id: 140,
parent: null,
title: "[",
location: "Bond St",
ticket_sla: "16 Hours",
reference_number: "1606210001",
ticket_sla_weighting: 2,
time_display: {
time: "<span class = "label label-danger">03:06:00</span>",
decimal_time: 6.5
}
]
}
The problem is that the datatable sorts correctly if I display the decimal_time, but as soon as I put the formatted time in the class, it sorts simply by the number of days, immediately to the left of the colon. (So 03:06:00 and 03:18:00 would not get sorted properly).
For Date/Time sorting in DataTable You need to use it's Sorting plug-ins
For Example,
You need to include this js files :
//cdnjs.cloudflare.com/ajax/libs/moment.js/2.8.4/moment.min.js
//cdn.datatables.net/plug-ins/1.10.12/sorting/datetime-moment.js
and then, In your jQuery use this as
$.fn.dataTable.moment( 'HH:mm MMM D, YY' ); // Pass your date time format as param
For Deeper reference please check :
Sorting Plugins
Ultimate date / time sorting plugin
Would anyone be willing to give me advice on how I can improve the performance of the following controller method?
def index
#contacts = Hash[current_user.company.contacts.map {|contact| [contact.id, ContactSerializer.new(contact).as_json[:contact]] }]
respond_to do |format|
format.json { render json: { contacts: #contacts } }
end
end
This returns the following data structure:
{
contacts: {
79: {
id: 79,
first_name: "Foo",
last_name: "Bar",
email: "t#t.co",
engagement: "0%",
company_id: 94,
created_at: " 9:41AM Jan 30, 2016",
updated_at: "10:57AM Feb 23, 2016",
published_response_count: 0,
groups: {
test: true,
test23: false,
Test222: false,
Last: false
},
invites: [
{
id: 112,
email: "t#t.co",
status: "Requested",
created_at: "Jan 30, 2016, 8:48 PM",
date_submitted: null,
response: null
}
],
responses: [ ],
promotions: [
{
id: 26,
company_id: 94,
key: "e5cb3bc80b58c29df8a61231d0",
updated_at: "Feb 11, 2016, 2:45 PM",
read: null,
social_media_posts: [ ]
}
]
},
81: {
id: 81,
first_name: "Foo2",
last_name: "Bar2",
email: "foobar2#foobar.com",
engagement: "0%",
company_id: 94,
created_at: "12:55PM Feb 04, 2016",
updated_at: " 4:25PM Feb 19, 2016",
published_response_count: 0,
groups: {
test: true,
test23: true,
Test222: false,
Last: false
},
invites: [
{
id: 116,
email: "foobar2#foobar.com",
status: "Requested",
created_at: "Feb 22, 2016, 9:10 PM",
date_submitted: null,
response: null
}
],
responses: [ ],
promotions: [
{
id: 26,
company_id: 94,
key: "e5cb3bc80b58c29df8a61231d0",
updated_at: "Feb 11, 2016, 2:45 PM",
read: null,
social_media_posts: [ ]
}
]
}
}
}
I need the index method to return a hash where the the keys are the contact IDs, as opposed to an Array, which is what would normally be returned. Additionally, I pass each contact through the serializer so that I get all associated data that my client needs.
This method works fine when there are only a few contacts, however when I have 100 or 1000, it really slows down. I benchmarked it with 100 contacts and it took 4 seconds to finish, which is abysmal. I'm wondering how I can improve my code to get the exact same output in a more performant manner. The key here is that the output needs to remain unchanged. I have no interest in modifying the client-side code (it depends on this data structure for numerous applications), so all changes need to occur on the server-side.
Here is my ContactSerializer for reference:
class ContactSerializer < ActiveModel::Serializer
attributes :id, :first_name, :last_name, :email, :engagement, :company_id, :created_at, :updated_at, :published_response_count, :groups
has_many :invites
has_many :responses
has_many :promotions
def groups
Hash[object.company.groups.map {|group| [group.name, object.groups.include?(group)] }]
end
def published_response_count
object.responses.where(published: true).count
end
def created_at
object.created_at.in_time_zone("Eastern Time (US & Canada)").strftime("%l:%M%p %b %d, %Y")
end
def updated_at
object.updated_at.in_time_zone("Eastern Time (US & Canada)").strftime("%l:%M%p %b %d, %Y")
end
def engagement
object.engagement
end
end
For what it's worth, I am fully aware that returning JSON data like this from Rails is not a great practice and have since moved away from it completely. Unfortunately this piece of code was written quite awhile ago and I can't afford the time to do a full rewrite of the client side to consume a standard output such as an array of contacts.
I started looking into the queries that we're being generated by ActiveRecord.
I realized after a while though that the queries were only accounting for a few ms of the total processing time. I started benchmarking the code starting with the base query company.contacts and then gradually adding on methods I had, such as map and then passing each contact into the serializer, and finally calling as_json[:contact] on the object returned by ContactSerializer.new(contact).
What I found is that calling as_json[:contact] on the serialized object was consuming an average of about 30ms (averaged over 100 runs) per contact. The reason I had that was to remove the root contact node from the JSON that was returned.
When I benchmarked my original code with 198 contacts, it took an average of 10400ms over 10 runs. When I removed as_json[:contact] and set root false on ContactSerializer, as described by "Abusing ActiveModel::Serializers for HAL", I was able to cut the time down from 10400ms to 87ms, while returning the exact same structure to the client, which is astounding.
It might be possible to shave a few ms off that with some query optimizations, but anything else at this point is just icing on the cake.
I need to return a leaderboard data in pages via JSON, which is the correct structure, is it this
{
pages: [
{
[
{user: John,
rating:11},
{user: Bob,
rating: 20},
{user: Andy,
rating: 30},
...
]
},
{
[
{user: Sally,
rating: 110},
{user: Peter,
rating: 115},
{user: Jim,
rating: 350},
...
]
},
...
]
}
Or is this (correct JSON)
{
"pages": [
[
{
"user": "John",
"rating": 11
},
{
"user": "Bob",
"rating": 20
},
{
"user": "Andy",
"rating": 30
}
],
[
{
"user": "Sally",
"rating": 110
},
{
"user": "Peter",
"rating": 115
},
{
"user": "Jim",
"rating": 350
}
]
]
}
UPDATE:
Thanks for all the prompt answers, and yes I did construct the JSON by hand which is obviously not a good idea as some of you have pointed out. The 2nd option is the proper JSON and I have updated it with the correct JSON structure for anyone else that might be reading this in the future.
The latter is correct, but you need to enclose all your strings with double quotes. You also used a period instead of a comma after the first closing square bracket.
You may wish to use JSONLint to validate your JSON.
Your first example doesn’t make much sense: pages is an array whose elements are objects but there are no key-value pairs. Your second example makes more sense: pages is an array where which element is in turn another array containing a list of objects.
Note that neither of your examples is valid JSON. As explained in the previous paragraph, your first example has objects with no key-value pairs. Furthermore, in both examples the strings aren’t quoted. In JSON, every string must be quoted, be it a key or a value.
You might want to check out this JSON validator :) http://www.jsonlint.com/
For starters, the first option is not valid JSON. The array:
{
pages: [
{
[ <== HERE
...would require a name. E.g.
{
pages: [
{
"SomeName": [
Also, assuming John, Bob, Andy etc, are strings, then they should be:
[
{user: "John",
rating:11},
{user: "Bob",
rating: 20},
{user: "Andy",
rating: 30},
...
]
Some would also argue that your dictionary member names should be enquoted.