Related
I have the following case:
I have some information stored in Neo4j. One property store a JSON document, and I would like to get the information inside this JSON document.
I have retrieved the data, even the JSON field using MATCH:
MATCH (n:Node) RETURN n.id, n.nodeInfo as JSONInfo
JSONInfo (which is a property of Node), has the JSON Information:
{
"TIMESTAMP":"2018-03-11T04:58:24Z",
"field1":"358716053191804",
"field2":"732111149743974",
"version_field", "3.9.1"
"field3":"0",
"field4":"0"
}
But, I just want to get field1, that has inside JSON property.
What is the best way to retrieve this field in the MATCH command?
Thanks in advance
First of all your example isn't valid JSON. Your version_field key and value should be separated by a colon, not a comma, and you need a comma after "3.9.1".
{ "TIMESTAMP":"2018-03-11T04:58:24Z", "field1":"358716053191804", "field2":"732111149743974", "version_field":"3.9.1", "field3":"0", "field4":"0" }
Once that's fixed, and the field is valid JSON, you can use JSON conversion functions from APOC Procedures to do the conversion from a JSON object to a map, then just use dot notation to access map properties:
MATCH (n:Node)
RETURN apoc.convert.fromJsonMap(n.nodeInfo).field1 as field1
Keep in mind that map properties in JSON map strings cannot be indexed, so the data within your JSON string property should be used for storage and retrieval, not for lookup.
I have a Json field stored in DB as text.
I want to look for records that contain Harvard as a value for those keys
json_data['infoOnProgram']['universityName']
How should I construct my where request to search on that Json field.
Student.where(json_data...
Thank you!
If there are a not very many records you can do this, which will be very expensive:
harvard_students = Student.map {|stud|
hash = JSON.parse(stud.json_data)
hash['infoOnProgram']['universityName'] == 'Harvard' ? stud : nil
}.compact
if you have lots and lots of records to iterate through you can limit your list with a regex query:
Simple
Student.where("json data like '%Harvard%'")
slightly better
Student.where("json data like '%infoOnProgram%universityName%Harvard%'")
The problem is without knowing how the data is structured it is hard to create a specific regex. You can use the where to limit and the iteration to confirm more precisely that these records are what you are looking for.
If you're sure you really do have valid JSON in your text column then you could cast it to jsonb (or json) and use the usual JSON operators and functions that PostgreSQL provides:
Student.where(
"json_data::jsonb -> 'infoOnProgram' ->> 'universityName' = ?",
'Harvard'
)
Student.where(
'json_data::jsonb -> :info ->> :name = :uni',
info: 'infoOnProgram',
name: 'universityName',
uni: 'Harvard'
)
Student.where(
'cast(json_data as jsonb) #>> array[:path] = :uni',
path: %w[infoOnProgram universityName],
uni: 'Harvard'
)
This is going to be an expensive query though as you won't be able to use any indexes. You really should start the process of changing the column's type to jsonb, that would validate the JSON, avoid having to type cast in queries, allow indexing, ...
I tried to display datas which is in Dictionary format. Below, three attempts are there. First attempt, output order is completely changed. Second attempt, output order is same as input. But, in third attempt, I declared variable as NSDictionary. Exact output I received. Why this changes in Dictionary? Kindly guide me. I searched for Swift's Dictionary tag. But I couldn't found out.
//First Attempt
var dict : Dictionary = ["name1" : "Loy", "name2" : "Roy"]
println(dict)
//output:
[name2: Roy, name1: Loy]
//Second Attempt
var dict : Dictionary = ["name2" : "Loy", "name1" : "Roy"]
println(dict)
//output:
[name2: Loy, name1: Roy]
-----------------------------------------------------------
//Third Attempt With NSDictionary
var dict : NSDictionary = ["name1" : "Loy", "name2" : "Roy"]
println(dict)
//output:
{
name1 = Loy;
name2 = Roy;
}
ANOTHER QUERY: I have used play ground to verify. My screen shot is below:
Here, In NSDictionary, I gave name5 as first, but in right side name2 is displaying, then, in println, it is displaying in ascending order. Why this is happening??
Here, In Dictionary, I gave name5 as first, but in right side name2 is displaying, then, in println, it is displaying, how it is taken on the Dictionary line. Why this is happening??
This is because of the definition of Dictionaries:
Dictionary
A dictionary stores associations between keys of the same type and values of the same type in an collection with no defined ordering.
There is no order, they might come out differently than they were put in. This is comparable to NSSet.
Edit:
NSDictionary
Dictionaries Collect Key-Value Pairs. Rather than simply maintaining an ordered or unordered collection of objects, an NSDictionary stores objects against given keys, which can then be used for retrieval.
There is also no order, however there is sorting on print for debugging purposes.
You can't sort a dictionary but you can sort its keys and loop through them as follow:
let myDictionary = ["name1" : "Loy", "name2" : "Roy", "name3" : "Tim", "name4" : "Steve"] // ["name1": "Loy", "name2": "Roy", "name3": "Tim", "name4": "Steve"]
let sorted = myDictionary.sorted {$0.key < $1.key} // or {$0.value < $1.value} to sort using the dictionary values
print(sorted) // "[(key: "name1", value: "Loy"), (key: "name2", value: "Roy"), (key: "name3", value: "Tim"), (key: "name4", value: "Steve")]\n"
for element in sorted {
print("Key = \(element.key) Value = \(element.value)" )
}
A little late for the party but if you want to maintain the order then use KeyValuePairs, the trade-off here is that if you use KeyValuePairs you lose the capability of maintaining unique elements in your list
var user: KeyValuePairs<String, String> {
return ["FirstName": "NSDumb",
"Address": "some address value here",
"Age":"30"]
}
prints
["FirstName": "NSDumb", "Address": "some address value", "Age": "30"]
Dictionaries, by nature, are not designed to be ordered, meaning that they're not supposed to be (although they can be!).
From the Dictionaries (Swift Standard Library documentation):
A dictionary is a type of hash table, providing fast access to the entries it contains. Each entry in the table is identified using its key, which is a hashable type such as a string or number. You use that key to retrieve the corresponding value, which can be any object. In other languages, similar data types are known as hashes or associated arrays.
This requires some basic knowledge of Data Structures, which I'll outline & oversimplify briefly.
Storing associated data without a dictionary
Consider for a minute if there was no Dictionary and you had to use an array of tuples instead, to store some information about different fruits and their colors, as another answer suggested:
let array = [
("Apple", "Red"),
("Banana", "Yellow"),
// ...
]
If you wanted to find the color of a fruit you'd have to loop through each element and check its value for the fruit, then return the color portion.
Dictionaries optimize their storage using hash functions to store their data using a unique hash that represents the key that is being stored. For swift this means turning our key—in this case a String—into an Int. Swift uses Int-based hashes, which we know because we all read the Hashable protocol documentation and we see that Hashable defines a hashValue property that returns an Int.
Storing associated data with a dictionary
The benefits of using a dictionary are that you get fast read access and fast write access to data; it makes "looking up" associated data easy and quick. Typically O(1) time complexity, although the apple docs don't specify, maybe because it depends on the key type's hash function implementation.
let dictionary = [
"Apple": "Red",
"Banana": "Yellow"
// ...
]
The trade off is that the order is typically not guaranteed to be preserved. Not guaranteed means that you might get lucky and it might be the same order, but it's not intended to be, so don't rely on it.
As an arbitrary example, maybe the string "Banana" gets hashed into the number 0, and "Apple" becomes 4. Since we now have an Int we could, under the hood, represent our dictionary as an array of size 5:
// what things *might* look like under, the hood, not our actual code
// comments represent the array index numbers
let privateArrayImplementationOfDictionary = [
"Yellow", // 0
nil, // 1
nil, // 2
nil, // 3
"Red", // 4
] // count = 5
You'll notice, we've converted our keys into array indices, and there are a bunch of blank spaces where we have nothing. Since we are using an array, we can insert data lightning fast, and retrieve it just as quickly.
Those nil spaces are reserved for more values that may come later, but this is also why when we try to get values out of a dictionary, they might be nil. So when we decide to add more values, something like:
dictionary["Lime"] = "Green" // pretend hashValue: 2
dictionary["Dragonfruit"] = "Pink" // pretend hashValue: 1
Our dictionary, under the hood, may look like this:
// what things *might* look like under, the hood, not our actual code
// comments represent the array index numbers
let privateArrayImplementationOfDictionary = [
"Yellow", // 0 ("Banana")
"Pink", // 1 ("Dragonfruit")
"Green", // 2 ("Lime")
nil, // 3 (unused space)
"Red", // 4 ("Apple")
] // count = 5
As you can see, the values are not stored at all in the order we entered them. In fact, the keys aren't even really there. This is because the hash function has change our keys into something else, a set of Int values that give us valid array indices for our actual storage mechanism, an array, which is hidden from the world.
I'm sure that was more information than you wanted and probably riddled with many inaccuracies, but it gives you the gist of how a dictionary works in practice and hopefully sounds better than, "that's just how it works."
When searching for the actual performance of Swift dictionaries, Is Swift dictionary ... indexed for performance? ... StackOverflow had some extra possible relevant details to offer.
If you're still interested to know more details about this, you can try implementing your own dictionary as an academic exercise. I'd also suggest picking up a book on Data Structures and Algorithms, there are many to choose from, unfortunately I don't have any suggestions for you.
The deeper you get into this topic the more you'll understand why you'll want to use one particular data structure over another.
Hope that helps!
✅ It is possible!
Although the Dictionary is not ordered, you can make it preserve the initial order by using the official OrderedDictionary from the original Swift Repo
The ordered collections currently contain:
Ordered Dictionary (That you are looking for)
Ordered Set
They said it is going to be merged in the Swift's source code soon (reference WWDC21)
Neither NSDictionary nor Swift::Dictionary orders its storage. The difference is that some NSDictionary objects sort their output when printing and Swift::Dictionary does not.
From the documentation of -[NSDictionary description]:
If each key in the dictionary is an NSString object, the entries are
listed in ascending order by key, otherwise the order in which the
entries are listed is undefined. This property is intended to produce
readable output for debugging purposes, not for serializing data.
From The Swift Programming Language:
A dictionary stores associations between keys of the same type and values of the same type in an collection with no defined ordering.
Basically, order of items as seen in output is arbitrary, dependant on internal implementation of data structure, and should not be relied on.
This is indeed an issue with dictionaries. However, there's a library available to make sure the order stays the way you initialised it.
OrderedDictionary is a lightweight implementation of an ordered dictionary data structure in Swift.
The OrderedDictionary structure is an immutable generic collection which combines the features of Dictionary and Array from the Swift standard library. Like Dictionary it stores key-value pairs and maps each key to a value. Like Array it stores those pairs sorted and accessible by a zero-based integer index.
Check it out here:
https://github.com/lukaskubanek/OrderedDictionary
For various reasons, I'm creating an app that takes a SQL query string as a URL parameter and passes it off to Postgres(similar to the CartDB SQL API, and CFPB's Qu). Rails then renders a JSON response of the results that come from Postgres.
Snippet from my controller:
#table = ActiveRecord::Base.connection.execute(#query)
render json: #table
This works fine. But when I use Postgres JSON functions (row_to_json, json_agg), it renders the nested JSON property as a string. For example, the following query:
query?q=SELECT max(municipal) AS series, json_agg(row_to_json((SELECT r FROM (SELECT sch_yr,grade_1 AS value ) r WHERE grade_1 IS NOT NULL))ORDER BY sch_yr ASC) AS values FROM ed_enroll WHERE grade_1 IS NOT NULL GROUP BY municipal
returns:
{
series: "Abington",
values: "[{"sch_yr":"2005-06","value":180}, {"sch_yr":"2005-06","value":180}, {"sch_yr":"2006-07","value":198}, {"sch_yr":"2006-07","value":198}, {"sch_yr":"2007-08","value":158}, {"sch_yr":"2007-08","value":158}, {"sch_yr":"2008-09","value":167}, {"sch_yr":"2008-09","value":167}, {"sch_yr":"2009-10","value":170}, {"sch_yr":"2009-10","value":170}, {"sch_yr":"2010-11","value":153}, {"sch_yr":"2010-11","value":153}, {"sch_yr":"2011-12","value":167}, {"sch_yr":"2011-12","value":167}]"
},
{
series: "Acton",
values: "[{"sch_yr":"2005-06","value":353}, {"sch_yr":"2005-06","value":353}, {"sch_yr":"2006-07","value":316}, {"sch_yr":"2006-07","value":316}, {"sch_yr":"2007-08","value":323}, {"sch_yr":"2007-08","value":323}, {"sch_yr":"2008-09","value":327}, {"sch_yr":"2008-09","value":327}, {"sch_yr":"2009-10","value":336}, {"sch_yr":"2009-10","value":336}, {"sch_yr":"2010-11","value":351}, {"sch_yr":"2010-11","value":351}, {"sch_yr":"2011-12","value":341}, {"sch_yr":"2011-12","value":341}]"
}
So, it only partially renders the JSON, running into problems when I have nested JSON arrays created with the Postgres functions in the query.
I'm not sure where to start with this problem. Any ideas? I am sure this is a problem with Rails.
ActiveRecord::Base.connection.execute doesn't know how to unpack database types into Ruby types so everything – numbers, booleans, JSON, everything – you get back from it will be a string. If you want sensible JSON to come out of your controller, you'll have to convert the data in #table to Ruby types by hand and then convert the Ruby-ified data to JSON in the usual fashion.
Your #table will actually be a PG::Result instance and those have methods such as ftype (get a column type) and fmod (get a type modifier for a column) that can help you figure out what sort of data is in each column in a PG::Result. You'd probably ask the PG::Result for the type and modifier for each column and then hand those to the format_type PostgreSQL function to get some intelligible type strings; then you'd map those type strings to conversion methods and use that mapping to unpack the strings you get back. If you dig around inside the ActiveRecord source, you'll see AR doing similar things. The AR source code is not for the faint hearted though, sorry but this is par for the course when you step outside the narrow confines of how AR things you should interact with databases.
You might want to rethink your "sling hunks of SQL around" approach. You'll probably have an easier time of things (and be able to whitelist when the queries do) if you can figure out a way to build the SQL yourself.
The PG::Result class (the type of #table), utilizes TypeMaps for type casts of result values to ruby objects. For your example, you could use PG::TypeMapByColumn as follows:
#table = ActiveRecord::Base.connection.execute(#query)
#table.type_map = PG::TypeMapByColumn.new [nil, PG::TextDecoder::JSON.new]
render json: #table
A more generic approach would be to use the PG::TypeMapByOid TypeMap class. This requires you to provide OIDs for each PG attribute type. A list of these can be found in pg_type.dat.
tm = PG::TypeMapByOid.new
tm.add_coder PG::TextDecoder::Integer.new oid: 23
tm.add_coder PG::TextDecoder::Boolean.new oid: 16
tm.add_coder PG::TextDecoder::JSON.new oid: 114
#table.type_map = tm
i'm using riak to store json documents right now, and i want to sort them based on some attribute, let's say there's a key, i.e
{
"someAttribute": "whatever",
"order": 1
}
so i want to sort the documents based on the "order".
I am currently retrieving the documents in riak with the erlang interface. i can retrieve the document back as a string, but i dont' really know what to do after that. i'm thinking the map function just reduces the json document itself, and in the reduce function, i'd make a check to see whether the item i'm looking at has a higher "order" than the head of the rest of the list, and if so append to beginning, and then return a lists:reverse.
despite my ideas above i've had zero results after almost an entire day, i'm so confused with the erlang interface in riak. can someone provide insight on how to write this map/reduce function, or just how to parse the json document?
As far as I know, You do not have access to Input list in Map. You emit from Map a document as 1 element list.
Inputs (all the docs to handle as {Bucket, Key}) -> Map (handle single doc) -> Reduce (whole list emitted from Map).
Maps are executed per each doc on many nodes whereas Reduce is done once on so called coordinator node (the one where query was called).
Solution:
Define Inputs (as a list or bucket)
Retrieve Value in Map and emit whole doc or {Id, Val_to_sort_by)
Sort in Reduce (using regular list:keysort)
This is not a map reduce solution but you should check out Riak Search.
so i "solved" the problem using javascript, still can't do it using erlang.
here is my query
{"inputs":"test",
"query":[{"map":{"language":"javascript",
"source":"function(value, keyData, arg){ var data = Riak.mapValuesJson(value)[0]; var obj = {}; obj[data.order] = data; return [ obj ];}"}},
{"reduce":{"language":"javascript",
"source":"function(values, arg){ return [ values.reduce(function(acc, item){ for(var order in item){ acc[order] = item[order]; } return acc; }) ];}",
"keep":true}}
]
}
so in the map phase, all i do is create a new array, obj, with the key as the order, and the value as the data itself. so visually, the obj is like this
{"1":{"firstName":"John","order":1}
in the reduce phase, i'm just putting it in the accumulator, so basically that's the sort if you think about it, because when you're done, everything will be put in order for you. so i put 2 json documents for testing, one is above, the ohter is just firstName: Billie, order 2. and here is my result for the query above
[{"1":{"firstName":"John","order":1},"2":{"firstName":"Billie","order":2}}]
so it works! . but i still need to do this in ERLANG, any insights?