I am trying to add a property to a node using
n.item = apoc.convert.toJson(itemObject)
Where
itemArrayObjects = {"source":"Blogspot.com","author":"noreply#blogger.com (Unknown)","title":"Elon Musk reveals who bitcoin's creator Satoshi Nakamoto might be","content":"Musk.MARK RALSTON/AFP via Getty Images\r\nElon Musk seems to agree with many that hyper-secret cryptocurrency expert Nick Szabo could be Satoshi Nakamoto, the mysterious creator of the digital currency… [+1467 chars]","publishedAt":"2021-12-29T20:41:00Z","url":"https://techncruncher.blogspot.com/2021/12/elon-musk-reveals-who-bitcoins-creator.html"}
this results in
Neo4jError: Failed to invoke function `apoc.convert.toJson`: Caused by: java.lang.NullPointerException
In the Neo4j Browser this works:
RETURN apoc.convert.toJson({d:"ddddd", e:"eeee"})
but this doesn't work:
RETURN apoc.convert.toJson({"a": "aaaaaa", "b": "bbbbbb"})
If I assign the values to a cypher :param like this:
:param items =>[{source:"Blogspot.com",author:"noreply#blogger.com (Unknown)",title:"Elon Musk reveals who bitcoin's creator Satoshi Nakamoto might be",content:"Musk.MARK RALSTON/AFP via Getty Images\r\nElon Musk seems to agree with many that hyper-secret cryptocurrency expert Nick Szabo could be Satoshi Nakamoto, the mysterious creator of the digital currency… [+1467 chars]",publishedAt:"2021-12-29T20:41:00Z",url:"https://techncruncher.blogspot.com/2021/12/elon-musk-reveals-who-bitcoins-creator.html"},{d:"xxddddd",e:"eeee"},{d:"ddddd",e:"eeee"}]
I get this as :params
{
"items": [
{
"publishedAt": "2021-12-29T20:41:00Z",
"author": "noreply#blogger.com (Unknown)",
"source": "Blogspot.com",
"title": "Elon Musk reveals who bitcoin's creator Satoshi Nakamoto might be",
"url": "https://techncruncher.blogspot.com/2021/12/elon-musk-reveals-who-bitcoins-creator.html",
"content": "Musk.MARK RALSTON/AFP via Getty Images
Elon Musk seems to agree with many that hyper-secret cryptocurrency expert Nick Szabo could be Satoshi Nakamoto, the mysterious creator of the digital currency… [+1467 chars]"
},
{
"d": "xxddddd",
"e": "eeee"
},
{
"d": "ddddd",
"e": "eeee"
}
]
}
Notice the keys are double quoted "" as they rightly should
and this works:
return apoc.convert.toJson($items)
So it appears some behind the scenes conversions going on. It also appears to be some inconsistency as it works sometime without changes.
can anyone shed some light on this?
EDIT: I am actually using neo4j Desktop 4.2.1 and APOC 4.2.0 locally and neo4j 4.4.2 docker image with apoc 4.4.0.1 on Digital Ocean. The inconsistency is that for the most part this works locally.
Apparently there was a bug in apoc v4.4.0.1 as it relates apoc.convert.Json()....a fix was made in v4.4.0.2
Related
I did a little Basic and Ada programming 25 years ago, but never got any deeper ... now my mind isn't as sharp as it used to be so I bought a few courses about Dart and Flutter on Udemy, but instructors don't really teach programming... They start advanced topics like complex data structures in the first lessons in 3 minute long video and I'm completely lost.
I'm trying to make some sense on my own about nested collections, researching on the internet, but i can't figure out how to access double nested collections (list/maps inside a list of maps). Can you guys share some links / courses on the topic.
Here is some sample code, I managed to access to the first map data, but I don't know how to get the items in the 'ingredient' list, neither the data in the 'macros' map... How can i now the 'protein' in my dishes? Thanks in advance.
void main() {
var recipes = [
{
'name': 'Hoisin Chicken Lettuce Cups',
'type': 'Salad',
'ingredients': ['mango', 'hoisin sauce', 'chicken breasts', 'romaine lettuce', 'cress'],
'calories': 289,
'macros': {
'fat': 9.6,
'sat_fat': 1.8,
'protein': 31.3,
'carbs': 20.3,
'sugar': 19.2,
'salt': 1.3,
'fibre': 2.5,
},
},
{
'name': 'Smoked Salmon Plates',
'type': 'Salad',
'ingredients': ['cucumber', 'dill', 'smoked salmon', 'avocado', 'cottage cheese'],
'calories': 246,
'macros': {
'fat': 17.7,
'sat_fat': 3.9,
'protein': 17.1,
'carbs': 4.4,
'sugar': 3.2,
'salt': 1.5,
'fibre': 1.2,
},
},
];
// Managing to accest collection data... How to get protein amount?
// How to know 'macros' lenght? ...
print(recipes);
print(recipes.length);
print('---------------------------------------');
print(recipes[1]);
print(recipes[1].length);
print(recipes[1]['ingredients']);
print(recipes[1]['macros']);
print('---------------------------------------');
}
I am using the slack API to get the full list of emoji, so that when I get a message, I will just replace :squirrel: with the icon.
The method https://slack.com/api/emoji.list works like a charm, but returns 30 icons only. I think this is correct since in the documentation page (https://api.slack.com/methods/emoji.list) they say:
This method lists the custom emoji for a team.
Fair enough, but how can I get the full list of the associations icon-name / icon URL ?
I finally managed to get all the icons and to use them and I post here the solution for anyone that would like to use do similar:
First of all, I got the Slack Custom Emoji through this slack URL
Since at step 1 we get only Custom Emojis, it is useful to know that slack uses standard emoji defined in unicode characters, mapped through custom handles like :smiley: or :horse:. The good thing is that we can find, linked through slack page a link to a JSON object with all the emoji mappings. This file is HUGE, but has everything we need.
In the file you'll find an array of javascript object like the one below:
{
"name":"SMILING FACE WITH OPEN MOUTH",
"unified":"1F603",
"variations":[],
"docomo":"E6F0",
"au":"E471",
"softbank":"E057",
"google":"FE330",
"image":"1f603.png",
"sheet_x":26,
"sheet_y":18,"
short_name":"smiley",
"short_names":["smiley"],
"text":":)",
"texts":["=)","=-)"],
"category":"People",
"sort_order":5,
"has_img_apple":true,
"has_img_google":true,
"has_img_twitter":true,
"has_img_emojione":true
}
I used the following information:
shortnames are the names that are used in slack (you'll need to turn smiley into :smiley: )
unified is the unicode character to use (to use it directly in an HTML page you'll need to add &#x so in this case you'll have to use 😃 which is rendered 😃
Using this information you will be able to create a slack-to-html function to decode emojis and display them wherever you want
Not entirely sure if this is what you are looking for, but if it's just about mapping images to slack-style names, this is a pretty good library:
https://github.com/iamcal/emoji-data
So, building on the example in their README:
The emoji with the Slack style short name point_uphas the hex value 261d, and can thus be found here: https://github.com/iamcal/emoji-data/blob/master/img-apple-160/261d.png
(Apple, because the default slack emoji are the apple emoji)
Just extending on #Luca's awesome solution, I've created a shortnames => html unicode javascript dictionary...
Download: Slack emoticons to unicode html mapping.
Generated - 17th August 2018 from the source https://raw.githubusercontent.com/iamcal/emoji-data/master/emoji.json
Example:
{
"+1": "👍",
"-1": "👎",
"100": "💯",
"1234": "🔢",
"8ball": "🎱",
"ab": "🆎",
"abc": "🔤",
"abcd": "🔡",
"accept": "🉑",
...
"zebra_face": "🦓",
"zipper_mouth_face": "🤐",
"zombie": "🧟",
"zzz": "💤"
}
Which becomes...
{ "+1": "👍", "-1": "👎", "100": "💯",
"1234": "🔢", "8ball": "🎱", "ab": "🆎",
"abc": "🔤", "abcd": "🔡", "accept": "🉑",
... "zebra_face": "🦓", "zipper_mouth_face": "🤐",
"zombie": "🧟", "zzz": "💤" }
As far as I know, there is no API endpoint or comprehensive list of supported emoji/keywords available. I was able to grab the full set (including custom emoji for the workspace) by inspecting the Slack emoji picker using React Developer Tools (Chrome extension).
Here's an example (JSON):
...
{
"name": "beers",
"unicode": "1f37b",
"id": "E1f37b",
"keywords": ["bar", "beer", "clink", "drink", "mug", "ale", "food"]
},
{
"name": "baby_bottle",
"unicode": "1f37c",
"id": "E1f37c",
"keywords": ["baby", "bottle", "drink", "milk", "infant"]
},
{
"name": "knife_fork_plate",
"unicode": "1f37d-fe0f",
"id": "E1f37d-fe0f",
"keywords": ["cooking", "fork", "knife", "plate"]
},
{
"name": "champagne",
"unicode": "1f37e",
"id": "E1f37e",
"keywords": ["bar", "bottle", "cork", "drink", "popping"]
},
{ "name": "popcorn", "unicode": "1f37f", "id": "E1f37f", "keywords": [] },
...
Full dump (as of 12/20/2020, excluding custom emoji): https://gist.github.com/impressiver/87b5b9682d935efba8936898fbfe1919
Since it seemed impossible to find a complete and up-to-date source for these emoji names I came up with this browser console script. Just let it run in Slack and when it finishes it will download a full emoji.json. At time of writing it contains 2485 entries.
https://gist.github.com/8461e125072c2806301403a4e1eca891
This looks like this:
{
"+1": "https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f44d#2x.png",
"+1::skin-tone-2": "https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f44d-1f3fb#2x.png",
"+1::skin-tone-3": "https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f44d-1f3fc#2x.png",
"+1::skin-tone-4": "https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f44d-1f3fd#2x.png",
If you want the actual emoji instead of the image you can parse that out of the image file name, e.g. 1f44d-1f3fd#2x.png is "\u1f44d\u1f3fd".
Disclaimer: I'm a novice.
I want to simulate a join for my mongodb embedded document. If I have an embedded list:
{
_id: ObjectId("5320f6c34b6576d373000000"),
user_id: "52f581096b657612fe020000",
list: "52f4fd9f52e39bc0c15674ea"
{
player_1: "52f4fd9f52e39bc0c15674ex",
player_2: "52f4fd9f52e39bc0c15674ey",
player_3: "52f4fd9f52e39bc0c15674ez"
}
}
And a player collection with each player being something like:
{
_id: ObjectId("52f4fd9f52e39bc0c15674ex"),
college: "Louisville",
headshot: "player.png",
height: "6'2",
name: "Wayne Brady",
position: "QB",
weight: 205
}
I want to end up with:
{
_id: ObjectId("5320f6c34b6576d373000000"),
user_id: "52f581096b657612fe020000",
list: "52f4fd9f52e39bc0c15674ea"
{
player_1:
{
_id: ObjectId("52f4fd9f52e39bc0c15674ex"),
college: "Louisville",
headshot: "player.png",
height: "6'2",
name: "Wayne Brady",
position: "QB",
weight: 205
},
etc...
}
}
So I can call User.lists.first.player_1.name.
This is what makes sense in my mind since I'm new to rails...and I don't want to embed players in each user's list because I'd have so many redundancies...
Advice? Is this possible, if so how? Is it a good idea, or is there a better way?
So have have a typical relational model, let's call it "one to many", which you have users or "user teams" and a whole pool of players. And in typical modelling form you want to "de-normalize" this to avoid duplication.
But here's the thing, MongoDB does not do joins. Joins are not "webscale" in the current parlance. So it leaves you thinking what to do. Or does MongoDB do joins?
db.eval(function() {
var user = db.user.findOne({ "user_id": "52f581096b657612fe020000" });
for ( k in user.list ) {
var player = db.player.findOne({ "_id": user.list[k] });
user.list[k] = player;
}
return user;
});
Which "arguably" is "kind of a join". And it was all done on the server, right?
But DO NOT DO THAT. While db.eval() has uses, something that you are going to query regularly is not one of the practical uses. Read the documentation, which shows the warnings. In particular, all JavaScript is running in a single thread so that will lock things up very quickly.
Now client side, you are more or less doing the same thing. And you ODM of choice is likely again doing "the same thing", though it is usually hiding it away in some manner so you don't see that part. Likewise the same could likely be said of your SQL ORM, which was also "sneaking off behind your back" and querying the database while you just accessed the objects in your code.
As for mapReduce. Well the problem with the data you present is that there is nothing to "reduce". There is a technique known as in "incremental mapReduce" but it would not be well suited to this type of data. A topic in itself, but you would basically need all the "users" associated to the "players" as well, stored in the "player data" to make that any kind of viability. And it's ultimately just another way of "cheating" joins.
This is the space in which MongoDB exists.
So rather than going and doing all this fetching or joining, it allows the concept of being able to "pre-join" your data as it were. And the point of this is to allow faster, and more atomic reads and writes. And this is known as embedding.
Looking at your data, there should not be a problem with embedding at all. Consider the points:
Presumably you are modelling "fantasy teams" for a given user. It would be fair to day that a "team" does not consist of an infinite number of players.
Aside from other things your "A1" usage is likely to be "displaying" the players associated with that "user team". And in so much as, you want to "display" as much information as possible, and keep that to a single read operation. You also want to easily add "players" to the "user team".
While a "player" may have "extended information", and possibly even some global statistics or scores, that information may well be not what you want to access, while associated to the "user team" that often. It can probably be written independently, and only read when looking at the "player detail".
Those are three good cases to support embedding. Sure you would be duplicating information stored against each user team, opposed to just a small "key" reference. And sure that information is likely to exist elsewhere in the full "player detail" and that would be duplication as well.
But the point of the "duplication" here is to optimize. So here it would seem valid to embed "some of the data", not all, but what you regularly use in your main operations. Considering the "player's" name, position, height and weight are not likely to change on a regular basis or not even at all in the context, then that seems a reasonable trade-off.
{
"_id": ObjectId("5320f6c34b6576d373000000"),
"user_id": ObjectId("52f581096b657612fe020000"),
"list": [
{
"_id": ObjectId("52f4fd9f52e39bc0c15674ex"),
"label": "Player1",
"college": "Louisville",
"headshot": "player.png",
"height": "6'2",
"name": "Wayne Brady",
"position": "QB",
"weight": 205
},
{
"label": "Player2",
(...)
}
]
},
That's not that bad. And it would take a lot to break the 16MB limit. And considering this seems to be a "user team" then it could probably do with some information from the "user" as well.
You also get a lot of power out of this when data is kept together like this, to find the top "player" picked by each user:
db.userteams.aggregate([
// Unwind the array
{ "$unwind": "$list" },
// Group and use the player name
{ "$group": {
"_id": {
"user_id": "$user_id",
"player": "$list.name",
},
"count": { "$sum": 1 }
}},
// Sort the results descending by popularity
{ "$sort": { "_id.user_id": 1, "count": -1 } },
// Group to limit the first one
{ "$group": {
"_id": "$_id.user_id",
"player": { "$first": "$_id.player" },
"picks": { "$first:" "$count" }
}}
])
Which admittedly is a reasonably trivial use of a name in this case, but it is an example of using information that has become available by the use of some embedding.
Of course you really believe that you need everything to be properly normalized, then do it that way, and live with the patterns you would need to access it. But this offers a perspective of doing this another way.
So don't over-concern yourself with embedding everything, and lose a little fear on embedding some things. There are no "get out of jail free cards" for using something not suited to relational modeling in a standard relational way. Choose something that suits your needs.
Is there any mechanism for controlling the order of properties?
I cannot reproduce this in http://www.neo4j.org/console
Using Neo4j 1.9.2 Community if I do the following:
CREATE (m1 {`$type`: {moduleTypeName}, Name: 'M1', ModelNumber: 'MN1'})
Then later I get this node back from a cypher query using the REST cypher endpoint I get back...
{
"extensions": {},
"paged_traverse": "http://localhost:7575/db/data/node/3777/paged/traverse/{returnType}{?pageSize,leaseTime}",
"outgoing_relationships": "http://localhost:7575/db/data/node/3777/relationships/out",
"traverse": "http://localhost:7575/db/data/node/3777/traverse/{returnType}",
"all_typed_relationships": "http://localhost:7575/db/data/node/3777/relationships/all/{-list|&|types}",
"property": "http://localhost:7575/db/data/node/3777/properties/{key}",
"all_relationships": "http://localhost:7575/db/data/node/3777/relationships/all",
"self": "http://localhost:7575/db/data/node/3777",
"properties": "http://localhost:7575/db/data/node/3777/properties",
"outgoing_typed_relationships": "http://localhost:7575/db/data/node/3777/relationships/out/{-list|&|types}",
"incoming_relationships": "http://localhost:7575/db/data/node/3777/relationships/in",
"incoming_typed_relationships": "http://localhost:7575/db/data/node/3777/relationships/in/{-list|&|types}",
"create_relationship": "http://localhost:7575/db/data/node/3777/relationships",
"data": {
"ModelNumber": "MN1",
"$type": "ModuleType",
"Name": "M1"
}
}
I'm using http://james.newtonking.com/pages/json-net.aspx to parse JSON and for it to automatically infer an object type, the $type property must be first. It makes sense when parsing the JSON in a stream when you don't want to load the entire thing into memory first.
It does not appear to be alphabetical, and it does not seem to be random either. It seems that the order is consistent for different object types, but inconsistent between them.
I have pulled the node in the Shell as well and so it seems that the order does not depend on how I get the node, but is not related to the order in which I create the node either.
Properties have no guaranteed order. Do not take any assumptions on a 'maybe' ordering. An upcoming version might change this assumed behaviour and break your code.
I guess it is simpler in Cypher to not return the node itself in favour of a list of properties, e.g.
START node=node(<myid>)
RETURN node.`$type`, node.ModelNumber, node.Name
This has defined columns.
Definitively it seems not to have that functionality.
My workaround for it it is to alias the properties with a prefix in the format aXX_, as in a01_, a02, a03_ and then remove it in the code.
Not pretty, not great, but it works as neo4j respects numerical order.
It needs a letter character at the beggining though, hence the "a" before the numbers.
I had a problem with ElasticSearch and Rails, where some data was not indexed properly because of attr_protected. Where does Elastic Search store the indexed data? It would be useful to check if the actual indexed data is wrong.
Checking the mapping with Tire.index('models').mapping does not help, the field is listed.
Probably the easiest way to explore your ElasticSearch cluster is to use elasticsearch-head.
You can install it by doing:
cd elasticsearch/
./bin/plugin -install mobz/elasticsearch-head
Then (assuming ElasticSearch is already running on your local machine), open a browser window to:
http://localhost:9200/_plugin/head/
Alternatively, you can just use curl from the command line, eg:
Check the mapping for an index:
curl -XGET 'http://127.0.0.1:9200/my_index/_mapping?pretty=1'
Get some sample docs:
curl -XGET 'http://127.0.0.1:9200/my_index/_search?pretty=1'
See the actual terms stored in a particular field (ie how that field has been analyzed):
curl -XGET 'http://127.0.0.1:9200/my_index/_search?pretty=1' -d '
{
"facets" : {
"my_terms" : {
"terms" : {
"size" : 50,
"field" : "foo"
}
}
}
}
More available here: http://www.elasticsearch.org/guide
UPDATE : Sense plugin in Marvel
By far the easiest way of writing curl-style commands for Elasticsearch is the Sense plugin in Marvel.
It comes with source highlighting, pretty indenting and autocomplete.
Note: Sense was originally a standalone chrome plugin but is now part of the Marvel project.
Absolutely the easiest way to see your indexed data is to view it in your browser. No downloads or installation needed.
I'm going to assume your elasticsearch host is http://127.0.0.1:9200.
Step 1
Navigate to http://127.0.0.1:9200/_cat/indices?v to list your indices. You'll see something like this:
Step 2
Try accessing the desired index:
http://127.0.0.1:9200/products_development_20160517164519304
The output will look something like this:
Notice the aliases, meaning we can as well access the index at:
http://127.0.0.1:9200/products_development
Step 3
Navigate to http://127.0.0.1:9200/products_development/_search?pretty to see your data:
ElasticSearch data browser
Search, charts, one-click setup....
Aggregation Solution
Solving the problem by grouping the data - DrTech's answer used facets in managing this but, will be deprecated according to Elasticsearch 1.0 reference.
Warning
Facets are deprecated and will be removed in a future release. You are encouraged to
migrate to aggregations instead.
Facets are replaced by aggregates - Introduced in an accessible manner in the Elasticsearch Guide - which loads an example into sense..
Short Solution
The solution is the same except aggregations require aggs instead of facets and with a count of 0 which sets limit to max integer - the example code requires the Marvel Plugin
# Basic aggregation
GET /houses/occupier/_search?search_type=count
{
"aggs" : {
"indexed_occupier_names" : { <= Whatever you want this to be
"terms" : {
"field" : "first_name", <= Name of the field you want to aggregate
"size" : 0
}
}
}
}
Full Solution
Here is the Sense code to test it out - example of a houses index, with an occupier type, and a field first_name:
DELETE /houses
# Index example docs
POST /houses/occupier/_bulk
{ "index": {}}
{ "first_name": "john" }
{ "index": {}}
{ "first_name": "john" }
{ "index": {}}
{ "first_name": "mark" }
# Basic aggregation
GET /houses/occupier/_search?search_type=count
{
"aggs" : {
"indexed_occupier_names" : {
"terms" : {
"field" : "first_name",
"size" : 0
}
}
}
}
Response
Response showing the relevant aggregation code. With two keys in the index, John and Mark.
....
"aggregations": {
"indexed_occupier_names": {
"buckets": [
{
"key": "john",
"doc_count": 2 <= 2 documents matching
},
{
"key": "mark",
"doc_count": 1 <= 1 document matching
}
]
}
}
....
A tool that helps me a lot to debug ElasticSearch is ElasticHQ. Basically, it is an HTML file with some JavaScript. No need to install anywhere, let alone in ES itself: just download it, unzip int and open the HTML file with a browser.
Not sure it is the best tool for ES heavy users. Yet, it is really practical to whoever is in a hurry to see the entries.
Kibana is also a good solution. It is a data visualization platform for Elastic.If installed it runs by default on port 5601.
Out of the many things it provides. It has "Dev Tools" where we can do your debugging.
For example you can check your available indexes here using the command
GET /_cat/indices
If you are using Google Chrome then you can simply use this extension named as Sense it is also a tool if you use Marvel.
https://chrome.google.com/webstore/detail/sense-beta/lhjgkmllcaadmopgmanpapmpjgmfcfig
Following #JanKlimo example, on terminal all you have to do is:
to see all the Index:
$ curl -XGET 'http://127.0.0.1:9200/_cat/indices?v'
to see content of Index products_development_20160517164519304:
$ curl -XGET 'http://127.0.0.1:9200/products_development_20160517164519304/_search?pretty=1'