Is there any mechanism for controlling the order of properties?
I cannot reproduce this in http://www.neo4j.org/console
Using Neo4j 1.9.2 Community if I do the following:
CREATE (m1 {`$type`: {moduleTypeName}, Name: 'M1', ModelNumber: 'MN1'})
Then later I get this node back from a cypher query using the REST cypher endpoint I get back...
{
"extensions": {},
"paged_traverse": "http://localhost:7575/db/data/node/3777/paged/traverse/{returnType}{?pageSize,leaseTime}",
"outgoing_relationships": "http://localhost:7575/db/data/node/3777/relationships/out",
"traverse": "http://localhost:7575/db/data/node/3777/traverse/{returnType}",
"all_typed_relationships": "http://localhost:7575/db/data/node/3777/relationships/all/{-list|&|types}",
"property": "http://localhost:7575/db/data/node/3777/properties/{key}",
"all_relationships": "http://localhost:7575/db/data/node/3777/relationships/all",
"self": "http://localhost:7575/db/data/node/3777",
"properties": "http://localhost:7575/db/data/node/3777/properties",
"outgoing_typed_relationships": "http://localhost:7575/db/data/node/3777/relationships/out/{-list|&|types}",
"incoming_relationships": "http://localhost:7575/db/data/node/3777/relationships/in",
"incoming_typed_relationships": "http://localhost:7575/db/data/node/3777/relationships/in/{-list|&|types}",
"create_relationship": "http://localhost:7575/db/data/node/3777/relationships",
"data": {
"ModelNumber": "MN1",
"$type": "ModuleType",
"Name": "M1"
}
}
I'm using http://james.newtonking.com/pages/json-net.aspx to parse JSON and for it to automatically infer an object type, the $type property must be first. It makes sense when parsing the JSON in a stream when you don't want to load the entire thing into memory first.
It does not appear to be alphabetical, and it does not seem to be random either. It seems that the order is consistent for different object types, but inconsistent between them.
I have pulled the node in the Shell as well and so it seems that the order does not depend on how I get the node, but is not related to the order in which I create the node either.
Properties have no guaranteed order. Do not take any assumptions on a 'maybe' ordering. An upcoming version might change this assumed behaviour and break your code.
I guess it is simpler in Cypher to not return the node itself in favour of a list of properties, e.g.
START node=node(<myid>)
RETURN node.`$type`, node.ModelNumber, node.Name
This has defined columns.
Definitively it seems not to have that functionality.
My workaround for it it is to alias the properties with a prefix in the format aXX_, as in a01_, a02, a03_ and then remove it in the code.
Not pretty, not great, but it works as neo4j respects numerical order.
It needs a letter character at the beggining though, hence the "a" before the numbers.
Related
How do I add a term to a listItem in Microsoft Graph API?
For simple String types (ProductSegment in the example) I do the following:
PATCH https://graph.microsoft.com/v1.0/sites/{{sharepoint_site_id}}/lists/{{sharepoint_list_id}}/items/{{num}}/fields
{
"DisplayedName": "asdasfsvsvdvsdbvdfb",
"DocumentType": "FLYER",
"ProductSegment": ["SEG1"],
"TEST_x0020_2_x0020_ProductSegment": [{
"TermGuid": "c252c37d-1fa3-4860-8d3e-ff2cdde1f673"
}],
"Active": true,
"ProductSegment#odata.type": "Collection(Edm.String)",
"TEST_x0020_2_x0020_ProductSegment#odata.type": "Collection(Edm.Term)"
}
Obviously it won't work for TEST_x0020_2_x0020_ProductSegment. But I just cannot find any hints in the documentation.
I got one step closer thanks to the duplicated issue. First I found the name (not the id) of the hidden field TEST 2 ProductSegment_0 (notice the _0 suffix). Then assembled the term value to send: -1;#MyLabel|c352c37d-1fa3-4860-8d3e-ff2cdde1f673.
PATCH https://graph.microsoft.com/v1.0/sites/{{sharepoint_site_id}}/lists/{{sharepoint_list_id}}/items/{{num}}/fields
{
"DisplayedName": "asdasfsvsvdvsdbvdfb",
"DocumentType": "FLYER",
"ProductSegment": ["SEG1"],
"i9da5ea20ec548bfb2097f0aefe49df8": "-1;#MyLabel|c352c37d-1fa3-4860-8d3e-ff2cdde1f673",
"Active": true,
"ProductSegment#odata.type": "Collection(Edm.String)"
}
and so I can add one item. I would need to add multiple, so I wanted to add the values to an array and set the field type (i9da5ea20ec548bfb2097f0aefe49df8#odata.type) to Collection(Edm.String).
Now I get an error with the code generalException as opposed to an invalidRequest.
As far as I know, graph API does not support updating SharePoint taxonomy. For now, you can go with classic SharePoint REST API for example to accomplish "advanced" things like updating taxonomy terms. Probably a duplicate of: Can't Update Sharepoint Managed Meta Data Field from Microsoft Graph Explorer
Finally I got it.
Thanks #Nikolay for the linked issue.
As I also added this to the end of the question, first you need the name (not the id!) of the hidden field TEST 2 ProductSegment_0 (notice the _0 suffix). Then assemble the term values to send: -1;#MyLabel|c352c37d-1fa3-4860-8d3e-ff2cdde1f673 and -1;#SecondLabel|1ef2af46-1fa3-4860-8d3e-ff2cdde1f673, and separate them with ;# (actually the content of the label is irrelevant but some string needs to be there).
Looks utterly ridiculous but works.
PATCH https://graph.microsoft.com/v1.0/sites/{{sharepoint_site_id}}/lists/{{sharepoint_list_id}}/items/{{num}}/fields
{
"DisplayedName": "asdasfsvsvdvsdbvdfb",
"DocumentType": "FLYER",
"ProductSegment": ["SEG1"],
"i9da5ea20ec548bfb2097f0aefe49df8": "-1;#MyLabel|c352c37d-1fa3-4860-8d3e-ff2cdde1f673";#-1;#SecondLabel|1ef2af46-1fa3-4860-8d3e-ff2cdde1f673,
"Active": true,
"ProductSegment#odata.type": "Collection(Edm.String)"
}
The log4j2 PatternLayout offers a %notEmpty conversion pattern that allows you to skip sections of the pattern that refer to empty variables.
Is there any way to do something similar for JsonTemplateLayout, specifically for thread context data (MDC)? It correctly (IMO) suppresses null fields, but it doesn't do the same with empty ones.
E.g., given the following in my JSON template:
"application": {
"name": { "key": "x-app", "$resolver": "mdc" },
"context": { "key": "x-app-context", "$resolver": "mdc" },
"instance": {
"name": { "key": "x-appinst", "$resolver": "mdc" },
"context": { "key": "x-appinst-context", "$resolver": "mdc" }
}
}
is there a way to prevent blocks like this from being logged, where the only data in the subtree is the empty string values for context?
"application":{"context":"","instance":{"context":""}}
(Yes, ideally I'd prevent those empty strings being put into the context in the first place, but this isn't my app, I'm just configuring it.)
JsonTemplateLayout author speaking here. Currently, JsonTemplateLayout doesn't support blank property exclusion for the following reasons:
The definition of empty/blank is ambiguous. One might have, null, {}, "\s*", [], [[]], [{}], etc. as valid JSON values. Which one of these are empty/blank? Let's assume we have agreed on a certain behavior. Will it apply to the rest of its users?
Checking if a value is empty/blank incurs an extra runtime cost.
Most of the time you don't care. You persist logs in a storage system, e.g., ELK stack, and there blank value elimination is provided out of the box by the storage engine in the most efficient way.
Would you mind sharing your use case, please? Why do you want to prevent the emission of "context": "" properties? If you deliver your logs to Elasticsearch, there you can easily exclude such fields via appropriate index mappings.
Near as I can tell, no. I would suggest you create a Jira issue to get that addressed.
I am currently working on a POC by using ThingsBoard PE.
Our raw data contains [Asset] [Attributes].
Data flow:
[IoT cloud] --https webhook carry raw data--> [ThingsBoard PE HTTP INTEGRATION] --uplink--> [ThingsBoard PE Data Converter]
My question is: is it possible to apply [Rule Chain] after [ThingsBoard PE Data Converter]? Therefore, the device can auto create relationship with [Asset] by the [Attribute], instead of [AssetName].
Example data after data converter process:
{
"deviceName": "ABC",
"deviceType": "temperature",
"attributes": {
"asset_id": 6 // <-- the id is used in asset attribute
},
"telemetry": {
"temperature": 39.43
}
}
Answering your two, separate questions:
is it possible to apply [Rule Chain] after [ThingsBoard PE Data Converter]?
Yes it is possible. Once your data is successfully integrated and you are receiving it, you can access it using the [Input] Rule Node (the default green one that is always there when you create a Rule) and route it to any other node you need.
Therefore, the device can auto create relationship with [Asset] by the [Attribute], instead of [AssetName].
So, you want the relationship to take your custom attribute and use that as the pattern that identifies the Asset you want to create the relationship from.
The PE edition already has the Create Relation Node. However, seems that as it is one is not able to do what you seek (has no option to specify custom Asset id).
However, two options you got are:
Create a Custom Rule Node that does what you want. For that try checking the Rule Node Development page from Thingsboard. You can use the Create Relation Node as base and work from there. This can be a longer solution than doing...
Enrich your incoming message's metadata, adding your desired attribute. The Create Relation Node allows you to use variables on your message's metadata in your Name and Type patterns, as seen from this screenshot I took from that node:
This allows us a workaround to what you want to do: Add a Script Transformation Node that adds attributes.asset_id to the metadata, for example as metadata.asset_id, so you can then use it as ${asset_id} on your Name and Type patterns.
For example, your Transform() method of such Script Transformation Node should look something like this:
function Transform(msg, metadata, msgType){
//assumming your desired id is msg.attributes.asset_id, add it to the metadata
metadata.asset_id = msg.attributes.asset_id;
//return the message, in your case to the Create Relation Node
return {msg: msg, metadata:metadata, msgType:msgType};
}
Finally, your Rule should be connected like this:
[Input] -> [Script Node] -> [Create Relation Node] -> [...whatever else you like]
The situation is as follows: I have a table in my database that recieves about 3 million rows each day. We want to archive this table on a regular base, so that only the 8 most recents weeks are in the table. The rest of the data can be archived tot AZure Data lake.
I allready found out how to do this by one day at a time. But now I want to run this pipeline each week for the first seven days in the table. I assume I should do this with the "For Each" component. It should itterate along the seven distinct dates that are present in the dataset I want to backup. This dataset is copied from the source table to an archive table on forehand.
It's not difficult to get the distinct dates with a SQL query, but how to get the result of this query into an array that is used for the "For Each" component?
The issue is solved thanks to a co-worker.
What we have to do is assign a parameter to the dataset of the sink. Does not matter how you name this and you do not have to assign a value to it. But let's assume this parameter is called "date"
After that you can use this parameter in the filename of the sink (also in dataset) with by using "#dataset().Date".
After that you go back to the copyactivity and in the sink you assign a dataset property to #item().DateSelect. (DateSelect is the field name from the array that is passed to the For Each activity)
See also the answer from Bo Xioa as part of the answer
This way it works perfectly. It's just a shame that this is not well documented
You can use lookup activity to fetch the column content, and the output will be like
{
"count": "2",
"value": [
{
"Id": "1",
"TableName" : "Table1"
},
{
"Id": "2",
"TableName" : "Table2"
}
]
}
Then you can pass the value array to the Foreach activity items field by using the pattern of #activity('MyLookupActivity').output.value
ref doc: Use the Lookup activity result in a subsequent activity
I post this as an answer, because the error does not fit into a comment :D
have seen antoher option to accomplish this. That is by executing a pipeline from another pipeline. And in that way I can define the dates that I should iterate over as a parameter in the second pipeline (learn.microsoft.com/en-us/azure/data-factory/…). But unformtunately this leads to the same rsult as when just using the foreach parameter. Because in the filename of my data lake file I have to use: #{item().columname}. I can see in the monitoring view that the right values are passed in the iteration steps, but I keep getting an error:
{
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The request to 'Unknown' failed and the status code is 'BadRequest', request id is ''. {\"error\":{\"code\":\"BadRequest\",\"message\":\"A potentially dangerous Request.Path value was detected from the client (:). Trace: cf3b4c3f-1681-4073-b225-17e1c07ec76d Time: 2018-08-02T05:16:13.2141897-07:00\"}} ,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (400) Bad Request.,Source=System,'",
"failureType": "UserError",
"target": "CopyDancerDatatoADL"
}
I am trying to store network layout in Couch DB, but my solution provides rather randomized graph.
I store a nodes with a document:
{_id ,
nodeName,
group}
and storing links in traditional:
{_id, source_id, target_id, value}
Following multiple tutorials on handling joins and multiple relationship in Couch DB I created view:
function(doc) {
if(doc.type == 'connection') {
if (doc.source_id)
emit("source", {'_id': doc.source_id});
if(doc.target_id)
emit("target", {'_id': doc.target_id});
}
}
which should have emitted sequence of source and target id, then I pass it to the list function with include_docs=true, assumes that source and target come in pairs stitches everything back in a structure like this:
{
"nodes":[
{"nodeName":"Name 1","group":"1"},
{"nodeName":"Name 2","group":"1"},
],
"links": [
{"source":7,"target":0,"value":1},
{"source":7,"target":5,"value":1}
]
}
Although my list produce a proper JSON, view map returns number of rows of source docs and then target docs.
So far I don't have any ideas how to make this thing working properly - I am happy to fetch additional values from document _id in the list, but so far I havn't find any good examples.
Alternative ways of achieving the same goal are welcome. _id values are standard for CouchDB so far.
Update: while writing a question I came up with different view which sorted my immediate problem, but I still would like to see other options.
updated map:
function(doc) {
if(doc.type == 'connection') {
if (doc.source_id)
emit([doc._id,0,"source"], {'_id': doc.source_id});
if(doc.target_id)
emit([doc._id,1,"target"], {'_id': doc.target_id});
}
}
Your updated map function makes more sense. However, you don't need 0 and 1 in your key since you have already "source"and "target".