I have a file having json records and want to remove some keys form json records before sending them to fluentd output. I can use 'record_transformer Filter Plugin' to remove keys but it removes only top level keys from json. How can I use nested keys.
Sample input:
{
"key1": 1,
"key2": 2,
"key3": {
"nested_key1": 1,
"nested_key2": 1,
"nested_key3": 1,
"nested_key4": {
"double_nested_key1": 1,
"double_nested_key2": 2
}
}
}
Output sample:
{
"key1": 1,
"key2": 2,
"key3": {
"nested_key1": 1,
"nested_key2": 1,
"nested_key4": {
"double_nested_key2": 2
}
}
}
https://docs.fluentd.org/v1.0/articles/api-plugin-helper-record_accessor#syntax
<filter fluentd-log.**>
#type record_transformer
remove_keys a, $.b.c, $.b.c.d
</filter>
Ever since version 1.1 there is a new syntax to delete nested:
remove_keys $.jsonPayload.msg
which will delete record[jsonPayload][msg]
According to this, you can use something like:
<record>
for_remove ${record["docker"].delete("container_id")}
</record>
remove_keys for_remove
for structure like:
"docker": {
"container_id": "c1853368630a300cac78890a8599c77cea23a3fe57961e4080722c74a3ba7430"
},
as result you will get:
"docker": {},
any of response didn't work for me.
remove-keys delete all if not nested!
Related
I am trying to contain array as a document value.
I succeed it in "tags" field as below;
This document contains array of strings.
curl -XGET localhost:9200/MY_INDEX/_doc/132328908
#=> {
"_index":"MY_INDEX",
"_type":"_doc",
"_id":"132328908",
"found":true,
"_source": {
"tags": ["food"]
}
}
However, when I am putting items in the same way as above,
the document is SOMETIMES like that;
curl -XGET localhost:9200/MY_INDEX/_doc/328098989
#=> {
"_index":"MY_INDEX",
"_type":"_doc",
"_id":"328098989",
"found":true,
"_source": {
"tags": "[\"food\"]"
}
}
This is string of array, not array of strings, which I expected.
"tags": "[\"food\"]"
It seems that this situation happens randomly and I could not predict it.
How could it happen?
Note:
・I use elasticsearch-ruby client to index a document.
This is my actual code;
es_client = Elasticsearch::Client.new url: MY_ENDPOINT
es_client.index(
index: MY_INDEX,
id: random_id, # defined elsewhere
body: {
doc: {
"tags": ["food"]
},
}
)
Thank you in advance.
Restassured: How Can we compare each element in Json array to one particular Same value in Java using Hemcrest Matchers, not using Foreach loop.
{
"id": 52352,
"name": "Great Apartments",
"floorplans": [
{
"id": 5342622,
"name": "THE STUDIO",
"fpCustomAmenities": [
{
"displaySequence": 2,
"amenityPartnerId": "gadasd",
"display": true,
"leased": true
},
{
"displaySequence": 13,
"amenityPartnerId": "sdfsfd",
"display": true,
"leased": true
}
]
},
{
"id": 4321020,
"name": "THE First Bed",
"fpCustomAmenities": [
{
"displaySequence": 4,
"amenityPartnerId": "gadasd",
"display": true,
"leased": true
},
{
"displaySequence": 15,
"amenityPartnerId": "hsfdsdf",
"display": true,
"leased": true
}
]
}
]
}
I want to compare that Leased=true for all the leased nodes at all the levels in the json response...
I have working code...
List<List<Boolean>> displayedvaluesfpStandardAmenities =
when().get(baseUrl + restUrl).
then().statusCode(200).log().ifError().
extract().body().jsonPath().getList("floorplans.fpCustomAmenities.display");
for (List<Boolean> displayedStandardList : displayedvaluesfpStandardAmenities) {
for (Boolean isDisplayedTrue : displayedStandardList) {
softAssert.assertTrue(isDisplayedTrue);
}
}
But the issue is I need the code to be in simple format using either Hemcrest Matchers or Restaussred Matchers and try simplistic way like Below, ( which is not working)
when().get(baseUrl + restUrl).
then().assertThat().body("floorplans.fpCustomAmenities.display",equalTo("true"));
The error I am getting is
java.lang.AssertionError: 1 expectation failed.
JSON path floorplans.fpCustomAmenities.display doesn't match.
Expected: true
Actual: <[[true, true], [true, true]]>
So what I need is the that all thes 'display' nodes in the json response where ever it is need to compared with "true", so that my test can Pass.
I have an alternate solution like mentioned above, but All I need is working solution using matchers.
Assuming fpCustomAmenities arrays are not empty, you can use the following solution;
when().get(baseUrl + restUrl).then()
.body("floorplans.findAll { it }.fpCustomAmenities" + // 1st line
".findAll { it }.leased.each{ a -> println a }" + // 2nd line
".grep{ it.contains(false) }.size()", equalTo(0)); // 3rd line
Here from the 1st line, we return each object in fpCustomAmenities array.
From the 2nd line we get boolean value of leased in each fpCustomAmenities object to a boolean array ([true, true]).
Each boolean array is printed from .each{ a -> println a }. I added it only to explain the answer. It is not relevant to the solution.
From 3rd line we check whether, if there is a false in each boolean array. grep() will return only the arrays which has a false. And then we get the filtered array count. Then we check whether it is equal to 0.
Check groovy documentation for more details.
Or
This solution does not use any Matchers. But this works.
String responseBody = when().get(baseUrl + restUrl).
then().extract().response().getBody().asPrettyString();
Assert.assertFalse(responseBody.contains("\"leased\": false"));
I am trying to get only the matched data from nested array of elastic search class. but I am not able to get it..the whole nested array data is being returned as output.
this is my Query:-
QueryBuilders.nestedQuery("questions",
QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("questions.questionTypeId", quesTypeId)), ScoreMode.None)
.innerHit(new InnerHitBuilder());
I am using querybuilders to get data from nested class.Its working fine but not able to get only the matched data.
Request Body :
{
"questionTypeId" : "MCMC"
}
when questionTypeId = "MCMC"
this is the output i am getting..Here I want to exclude the output for which the questionTypeId = "SCMC".
output :
{
"id": "46",
"subjectId": 1,
"topicId": 1,
"subtopicId": 1,
"languageId": 1,
"difficultyId": 4,
"isConceptual": false,
"examCatId": 3,
"examId": 1,
"usedIn": 1,
"questions": [
{
"id": "46_31",
"pid": 31,
"questionId": "QID41336691",
"childId": "CID1",
"questionTypeId": "MCMC",
"instruction": "This is a single correct multiple choice question.",
"question": "Who holds the most english premier league titles?",
"solution": "Manchester United",
"status": 1000,
"questionTranslation": []
},
{
"id": "46_33",
"pid": 33,
"questionId": "QID41336677",
"childId": "CID1",
"questionTypeId": "SCMC",
"instruction": "This is a single correct multiple choice question.",
"question": "Who holds the most english premier league titles?",
"solution": "Manchester United",
"status": 1000,
"questionTranslation": []
}
]
}
As you have tagged this with spring-data-elasticsearch:
Support to return inner hits was recently added to version 4.1.M1 and so will be included in the next released version. Then in a SearchHit you will get the complete top level document, but in the innerHits property only the matching inner hits will be returned.
I have this relatively complex search query that's already being built and working with perfect sorting.
But I think here searching is slow just because of script so all I want to remove script and write query accordingly.
current code :-
"sort": [
{
"_script": {
"type": "number",
"script": {
"lang": "painless",
"source": "double pscore = 0;for(id in params.boost_ids){if(params._source.midoffice_master_id == id){pscore = -999999999;}}return pscore;",
"params": {
"boost_ids": [
3,
4,
5
]
}
}
}
}]
Above code explaination:-
For example, if a match query would give a result like:
[{m_id: 1, name: A}, {m_id: 2, name: B}, {m_id: 3, name: C}, {m_id: 4, name: D}, ...]
So I want to boost document with m_id array [3, 4, 5] which would then transform the result into:
[{m_id: 3, name: C}, {m_id: 4, name: D}, {m_id: 1, name: A}, {m_id: 2, name: B}, ...]
You can make use of the below query using Function Score Query(for boosting) and Terms Query (used to query array of values)
Note that the logic I've mentioned is in the should clause of the bool query.
POST <your_index_name>/_search
{
"query": {
"bool": {
"must": [
{
"match_all": {} //just a sample must clause to retrieve all docs
}
],
"should": [
{
"function_score": { <---- Function Score Query
"query": {
"terms": { <---- Terms Query
"m_id": [
3,4,5
]
}
},
"boost": 100 <---- Boosting value
}
}
]
}
}
}
So basically, you can remove the sort logic completely and add the above function query in your should clause, which would give you the results in the order you are looking for.
Note that you'd have to find a way to add the logic correctly in case if you have much complex query, and if you are struggling with anything, do let me know. I'd be happy to help!!
Hope this helps!
I'm facing an issue importing a JSON file using the apoc.load.json procedure.
The expected relationship I'm trying to capture:
University --child--> Class --child--> Student
Output:
Neo.ClientError.Statement.SyntaxError: Variable `value` not defined (line 1, column 8 (offset: 7))
"UNWIND value.university AS university"
Here is the performed command sequence:
CALL apoc.load.json("FILE:///C:/tmp/input.json") YIELD value
UNWIND value.university AS university
UNWIND university.class AS class
UNWIND class.student AS student
MERGE (u:UniversityCategory {name:university.name})
MERGE (c:ClassCategory {name:class.name})
MERGE (s:StudentCategory {instr:student.name})
ON CREATE SET i.ID = instructions.ID
ON CREATE SET i.GPA = instructions.GPA
MERGE (u)-[:CHILD]->(c)
MERGE (c)-[:CHILD]->(s)
Here is the JSON file structure:
{
"university": [{
"name": "universityA",
"class": [{
"name": "class_1",
"student": [{
"name": "student_1",
"ID": "1234",
"GPA": "3.8"
},
{
"name": "student_2",
"ID": "12345",
"GPA": "3.4"
}
]
},
{
"name": "class_2",
"student": [{
"name": "student_3",
"ID": "14",
"GPA": "3.0"
}]
}
]
}]
}
My apoc.load.json command appears to work because I see the structured JSON file in the browser window. The next steps are suspect but I think I'm close to defining the relationships.
Resolved my issue.
The commands I expressed above are all correct but the load.apoc.json command has to be run in the SAME query as the rest versus sequentially.