Datadog Grok Parsing - extracting array attribute from nested JSON - parsing

Using Datadog Parsing rule, I am trying to extract an attribute which is present inside an array.
Sample I've been work on:
{"messageHeader":{"serviceId":"aad1","requestType":"MM"},"data":{"code":"400","reason":"Bad Request","message":[{"responseCode":"E111","description":"Unable to reserve request type"}]}}
what I wanted to achieve is:
{
"description": "Unable to reserve request type",
"responseCode": "E111"
}
I tried using this method - https://stackoverflow.com/a/62096791/8552537
but in log explorer I'm not getting the result.
Following the method explained in given link. First Grok parser:-
parsing_rule1 %{data::json}
Output:
{
"data": {
"reason": "Bad Request",
"code": "400",
"message": [
{
"description": "Unable to reserve request type",
"responseCode": "E111"
}
]
},
"messageHeader": {
"requestType": "MM",
"serviceId": "aad1"
}
}
To create a custom log-based metrics, First we need to create a facet for responseCode but facet type doesn't support an array.
Now the goal is to pull out the details from that message field. With Grok parsers you can specify a specific attribute to parse further.
So in that same pipeline I've added another grok parser processor, right after our first.
And then configured the advanced options section to run on data.message, since that is what we called the attribute.
helper grok parser
Result:
helper grok parser result

Related

Filter event objects by body content

I'm trying to query calendar events using the Graph API and I would like to filter them by category and body content.
The goal is to get any event object that is in category "Test" and contains the string "FooBar" in its body content.
I tried to query the Graph API with the following request:
https://graph.microsoft.com/v1.0/me/events?$filter=(categories/any(x:x eq 'Test') and contains(body/content, 'FooBar'))
The response is a 500 error message:
{
"error": {
"code": "ErrorInternalServerError",
"message": "An internal server error occurred. The operation failed.",
//...
}
}
The "categories" filter clause works fine on its own but as soon as I put the "body/content" clause back in I get the aforementioned error response.
The json object I get from the Graph API looks like this (stripped down for better readability)
{
"value": [
{
"categories": [
"Test"
],
"bodyPreview": "FooBar",
"body": {
"contentType": "html",
"content": "<html><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\"></head><body><div class=\"BodyFragment\"></div><div class=\"BodyFragment\"><font size=\"2\"><span style=\"font-size:11pt\"><div class=\"PlainText\">FooBar</div></span></font></div></body></html>"
}
}
]
}
Is there something wrong with my filter clause or might this be an issue with Graph API itself?
I first tried to filter by bodyPreview but the error response I got clearly said that this field cannot be filtered by so I guess body/content should be possible.
The filtering by body/content is not supported for events.
I've tried similar query for messages and it works fine, so your filter clause is correct.
GET https://graph.microsoft.com/v1.0/me/messages?$filter=(categories/any(x:x eq 'Test') and contains(body/content, 'FooBar'))

How could I manage Graylog to parse my JSON logs correctly?

I have an rails app and I'm trying to configure logging to graylog. Pipeline consists of next steps:
1) Logs are written to file in JSON format by SemanticLogger gem. Log message consist of header info (first level tags) and payload with several levels of hierarchy:
{
"tag": "mortgage",
"app": "sneakers",
"pid": 3448,
"env": "production",
"host": "thesaurus-mortgage",
"thread": "91090300",
"level": "info",
"name": "Sneakers",
"payload": {
"class": "EgrnListenerWorker",
"method": "work",
"json": {
"resource": "kontur",
"action": "request_egrn_done",
"system_code": "thesaurus",
"id": 35883717,
"project_id": "mortgage",
"bank_id": "ab",
"params": {
"egrn": {
"zip": "rosreestr/kontur/kontur_4288_2018-10-11_021848.zip",
"pdf": "rosreestr/kontur/kontur_4288_2018-10-11_021848.pdf",
"xml": "rosreestr/kontur/kontur_4288_2018-10-11_021848.xml"
},
"code": "SUCCESS"
}
},
"valid_json": true
},
"created_at": "2018-10-11T17:44:58.262+00:00"
}
2) File is being read by Filebeat service and sent to Graylog.
And graylog could not parse correctly payload contents:
As you can see - keys are concatenated with ":" in one string in such manner: key1=value1:key2=value2. This is not what I am expected. It would be perfect if I could manage graylog to parse contents of payload into different fields with names payload.key1, payload.key2 and so on (so I could perform search on these fields)
ps: my log data is heterogeneous, i.e. payload contents depend on functionality it was produced by, so I expect that there would be a huge amount of different fields of a kind "payload.xxxxx" - is it ok?
This isn't exactly a filebeat question since filebeat only ships the logs in their original JSON format (zipped, if wanted).
From the Graylog Website: http://docs.graylog.org/en/2.4/pages/extractors.html
Using the JSON extractor
Since version 1.2, Graylog also supports extracting data from messages sent in JSON format.
Using the JSON extractor is easy: once a Graylog input receives
messages in JSON format, you can create an extractor by going to
System -> Inputs and clicking on the Manage extractors button for that
input. Next, you need to load a message to extract data from, and
select the field containing the JSON document. The following page let
you add some extra information to tell Graylog how it should extract
the information.
This should get you going.

AWS Lex Python Codehook references

I am still pretty new to Python (and coding) but I am trying to build my own Lex bot with a Lambda function. I have been following the tutorials and I can understand how it all works. Problem is when I am trying to write my own Lambda functions for Lex, I cannot find any references to help me write my code for example looking at the code below.
def get_slots(intent_request):
return intent_request['currentIntent']['slots']
What is "(intent_request)" and where would I find reference to this? Same for "['currentIntent'], how can I find out what it is and why it is there??
Sorry if this seems stupid to most people on here but I can't start writing code and continue to learn if I can't find any documentation to suggest what these are and why they are needed in order to write code for my own Lex bots.
Thanks in advance!!!
The intent_request is the incoming "request" or "event" from Lex to your Lambda Function. It holds all the necessary information about the user's input and your Lex bot's processing of that input (trigger certain intent, fill certain slots, confirmations, etc.)
This should be the documentation you are looking for.
Lambda Function Input Event and Response Format:
This section describes the structure of the event data that Amazon Lex provides to a Lambda function. Use this information to parse the input in your Lambda code. It also explains the format of the response that Amazon Lex expects your Lambda function to return.
And here is the Event/Request format:
{
"currentIntent": {
"name": "intent-name",
"slots": {
"slot name": "value",
"slot name": "value"
},
"slotDetails": {
"slot name": {
"resolutions" : [
{ "value": "resolved value" },
{ "value": "resolved value" }
],
"originalValue": "original text"
},
"slot name": {
"resolutions" : [
{ "value": "resolved value" },
{ "value": "resolved value" }
],
"originalValue": "original text"
}
},
"confirmationStatus": "None, Confirmed, or Denied (intent confirmation, if configured)"
},
"bot": {
"name": "bot name",
"alias": "bot alias",
"version": "bot version"
},
"userId": "User ID specified in the POST request to Amazon Lex.",
"inputTranscript": "Text used to process the request",
"invocationSource": "FulfillmentCodeHook or DialogCodeHook",
"outputDialogMode": "Text or Voice, based on ContentType request header in runtime API request",
"messageVersion": "1.0",
"sessionAttributes": {
"key": "value",
"key": "value"
},
"requestAttributes": {
"key": "value",
"key": "value"
}
}
The slots data is found inside currentIntent and that is inside of this whole intent_request object. That is why you are seeing the code: intent_request['currentIntent']['slots']
So to get session attributes you can find them here: intent_request['sessionAttributes']
Also extremely useful is the exact user input text:
intent_request['inputTranscript']

Unable to filter messages by recipient in Microsoft Graph Api. One or more invalid nodes

I am trying to get a list of messages that are filtered by recipient from Microsoft Graph API. The url I am using for the request is:
https://graph.microsoft.com/beta/me/messages?$filter=toRecipients/any(r: r/emailAddress/address eq '[Email Address]')
But I am getting this is the response:
{
"error": {
"code": "ErrorInvalidUrlQueryFilter",
"message": "The query filter contains one or more invalid nodes.",
"innerError": {
"request-id": "7db712c3-e337-49d9-aa8d-4a5d350d8480",
"date": "2016-09-28T16:58:34"
}
}
}
A successful request should look like this (with a lot more data that I have omitted).
{
"#odata.context": "https://graph.microsoft.com/beta/$metadata#users('99999999-9999-9999-9999-999999999999')/messages",
"#odata.nextLink": "https://graph.microsoft.com/beta/me/messages?$skip=10",
"value": [
{
"toRecipients": [
{
"emailAddress": {
"name": "[Name]",
"address": "[Email Address]"
}
}
],
}
]
}
The request works if I remove the filter, and I am able to perform requests with simpler filters.
Is there a problem with my URL, or is there another way to make the request?
Another way to make the request might be to not use filter and use search instead, depending on exactly what you want (and you may have already tried this):
https://graph.microsoft.com/beta/me/messages?$search="to:[Email Address]"
After several hours looking for the solution, I found in the office365 documentation that the property toRecipients is not filterable:
https://msdn.microsoft.com/en-us/office/office365/api/complex-types-for-mail-contacts-calendar#MessageResource
I guess that it's the same in the graph api. So the only solution is using search.

"additional properties" error in Survey Monkey API call

I'm trying to use the create_flow endpoint to the Survey Monkey API. It is sending back a status 3 message with the following error:
additional properties not defined by 'properties' are not allowed in field '_data'
I'm able to do successfully use all other API endpoints and have a valid API key and durable OAuth token.
Here's an example JSON body that I'm sending to: https://api.surveymonkey.net/v2/batch/create_flow?api_key=apikeyhere
{
"survey": {
"template_id": "566",
"survey_title": "test1",
"collector": {
"type": "email",
"name": "collector1",
"recipients": [
{
"email": "email#example.com"
}
]
},
"email_message": {
"reply_email": "myemail#example.com",
"subject": "this is a test"
}
}
Note: JSON formatting here is being generated automatically using RJSONIO
Any ideas what might be causing the error? It seems like all fields are correctly named and where they're supposed to be, so I'm not sure what the problem is.
It's a bad error message unfortunately - it's a known issue. It means you are providing extra keys that are not part of the create_flow schema.
The issue here is that the "email_message" and "collector" keys have been nested inside of "survey", instead of being in the main JSON body like the "survey" key. Move them out a level and it should work.

Resources