Fluentd JSON treat the whole subfields as string to have them in one field - fluentd

I have a JSON with several fields, I want to tell to Fluentd to treat "data" and whole their subfields as one string field and to avoid Elasticsearch to index them
I need to handle the events in Fluentd instead of Elasticsearch
Sample JSON
{
"timestamp": "2022-04-22T00:00:05+02:00",
"type": "example",
"info": {
"severity": "info",
"ip": "8.8.8.8",
"by_user": 3,
"host": "hostname.net",
"method": "unknown"
},
"data": {
"action": "update",
"type": "data_type",
"msg": "kindly message",
"extra": {
"config1": {
"more_fields": false
},
"config2": {
"more_fields": false
}
}
}
}
Field to ingest and index:
"timestamp", "type", "info" (all subfields), "data" ("action", "type", "msg") and treat as string the fields below "data" ("extra" and all their subfields)
I appreciate the help you can provide to make this works

Related

Swagger API Special character issue

I am new in swagger and want to deploy an API which is having query string. This is the API I am getting encoded URL after passing the parameter in the GET method.
API URL with Parameter should be:-
baseurl/v1/auth/getOTP?email=somename#email.com
but I am getting something like:-
baseurl/v1/auth/getOTP?email=somename%40email.com
what I have tried is:-
"/auth/getOTP": {
"get": {
"tags": [
"pet"
],
"summary": "",
"description": "",
"operationId": "findPetsByStatus",
"produces": [
"application/json",
"application/xml"
],
"parameters": [
{
"name": "email",
"in": "path",
"description": "",
"required": true,
"type": "string",
}
],
"responses": {
"200": {
"description": "successful operation",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/Pet"
}
}
},
"400": {
"description": "Invalid value"
}
},
"security": [
{
"petstore_auth": [
"write:pets",
"read:pets"
]
}
]
}
},
Swagger OpenAPI has Specified: in this GitHub Issue-1840, It is specifically disallowed on in: path parameters for the same reason they don't allow optional path parameters, e.g. {/foo} By having variable path segments it become difficult to resolve a URL to an operation.
If you want to use some kind of hack then follow this GitHub Issue-230.
If you really needed then Its support in in: query parameters, as below,
The allowReserved keyword specifies whether the reserved characters :/?#[]#!$&'()*+,;= in parameter values are allowed to be sent as they are, or should be percent-encoded. By default, allowReserved is false, and reserved characters are percent-encoded.
Here you need to set it to true,
"parameters": [
{
"name": "email",
"in": "query",
"description": "",
"required": true,
"type": "string",
"allowReserved": true
}
],
Look at the Example Swagger Describing Parameters
and For more details follow Swagger Serialization.

How can I generate a sample request for swagger definition with additionalProperties?

I have this definition in a swagger document:
"/myEndpoint": {
"post": {
"summary": "handle",
"operationId": "handleUsingGET",
"produces": ["application/json", "application/vnd.spring-boot.actuator.v2+json"],
"parameters": [{
"in": "body",
"name": "body",
"description": "body",
"required": false,
"schema": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
],
"responses": { ... }
}
I need to create a request with a body but I'm not sure how to do that with this schema definition. I have read this question so I know that additionalProperties is used to represent a Dictionary. But there is no property name or anything specified here, so can we say any key-value pairs would be valid to sent in the body? Or is this definition not complete?

Storing a List in a Vertex using Gremlin and Azure Cosmos Graph

I am trying to store a list of changes made to a Vertex in the Vertex itself. Ideally I would want something like this:
{
"id": "95fcfa87-1c03-436d-b3ca-340cea926ee9",
"label": "person",
"type": "vertex",
"log": [{
"user": "user#user.dk",
"action": "update",
"timestamp": "22-03-2017",
"field": "firstName",
"oldValue": "Marco"
}
]
}
Using this method chain I am able to a achieve the following structure
graph.addV('person')
.property('firstName', 'Thomas')
.property(list, 'log', '22-03-2017')
.properties('log')
.hasValue('22-03-2017', '21-03-2017')
.property('user','user#user.dk')
.property('action', 'update')
.property('field', 'firstName')
.property('oldValue', 'Marco')
{
"id": "95fcfa87-1c03-436d-b3ca-340cea926ee9",
"label": "person",
"type": "vertex",
"properties": {
"firstName": [{
"id": "f23482a9-48bc-44e0-b783-3b74a2439a11",
"value": "Thomas"
}
],
"log": [{
"id": "5cfa35e1-e453-42e2-99b1-eb64cd853f22",
"value": "22-03-2017",
"properties": {
"user": "user#user.dk",
"action": "update",
"field": "firstName",
"oldValue": "Marco"
}
}
]
}
}
However this seems overly complex, as I will have to store a value and add properties to it.
Is it possible to add anonymous objects (i.e. without id and value) with the above mentioned data?
Not an actual solution to storing proper objects in a history log, but if you just use it as a log and don't have to access or query it by its properties, you could just put the serialised JSON in the value?
Something like along these lines should approximate the structure you're requesting:
dynamic entry = new JObject();
entry.user = "user#user.dk";
entry.action = "update";
entry.timestamp = "22-03-2017 12:34:56";
entry.field = "firstName";
entry.oldValue = "Marco";
graph.addV('person')
.property('firstName', 'Thomas')
.property(list, 'log', entry.ToString());
{
"id": "95fcfa87-1c03-436d-b3ca-340cea926ee9",
"label": "person",
"type": "vertex",
"properties": {
"firstName": [{
"id": "f23482a9-48bc-44e0-b783-3b74a2439a11",
"value": "Thomas"
}
],
"log": [{
"id": "5cfa35e1-e453-42e2-99b1-eb64cd853f22",
"value": "{\"user\":\"user#user.dk\",\"action\":\"update\",\"timestamp\":\"22-03-2017\",\"field\":\"firstName\",\"oldValue\":\"Marco\"}"
}
]
}
}
These log entries can easily be read, deserialised, used, and presented, but will not do much for queriability.

Breeze EntityManager cache not cleared after successful server save

I've been trying for a few days now to get the Breeze 1.4.9 to work with a rails back end in a different manner than the Breeze Ruby SPA sample. I would rather send bulk save changes instead of trying to send RESTful calls to the server on every entity change. To that end, I've written a rails controller/model methods that will parse out all the different entities in a Breeze SaveChanges POST and act accordingly. Everything works great except that the response to SaveChanges POST doesn't seem to satisfy all the checks for Breeze and EntityManager.hasChanges() is still true even after the response is processed successfully.
Here's a typical cycle:
Breeze requests my hand crafted metadata and parses it fine:
{
"metadataVersion": "1.0.5",
"namingConvention": "rubyNamingConvention",
"localQueryComparisonOptions": "caseInsensitiveSQL",
"dataServices": [
{
"serviceName": "breeze\/Breeze\/",
"hasServerMetadata": true,
"jsonResultsAdapter": "webApi_default",
"useJsonp": false
}
],
"structuralTypes": [
{
"shortName": "VarianceReason",
"namespace": "Icon",
"autoGeneratedKeyType": "Identity",
"defaultResourceName": "VarianceReasons",
"dataProperties": [
{
"name": "id",
"dataType": "Int32",
"isNullable": false,
"defaultValue": 0,
"isPartOfKey": true,
"validators": [
{
"name": "required"
},
{
"name": "int32"
}
]
},
{
"name": "name",
"dataType": "String",
"isNullable": false,
"defaultValue": "",
"maxLength": 256,
"validators": [
{
"name": "required"
},
{
"maxLength": 256,
"name": "maxLength"
}
]
},
{
"name": "createdAt",
"dataType": "DateTime",
"isNullable": false,
"defaultValue": "1900-01-01T08:00:00.000Z",
"validators": [
{
"name": "required"
},
{
"name": "date"
}
]
},
{
"name": "updatedAt",
"dataType": "DateTime",
"isNullable": false,
"defaultValue": "1900-01-01T08:00:00.000Z",
"validators": [
{
"name": "required"
},
{
"name": "date"
}
]
}
]
}
],
"resourceEntityTypeMap": {
"VarianceReasons": "VarianceReason:#Icon"
}
}
I make an entity change in Breeze and it POSTs the below to rails when I call em.SaveChanges():
{
"entities":[
{
"id":-1,
"name":"anyuthingasd",
"created_at":"1900-01-01T08:00:00.000Z",
"updated_at":"1900-01-01T08:00:00.000Z",
"entityAspect":{
"entityTypeName":"VarianceReason:#Icon",
"defaultResourceName":"VarianceReasons",
"entityState":"Added",
"originalValuesMap":{
},
"autoGeneratedKey":{
"propertyName":"id",
"autoGeneratedKeyType":"Identity"
}
}
}
],
"saveOptions":{
}
}
Rails then responds with:
{
"KeyMappings":[
{
"EntityTypeName":"VarianceReason:#Icon",
"TempValue":-1,
"RealValue":16
}
],
"Entities":[
{
"id":16,
"name":"anyuthingasd",
"created_at":"2014-05-02T14:21:24.221Z",
"updated_at":"2014-05-02T14:21:24.221Z",
"Entity":null
}
]
}
Breeze then merges in the new id key mapping but doesn't clear the cache, so next time I make another entity change it still has the first change which has already persisted to the server and the new change. Can anyone tell me what I'm not responding with from the rails side that makes Breeze EntityManager not satisfied? I'm trying to trace through the 15k lines of code but can't say I'm a JS ninja.
We really do need to show folks how to build a data service adapter for whatever service they've got.
In this case, it appears you chose to implement something like the SaveChanges method in C# on the Web API. In other words, you've chosen to emulate the out-of-the-box Breeze protocol. That's cool! And non-trivial too so kudos to you.
I think what's missing from the entity JSON in your save response is the EntityType name. Breeze can't find the corresponding cached entities without knowing their types and thus cannot update their change-states.
Again, because you've decided to use the default Web API data service adapter, you'll want to return a response that adapter expects. That adapter defines a "jsonResultsAdapter" that expects each JSON entity data object to have a $type property specifying the full type name (namespace.typename).
In your example, I think you'd want to return
...
"Entities":[
{
"$type": "Icon.VarianceReason",
"id":16,
"name":"anyuthingasd",
"created_at":"2014-05-02T14:21:24.221Z",
"updated_at":"2014-05-02T14:21:24.221Z",
}
]
How about an example?
I suspect that you may not have easy access to a server with Web API that can show you what a save response looks like with the default adapter. Therefore, I've pasted below a Todo app's saveChanges request and response for a change-set that includes a new, a modified, and a deleted TodoItem.
The Request
Below is the payload of the POST request to the "SaveChanges" endpoint. It is probably way more verbose than you need (more verbose than I'd need). Just to pick one example, the "autoGeneratedKey" is of no interest to the server whatsoever.
I'm just showing you what the default data service adapter sends. Someday you'll write your own to do it the way you want it. For now I suppose there is no harm in sending too much crappola ... as long as you're happy to ignore it on the Rails end :-)
{
"entities": [
{
"Id": 5,
"Description": "Cheese",
"CreatedAt": "2012-08-22T09:05:00.000Z",
"IsDone": true,
"IsArchived": false,
"entityAspect": {
"entityTypeName": "TodoItem:#Todo.Models",
"defaultResourceName": "Todos",
"entityState": "Deleted",
"originalValuesMap": {
},
"autoGeneratedKey": {
"propertyName": "Id",
"autoGeneratedKeyType": "Identity"
}
}
},
{
"Id": 6,
"Description": "Modified Todo",
"CreatedAt": "2012-08-22T09:06:00.000Z",
"IsDone": false,
"IsArchived": false,
"entityAspect": {
"entityTypeName": "TodoItem:#Todo.Models",
"defaultResourceName": "Todos",
"entityState": "Modified",
"originalValuesMap": {
"Description": "Wine"
},
"autoGeneratedKey": {
"propertyName": "Id",
"autoGeneratedKeyType": "Identity"
}
}
},
{
"Id": -1,
"Description": "New Todo",
"CreatedAt": "2014-05-02T17:34:00.904Z",
"IsDone": false,
"IsArchived": false,
"entityAspect": {
"entityTypeName": "TodoItem:#Todo.Models",
"defaultResourceName": "Todos",
"entityState": "Added",
"originalValuesMap": {
},
"autoGeneratedKey": {
"propertyName": "Id",
"autoGeneratedKeyType": "Identity"
}
}
}
],
"saveOptions": {
}
}
The Response
The $id property is a node counter. It's useful when you have repeated entities so you don't have to worry about cycles or repeated entity data in your payload (an object with a $ref property is the placeholder for the repeated entity). You can ignore $id if you don't need this feature (and you rarely would need it in a save result).
Notice that the $type is in the .NET "CSDL" type format "namespace.typename", not the Breeze type format "typename:#namespace". This is an artifact of the data service adapter's jsonResultsAdapter ... which you can change to better suit your Rails implementation. None of this is cast in stone. I'm just reporting what these adapters do as delivered.
You can ignore the assembly name (", Todo-Angular") in the $type value; Breeze doesn't care about it.
Notice that the deleted "Cheese" entity was returned with all of its contents. I bet you don't have to do that. You could get away with returning a stripped down version that simply lets the client know Rails got the message:
{
"$id": "2",
"$type": "Todo.Models.TodoItem, Todo-Angular",
"Id": 5
},
And now ... the complete JSON response body:
{
"$id": "1",
"$type": "Breeze.ContextProvider.SaveResult, Breeze.ContextProvider",
"Entities": [
{
"$id": "2",
"$type": "Todo.Models.TodoItem, Todo-Angular",
"Id": 5,
"Description": "Cheese",
"CreatedAt": "2012-08-22T09:05:00.000Z",
"IsDone": true,
"IsArchived": false
},
{
"$id": "3",
"$type": "Todo.Models.TodoItem, Todo-Angular",
"Id": 6,
"Description": "Modified Todo",
"CreatedAt": "2012-08-22T09:06:00.000Z",
"IsDone": false,
"IsArchived": false
},
{
"$id": "4",
"$type": "Todo.Models.TodoItem, Todo-Angular",
"Id": 7,
"Description": "New Todo",
"CreatedAt": "2014-05-02T17:34:00.904Z",
"IsDone": false,
"IsArchived": false
}
],
"KeyMappings": [
{
"$id": "5",
"$type": "Breeze.ContextProvider.KeyMapping, Breeze.ContextProvider",
"EntityTypeName": "Todo.Models.TodoItem",
"TempValue": -1,
"RealValue": 7
}
],
"Errors": null
}

How do I use the swagger models section?

Inside the Swagger API Documentation there is inside the json beside the apis array a model object entry but no documentation about it. How can I use this "models" part?
{
apiVersion: "0.2",
swaggerVersion: "1.1",
basePath: "http://petstore.swagger.wordnik.com/api",
resourcePath: "/pet.{format}"
...
apis: [...]
models: {...}
}
Models are nothing but like your POJO classes in java which have variables and properties. In models section you can define your own custom class and you can refer it as data type.
If you see below
{
"path": "/pet.{format}",
"description": "Operations about pets",
"operations": [
{
"httpMethod": "POST",
"summary": "Add a new pet to the store",
"responseClass": "void",
"nickname": "addPet",
"parameters": [
{
"description": "Pet object that needs to be added to the store",
"paramType": "body",
"required": true,
"allowMultiple": false,
"dataType": "Pet"
}
],
"errorResponses": [
{
"code": 405,
"reason": "Invalid input"
}
]
}
Here in parameter section it have one parameter who's dataType is Pet and pet is defined in models as below
{
"models": {
"Pet": {
"id": "Pet",
"properties": {
"id": {
"type": "long"
},
"status": {
"allowableValues": {
"valueType": "LIST",
"values": [
"available",
"pending",
"sold"
]
},
"description": "pet status in the store",
"type": "string"
},
"name": {
"type": "string"
},
"photoUrls": {
"items": {
"type": "string"
},
"type": "Array"
}
}
}
}}
You can have nested models , for more information see Swagger PetStore example
So models are nothing but like classes.

Resources