I am using version fluent-plugin-kafka version 0.12.3.
I have an application that outputs its logs in JSON format, but my console consumer shows that the logs are escaped.
for instance, the application output the following log line:
{
"msg": "ok"
}
when i look at the log using the console consumer, it outputs:
{
"container_id": "7e...",
"container_name": "/app",
"source": "stdout",
"log": "{\"msg\": \"ok\"}"
}
how should fluent-plugin-kafka should be configured so the application log will be nested as a json without escaping it?
desired output:
{
"container_id": "7e...",
"container_name": "/app",
"source": "stdout",
"log": {
"msg": "ok"
}
}
using record transformer filter i was able to mutate the incoming data.
Related
I'm trying to make a Git push request to our Azure Devops server via the API. The address is https://MYSITE.visualstudio.com/MYPROJECT/_apis/git/repositories/2b34d4f7-2c1f-42e7-8861-u0ba34f72b40/pushes?api-version=5.1 and the body is as follows:
{
"commits": [
{
"comment": "Just a dummy commit",
"changes": [
{
"changeType": "edit",
"item": {
"path": "/src/MYPROJECT/MYPROJECT.csproj"
},
"newContent": {
"content": "beans",
"contentType": "rawText"
}
}
]
}
],
"refUpdates": [
{
"name": "refs/heads/TestDummyPRs/upgradeProjectToLatest",
"oldObjectId": "058da4f3328cb1048cb43faf3b5158bc3b025615"
}
]
}
I'm getting the following error:
Web Request Failed after 4 attempts. Request: https://MYSITE.visualstudio.com/MYPROJECT/_apis/git/repositories/2b34d4f7-2c1f-42e7-8861-u0ba34f72b40/pushes?api-version=5.1. Status: BadRequest. Response: Invalid status code [BadRequest]. Response: {"$id":"1","innerException":null,"message":"The parameters are incorrect. A posted push must contain exactly one commit and one refUpdate.\r\nParameter name: newPush","typeName":"Microsoft.TeamFoundation.SourceControl.WebServer.InvalidArgumentValueException, Microsoft.TeamFoundation.SourceControl.WebServer","typeKey":"InvalidArgumentValueException","errorCode":0,"eventId":0}
"A posted push must contain exactly one commit and one refUpdate" doesn't seem entirely reasonable as that's exactly what I have in my body. Does anybody know what might be going on here?
Note that I am having no issues making other web requests, such as creating branches or retrieving file contents.
I expected my web request to proceed smoothly, and to create a Push containing the specified commit to the specified refUpdate.
I have made a manual push for via the Azure Devops web interface and caught the network traffic, and I grabbed the following JSON request out of it:
{
"commits": [
{
"changes": [
{
"changeType": 2,
"item": {
"path": "/src/MYPROJECT/MYPROJECT.csproj"
},
"newContent": {
"content": "beans",
"contentType": 0
}
}
],
"comment": "Just a dummy commit"
}
],
"refUpdates": [
{
"name": ""refs/heads/TestDummyPRs/upgradeProjectToLatest",
"oldObjectId": "058da4f3328cb1048cb43faf3b5158bc3b025615"
}
]
}
This seems to be meaningfully identical to the Push I'm making from my code, other than the enum fields using numerical values instead of text. I have tried my code with numerical values for enums, but that didn't change anything about the error.
I found the issue. The web request from my application was being sent with UTF-16 encoding, whereas Postman had defaulted to UTF-8 encoding. I changed my application to use UTF-8 and it worked.
I have an rails app and I'm trying to configure logging to graylog. Pipeline consists of next steps:
1) Logs are written to file in JSON format by SemanticLogger gem. Log message consist of header info (first level tags) and payload with several levels of hierarchy:
{
"tag": "mortgage",
"app": "sneakers",
"pid": 3448,
"env": "production",
"host": "thesaurus-mortgage",
"thread": "91090300",
"level": "info",
"name": "Sneakers",
"payload": {
"class": "EgrnListenerWorker",
"method": "work",
"json": {
"resource": "kontur",
"action": "request_egrn_done",
"system_code": "thesaurus",
"id": 35883717,
"project_id": "mortgage",
"bank_id": "ab",
"params": {
"egrn": {
"zip": "rosreestr/kontur/kontur_4288_2018-10-11_021848.zip",
"pdf": "rosreestr/kontur/kontur_4288_2018-10-11_021848.pdf",
"xml": "rosreestr/kontur/kontur_4288_2018-10-11_021848.xml"
},
"code": "SUCCESS"
}
},
"valid_json": true
},
"created_at": "2018-10-11T17:44:58.262+00:00"
}
2) File is being read by Filebeat service and sent to Graylog.
And graylog could not parse correctly payload contents:
As you can see - keys are concatenated with ":" in one string in such manner: key1=value1:key2=value2. This is not what I am expected. It would be perfect if I could manage graylog to parse contents of payload into different fields with names payload.key1, payload.key2 and so on (so I could perform search on these fields)
ps: my log data is heterogeneous, i.e. payload contents depend on functionality it was produced by, so I expect that there would be a huge amount of different fields of a kind "payload.xxxxx" - is it ok?
This isn't exactly a filebeat question since filebeat only ships the logs in their original JSON format (zipped, if wanted).
From the Graylog Website: http://docs.graylog.org/en/2.4/pages/extractors.html
Using the JSON extractor
Since version 1.2, Graylog also supports extracting data from messages sent in JSON format.
Using the JSON extractor is easy: once a Graylog input receives
messages in JSON format, you can create an extractor by going to
System -> Inputs and clicking on the Manage extractors button for that
input. Next, you need to load a message to extract data from, and
select the field containing the JSON document. The following page let
you add some extra information to tell Graylog how it should extract
the information.
This should get you going.
I've upgraded to Cloud Endpoints 2.0 which no longer supports RPC. Therefore, I generated a new discovery document and used the service generator with the REST discovery doc as input in order to generate the client library for my iOS app.
Using the new REST discovery doc I am getting the following error when trying to generate the library:
~/workspace/google-api-objectivec-client-for-rest/Source/Tools/ServiceGenerator/build/Release/ServiceGenerator discovery/servUsApi-v1-rest.discovery --outputDir GTLAPI --gtlrFrameworkName GoogleAPIClientForREST
ERROR: Failure, exception: Looking at parameter 'creditKickbackKash:creditAmount', found a type/format pair of 'number/(null)', and don't how to map that to Objective-C
I was able to manually fix this by adding (in numerous places) in the discovery doc, the "format": "double" key and value for all double parameters. Notice creditAmount below is missing a format, like all other doubles.
The generated discovery doc looks like this:
"creditKickbackKash": {
"httpMethod": "PUT",
"id": "servUsApi.admin.creditKickbackKash",
"parameterOrder": [
"userId",
"creditAmount"
],
"parameters": {
"userId": {
"format": "int64",
"location": "path",
"required": true,
"type": "string"
},
"creditAmount": {
"location": "path",
"required": true,
"type": "number"
}
},
"path": "creditKickbackKash/{userId}/{creditAmount}",
"response": {
"$ref": "ResultDTO"
},
"scopes": [
"https://www.googleapis.com/auth/userinfo.email"
]
}
Is anyone else having this issue? How can I get the discovery document generation to properly format the document including double number types?
I had the same problem. I rolled back from 1.9.50 to 1.9.48 and the problem is gone.
I have an existing rails3 application with SendGrid Service integration based on SMTP relay, and subscribed to Event Notification Webhooks.
I've got reminder email from Service, informing about impending change to SendGrid's service, and I need make some changes to match new version of Event Notification Webhooks v3.
Email I got contains instructions about How To Upgrade:
Review the documentation to understand how to consume this new version's events.
Implement changes on your side to handle the new in-formatting of event data.
Make changes on SendGrid's configuration page
I wounder, what changes now need I implement on my site? I read manual for v3, but there no information about specifications differences between old and new versions.
Previously, the SendGrid event webhook sent improper batched JSON, with documents seperated by line breaks:
{ "email": "john.doe#sendgrid.com", "timestamp": 1337197600, "smtp-id": "<4FB4041F.6080505#sendgrid.com>", "event": "processed" }
{ "email": "john.doe#sendgrid.com", "timestamp": 1337966815, "category": "newuser", "event": "click", "url": "http://sendgrid.com" }
{ "email": "john.doe#sendgrid.com", "timestamp": 1337969592, "smtp-id": "<20120525181309.C1A9B40405B3#Example-Mac.local>", "event": "processed" }
Today, with v3 it sends proper JSON as such:
[
{
"email": "john.doe#sendgrid.com",
"timestamp": 1337197600,
"smtp-id": "<4FB4041F.6080505#sendgrid.com>",
"event": "processed"
},
{
"email": "john.doe#sendgrid.com",
"timestamp": 1337966815,
"category": "newuser",
"event": "click",
"url": "http://sendgrid.com"
},
{
"email": "john.doe#sendgrid.com",
"timestamp": 1337969592,
"smtp-id": "<20120525181309.C1A9B40405B3#Example-Mac.local>",
"event": "processed"
}
]
If you'd previously been dealing with non-batched events you'll need to adjust your code so that it interprets an array, rather than just a single document, and then loops through said array. Put into code:
# This
consume_data(params)
# Becomes this
params.each { |doc|
consume_data(doc)
}
I have the following cURL cmd running which returns all tasks for a specific userID within a specific workspaceID, but the "completed=false" part of the cmd doesn't influence the data returned. I am trying to exclude all "completed" tasks. Everything else is working as I expected. Below is the current cmd string and a snippet of the data returned back. As you can see one of the returned tasks indicates "completed" is true while the other indicates false.
Anyone see what I may be missing? Thanks in advance for your help.
cURL cmd:
curl -u <api-key>: "https://app.asana.com/api/1.0/tasks?workspace=<workspace>&assignee=<teamMemberId>&completed=false&opt_fields=name,assignee,assignee_status,notes,created_at,modified_at,completed_at,due_on,completed&opt_pretty"
output (partial)
{
"data": [
{
"id": <ID>,
"created_at": "2014-03-11T03:34:52.002Z",
"modified_at": "2014-03-11T23:33:07.544Z",
"name": "<nameoftask>",
"notes": "",
"assignee": {
"id": <ID>
},
"completed": true,
"assignee_status": "upcoming",
"completed_at": "2014-03-11T23:33:06.729Z",
"due_on": "2014-03-11"
},
{
"id": <ID>,
"created_at": "2014-03-11T23:33:07.196Z",
"modified_at": "2014-03-11T23:33:07.196Z",
"name": "<nameoftask>",
"notes": "",
"assignee": {
"id": ID>
},
"completed": false,
"assignee_status": "upcoming",
"completed_at": null,
"due_on": "2014-03-14"
},
TL;DR: You want completed_since=now, not completed=false.
The GET /tasks endpoint doesn't take a parameter completed - it does however take completed_since, which can take a timestamp (as provided by JS's .toISOString()) or the string "now". It always includes incomplete tasks. This is documented in the API Docs on Tasks (specifically the section on "Querying for tasks").
We're looking to improve the way you can filter tasks, but for right now what's documented there is all there is.