parsing JSON file using telegraf input plugin : unexpected Output - influxdb

I’m new to telegraf and influxdb, and currently looking forward to exploring telegraf, but unfortunetly, I have some difficulty getting started, I will try to explain my problem below:
Objectif: parsing JSON file using telegraf input plugin.
Input : https://wetransfer.com/downloads/0abf7c609d000a7c9300dc20ee0f565120200624164841/ab22bf ( JSON file used )
The input json file is a repetition of the same structure that starts from params and ends at it.
you find below the main part of the input file :
{
"events":[
{
"params":[
{
"name":"element_type",
"value":"Home_Menu"
},
{
"name":"element_id",
"value":""
},
{
"name":"uuid",
"value":"981CD435-E6BC-01E6-4FDC-B57B5CFA9824"
},
{
"name":"section_label",
"value":"HOME"
},
{
"name":"element_label",
"value":""
}
],
"libVersion":"4.2.5",
"context":{
"locale":"ro-RO",
"country":"RO",
"application_name":"spresso",
"application_version":"2.1.8",
"model":"iPhone11,8",
"os_version":"13.5",
"platform":"iOS",
"application_lang_market":"ro_RO",
"platform_width":"320",
"device_type":"mobile",
"platform_height":"480"
},
"date":"2020-05-31T09:38:55.087+03:00",
"ssid":"spresso",
"type":"MOBILEPAGELOAD",
"user":{
"anonymousid":"6BC6DC89-EEDA-4EB6-B6AD-A213A65941AF",
"userid":"2398839"
},
"reception_date":"2020-06-01T03:02:49.613Z",
"event_version":"v1"
}
Issue : Following the documentation, I tried to define a simple telegraf.conf file as below:
[[outputs.influxdb_v2]]
…
[[inputs.file]]
files = ["/home/mouhcine/json/file.json"]
json_name_key = "My_json"
#... Listing all the string fields in the json.(I put only these for simplicity reason).
json_string_fields = ["ssid","type","userid","name","value","country","model"]
data_format = "json"
json_query= "events"
Basically declaring string fields in the telegraf.conf file would do it, but I couldn’t get all the fields that are subset in the json file, like for example what’s inside ( params or context ).
So finally, I get to parse fields with the same level of hierarchy as ssid, type, libVersion, but not the ones inside ( params, context, user).
Output : Screen2 ( attachment ).
OUTPUT
By curiosity, I tried to test the documentation’s example, in order to verify whether I get the same expected result, and the answer is no :/, I don’t get to parse the string field in the subset of the file.
The doc’s example below:
Input :
{
"a": 5,
"b": {
"c": 6,
"my_field": "description"
},
"my_tag_1": "foo",
"name": "my_json"
}
telegraf.conf
[[outputs.influxdb_v2]]
…
[[inputs.file]]
files = ["/home/mouhcine/json/influx.json"]
json_name_key = "name"
tag_keys = ["my_tag_1"]
json_string_fields = ["my_field"]
data_format = "json"
Expected Output : my_json,my_tag_1=foo a=5,b_c=6,my_field="description"
The Result I get : "my_field" is missing.
Output: Screen 1 ( attachement ).
OUTPUT
By the way, I use the influxdb cloud 2, and I apologize for the long description of this little problem, I would appreciate some help please :), Thank you so much in advance.

Related

How to pass different values to Pipeline Parameters

suppose if i am doing hyper parameter tuning to one of my model, lets say, i am using AdaBoostClassifier() and want to pass different base_estimator, so i pass SVC & DecisionTreeClassifier as estimator
_parameters=[
{
'mdl':[AdaBoostClassifier(random_state=23)],
'mdl__learning_rate':np.linspace(0,1,20),
'mdl__base_estimator':[SVC(),DecisionTreeClassifier()]
}
]
now, i want to pass different values to ccp_alpha of DecisionTreeClassifier, something like this
'mdl__base_estimator':[LinearRegression(),DecisionTreeClassifier(ccp_alpha=[0.1,0.2,0.3,0.4])]
how can i do that, i tried passing it like this, but it is not working, here is my entire code
pipeline=Pipeline(
[
('scal',StandardScaler()),
('mdl','passthrough')
]
)
_parameters=[
{
'mdl':[DecisionTreeClassifier(random_state=42)] ,
'mdl__max_depth':np.linspace(2,30,2),
'mdl__min_samples_split':np.linspace(1,10,1),
'mdl__max_features':np.linspace(1,100,1),
'mdl__ccp_alpha':np.linspace(0,1,10)
}
,{
'mdl':[AdaBoostClassifier(random_state=23)],
'mdl__learning_rate':np.linspace(0,1,20),
'mdl__base_estimator':[SVC(),DecisionTreeClassifier(ccp_alpha=[0.3,0.4,0.5,0.7])]
}
]
grid_search=GridSearchCV(_pipeline,_parameters,cv=3,n_jobs=-1,scoring='f1')
grid_search.fit(x,y
)
This kind of splitting is why param_grid can be a list of dicts, as in your outer split; but it cannot easily handle the nested disjunction you have. Two approaches come to mind.
More disjoint grids:
_parameters=[
{
'mdl': [DecisionTreeClassifier(random_state=42)],
'mdl__max_depth': np.linspace(2,30,2),
'mdl__min_samples_split': np.linspace(1,10,1),
'mdl__max_features': np.linspace(1,100,1),
'mdl__ccp_alpha': np.linspace(0,1,10),
},
{
'mdl': [AdaBoostClassifier(random_state=23)],
'mdl__learning_rate': np.linspace(0,1,20),
'mdl__base_estimator': [SVC()],
},
{
'mdl': [AdaBoostClassifier(random_state=23)],
'mdl__learning_rate': np.linspace(0,1,20),
'mdl__base_estimator': [DecisionTreeClassifier()],
'mdl__base_estimator__ccp_alpha': [0.3,0.4,0.5,0.7],
},
]
Or list comprehension:
_parameters=[
{
'mdl': [DecisionTreeClassifier(random_state=42)],
'mdl__max_depth': np.linspace(2,30,2),
'mdl__min_samples_split': np.linspace(1,10,1),
'mdl__max_features': np.linspace(1,100,1),
'mdl__ccp_alpha': np.linspace(0,1,10),
},
{
'mdl': [AdaBoostClassifier(random_state=23)],
'mdl__learning_rate': np.linspace(0,1,20),
'mdl__base_estimator': [SVC()] + [DecisionTreeClassifier(ccp_alpha=a) for a in [0.3,0.4,0.5,0.7]],
},
]

Form data is not correct with Suave web server

I am trying to receive a confirmation from AWS' SNS system.
It sends a message through a POST to a webserver and I'm receiving it using Suave.
When I get the message, the form field is truncated, I am receiving:
"{\n \"Type\" : \"SubscriptionConfirmation\",\n \"MessageId\" : \"13dd68fa-2720-419a-8d8a-edc9f4466ea3\",\n \"Token\" : \"2336412f37fb687f5d51e6e2425e90ccf23f36200405c1942ea85f0874b382c47327c4bfd63203eace5240bb7b253428af362a04a61c8f98ab1718c679e9e27594529615adf5d86729374cce472768b91622851aced957c1dcddf21e3d82b48f5ff528c8dbc911179a3f26126a8d7f00\",\n \"TopicArn\" : \"arn:aws:sns:ap-northeast-1:960544703730:igorbot\",\n \"Message\" : \"You have chosen to subscribe to the topic arn:aws:sns:ap-northeast-1:960544703730:igorbot.\\nTo confirm the subscription, visit the SubscribeURL included in this message.\",\n \"SubscribeURL\" : \"https://sns.ap-northeast-1.amazonaws.com/?Action",
"ConfirmSubscription"
so it's an unfinished json...
but when I look at the rawForm field, I get the whole message:
{
"Type" : "SubscriptionConfirmation",
"MessageId" : "0645c009-f4fa-4fb7-9c94-127e98f5eb76",
"Token" : "2336412f37fb687f5d51e6e2425e90ccf23f36200406641a9f63b753dad1d31c61da2ebeea8cdaeed2e3c04f701bd08e2d2c9cac65676979e43c1089a96779f9b57a0e0f072013333db51472ca43c1e6a0854cf3af6769d95c7911d74c9e2bec22db93de90e537d480070891ddaaa548",
"TopicArn" : "arn:aws:sns:ap-northeast-1:960544703730:igorbot",
"Message" : "You have chosen to subscribe to the topic arn:aws:sns:ap-northeast-1:960544703730:igorbot.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL" : "https://sns.ap-northeast-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:ap-northeast-1:960544703730:igorbot&Token=2336412f37fb687f5d51e6e2425e90ccf23f36200406641a9f63b753dad1d31c61da2ebeea8cdaeed2e3c04f701bd08e2d2c9cac65676979e43c1089a96779f9b57a0e0f072013333db51472ca43c1e6a0854cf3af6769d95c7911d74c9e2bec22db93de90e537d480070891ddaaa548",
"Timestamp" : "2021-03-13T20:36:21.918Z",
"SignatureVersion" : "1",
"Signature" : "FBnpuGtkZmzox+5ryo1/4k1hgwmoeuvcptQ2dOyyneShVHovmdemMqo9JFTzbBFelTN7FMojX/sIjFs2dZoQYeqEgsQW9WqCiEstDQu0toHn7KKxapzIoGfjfh6Rikfy8Liv88RRNLC2DLtxWW2JWr5Mmwkjtro/pm7vyJhfp5G4qcAB3gtBOtVm+XOAai6rY7obcMojkmMr4jDd9UqutV6imyYDCH+PvUCnc7aKg6p4EmZO33VlRibIPa5PiN1Sj/mmNhyoeR4pGu+0Jci+utvonXaYPgtlDuEyoVcgUQ6lki1xiclIRpDm4FvOL5tvUSq+Jdjz3prlNDNM8AuQpQ==",
"SigningCertURL" : "https://sns.ap-northeast-1.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem"
}
At first I thought the output was truncated, but then I narrowed it down. Posting this line:
{
"SubscribeURL" : "https://sns.ap-northeast-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:ap-northeast-1:960544703730:igorbot&Token=2336412f37fb687f5d51e6e2425e90ccf23f36200406641a9f63b753dad1d31c61da2ebeea8cdaeed2e3c04f701bd08e2d2c9cac65676979e43c1089a96779f9b57a0e0f072013333db51472ca43c1e6a0854cf3af6769d95c7911d74c9e2bec22db93de90e537d480070891ddaaa548"
}
will fail and get truncated at the first '=' sign. Then:
{
"SubscribeURL" : "abc=3"
}
will cause Suave to fail. When sent as form data, it will not be converted to a string properly.
Is there any setting related to encoding, etc that would allow to prevent this? (since I can't ask AWS to escape all their '=' signs)
JSON and form data (application/x-www-form-urlencoded, to be specific) have different syntax. Form data looks like key1=value1&key2=value2, so Suave is breaking the string on & and = in order to parse it.
You can parse JSON using mapJson instead. Something like this should work:
open System.Runtime.Serialization
open Suave
open Suave.Json
[<DataContract>]
type Subscription =
{
[<field: DataMember(Name = "SubscribeURL")>]
SubscribeUrl : string;
}
[<EntryPoint>]
let main argv =
let app = mapJson (fun sub -> sprintf "URL: %s" sub.SubscribeUrl)
startWebServer defaultConfig app
0

What are the GLTF animations sampler input/output values?

I am reading the specification, but I can not understand the properties of the sampler.
This is the animation that I have
"animations" : [
{
"channels" : [
{
"sampler" : 0,
"target" : {
"node" : 0,
"path" : "translation"
}
}
],
"name" : "00001_2780.datAction",
"samplers" : [
{
"input" : 9,
"interpolation" : "CUBICSPLINE",
"output" : 10
}
]
},
{
"channels" : [
{
"sampler" : 0,
"target" : {
"node" : 1,
"path" : "translation"
}
}
],
"name" : "00002_2780.datAction",
"samplers" : [
{
"input" : 9,
"interpolation" : "CUBICSPLINE",
"output" : 11
}
]
}
],
What I can not understand is what are the values 9 and 10 for the first sample and 9 and 11 for the second
All that we have in the specification is
https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#animations
Each of the animation's samplers defines the input/output pair: a set of floating point scalar values representing linear time in seconds; and a set of vectors or scalars representing animated property.
And this makes it more unclear to me.
Is there a more detailed explanation about what input/output values are and what they represent. What will happen for example if I change the input from 9 to 99 or to 9.9 or to 0.9 or to 0. How will this change the animation?
Thanks
The numbers 9 and 10 here are glTF accessor index ID values. If you decode accessor index 9, you'll find the list of times for each of the keyframes of the animation. If you decode accessor 10, normally you would expect to find the list of values for the keyframes. But since this is CUBICSPLINE, accessor 10 will contain the in-tangent, value, and out-tangent for each keyframe.
One way to investigate glTF files like this is to use the glTF Tools extension for VSCode. You can right-click the input or output value and choose Go To Definition to get to the accessor in question, and choose Go To Definition again to decode it. (Disclaimer, I'm a contributor to glTF Tools).

Ruby, How to generate JSON format?

I need to send an input in the below format to an API, I'm running into problems producing the desired format which is below.
{ "application" :
"{\"attributes\":{\"type\":\"genesis__Applications__c\"},\"genesis__Days_Convention__c\":\"ACTUAL/ACTUAL\", \"RecordTypeID\":\"012260000004RHF\",
\"genesis__Interest_Calculation_Method__c\":\"Flat\",
\"genesis__Interest_Rate__c\":10.0000,
\"genesis__Loan_Amount__c\":22120.00,
\"Application_Completed__c\":false,
\"genesis__Payment_Frequency__c\":\"WEEKLY\",
\"genesis__Product_Type__c\":\"LOAN\", \"genesis__Term__c\":24,
\"genesis__Interest_Only_Period__c\":2,
\"genesis__Balloon_Payment__c\":100.00}", "relatedObjects" : "{
\"genesis__Account__c\" : {\"attributes\":
{\"type\":\"Account\"}, \"Name\":\"LONDON METALS HOLDINGS
LIMITED\" }, \"Loan_Product_Purpose__c\" : {\"attributes\":
{\"type\":\"Loan_Product_Purpose__c\"}, \"Name\":\"Equipment
Purchase\" }}" }
API is accepting above format as input.
Input specifications as shown.
We tried JSON builder that did not help instead producing the below format
which is not valid for API.
"\"{:attributes:{:type:\\"genesis__Applications__c\\"},
:genesis__Days_Convention__c:\\"ACTUAL/ACTUAL\\",
:RecordTypeID:\\"012260000004RHF\\",
:genesis__Interest_Calculation_Method__c:\\"Flat\\",
:genesis__Interest_Rate__c:10.0, :genesis__Loan_Amount__c:22120.0,
:Application_Completed__c:false,
:genesis__Payment_Frequency__c:\\"WEEKLY\\",
:genesis__Product_Type__c:\\"LOAN\\", :genesis__Term__c:24,
:genesis__Interest_Only_Period__c:2,
:genesis__Balloon_Payment__c:100.0}\""
edit1: Input Hash object
{:application=> {:attributes=>{:type=>"genesis__Applications__c"},
:genesis__Days_Convention__c=>"ACTUAL/ACTUAL",
:RecordTypeID=>"012260000004RHF",
:genesis__Interest_Calculation_Method__c=>"Flat",
:genesis__Interest_Rate__c=>10.0,
:genesis__Loan_Amount__c=>22120.0,
:Application_Completed__c=>false,
:genesis__Payment_Frequency__c=>"WEEKLY",
:genesis__Product_Type__c=>"LOAN", :genesis__Term__c=>24,
:genesis__Interest_Only_Period__c=>2,
:genesis__Balloon_Payment__c=>100.0}, :relatedObjects=>
{:genesis__Account__c=>{:attributes=>{:type=>"Account"},
:Name=>"LONDON METALS HOLDINGS LIMITED"},
:Loan_Product_Purpose__c=>{:attributes=>{:type=>"Loan_Product_Purpose__c"}, :Name=>"Equipment Purchase"}}}
How are you using JSON? It should work just fine
require 'json'
h = {:application=>
{:attributes=>{:type=>"genesis__Applications__c"},
:genesis__Days_Convention__c=>"ACTUAL/ACTUAL",
:RecordTypeID=>"012260000004RHF",
:genesis__Interest_Calculation_Method__c=>"Flat",
:genesis__Interest_Rate__c=>10.0,
:genesis__Loan_Amount__c=>22120.0,
:Application_Completed__c=>false,
:genesis__Payment_Frequency__c=>"WEEKLY",
:genesis__Product_Type__c=>"LOAN", :genesis__Term__c=>24,
:genesis__Interest_Only_Period__c=>2,
:genesis__Balloon_Payment__c=>100.0}, :relatedObjects=>
{:genesis__Account__c=>{:attributes=>{:type=>"Account"}, :Name=>"LONDON METALS HOLDINGS LIMITED"},
:Loan_Product_Purpose__c=>{:attributes=>{:type=>"Loan_Product_Purpose__c"}, :Name=>"Equipment Purchase"}}}
puts h.to_json
prints:
{"application":{"attributes":{"type":"genesis__Applications__c"},"genesis__Days_Convention__c":"ACTUAL/ACTUAL","RecordTypeID":"012260000004RHF","genesis__Interest_Calculation_Method__c":"Flat","genesis__Interest_Rate__c":10.0,"genesis__Loan_Amount__c":22120.0,"Application_Completed__c":false,"genesis__Payment_Frequency__c":"WEEKLY","genesis__Product_Type__c":"LOAN","genesis__Term__c":24,"genesis__Interest_Only_Period__c":2,"genesis__Balloon_Payment__c":100.0},"relatedObjects":{"genesis__Account__c":{"attributes":{"type":"Account"},"Name":"LONDON METALS HOLDINGS LIMITED"},"Loan_Product_Purpose__c":{"attributes":{"type":"Loan_Product_Purpose__c"},"Name":"Equipment Purchase"}}}
Ok, now I see the value of application and relatedObjects are strings (with json content). So try this instead:
require 'json'
application = {:attributes=>{:type=>"genesis__Applications__c"},
:genesis__Days_Convention__c=>"ACTUAL/ACTUAL",
:RecordTypeID=>"012260000004RHF",
:genesis__Interest_Calculation_Method__c=>"Flat",
:genesis__Interest_Rate__c=>10.0,
:genesis__Loan_Amount__c=>22120.0,
:Application_Completed__c=>false,
:genesis__Payment_Frequency__c=>"WEEKLY",
:genesis__Product_Type__c=>"LOAN", :genesis__Term__c=>24,
:genesis__Interest_Only_Period__c=>2,
:genesis__Balloon_Payment__c=>100.0}
relatedObjects = {:genesis__Account__c=>{:attributes=>{:type=>"Account"}, :Name=>"LONDON METALS HOLDINGS LIMITED"},
:Loan_Product_Purpose__c=>{:attributes=>{:type=>"Loan_Product_Purpose__c"}, :Name=>"Equipment Purchase"}}
h = {:application=> application.to_json,
:relatedObjects=> relatedObjects.to_json}
puts h.to_json

Highcharts not rendering data points

I'm pulling some data from a database that I'm trying to render into a Highcharts stock chart. The data is pulled from the database with PHP and passed to the chart with a $.get(..php/line-data.php) call, and the data retrieved is supposed to be the data that is rendered on the chart.
The data is being returned in the following manner, and I have verified this by logging data in the console. It appears as such, with the first value being the UNIX-to-Javascript converted date/time (x-axis), and the second being the value (y-axis):
[[1362639600000, 8],[1362726000000, 20],[1362985200000, 28],[1363071600000, 51],[1363158000000, 64],[1363244400000, 11],[1363330800000, 4],[1363503600000, 4],[1363590000000, 21],[1363676400000, 10],[1363762800000, 31],[1363849200000, 13],[1363935600000, 17],[1364194800000, 10],[1364454000000, 1],[1365058800000, 30],[1365145200000, 10],[1366009200000, 55],[1366182000000, 18],[1366268400000, 22],[1366354800000, 12]]
As an experiment, I tried just plugging this data straight into a basic demo Fiddle, and it seems to render fine.
FIDDLE HERE.
So what am I doing incorrectly? Everything seems to be set up correctly, but it's not rendering. This is what I see:
Here are the relevant portions of my code. Yes, I know that mysql_* is deprecated...I'll change it.
$.get('../php/line-data.php', function(data) {
window.chart = new Highcharts.StockChart({
chart : {
renderTo : 'total_mentions',
margin: [20, 10, 10, 10],
spacingTop: 0,
spacingBottom: 1,
spacingLeft: 0,
spacingRight: 0
},
series : [{
name : 'Total Mentions',
data: data,
type:'line',
lineWidth:1,
shadow:false,
states: {
hover: {
lineWidth:1
}
},
id : 'dataseries',
tooltip : {
valueDecimals: 4,
borderColor:'#DA7925',
borderRadius: 0,
borderWidth: 1,
shadow: false
},
color:'#DA7925',
fillOpacity:0.2
}]
[more options...etc.]
No problems with this code. It's pulling the correct data and echoing how I expect it to.
<?php
$expAddress = "URL";
$expUser = "USERNAME";
$expPwd = "PASSWORD";
$database = "DB";
$db = mysql_connect($expAddress, $expUser, $expPwd);
mysql_select_db($database, $db);
$ok = mysql_query("
SELECT
DATE(created_at) AS create_date,
COUNT(id) AS total
FROM
tweets
WHERE
subject LIKE 'word1'
OR
subject LIKE 'word2'
GROUP BY
DATE(created_at)");
if (!$ok) {
echo "<li>Mysql Error: ".mysql_error()."</li>";
}
else {
while($row = mysql_fetch_assoc($ok)){
extract($row);
$date = strtotime($create_date);
$date *= 1000;
$data[] = "[$date, $total]";
}
$tmp = join($data,',');
echo "[".$tmp."]";
}
?>
Have you tried parsing your data (string) into a javascript object before setting it to the series[i].data?
series : [{
data: JSON.parse(data)
}]
What you are getting from php through $.get is basically string and NOT a javascript array of array of numbers, which is what you want. It may look like that, but it is as simple as "5"!=5, but parseInt("5")==5 same is the case with json objects, you need to parse the string into such an object before javascript or highcharts can interpret it correctly, highcharts could do it on your behalf, but it is not designed that way.
Try his fiddle to get an idea of the data types in picture
var data="[[1362639600000, 8],[1362726000000, 20],[1362985200000, 28],[1363071600000, 51],[1363158000000, 64],[1363244400000, 11],[1363330800000, 4],[1363503600000, 4],[1363590000000, 21],[1363676400000, 10],[1363762800000, 31],[1363849200000, 13],[1363935600000, 17],[1364194800000, 10],[1364454000000, 1],[1365058800000, 30],[1365145200000, 10],[1366009200000, 55],[1366182000000, 18],[1366268400000, 22],[1366354800000, 12]]"
console.log(typeof data); //string
var parsedData=JSON.parse(data);
console.log(typeof parsedData); //object
console.log(typeof parsedData[0]); //object [1362639600000, 8]
console.log(typeof parsedData[0][0]); //number 1362639600000
When you paste the console value directly in the fiddle, you are actually pasting it as a valid javascript array, try using your console value wrapped by " quotes " and see that the exact issue is reproduced!!
Demo # jsFiddle
An alternate approach could be using the $.getJSON() method instead. jQuery does the parsing for you before it calls your callback method
Your problem is in either the output from the PHP script or when you receive the data in your Javascript (quite obvious).
First, don't do JSON by hand use json_encode (http://php.net/manual/en/function.json-encode.php). It's easier and it will guarantee that strings will be escaped properly.
Secondly, inspect your data variable with a debugger. You could also post the exact content of the variable to the question.
But basically, as long as it is working in the fiddle and not in your program you have not yet reproduced the error in your code properly in the fiddle.
For instance, you could replace data in your callback with the data you have in your fiddle to see if the code runs.

Resources