Is there a way to filter out the nested JSON string out into separate fields in fluentd?
Current JSON:
{
Value1: "something",
Value2: "something",
Message:{
Value3: "Something",
Value3: "Something"
}
}
What I want (or something similar):
{
Value1: "something",
Value2: "something",
Message.Value3: "Something",
Message.Value3: "Something"
}
The JSON doesn't have to be flatten like in above example but I do want the values to be in their own separate fields(columns) when they reach elasticsearch. In other wards, I want to split the single long Message string into multiple fields contained within it.
Try this:
<source>
...
</source>
<filter myapp**>
#type parser
key_name Message
format multi_format
<pattern>
format json # try parsing json in the first place
</pattern>
<pattern>
format none # leave as is if this is not json (plaintext)
</pattern>
reserve_data true # keep the original Message field in case anything go wrong
</filter>
<match myapp**>
...
</match>
Multi-format parser: https://github.com/repeatedly/fluent-plugin-multi-format-parser
Related
I got a jbuilder response like below
{
"name": "amb_devcernerpowerchart_com:patient_val2",
"value": "{\"value\"=>\"patient_value\", \"expiration\"=>31536000, \"created_datetime\"=>\"2019-12-09T12:09:59Z\"}"
}
I want to only value part of value i.e i want output like below.
{
"name": "amb_devcernerpowerchart_com:patient_val2",
"value": "value"=>"patient_value"
}
How do i get this?
Jbuilder file looks like this
json.name #component.preference_data.id
json.value #component.preference_data.value
From the question, it seems as though the data that you are receiving in the value field is a stringified JSON. You would have to parse the JSON and retrieve the required value as follows:
json.value JSON.parse(#component.preference_data.value)['value']
Basically, #component.preference_data.value returns a string, which is parsed into a JSON using JSON.parse. At the end of that, we get a hash whose 'value' field is retrieved.
I am trying to format date by iterating through the #data json formatted input shown as below.
#data= JSON.parse(request,symbolize_names: true)[:data]
#data = #data.each{ |key,val| k = Date.parse(key).strftime("%Y %m") }
But this way it is showing error "no implicit conversion of Symbol into String".
Could any one please help?
If you're iterating over a hash where the keys are symbols, then the error is telling you where and what's the problem. In order to parse a date, you must pass a string as argument, and as you haven't converted such object, then you're getting the error.
Try using to_s, to convert the symbol to string:
#data.each do |key, _|
puts Date.parse(key.to_s).strftime '%Y %m'
end
Note if you're inside a block, and you're not going to use the k variable you're creating, then you can avoid creating it, it won't be available outside the block. You're just printing the parsed date.
If you're not going to use the value block variable, then you can omit it.
As pointed #mu, you can omit the symbolize_names: true, and this way the keys will be strings, then, the conversion isn't needed:
require 'date'
require 'json'
request = '{
"data": {
"2017-11-22 00:22:26": "foo" ,
"2017-11-22 00:22:27": "bar"
}
}'
#data = JSON.parse(request)['data']
#data.each do |key, _|
puts Date.parse(key).strftime '%Y %m'
end
request is an approximation to your real data.
I'm trying to send a JSON API response from my rails app to Dasheroo which expects the following format:
{
my_statistic: { type: 'integer', value: 1, label: 'My Statistic' }
}
However it is not happy with my data structure generated by the following code:
In controller:
def count_foo_members
#foo = Foo.all.count
end
In count_foo_members.json.jbuilder:
json.foo_members do
json.type 'integer'
json.value #foo
json.label 'Foo Members'
end
If I open this route in my browser I can see the following:
{
"foo_members":{"type":"integer","value":1,"label":"Foo Members"}
}
From the results above, the only thing that I can see that could have an effect on the result is the fact that my JSON result has quotation marks around the JSON Key values.
My question thus is: How can I remove these quotation marks in Rails 4 and JBuilder?
JSON.parse(you_response) and you get standart hash.
You cannot remove the quotation marks from the keys. The responsibility is on the consumer (Dasheroo) to parse your JSON string into a JavaScript Object, which will "remove" the quotes from the keys.
Read json-object-with-or-without-quotes for further practical insight.
I am using Ruby version 1.9.3. Here is a simple version of the actual XML page that I want to get information from. I need to access it from a secure website which requires login credentials. I can't use Nokogiri because I wasn't able to log into the website using it.
<root>
<person>
<name>Jack</name>
<age>10</age>
</person>
<person>
<name>Jones</name>
</person>
<person>
<name>Jon</name>
<age>16</age>
</person>
</root>
As you can see sometimes the tag age does not appear. Using REXML with Ruby, I use the following code:
agent = Mechanize.new
xml = agent.get("https://securewebsite.com/page.xml")
document = REXML::Document.new(xml.body)
name = XPath.match(document, "//person/name").map {|x| x.text}
# => ["Jack", "Jones", "Jon"]
age = XPath.match(document, "//person/age").map {|x| x.text}
# => ["10", "16"]
The problem is that I can't associate the age with the correct name because the index are now out of order. For example at index 1, name[1] is Jones but age[1] is 16. But that is not true because the person tag for Jones does not have the age tag.
Is there any way that I can get the age array to output: # => ["10", nil ,"16"] so that I can associate the correct name with its corresponding age?
Or is there a better way? Let me know if further explanation is required.
The problem is that we are looking at age and name as completely separate collections of information. What we need to do is get information from person as a collection.
xml = "<your xml here />"
doc = Nokogiri::XML(xml)
persons = doc.xpath("//person")
persons_data = persons.map {|person|
{
name: person.xpath("./name").text,
age: person.xpath("./age").text
}
}
This gets the person nodes and then gets the related information from them giving a result:
puts persons_data.inspect #=> [
{:name=>"Jack", :age=>"10"},
{:name=>"Jones", :age=>""},
{:name=>"Jon", :age=>"16"}
]
So to get the name and age of the first person you would call
persons_data[0]["name"] #=> "Jack"
persons_data[0]["age"] #=> "10"
I'd do something like this:
require 'nokogiri'
doc = Nokogiri::XML(<<EOT)
<root>
<person>
<name>Jack</name>
<age>10</age>
</person>
<person>
<name>Jones</name>
</person>
<person>
<name>Jon</name>
<age>16</age>
</person>
</root>
EOT
people = doc.search('person').each_with_object({}){ |person, h|
age = person.at('age')
h[person.at('name').text] = age ? age.text : nil
}
people # => {"Jack"=>"10", "Jones"=>nil, "Jon"=>"16"}
At that point, if I only want the ages, I'd use values:
people.values # => ["10", nil, "16"]
Retrieving a single person's age is trivial then:
people['Jon'] # => "16"
people['Jack'] # => "10"
I get this error when I'm using the .to_h method: ``block in ': undefined method to_h'
My mistake. to_h is not in older Rubies, but it's not needed because of how I'm generating the hash being returned. I adjusted the code above which will work in any Ruby that implements each_with_object.
I'm writing a script that will add new translations to the en.yml file. However, when I'm dumping them back to the file, my strings are in the following format:
some_key: This is the value
I'm trying to make the output be:
some_key: "This is the value"
I'm writing the translations like this:
File.open(yaml_file, "w") do |f|
f.write(translations.to_yaml)
end
Where translations is the hash containing all the translations.
Is there any way of adding these quotes, besides manually parsing/rewriting the YAML file?
The plan (unquotes) scalar representation is the preferred version when the scalar type doesn't require escaping.
In your case, the String:
This is the value
doesn't need to be in quotes, thus, if you supply the following YAML:
key: "This is the value"
the processor may return:
key: This is the value
because they are totally equivalent. However, if you actually want to enter a quoted string as value, then you should use:
key: '"This is the value"'
or escape the double quote:
key: "\"This is the value\""
I gave a quick look at the Psych emitter code, the one invoked by the to_yaml, and there doesn't seem to be an option to force quoting on scalar.
I don't even see the option implemented in the scalar emitter code.
def visit_Psych_Nodes_Scalar o
#handler.scalar o.value, o.anchor, o.tag, o.plain, o.quoted, o.style
end
In other words, you cannot enforce quoting.
Updated for hash conversion
def value_stringify(hash)
hash.each do |k,v|
if v.kind_of? Hash
hash[k]= value_stringify(v)
else
hash[k]="\"#{v}\""
end
end
end
Now use the converted hash to store yaml.
File.open(yaml_file, "w") do |f|
f.write(value_stringify(translations).to_yaml)
end
Now it should work..
The format you get is valid YAML. However, if you really want this you could temporarily modify your data before converting it.
Normal:
{ foo: "bar" }.to_yaml
# => foo: bar
With an space after:
{ foo: "bar " }.to_yaml
# => foo: 'bar '
Note that you get single quotes and not double quotes. So if you temporarily modifying your data you could add in an placeholder which you remove later.
Example:
{ foo: "foo --REMOVE-- ", bar: "bar --REMOVE-- " }.to_yaml
.gsub(' --REMOVE-- ', '')
# => foo: 'foo'
# bar: 'bar'