I recently started to use ruby psych to generate yaml strings from hashes..
I have a concern however, in case I need to !Ref or !GetAtt for another resource, it would end up being wrapped in double quoted strings after the YAML. For example:
{
'FooPolicy' => {
'Type' => 'AWS::SQS::QueuePolicy',
'Properties' => {
'Queues' => '!Ref FooQueue',
'PolicyDocument' => {
'Statement' => {
'Action' => 'SQS:*',
'Effect' => 'Allow',
'Resource' => '!GetAtt FooQueue.Arn',
'Principal' => {
'AWS' => '${AWS::AccountId}'
}
}
}
}
}
}
The output is
FooPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues: "!Ref FooQueue"
PolicyDocument:
Statement:
Action: SQS:*
Effect: Allow
Resource: "!GetAtt FooQueue.Arn"
Principal:
AWS: "${AWS::AccountId}"
which is throwing malform message in Cloudformation validation... Has anyone run into this issue?
Related
Hello I'm new to Elasticsearch
I'm working with log files comming from filebeat and logstash and I'm trying to add a field "response_time", and then affect the difference between timestamps to It.
So I create a logstash's filter and I add it to logstash configuration file but when I restared the container I get the error bellow.
This is my logstash configuration file:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
ruby {
code => "event.set('indexDay', event.get('[#timestamp]').time.localtime('+01:00').strftime('%Y%m%d'))"
}
aggregate {
add_field => {
"response_time" => "timestamp2-timestamp1"
}
}
grok {
match => ["message","%{LOGLEVEL:loglevel},%{DATESTAMP_RFC2822:timestamp},%{NOTSPACE:event_type},%{NUMBER:capture_res_id},%{NUMBER:capture_pid},%{NUMBER:mti},%{NUMBER:node_id}
,%{UUID:msg_uuid},%{NOTSPACE:module},%{NUMBER :respCode}"]}
if [event_type] == "request_inc" {
aggregate {
msg_uuid => "%{UUID}"
timestamp1 => event.get('DATESTAMP_RFC2822')
code => "map['response_time'] = 0"
map_action => "create"
}
}
if [event_type] == "response_outg" {
aggregate {
msg_uuid => "%{UUID}"
event_type => event.set('event_type')
timestamp2 => "%{DATESTAMP_RFC2822}"
code => "map['response_time']"
map_action => "update"
end_of_task => true
timeout =>120
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
template => "/usr/share/logstash/templates/testblogstash.template.json"
template_name => "testblogstash"
template_overwrite => true
index => "testblogstash-%{indexDay}"
codec => json
}
stdout {
codec => rubydebug
}
}
And this is an exemple of my log file:
{"log_level":"INFO","timestamp":"2021-12-15T16:06:24.400087Z","event_type":"s_tart","ca_id":"11","c_pid":"114","mti":"00","node_id":"00","msg_uuid":"1234","module":"cmde"}
{"log_level":"INFO","timestamp":"2021-12-15T16:06:31.993057Z","event_type":"e_nd","mti":"00","node_id":"00","msg_uuid":"1234","module":"PWC-cmde","respCode":"1"}
This is the error from docker logs :
[2022-06-01T14:43:24,529][ERROR][logstash.agent ] Failed to execute
action {:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
[A-Za-z0-9_-], [ \t\r\n], "#", "{", [A-Za-z0-9_], "}" at line
25, column 24 (byte 689) after filter {\r\n json {\r\n source =>
"message"\r\n }\r\n ruby {\r\n code => "event.set('indexDay',
event.get('[#timestamp]').time.localtime('+01:00').strftime('%Y%m%d'))"\r\n
}\r\n aggregate {\r\n add_field => {\r\n "response_time" =>
"timestamp2-timestamp1"\r\n\t\t }\r\n\t\t}\r\n grok {\r\n match =>
["message","%{LOGLEVEL:loglevel},%{DATESTAMP_RFC2822:timestamp},%{NOTSPACE:event_type},%{NUMBER:capture_res_id},%{NUMBER:capture_pid},%{NUMBER:mti},%{NUMBER:node_id}\r\n\t,%{UUID:msg_uuid},%{NOTSPACE:module},%{NUMBER
:respCode}"]}\r\n if [event_type] == "request_inc" {\r\n aggregate
{\r\n\t msg_uuid => "%{UUID}"\r\n\t timestamp1 => event",
:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in
compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in
execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in block
in converge_state'"]}
...
[2022-06-01T14:43:29,460][INFO ][logstash.runner ] Logstash shut down.
I'm trying to parse out JSON data and create my own dictionary to show a subset of the data. The thing is, I'm noticing that my input data changes based on what is scanned (with nmap). Some elements might be an array value, whereas some might not. The combinations seem to be pretty broad.
For instance, here is the simplest input where only an IP address was found:
{
'host' => {
'address' => {
'addr' => '192.168.0.1'
},
'status' => {...}
}
}
But then, the IP and MAC address might be found:
{
'host' => {
'address' => [{
'addrtype' => 'ipv4',
'addr' => '192.168.0.1',
},{
'addrtype' => 'mac',
'mac' => '00:AA:BB:CC:DD:EE',
},
'status' => {...}
}]
}
Those are just a couple examples. Other variations I've seen:
`host.class` = Array
`address.class` = Hash
`host['status'].class` = Array
etc...
As I go through to parse the output, I am first checking if the element is an Array, if it is, I access the key/values one way, whereas if it's not an array, I essentially have to duplicate my code with a few tweaks to it, which doesn't seem very eloquent:
hash = {}
if hosts.class == Array
hosts.each do |host|
ip = if host['address'].class == Array
host['address'][0]['addr']
else
host['address']['addr']
end
hash[ip] = {}
end
else
ip = if hosts['address'].class == Array
hosts['address'][0]['addr']
else
hosts['address']['addr']
end
hash[ip] = {}
end
puts hash
end
In the end, I'm just trying to find a better/eloquent way to produce a hash like below, while accounts for the possibility that an element may/may not be an Array:
{
'192.168.0.1' => {
'mac' => '00:aa:bb:cc:dd:ee',
'vendor' => 'Apple',
'ports' => {
'80' => {
'status' => 'open',
'service' => 'httpd'
}
'443' => {
'status' => 'filtered',
'service' => 'httpd'
}
}
},
192.168.0.2 => {
...
}
}
If there a ruby method that I haven't run across yet that will make this more fluid?
Not really... but you can make it always an array eg by doing something like:
hosts = [hosts] unless hosts.is_a?(Array)
or similar... then just pass that to your now-non-duplicated code. :)
The 20 lines of code in your question can be reduced to a single line using Array#wrap instead of conditionals, and using Enumerable#map instead of Enumerable#each:
Array.wrap(hosts).map { |host| [Array.wrap(host['address']).first['addr'], {}] }.to_h
Now that's magic!
I use this method to list all routes of my rails application:
def routes
Rails.application.routes.routes.collect do |r|
{ path: r.path.spec.to_s }
end
end
My result is:
[
{ :path => '/my/path/' },
{ :path => '/my/path2/' },
...
]
I want to obtain also the http method used for that route. How to do it?
Exploring class documentation I couldn't find it.
The expected result is:
[
{ :path => '/my/path/', :method => :get },
{ :path => '/my/path2/', :method => :post },
...
]
There is a way to obtain the http method associated to a route? (or a list of methods)
ActionDispatch::Journey::Route has a verb method which returns a RegExp:
You could try this:
def routes
Rails.application.routes.routes.collect do |r|
{ path: r.path.spec.to_s, :verb => r.verb.source[/[a-z]+/i].to_sym }
end
end
I've been pouring over the google adwords api docs and I can't figure out how to format the selector to retrieve CPC information. I am using the google-adwords-api gem. Below is the method I'm working on inside my Adwords api module.
def self.traffic_estimator_service keyword
if !#adwords #If not already authenticated, do it first
Adwords.authenticate()
end
traffic_estimator_service = #adwords.service(:TrafficEstimatorService, API_VERSION)
selector = {
:xsi_type => 'KeywordEstimateRequest',
:match_type => 'EXACT',
:keyword => keyword
}
data = traffic_estimator_service.get(selector)
puts '---------------------------------'
puts data.inspect
puts '---------------------------------'
end
Of course I never get to the data2.inspect line because of the api errors. ie:
AdsCommon::Errors::UnexpectedParametersError (AdsCommon::Errors::UnexpectedParametersError: [:match_type]):
I've moved things around and tried multiple things inside the selector hash. Can someone give me an example of what this selector hash should look like?
selector = {
:campaign_estimate_requests => [
{
:xsi_type => 'CampaignEstimateRequest',
:ad_group_estimate_requests => {
:xsi_type => 'AdGroupEstimateRequest',
:keyword_estimate_requests => [
{
:max_cpc => {
:xsi_type => 'Money',
:micro_amount => 1_000_000
},
:xsi_type => 'KeywordEstimateRequest',
:keyword => {
:xsi_type => 'Keyword',
:text => keyword,
:match_type => 'EXACT'
}
}
]
}
}
]
}
When supplying savon with:
hash = {
"Objects" => { //stuff here },
:attributes! => { "Objects" => {"xsi:type" => "Something"}}
}
I get:
<Objects>...</Objects>
When supplying savon with anything else i get the expected result:
hash = {
"foo" => { //stuff here },
:attributes! => { "foo" => {"xsi:type" => "Something"}}
}
I get:
<foo xsi:type="Something"></foo>
I must use the string "Objects" as the key. I am coding to a 3rd party SOAP web service. I cannot use a symbol because the first letter would become a lower cap.
thanks,
You have to change :attributes! to :#xsi:type=>"Something" within the hash where you want the attribute
Like:
"foo"=>{:#xsi:type=>'something', //stuff here}