I have been searching online, but cannot quite figure this out.
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{URIPATH:url}
}
I need to get contents out of the url and put stuff it in elastic search.
Logs have urls like this
URL 1 =
/NEED_A/Constant_A/Constant_B/Constant_C/Need_B/Constant_D/Need_C/Need_D
URL 2 =
/NEED_A/Constant_A /Constant_B/Constant_C/Need_B/Constant_D
URL 3 =
/Wierd_A
Need_A, NEED_B, NEED_C, Need_D, Wierd_A should go in respective fields.
I have been trying to find a if else-if loop, but not really gotten anything yet.
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} {URIPATH:url} %]
}
if[url] == "/*/Constant_A/Constant_B/Constant_C/*/Constant_D/*/*" {
\/%{WORD:NEED_A}\/.*\/.*\/.*\/%{WORD:NEED_B}\/.*\/%{WORD:NEED_C}
}
else-if[url] == "/*/Constant_A/Constant_B/Constant_C/*/Constant_D" {
\/%{WORD:NEED_A}\/.*\/.*\/.*\/%{WORD:NEED_B}\/.*\/
}
else-if
//something similar for url 3
}
//move on if nothing matches
Any thoughts?
logstash isn't a looping type system.
If you want to run multiple patterns against your input, just list them:
grok {
match => {
"message" => [
"%{PATTERN1}",
"%{PATTERN2}"
]
]
}
Related
So I have objects that I'm trying to pass to a json-acccepting restful interface using parms and #connection, but the source object and the target are in different formats, so I need to rename them.
So I'm trying to do something like this:
parms = {
:foo => anBunchOfBars.bar[
:barID => bar.Identifier
:bartab => bar.finances.owed
:barbell => bar.equipment.first
]
}
... to generate an outgoing JSON similar to the following:
{
"foo": [
{
"barID": "Irish Pub",
"bartab": "30 Yen",
"barbell": "Of the ball."
},
{
"barID": "One Gold Bar",
"bartab": "100 cents",
"barbell": "fry."
}
]
}
But I can't find the proper syntax. Examples I've found seem to only cycle through a core list and leave out naming items on nested traits (or skip nested traits entirely), and I haven't seen anything that shows how to pull values from a looped item.
What's the proper syntax to format the parms value?
Assuming that anBunchOfBars is an array of bars:
bars = anBunchOfBars.map do |bar|
{
barID: bar.Identifier
bartab: bar.finances.owed
barbell: bar.equipment.first
}
end
params = { foo: bars }
I'm using json-compare gem to compare two different json files.
Example file 1:
{"suggestions": [
{
"id1": 1,
"title1": "Test",
"body1": "Test"
}
]
}
Example file 2:
{"suggestions": [
{
"id2": 1,
"title2": "Test",
"body2": "Test"
}
]
}
The gem works well and spits out a hash that looks like this:
{:update=>
{"suggestions" =>
{:update=>
{0=>
{:append=>
{"id2"=>1, "title2"=>"Test", "body2"=>"Test"},
:remove=>
{"id1"=>1, "title1"=>"Test", "body1"=>"Test"},
}
}
}
}
}
How can I parse this and return all the places where json Keys were changed? For the sake of simplicity, how would I put to the console:
id1 changed to id2
title1 changed to title2
body1 changed to body2
For the purpose of what I'm building I don't need to know changes to the values. I just need to know that id1 became id2, etc.
Except if you are relaying on key ordering there is no way to tell that id1 got replaced by id2 and title2 by title1, or that id1 became title1 and id2 became title2. Sounds like you would need specific logic related to the actual key names (in this example searching for different integer suffixes).
Maybe this can be enough for the purpose:
def find_what_changed_in(mhash, result = [])
result << mhash
return if mhash.keys == [:append, :remove]
mhash.keys.each { |k| find_what_changed_in(mhash[k], result) }
result.last
end
find_what_changed_in(changes)
#=> {:append=>{"id2"=>1, "title2"=>"Test", "body2"=>"Test"}, :remove=>{"id1"=>1, "title1"=>"Test", "body1"=>"Test"}}
Where:
changes = {:update=>
{"suggestions" =>
{:update=>
{0=>
{:append=>
{"id2"=>1, "title2"=>"Test", "body2"=>"Test"},
:remove=>
{"id1"=>1, "title1"=>"Test", "body1"=>"Test"},
...
$message_text = message.text
module STAPI
HTTP = GraphQL::Client::HTTP.new("API_ADDRESS")
Schema = GraphQL::Client.load_schema(HTTP)
Client = GraphQL::Client.new(schema: Schema,execute: HTTP)
end
module Sticker
Query = STAPI::Client.parse <<-'GRAPHQL'
{
stickers(query: "#{$message_text}", first: 21, after: "") {
edges {
node {
fileUrl(fullWatermarked: true)
}
}
}
}
GRAPHQL
end
result = STAPI::Client.query(Sticker::Query)
First of all this is an messenger bot and with this messenger bot i am taking users input and i am searching the input in our database then i post something related with the input.
So, in the code i can't search beacuse, in my opinion query: "#{$message_text}" is broken.
I take users input with message.text . This input comes like this 'Hello' (with '). But i need to give query: "#{$message_text}" part like this "Hello"(with "). how can i give messenger.text to query part like this "Hello"
I believe you already know how to make this work based on what you wrote in your other question. For others facing the same problem, take a look at variables. Your example would become as follows:
$message_text = message.text
module STAPI
HTTP = GraphQL::Client::HTTP.new("API_ADDRESS")
Schema = GraphQL::Client.load_schema(HTTP)
Client = GraphQL::Client.new(schema: Schema,execute: HTTP)
end
module Sticker
Query = STAPI::Client.parse <<-'GRAPHQL'
query Q($message: String) {
stickers(query: $message, first: 21, after: "") {
edges {
node {
fileUrl(fullWatermarked: true)
}
}
}
}
GRAPHQL
end
result = STAPI::Client.query(Sticker::Query, variables: {message_text: $message_text})
I have a domain class
class Url {
UUID id
String url
static hasMany = [
indications:UrlIndication
]
...
}
And
class UrlIndication {
UUID id
String name
static belongsTo = Url
...
}
I want to choose urls so that it has all the necessary UrlIndication elements in a given list indicationsId.
For that I use an association and criteria like this one:
indications {
and {
indicationsId.each{
indication->
eq ('id',UUID.fromString(indication as String))
}
}
}
However, all I got is an empty result. Can you suggest any modifications/ other methods so that I can do this? Thanks in advance
Your query returned an empty list because it's the equivalent of the expression (pseudo-code): if 1 = 1 and 1 = 2 and 1 = 3
Such an expression would always be false. in or inList would not work for the reason #innovatism described.
In theory, Criteria's eqAll() or HQL's = ALL would work. But, I don't know for sure because I could not get either one to work.
What will work is to use inList to return a subset of Urls: those which contain at least one of the UrlIndication IDs. Then use Groovy's containsAll() to finish the job.
def ids = indicationsId.collect { UUID.fromString(it as String) }
Url.createCriteria()
.buildCriteria {
indications {
inList 'id', ids
}
}
.setResultTransformer(org.hibernate.Criteria.DISTINCT_ROOT_ENTITY)
.list()
.findAll {
it.indications.id.containsAll(ids)
}
Since the query has the potential to return duplicate Url instances, the ResultTransformer is set to return a unique list.
Finally, findAll() is used along with containsAll() to filter the list further.
Using eqAll (maybe)
Something like the following might work. Something funky is going on with Grails' HibernateCriteriaBuilder that causes the eqAll method to look up properties in the root entity; completely ignoring the sub criteria. So the following uses Hibernate directly. It didn't work for me, but it's as close as I could get. And it gave me a head-ache!
Url.createCriteria().buildCriteria {}
.createCriteria('indications', 'i')
.add(org.hibernate.criterion.Property.forName('i.id').eqAll(org.hibernate.criterion.DetachedCriteria.forClass(UrlIndication)
.add(org.hibernate.criterion.Restrictions.in('id', ids))
.setProjection(org.hibernate.criterion.Property.forName('id'))
))
.setResultTransformer(org.hibernate.Criteria.DISTINCT_ROOT_ENTITY)
.list()
The problem I had is I could not get Restrictions.in to work. Restrictions.eq works fine.
the in clause should do:
indications {
'in' 'id', indicationsId.collect{ UUID.fromString indication.toString() }
}
I am reading data from files by defining path as *.log etc,
Files names are like app1a_test2_heep.log , cdc2a_test3_heep.log etc
How to configure logstash so that the part of string that is string before underscore (app1a, cdc2a..) should be grepped and added to host field and removing the default host.
Eg:
fileName: app1a_test2_heep.log
host => app1a
Thanks in advance,
Ravi
Ruby filter can do what you want.
input {
file {
path => "/home/benlim/app1a_test2_heep.log"
}
}
filter {
ruby {
code => "
filename = event['path'].split('/').last
event['host'] = filename.split('_').first
"
}
}
output {
stdout { codec => rubydebug }
}
First, get the filename from the path. Then, get the hostname.
You can gsub (substitute) in mutate filter. It takes 3 arguments - field, what to sub and on what. You can use regexp here.
filter {
mutate {
gsub => [
# replace all after slashes with nothing
"host", "[_.*]", ""
]
}
}