I have this field in the raw log:
"user ip port" : 192.xxx.xx.xx:8080
I want to process those fields using grok like this
if "user ip port" {
grok{
match => { "c&c_ip_port" => ["^%{DATA:ip}\:%{DATA:port}$"] }
}
}
How do I select those fields in the if statement ?
already try using ["user device ip"] and [user ip port] but the field wont process by the grok.
Thanks
Related
A normal event could be like this:
2015-11-20 18:50:33,739 [TRE01_0101] [76] [10.117.10.220]
but sometimes I have a log with "default" IP:
2015-11-04 23:14:27,469 [TRE01_0101] [40] [default]
If I have defined in grok a [SYNTAX:SEMANTIC] pattern as follows:
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{DATA:instance}\] \[%{NUMBER:numeric}\] \[%{IP:client}\]}"}
}
How can I parse a log that contains dafault as IP?
Now I'm getting a _grokparsefailure because "default" is not an "IP SYNTAX".
Thanks in advance
You can group things together and then make them conditional:
(%{IP:client}|default)
Is there a way to make a Sensu check that takes a .log file as input and parses it and returns selected info to InfluxDB.
Im very new to this so maybe I didnt describe my problem the best way.
I found the best way to do this is with Logstash (mostly because I use ELK for general log aggregation anyway).
Set up a Logstash server.
https://www.elastic.co/products/logstash
Install logstash-forwarder on the client(s). Configure logstash-forwarder to read the logs you want and to send them to your logstash server.
https://github.com/elastic/logstash-forwarder
In the Logstash server's config;
Define a lumberjack input for the log you want to send to sensu (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-lumberjack.html).
Eg:
input {
lumberjack {
port => 5555
type => "logs"
tags => ["lumberjack", "influxdb"]
}
}
Do your processing/filtering.
Eg:
filter {
if ("influxdb" in [tags]) {
...
}
}
Define an InfluxDB output (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-influxdb.html).
Eg:
output {
influxdb {
...
}
}
This method would skip Sensu all together. If you do want to send the logs to Sensu and see the output in Uchiwa it would involve setting up some Sensu-friendly info in your logstash filter:
filter {
if ("influxdb" in [tags]) {
add_field => { "name" => "SensuCheckName" }
add_field => { "handler" => "SensuHandlerName" }
add_field => { "output" => "the stuff you want to send to sensu" }
add_field => { "status" => "1" }
}
}
And sending the logs to sensu's RabbitMQ transport (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-rabbitmq.html):
output {
rabbitmq {
exchange => "results"
exchange_type => "direct"
host => "192.168.0.5 or whatever it is"
vhost => "/sensu"
user => "sensuUser"
password => "whateverItIs"
}
}
Define a Sensu handler for this (name above in logstash filter) and do any extra processing there before passing it to InfluxDB.
If you haven't got Sensu sending data to InfuxBD set up already, go here: https://github.com/sensu-plugins/sensu-plugins-influxdb
I'm creating an expect script to let me use macros in my managing of cisco devices.
after all the connections are done, my script waits for the "#" so I know the user is ready for input. if I define my procedure AFTER interact, script works fine, but obviously it hasn't been declared yet so the script fails. if I define it BEFORE interact, I time out like it's actually "expecting"
why would this procedure "run" without even being called?
proc portSec {port} {
send "show port interface $int\r"
expect {
-nocase -re "(invalid)|(ambig}" {
puts "\nInvalid Interface\n"
return
}
-nocase -re "(\[0-9\]+.\[^ \]+.\[^ \]+):\[^ \]" {
set mac $expect_out(1,string)
}
}
~~~~expect "#" ~~~~~~
send "show port address \| i $mac\r"
expect "#"
}
interact {
"!p" {
send_user "\nWelcome to macro mode!\nWhich interface would you like to clear port security?: "
stty echo
expect_user -re "(.*)\r" {
set port $expect_out(1,string)
}
stty -echo
portSec $port
send "\r"
}
}
Here is the debug
expect: does " \r\nYour password will expire in 1 week, 5 days, 3 hours,
44 minutes and 56 seconds
\r\r\n\r\nHOSTNAME line 1 \r\n\r\nHOSTNAME#" (spawn_id exp4)
match glob pattern "#"? yes
expect: set expect_out(0,string) "#"
expect: set expect_out(spawn_id) "exp4"
expect: set expect_out(buffer) " \r\nYour password will expire in 1 week,
5 days, 3 hours, 44 minutes and 56 seconds\r\r\n\r\nHOSTNAME line 1
n\r\nHOSTNAME#"
expect: does "" (spawn_id exp4) match glob pattern "#"? no
expect: timed out
can't read "mac": no such variable
while executing
"send "show port address \| i $mac\r""
(file "./ios.exp" line 78)
stepping through the debugger, I tilded "~~~" the line where it's expecting
Oh!!! Poor me. How come I missed this mistake!!! :-D
After banging my head for a lot, found that you have used close braces by mistake in the expect statement.
-nocase -re "(invalid)|(ambig}" {
Changing it to bracket, solves this issue. :-)
proc portSec {port} {
send "show port interface $int\r"
expect {
-nocase -re "(invalid)|(ambig)" {
puts "\nInvalid Interface\n"
return
}
-nocase -re {\[0-9\]+.\[^ \]+.\[^ \]+):\[^ \]} {
set mac $expect_out(1,string)
}
}
expect "#";#
send "show port address \| i $mac\r"
expect "#"
}
It all makes sense now. But, why Expect didn't throw any error as such? Then Why it has to accept the remaining part of braces as correct one ?
Well, Because of the misplaced close brace, Expect assumes that the it closes the first expect statement. The code segment remaining are
# two code segments enclosed in braces
# 'Expect' assumes these two as pattern-action pair
# which is why no errors thrown for these blocks.
{
puts "\nInvalid Interface\n"
return
}
-nocase -re "(\[0-9\]+.\[^ \]+.\[^ \]+):\[^ \]" {
set mac $expect_out(1,string)
}
};# This one closes the proc 'portSec'
####### Below code is now in global scope #########3
expect "#"; # Now, wait for '#'
# Send this command now, but wait...
# I don't know about the variable 'mac'
send "show port address \| i $mac\r"
expect "#"
Thereby throwing the error message as can't read "mac": no such variable.
Final product:
#port security clear feature
proc clearPortSec {} {
send_user "\n=====\nWelcome to the Port Security Macro!!!!!\nWhich interface would you like to clear port security?: "
#interact seems to turn of user echoing... turn it back on so the user can make changes
stty echo
expect_user -re "(.*)\r" {
set port $expect_out(1,string)
}
stty -echo
#send ctrl+u to clear the line, and send ctrl+z so we know we are in enable mode
send "\025\032\r"
expect "#"
#verify it is explicitly an access port, if not we don't do the operations to prevent trunks from being shut
send "sho run int $port \| i switchport mode access\r"
expect {
-nocase -re "\r\n\[ \]*switchport mode access\[ \]*\r\n.*#" {
send "\rshow port interface $port\r"
expect {
#grabs offending mac address on this port
-nocase -re "(\[0-9\]+.\[^ \]+.\[^ \]+):\[^ \]" {
set mac $expect_out(1,string)
expect "#"
#finds where switch learned the offending mac
send "show port address \| i $mac\r"
expect {
#case where switch learned mac from another interface
-nocase -re "sticky +(\[^ \]+)" {
set offendport $expect_out(1,string)
puts "\n=====\nMac was found on interface $offendport. Clearing $offendport\n====="
#clear port mac was found on
expect "#"
send "\rconf t\r"; expect "#"
send "interface $offendport\r"; expect "#"
send "no switchport port-security mac-address sticky\r"; expect "#"
send "switchport port-security mac-address sticky\r"; expect "#"
send "shut\r"; expect "#"
send "no shut\r;" expect "#"
#switch to original port
send "interface $port\r"; expect "#"
send "no switchport port-security mac-address sticky\r"; expect "#"
send "switchport port-security mac-address sticky\r"; expect "#"
send "shut\r"; expect "#"
send "no shut\r"; expect "#"
#end all operation and print status of each port
send "end\r"; expect "#"
send "wr\r"; expect "#"
puts "\n=====\nSleeping a few seconds..\nPrevious port will NOT be shut. Please shut if necessary\n====="
send "show int $offendport status\r"; expect "#"
send "show int $port status\r"; expect "#"
}
#case where switch never learned mac
"#" {
puts "\n=====\nMac is no where else on the switch, clearing existing sticky mac from $port\n====="
send "\rconf t\r"; expect "#"
send "interface $port\r"; expect "#"
send "no switchport port-security mac-address sticky\r"; expect "#"
send "switchport port-security mac-address sticky\r"; expect "#"
send "shut\r"; expect "#"
send "no shut\r"; expect "#"
send "end\r"; expect "#"
send "wr\r"; expect "#"
puts "\n=====\nSleeping a few seconds..\n====="
sleep 3
send "show int $port status\r"
}
}
}
}
}
#if we get back '%' - input error, if we get back '#' switchport mode access doesn't exist
-re "\[%#\]" {
puts "\n=====\nInvalid Interface\nThis script ignores all interfaces that don't have \"Switchport mode access\"\n====="
send "\r"
}
}
}
# Don't check keys
spawn ssh -o StrictHostKeyChecking=no $username\#$hostname
expect {
"assword:" {
interact -o -re "\[#>\]" return {
puts "\nLog in Complete"
}
}
timeout {
puts "=====\n$hostname timeout\n====="
exit 1
}
eof {
exit 1
}
}
send "\n"
expect "#"
set timeout -1
interact {
-re "!\[Hh\]" {
puts "\n=====\nMacro list\n=====\nPort Security: !p\n\n"
send "\r"
expect "#"
}
"!p" { clearPortSec }
}
I have a docker container that log to stdout/stderr. Docker save it's output into /var/lib/docker/containers//-logs.json
The log has lines with the following structure
{"log":"This is a message","stream":"stderr","time":"2015-03-12T19:27:27.310818102Z"}
which input/codec/filter should I use to get only the log field as the message ?
Thanks!
Use the json codec to parse the JSON string (you could instead use the json filter), then rename the "log" field to "message" with the mutate filter and finally use the date filter to parse the "time" field.
filter {
mutate {
rename => ["log", "message"]
}
date {
match => ["time", "ISO8601"]
remove_field => ["time"]
}
}
I have a tcl script to log into devices and print SUCCESS. This is the script:
The file: (the first IP is valid, and can be logged into, the next 3 are fake).
192.38.133.145
178.18.34.48
183.24.56.3
145.234.67.145
The script:
#!/bin/expect
package require Expect
set file [open "hosts" r]
set f_data [read $file]
set data [split $f_data "\n"]
foreach host $data {
set timeout 8
if {$host > 0} {
## GETS THE HOST IP##
set host_ip $host
##LOGS INTO THE DEVICE##
spawn ssh test#$host_ip
expect {
"password:" {
puts "SUCCESS"
} timeout {
puts "Could not connect to host: ${host_ip}"
#break
}
}
send "password\r"
expect ">"
send "en\r"
}
}
If I do not include the break, I get the message could not connect to host, but instead of looping to the next host, it sends "en\r".
when I do include the break, it gives the message that it cannot reach the host (the second IP, which is expected) and the script ends there (it does not process the 3rd IP). How do I cannot seem to get it to process the 3rd and 4th IPs.
I used the method suggested by potrzebie in this thread: TCL: loops How to get out of inner most loop to outside?
and still cannot get it to work
break should work. The expect man page has this to say in the documentation for the expect command:
Actions such as break and continue cause control structures (i.e., for, proc) to behave in the usual way.
I'd write your loop like this:
foreach host $data {
# very basic data validation: an ipv4 address contains some dots
if {[string match {*.*.*.*} $host]} {
spawn ssh test#$host
expect {
"password:" {
puts "SUCCESS"
send "password\r"
exp_continue
}
timeout {
puts "Could not connect to host: $host"
continue
}
">" {
# I see a prompt. Fall out of this "expect loop"
}
}
send "en\r"
expect ">"
# do something to close the connection
send "exit\r"
expect eof
}
}