Show a match (grep) and show one column of the matched line - grep

Say I have a following text file (Nmap's .gnmap) and what I want is to show my match and corresponding IP addres. I.e. only show 22/open and 2000/open and also IP of those two ports. I need this result:
10.10.10.1 22/open 2000/open
10.10.10.2 2000/open
To get IPs is simple grep -iE "22/open|2000/open" file, but how do i also display IP? I need following output (exact extra characters does not matter, as long as each line contains IP and port):
Example source file:
Host: 10.10.10.1 () Ports: 22/open/tcp//ssh///, 80/open/tcp//http///, 113/closed/tcp//ident///, 443/open/tcp//https///, 541/open/tcp//uucp-rlogin///, 2000/open/tcp//cisco-sccp/// Ignored State: filtered (4994)
Host: 10.10.10.2 () Ports: 113/closed/tcp//ident///, 2000/open/tcp//cisco-sccp/// Ignored State: filtered (4998)

Could you please try following.
awk '
match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/){
ip_found=substr($0,RSTART,RLENGTH)
}
{
while(match($0,/22\/open|2000\/open/)){
val=(val?val OFS:"")substr($0,RSTART,RLENGTH)
$0=substr($0,RSTART+RLENGTH)
}
if(ip_found){
print ip_found,val
}
ip_found=val=""
}
' Input_file
Output will be as follows.
10.10.10.1 22/open 2000/open
10.10.10.2 2000/open

Related

Extract Unique IP sources from pcap file

My approach is to:
1 - Use tshark and export the list in a txt file.
tshark -r file.cap -T fields -e ip.src > output.txt
2 - Use sort to delete the double ips
sort output.txt | uniq > uniqueip.txt
3 - use uniqueip.txt to count the lines with
wc -l output.txt
I noticed right after i get the output.txt has some strange formatting where some ips are in line? why are they not in a new line?
This it the output.txt
"58.176.204.64"
"180.168.211.204"
"103.248.63.253"
"216.245.214.196,146.231.254.240"
"112.104.105.79"
"216.245.214.196,146.231.254.131"
"112.104.105.79"
"10.0.61.65,146.231.254.12"
The reason why some lines contain more than 1 IP address separated by a comma is because the packet itself contains more than 1 IP header. Such is the case for tunneling protocols or for ICMP error packets whose payload contains the original IP header that caused the ICMP error packet to be sent in the first place, or for other types of packets as well.

Parse YAML using shell

I have a yaml something of this sort and need to parse and get the value of 'url' here.
name: Yaml_Test
servers:
- name: host_ip
host: host_name
port: 443
scheme: https
variables:
- constant:
name: url
value: https://url_here
With the current versions of both kislyuk/yq and mikefarah/yq, use select to filter the array items:
.variables[].constant | select(.name == "url").value
https://url_here
Using kislyuk/yq, you may want to add the -r option to output raw text:
yq -r '…' file.yaml
Using an older version of mikefarah/yq, you need to specify the command e or eval:
yq e '…' file.yaml
Edit: Although not recommended, here's as requested in a comment a grep-only solution which heavily relies on how the YAML document is represented textually, based on the sample provided (i.e. no other occurrence of name: url etc.):
grep -A1 'name: url' file.yaml | grep -o 'http.*'
https://url_here

Fluent-bit Filter seems to only work when Match is *

I'm using docker-compose, that generates over 20 services. Most of them are similar, but parses different datetime format or values differ slightly. My logging idea is logging everything to systemd and then getting it over fluent-bit.
most of the services in docker-compose look something like this (tag beginning gets different names based on parser I will want to use later):
A-service:
image: A-service
restart: always
network_mode: host
depends_on:
- kafka
- schema-registry
environment:
- KAFKA_BROKERS=127.0.0.1:9092
- SCHEMA_REGISTRY_URL=127.0.0.1:8081
logging:
driver: journald
options:
tag: "dockerC/{{.ImageName}}/{{.Name}}/{{.ID}}"
B-service:
image: B-service
restart: always
network_mode: host
depends_on:
- kafka
- schema-registry
environment:
- KAFKA_BROKERS=127.0.0.1:9092
- SCHEMA_REGISTRY_URL=127.0.0.1:8081
logging:
driver: journald
options:
tag: "dockerJ/{{.ImageName}}/{{.Name}}/{{.ID}}"
fluent-bit.conf:
[SERVICE]
Flush 5
Daemon Off
Log_Level info
parsers_file parsers.conf
[INPUT]
Name systemd
Tag *
Path /run/log/journal
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
[FILTER]
Name parser
Parser dockerJ
Match dockerJ*
Key_Name MESSAGE
Reserve_Data On
Preserve_Key On
[FILTER]
Name parser
Parser dockerC
Match dockerC*
Key_Name MESSAGE
Reserve_Data On
Preserve_Key On
[OUTPUT]
Name es
Match *
Index fluent_bit
Type json
Retry_Limit false
Host ${ELASTICSEARCH_HOST}
Port ${ELASTICSEARCH_PORT}
HTTP_User ${ELASTICSEARCH_USERNAME}
HTTP_Passwd ${ELASTICSEARCH_PASSWORD}
tls off
tls.verify Off
parsers.conf
[PARSER]
Name dockerJ
Format json
Time_Key timeStamp
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped_utf8 MESSAGE do_next
Decode_Field json MESSAGE
[PARSER]
Name dockerC
Format json
Time_Key time
Time_Format %Y/%m/%d %H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped_utf8 MESSAGE do_next
Decode_Field json MESSAGE
if I change Filter Match:
Match dockerC* -> Match *
Match dockerJ* -> Match *
It matches and JSON gets parsed without any problem in es, but I get problems due to different time formats later in my elastic search or fluent-bit invalid time format error
I could edit and make like 8 different [INPUT] fields with different tags, but it seems just a waist of computer resources to do so.
So my question would be: how to actually use tags/filters and send messages based on Tags that are set outside the scope of fluent-bit (like in this case - docker-compose.yml)?
Systemd_Filter _SYSTEMD_UNIT=docker.service sets the TAG as "docker.service" and not the tag field I was expecting. To use the TAG I want - I have to manually change every TAG. Which is achievable my adding rewrite_tag filter:
[FILTER]
Name rewrite_tag
Match docker.service*
Rule $NAME_OF_THE_TAG_KEY .* $NAME_OF_THE_TAG_KEY false
Emitter_Name re_emitted
since I want to change every field I just added .* regex that matches to anything

Lua. look for an associated (with MAC) IP in dhcp.leases

I am looking for a Lua replacement for the following bash script:
MAC="d4:be:d9:3a:78:88"
IP=`grep $MAC /tmp/dhcp.leases | awk {'print $3'}`
echo $IP
/tmp/dhcp.leases looks like:
1497518739 d4:be:d9:3a:78:88 192.168.96.180 DESKTOP-2VECMJ7 01:d4:be:d9:3a:78:88
After reading the file into a Lua string, extracting the IP address is simple:
function extract(text,mac)
return text:match(" "..mac.." (.-) ")
end

output as a single row from dhcp output

I'm running dhcp query through powershell. I am getting the information I want to see but not in a format I need.
Sample:
data I'm working with the command:
netsh dhcp server \\dhcpserver scope 10.0.1.0 show clientsvq
gets output of
10.0.1.21 - 255.255.255.0 -00-23-7d-e9-45-58 -2/19/2016 5:13:50 PM -D -BUILD-01.example.com -NonQuarantined -INACTIVE -No
10.0.1.22 - 255.255.255.0 -00-23-7d-e9-54-38 -2/19/2016 5:13:55 PM -D -BUILD-02.example.com -NonQuarantined -INACTIVE -No
10.0.1.23 - 255.255.255.0 -00-23-7d-e8-cf-80 -6/11/2016 11:30:30 AM -D -BUILD-03.example.com -NonQuarantined -INACTIVE -No
What I want as an output is this:
10.0.1.21 00237de94558 BUILD-01.example.com
10.0.1.22 00237de95438 BUILD-02.example.com
10.0.1.23 00237de8cf80 BUILD-03.example.com
tried formatting the output by extracting the desired column and stripping off the special characters, command:
netsh dhcp server \\dhcpserver scope 10.0.1.0 show clientsvq | findstr example.com | foreach {"$(($_ -split '\s+',10)[0,3])" -replace "\-",""; "$(($_ -split '\s+',10)[8])" -replace "^-","";}
I get the following output format:
10.0.1.21 00237de94558
BUILD-01.example.com
10.0.1.22 00237de95438
BUILD-02.example.com
10.0.1.23 00237de8cf80
BUILD-03.example.com
Can someone point me out where my logic is flawed? I can't seemed to get to the much needed correct output. Thanks in advance.
Here's another way to parse the output and return rich objects instead of text:
netsh dhcp server \\dhcpserver scope 10.0.1.0 show clientsvq | foreach {
$values = ($_ -split '\s+-')[0,2,5]
if($values[0] -as [ipaddress])
{
New-Object PSObject -Property #{
IPAddress = [ipaddress]$values[0].Trim()
MAC = $values[1].Trim() -replace '-'
Name = $values[2].Trim()
}
}
}
Here's a version that seems to work. I simplified what you attempted by storing the array returned by the -split operator in a variable to avoid the unwinding that would happen if it was sent down the pipeline. This way we can access each column in a straightforward way. Note, to simplify, I just replaced your dhcp command with $dhcpcommand:
$dhcpcommand | select-string example.com | %{$cols = #($_ -split '\s+'); "$($cols[0]) $($cols[3] -replace '-','') $($cols[8])"}

Resources