So I have this log and I was trying to parse it using benthos grok. What I need to do is return json 5 elements:
• Timestamp
• Connection direction (inbound/outbound)
• Source IP
• Destination IP
• Source Port
in json format of this log:
<134>1 2023-01-21T17:18:05Z CHKPGWMGMT CheckPoint 16575 - [action:"Accept"; flags:"411908"; ifdir:"outbound"; ifname:"eth0"; logid:"0"; loguid:"{0x80c5f24,0x273f572f,0x1a6c6aae,0x5f835b6e}"; origin:"10.21.10.2"; originsicname:"cn=cp_mgmt,o=CHKPGWMGMT..f6b99b"; sequencenum:"4"; time:"1674314285"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F7CAC520-C428-484E-8004-06A1FAC151A3};mgmt=CHKPGWMGMT;date=1667399823;policy_name=Standard]"; dst:"10.21.10.2"; inzone:"Local"; layer_name:"Network"; layer_uuid:"8a994dd3-993e-4c0c-92a1-a8630b153f4c"; match_id:"1"; parent_rule:"0"; rule_action:"Accept"; rule_uid:"102f52bf-da21-49cd-b2e2-6affe347215d"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"46540"; service:"1433"; service_id:"https"; src:"10.21.9.1"]
input:
type: file
file:
paths: [./intput.txt]
codec: lines
pipeline:
processors:
- grok:
expressions:
- '%{NGFWLOGFILE}'
pattern_definitions:
NGFWLOGFILE: '%{NOTSPACE:interfaceid} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:Letters} %{NOTSPACE:Mhm} %{NOTSPACE:Skaicius} %{NOTSPACE:AA} %{NOTSPACE:Action}'
# - mapping: |
# root.timestamp = this.timestamp
# root.Action = this.Action
output:
stdout: {}
#output:
# label: ""
# file:
# path: "Output.txt"
# codec: lines
So I tried using grok to parse the log to json format and mapping to filter the part I want to get.
The part there I got stuck is pattern_definitions how to extract data from the list which already has names at the log file or I should use some better approach to the task?
Grok translates to a regular expression under the covers, so I don't think it has any notion of lists and such. Try this:
input:
generate:
count: 1
interval: 0s
mapping: |
root = """<134>1 2023-01-21T17:18:05Z CHKPGWMGMT CheckPoint 16575 - [action:"Accept"; flags:"411908"; ifdir:"outbound"; ifname:"eth0"; logid:"0"; loguid:"{0x80c5f24,0x273f572f,0x1a6c6aae,0x5f835b6e}"; origin:"10.21.10.2"; originsicname:"cn=cp_mgmt,o=CHKPGWMGMT..f6b99b"; sequencenum:"4"; time:"1674314285"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F7CAC520-C428-484E-8004-06A1FAC151A3};mgmt=CHKPGWMGMT;date=1667399823;policy_name=Standard]"; dst:"10.21.10.2"; inzone:"Local"; layer_name:"Network"; layer_uuid:"8a994dd3-993e-4c0c-92a1-a8630b153f4c"; match_id:"1"; parent_rule:"0"; rule_action:"Accept"; rule_uid:"102f52bf-da21-49cd-b2e2-6affe347215d"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"46540"; service:"1433"; service_id:"https"; src:"10.21.9.1"]"""
pipeline:
processors:
- grok:
expressions:
- "%{NGFWLOGFILE}"
pattern_definitions:
NGFWLOGFILE: |-
%{NOTSPACE:interfaceid} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:Letters} %{NOTSPACE:Mhm} %{NOTSPACE:Skaicius} %{NOTSPACE:AA} \[%{GREEDYDATA}; ifdir:"%{DATA:connectionDirection}"; %{GREEDYDATA}; dst:"%{DATA:destinationIP}"; %{GREEDYDATA}; s_port:"%{DATA:sourcePort}"; %{GREEDYDATA}; src:"%{DATA:sourceIP}"\]
output:
stdout: {}
Related
I have a yaml file with the following structure:
transfers:
- name: xyz
cloud: aws
subheading:
impact: Low
reason: ---
artifacts:
- name: name1
type: type1
source:
hash: a1b2C3dd4 ---> VALUE TO OVERWRITE
I would like to overwrite the existing hash value with a value of the latest GIT_COMMIT.
I have tried the method from the following question: write yaml file in jenkins with groovy. However, the value of hash[0][0] remains unchanged. This is the case even when I replace env.GIT_COMMIT with a test hash string "testHash123". I'm unsure why this is the case?
def filename = ('path/to/file.yaml')
def datas = readYaml file: filename
//change hash
datas.transfers['artifacts'].source.hash[0][0] = env.GIT_COMMIT
writeYaml file: filename, data: datas, overwrite: true
Please try the following.
datas.transfers[0]['artifacts'][0]['source'].hash = env.GIT_COMMIT
The easiest way to figure this out is by printing, so you can understand the structure.
[transfers:[[name:xyz, cloud:aws, subheading:[impact:Low, reason:xxxx], artifacts:[[name:name1, type:type1, source:[hash:a1b2C3dd4]]]]]]
As you can see above the transfer is a sequence, so you need to extract the correct segment with an index.
I have a yaml file(config.yaml) with tags/structure similar to what is mentioned below. I need to add a new tenant(tenant3) to the list of the existing tenants. How do I achieve it using the pipeline/groovy script? Any help/lead would be appreciated.
consumer_services:
- security
- token
id: 10000
tenants:
tenant_1:
state: all
web_token: true
cluster_pairs:
- cluster1
datacenter: local
client: CLIENT_TEST
tenant_2:
state: all
web_token: true
cluster_pairs:
- cluster2
datacenter: local
client: CLIENT_TEST
base_network:
subnets:
- 10.160.10.10
- 10.179.1.09
I think you need to do something like that:
#Grab('org.yaml:snakeyaml:1.17')
import org.yaml.snakeyaml.DumperOptions
import org.yaml.snakeyaml.Yaml
def options = new DumperOptions()
options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK)
Yaml yaml = new Yaml(options)
// load existing structure
def structure = yaml.load(new File("original.yml").text)
// modify the structure
structure.tenants.tenant_3 =
[
state : 'all',
web_token : true,
cluster_pairs: ['cluster3'],
datacenter : 'local',
client : 'CLIENT_TEST'
]
// save to the new file
new File("modified.yml").write(yaml.dump(structure))
So the steps are:
load the data from file into the memory
modify the structure in a way you like
store the modified structure to the file
I hope it will help.
I'm using OpenApi 3. A tool I use, Owasp Zap looks at the OpenAPI doc and creates fake requests. When it gets a 404, it complains that it doesn't have the media type that the OpenAPI promises.
But I didn't write anything in the OpenAPI doc about how 404s are handled. Obviously I can't write an infinite number of bad end points & document that they return 404s.
What is the right way to record this in the OpenAPI yaml or json?
Here is a minimal yaml file... I know for sure that this file does say anything about 404, ie. 404s aren't in the contract so tools are complaining that 404s are valid responses, but 404 is what a site should return when a resource is missing
---
"openapi": "3.0.0"
paths:
/Foo/:
get:
responses:
"200":
content:
application/json:
schema:
$ref: "#/components/schemas/Foo"
default:
description: Errors
content:
application/json:
schema:
$ref: "#/components/schemas/Error"
components:
schemas:
Foo:
type: object
required:
- name
properties:
name:
type: string
Error:
type: object
required:
- error
properties:
error:
type: string
message:
type: string
data:
type: object
This has been proposed already but not implemented: https://github.com/OAI/OpenAPI-Specification/issues/521
In the comments someone gave a suggestion: https://github.com/OAI/OpenAPI-Specification/issues/521#issuecomment-513055351, which reduces a little your code, but you would still have to insert N*M entries for N paths * M methods.
Since we don't have the ability to make the specification change to our needs, all that remains is we adapting ourselves.
From your profile, you seem to be a windows user. You can for example, create a new explorer context menu to your .yaml files (Add menu item to windows context menu only for specific filetype, Adding a context menu item in Windows for a specific file extension), and make it run a script that auto-fills your file.
Here, an example python script called yamlfill404.py that would be used in the context call in a way like path/to/pythonexecutable/python.exe path/to/python/script/yamlfill404.py %1, where %1 is the path to the file being right clicked.
Python file:
import yaml
from sys import argv
import re
order = ['openapi','paths','components']
level0re = re.compile('(?<=\n)[^ ][^:]+')
def _propfill(rootnode, nodes, value):
if len(nodes) == 1:
rootnode[nodes[0]] = value
if len(nodes) > 1:
nextnode = rootnode.get(nodes[0])
if rootnode.get(nodes[0]) is None:
nextnode = {}
rootnode[nodes[0]] = nextnode
_propfill(nextnode, nodes[1:], value)
def propfill(rootnode, nodepath, value):
_propfill(rootnode, [n.replace('__slash__','/') for n in nodepath.replace('\/','__slash__').split('/')], value)
def yamlfill(filepath):
with open(filepath, 'r') as file:
yamltree = yaml.safe_load(file)
#propfill(yamltree, 'components/schemas/notFoundResponse/...', '')
propfill(yamltree, 'components/responses/notFound/description', 'Not found response')
propfill(yamltree, 'components/responses/notFound/content/application\/json/schema/$ref', '#/components/schemas/notFoundResponse')
responses = [mv['responses'] if 'responses' in mv else [] for pk,pv in (yamltree['paths'].items() if 'paths' in yamltree else []) for mk,mv in pv.items()]
for response in responses:
propfill(response, '404/$ref', '#/components/responses/notFound')
yamlstring = yaml.dump(yamltree)
offsets = [i[1] for i in sorted([(order.index(f.group(0)) if f.group(0) in order else len(order),f.start()-1) for f in [f for f in level0re.finditer('\n'+yamlstring)]])]
offsets = [(offset,(sorted([o for o in offsets if o > offset]+[len(yamlstring)-1])[0])) for offset in offsets]
with open(filepath[:-5]+'_404.yaml', 'w') as file:
file.write(''.join(['\n'+yamlstring[o[0]:o[1]] for o in offsets]).strip())
yamlfill(argv[-1])
It processes the %1, which would be path/to/original.yaml and saves it as path/to/original_404.yaml (but you can change it to overwrite the original).
This example script changes the yaml formating (quotes type, spacing, ordering etc), because of the library used pyyaml. I had to reorder the file with the order = ['openapi','paths','components'], because it loses ordering. For less instrusion, maybe a more manual insertion would be better suited. Maybe one that uses only regex. Maye using awk, there are plenty of ways.
Unfortunately it is just a hack not not a solution.
I've some YAML files which I wanted to apply to create Custom Resources. But before applying it I want to change the spec and ENVs of the YAML snippet. So what could be the best way to do this?
What I'm doing now is:
Let suppose this is the YAML
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: default
spec:
appinfo:
appns: 'default'
applabel: 'app=nginx'
appkind: 'deployment'
# It can be true/false
annotationCheck: 'false'
# It can be active/stop
engineState: 'active'
chaosServiceAccount: pod-delete-sa
monitoring: false
# It can be delete/retain
jobCleanUpPolicy: 'delete'
experiments:
- name: pod-delete
spec:
components:
env:
# set chaos duration (in sec) as desired
- name: TOTAL_CHAOS_DURATION
value: '30'
# set chaos interval (in sec) as desired
- name: CHAOS_INTERVAL
value: '10'
# pod failures without '--force' & default terminationGracePeriodSeconds
- name: FORCE
value: 'false'
I Download this file from the raw link. (1 function for this)
Replace a field like jobCleanUpPolicy: 'delete' to jobCleanUpPolicy: 'retain' (1 fn)
Replace the next line when a match is found like value: 'false' to value: 'true' for FORCE ENV. (1 fn)
Applying the final manifest created! (1 function)
Can this be optimized?
If you create a struct representing the resource (or, even better, can import the package which defines the CRD) you could take the yaml string, marshal it into the struct and then edit the fields directly as fields of a struct instead
Hi You may use LitmusPortal to use this.Refer
I am looking at a situation where I'd like to bring some structure to what would be a string in an typical language. And wondering how to use Rebol's parts box to do it.
So let's say I've got a line that looks like this in the original language I'm trying to dialect:
something = ("/foo/mumble" "/foo/${BAR}/baz")
I want to use Rebol's primitives, so certainly a file path. Here is a random example of what I thought of off the top of my head:
something: [%/foo/mumble [%/foo/ BAR %/baz]]
If it were code you'd use REJOIN or COMBINE. But this is not designed to be executed, it's more like a configuration file. You're not supposed to be running arbitrary code, just getting a list of files.
I'm not sure about how feasible it is to stick with strings and yet still have these type as FILE!. Not all characters work in a FILE!, for instance:
>> load "%/foo/${BAR}/baz"
== [%/foo/$ "BAR" /baz]
It makes me wonder what my options are in Rebol data that's supposed to represent a configuration file. I can use plain old strings and do substitutions like other things do. Maybe REWORD with an OBJECT block to represent the environment?
What is the 'reword' function in Rebol and how do I use it?
In any case, I want to know how to represent a filename in a declarative context with environment variable substitutions like this.
I should use file! Your example need "" after %
f: load {%"/foo/${BAR}/baz"}
replace f "${BAR}" "MYVALUE" ;== %/foo/MYVALUE/baz
you could use path! with parens.
the only issue is the root, for which you can use another character to replace the "%" used for files... let's use '! (note this should be a word 'valid character).
when calling to-block on a path! type, it returns each part as its own token... useful.
to-block '!/path/(foo)/file.txt
== [! path (foo) file.txt]
here is a little script which loads three paths and uses parens as a constructed part of the path and uses tags to escape path-illegal characters (like a space!)
environments: make object! [
foo: "FU"
bar: "BR"
]
paths: [
!/path/(foo)/file.txt
!/root/<escape weird chars $>/(bar ".txt")
!/("__" foo)/path/(bar)
]
parse paths [
some [
(print "------" )
set data path! here: ( insert/only here to-block data to-block data )
(out-path: copy %"" )
into [
path-parts: (?? path-parts)
'!
some [
[ set data [word! | tag! | number!] (
append out-path rejoin ["/" to-string data]
)]
|
into [
( append out-path "/")
some [
set data word! ( append out-path rejoin [to-string get in environments data] )
| set data skip ( append out-path rejoin [ to-string data])
]
]
| here: set data skip (to-error rejoin ["invalid path token (" type? data ") here: " mold here])
]
]
(?? out-path)
]
]
Note this works both in Rebol3 and Rebol2
output is as follows:
------
path-parts: [! path (foo) file.txt]
out-path: %/path/FU/file.txt
------
path-parts: [! root <escape weird chars $> (bar ".txt")]
out-path: %/root/escape%20weird%20chars%20$/BR.txt
------
path-parts: [! ("__" foo) path (bar)]
out-path: %/__FU/path/BR
------