OPA/Rego execute function for each element of an array - open-policy-agent

I am new at OPA/Rego and I am trying to write a policy to check if an Azure Network Security Group contains all the rules that I define on an array
package sample
default compliant = false
toSet(arr) = {x | x := arr[_]}
checkProperty(rule, index, propertySingular, propertyPlural) = true
{
object.get(input.properties.securityRules[index].properties, propertySingular, "") == object.get(rule, propertySingular, "")
count(toSet(object.get(input.properties.securityRules[index].properties, propertyPlural, [])) - toSet(object.get(rule, propertyPlural, []))) == 0
}
existRule(rule) = true
{
input.properties.securityRules[i].name == rule.name
input.properties.securityRules[i].properties.provisioningState == rule.provisioningState
input.properties.securityRules[i].properties.description == rule.description
input.properties.securityRules[i].properties.protocol == rule.protocol
checkProperty(rule, i, "sourcePortRange", "sourcePortRanges")
checkProperty(rule, i, "destinationPortRange", "destinationPortRanges")
checkProperty(rule, i, "sourceAddressPrefix", "sourceAddressPrefixes")
checkProperty(rule, i, "destinationAddressPrefix", "destinationAddressPrefixes")
input.properties.securityRules[i].properties.access == rule.access
input.properties.securityRules[i].properties.priority == rule.priority
input.properties.securityRules[i].properties.direction == rule.direction
}
compliant
{
rules := [
{
"name": "name1",
"provisioningState": "Succeeded",
"description": "description1",
"protocol": "*",
"sourcePortRange": "*",
"destinationPortRange": "53",
"destinationAddressPrefix": "*",
"access": "Allow",
"priority": 1,
"direction": "Inbound",
"sourceAddressPrefixes":
[
"xx.xx.xx.xx",
"xx.xx.xx.xx",
"xx.xx.xx.xx"
],
},
{
"name": "name2",
"provisioningState": "Succeeded",
"description": "description2",
"protocol": "*",
"sourcePortRange": "*",
"destinationPortRange": "54",
"sourceAddressPrefix": "*",
"access": "Allow",
"priority": 2,
"direction": "Outbound",
"destinationAddressPrefixes":
[
"xx.xx.xx.xx",
"xx.xx.xx.xx",
"xx.xx.xx.xx"
]
}
]
#checks
existRule(rules[i])
}
The issue seem to be that when execute existRule(rules[i]) if one of the rules match it returns true, don't mather if other rules doesn't
If I replace existRule(rules[i]) with existRule(rules[0]) or existRule(rules[1]), it return true or false depending on if the rule on that position matchs.
Is there any way to get the result of the execution of existRule(rules[i]) for all the elements of the array?
I already tried result := [existRule(rules[i])] but it only return one element with true

Sure! Use a list comprehension and call the function inside of it, then compare the size of the result to what you had before. Given your example, you would replace existRule(rules[i]) with something like this:
compliantRules := [rule | rule := rules[_]
existRule(rule)]
count(compliantRules) == count(rules)

Related

Telegraf splits input data into different outputs

I want to write Telegraf config file which will:
Uses openweathermap input or custom http request result
{
"fields": {
...
"humidity": 97,
"temperature": -11.34,
...
},
"name": "weather",
"tags": {...},
"timestamp": 1675786146
}
Splits result on two similar JSONs:
{
"sensorID": "owm",
"timestamp": 1675786146,
"value": 97,
"type": "humidity"
}
and
{
"sensorID": "owm",
"timestamp": 1675786146,
"value": -11.34,
"type": "temperature"
}
Sends this JSONs into MQTT queue
Is it possible or I must create two different configs and make two api calls?
I found next configuration which solves my problem:
[[outputs.mqtt]]
servers = ["${MQTT_URL}", ]
topic_prefix = "owm/data"
data_format = "json"
json_transformation = '{"sensorID":"owm","type":"temperature","value":fields.main_temp,"timestamp":timestamp}'
[[outputs.mqtt]]
servers = ["${MQTT_URL}", ]
topic_prefix = "owm/data"
data_format = "json"
json_transformation = '{"sensorID":"owm","type":"humidity","value":fields.main_humidity,"timestamp":timestamp}'
[[inputs.http]]
urls = [
"https://api.openweathermap.org/data/2.5/weather?lat={$LAT}&lon={$LON}2&appid=${API_KEY}&units=metric"
]
data_format = "json"
Here we:
Retrieve data from OWM in input plugin.
Transform received data structure in needed structure in two different output plugins. We use this language https://jsonata.org/ for this aim.

Loop through a nested json object in OPA

I am very new to OPA and trying to loop through the below input data
input =
"data":{
"list":[
{"Name": "abc", "Status": "Done"},
{"Name": "def", "Status": "Done"},
{"Name": "ghi", "Status": "pending"},
{"Name": "jkl", "Status": ""},
{"Name": "mno", "Status": null},
]
}
and return two lists based on that input, one list that would return the names of all objects that has the status as 'Done' and another list with status not equal to 'Done', here is the rego code I was trying, I used somewhat python syntax to convey the code as I was not sure how the opa syntax would look like
package play
default done_list := []
default other_list := []
done_list {
some i
input.data.list[i].Status == "Done"
done_list.append(input.data.list[i].Name) #python syntax and looking for something similar in opa
}
other_list{
some j
input.data.list[j].Status != "Done"
other_list.append(input.data.list[j].Name)
}
#or maybe use a list comprehension like this?
#not even sure if this makes sense but ...
done_list = [x.Name | x := input.data.list[_]; x.Status = "Done"]
other_list = [x.Name | x := input.data.list[_]; x.Status != "Done"]
test_return{
"done_list": done_list,
"other_list": other_list
}
and the output I'm looking for is something like
{
"done_list": ["abc", "def"],
"other_list": ["ghi","jkl","mno"]
}
Any help is greatly appreciated. Thanks in advance.
I think you're best to use a list comprehension as you guessed :)
package play
doneStatus := "Done"
result := {
"done_list": [e |
item := input.data.list[_]
item.Status == doneStatus
e := item.Name
],
"other_list": [e |
item := input.data.list[_]
item.Status != doneStatus
e := item.Name
],
}
I've created a Rego playground which shows this: https://play.openpolicyagent.org/p/dBKUXFO3v2

Need to print certain letters after a particular word using shell script

I am new to shell scripting and I want to print the IPs inside current cidr range and proposed cidr range below is the output
{
"acknowledgeRequiredBy": xxxxxx000,
"acknowledged": false,
"acknowledgedBy": "name#email.com",
"acknowledgedOn": xxxxxx00000,
"contacts": [
"dl#email.com",
"dl2#gmail.com"
],
"currentCidrs": [
"1.2.3.4/24",
"5.6.7.8/24",
"9.10.11.12/24",
"13.14.15.16/24",
"17.18.19.20/24",
"21.22.23.24/24",
"25.26.27.28/24",
],
"id": 1x4x3xx,
"latestTicketId": 0000,
"mapAlias": "sample name",
"mcmMapRuleId": 111xxx,
"proposedCidrs": [
"25.26.27.28/24",
"1.2.3.4/24",
"5.6.7.8/24",
"9.10.11.12/24",
],
"ruleName": "namerule.com",
"service": "S",
"shared": na,
"sureRouteName": "example",
I want the output as only the IPs please help me with the logic

Open Policy Agent - Improve performance of a grouping comprehension

I have a id and role mapping in below format
{
"ra": [
{
"id": 168,
"code": "AFAP"
},
{
"id": 180,
"code": "ABC"
},
{
"id": 180,
"code": "ABCMND"
}
]
}
I need the output to be like below
{
"roleactions": {
"168": [
"AFAP"
],
"180": [
"ABC",
"ABCMND",
"DCRMP"
]
}
}
So i wrote the below code
roleactions = r_map {
r := data.ra
r_map := {id: list |
some i
id := r[i].id
list := [obj |
some j
r[j].id == id
obj := r[j].code
]
}
}
But when I run this for it almost takes 5-6 seconds
Found 1 result in 5682526.465 µs.
Can someone guide on how to make write this policy map to improve performance?
OPA can evaluate comprehensions like this in linear-time: https://www.openpolicyagent.org/docs/latest/policy-performance/#comprehension-indexing. The problem in this case is that the local variable r is not safe when considering the comprehensions in isolation.
If you refactor the comprehension like below, the runtime should be linear:
roleactions := r_map {
r_map := {id: list |
some i
id := data.ra[i].id
list := [obj |
some j
data.ra[j].id == id
obj := data.ra[j].code
]
}
}

difference between detect-secrets and detect-secrets-hook results

I'm evaluating detect-secrets and I'm not sure why I get different results from detect-secrets and the hook.
Here is a log of a simplification:
$ cat docs/how-to-2.md
AZ_STORAGE_CS="DefaultEndpointsProtocol=https;AccountName=storageaccount1234;AccountKey=1OM7c6u5Ocp/zyUMWcRChowzd8czZmxPhzHZ8o45X7tAryr6JFF79+zerFFQS34KzVTK0yadoZGkvZh42A==;EndpointSuffix=core.windows.net"
$ detect-secrets scan --string $(cat docs/how-to-2.md)
AWSKeyDetector : False
ArtifactoryDetector : False
Base64HighEntropyString: True (5.367)
BasicAuthDetector : False
CloudantDetector : False
HexHighEntropyString : False
IbmCloudIamDetector : False
IbmCosHmacDetector : False
JwtTokenDetector : False
KeywordDetector : False
MailchimpDetector : False
PrivateKeyDetector : False
SlackDetector : False
SoftlayerDetector : False
StripeDetector : False
TwilioKeyDetector : False
$ detect-secrets-hook docs/how-to-2.md
$ detect-secrets-hook --baseline .secrets.baseline docs/how-to-2.md
I would have expected that detect-secrets-hook would tell me about that Azure storage account key that has high entropy.
more details about the baseline:
$ cat .secrets.baseline
{
"custom_plugin_paths": [],
"exclude": {
"files": null,
"lines": null
},
"generated_at": "2020-10-09T10:06:54Z",
"plugins_used": [
{
"name": "AWSKeyDetector"
},
{
"name": "ArtifactoryDetector"
},
{
"base64_limit": 4.5,
"name": "Base64HighEntropyString"
},
{
"name": "BasicAuthDetector"
},
{
"name": "CloudantDetector"
},
{
"hex_limit": 3,
"name": "HexHighEntropyString"
},
{
"name": "IbmCloudIamDetector"
},
{
"name": "IbmCosHmacDetector"
},
{
"name": "JwtTokenDetector"
},
{
"keyword_exclude": null,
"name": "KeywordDetector"
},
{
"name": "MailchimpDetector"
},
{
"name": "PrivateKeyDetector"
},
{
"name": "SlackDetector"
},
{
"name": "SoftlayerDetector"
},
{
"name": "StripeDetector"
},
{
"name": "TwilioKeyDetector"
}
],
"results": {
".devcontainer/Dockerfile": [
{
###obfuscated###
}
],
"deployment/export-sp.sh": [
{
###obfuscated###
}
],
"docs/pip-install-from-artifacts-feeds.md": [
{
###obfuscated###
}
]
},
"version": "0.14.3",
"word_list": {
"file": null,
"hash": null
}
}
This is definitely peculiar behavior, but after some investigation, I realize that you've stumbled upon an edge case of the tool.
tl;dr
HighEntropyStringPlugin supports a limited set of characters (not including ;)
To reduce false positives, HighEntropyStringPlugin leverages the heuristic that strings are quoted in certain contexts.
To improve UI, inline string scanning does not require quoted strings.
Therefore, the functionality differs: when run through detect-secrets-hook, it does not parse the string accordingly due to the existence of ;. However, when run through detect-secrets scan --string, it does not require quotes, and breaks the string up.
Detailed Explanation
HighEntropyString tests are pretty noisy, if not aggressively pruned for false positives. One way it attempts to do this is via applying a rather strict regex (source), which requires it to be inside quotes. However, for certain contexts, this quoted requirement is removed (e.g. YAML files, and inline string scanning).
When this quoted requirement is removed, we get the following breakdown:
>>> line = 'AZ_STORAGE_CS="DefaultEndpointsProtocol=https;AccountName=storageaccount1234;AccountKey=1OM7c6u5Ocp/zyUMWcRChowzd8czZmxPhzHZ8o45X7tAryr6JFF79+zerFFQS34KzVTK0yadoZGkvZh42A==;EndpointSuffix=core.windows.net"'
>>> with self.non_quoted_string_regex(is_exact_match=False):
... self.regex.findall(line)
['AZ_STORAGE_CS=', 'DefaultEndpointsProtocol=https', 'AccountName=storageaccount1234', 'AccountKey=1OM7c6u5Ocp/zyUMWcRChowzd8czZmxPhzHZ8o45X7tAryr6JFF79+zerFFQS34KzVTK0yadoZGkvZh42A==', 'EndpointSuffix=core', 'windows', 'net']
When we do so, we can see that the AccountKey=1OM7c6u5Ocp/zyUMWcRChowzd8czZmxPhzHZ8o45X7tAryr6JFF79+zerFFQS34KzVTK0yadoZGkvZh42A== would trigger the base64 plugin, as demonstrated below:
$ detect-secrets scan --string 'AccountKey=1OM7c6u5Ocp/zyUMWcRChowzd8czZmxPhzHZ8o45X7tAryr6JFF79+zerFFQS34KzVTK0yadoZGkvZh42A=='
AWSKeyDetector : False
ArtifactoryDetector : False
Base64HighEntropyString: True (5.367)
BasicAuthDetector : False
CloudantDetector : False
HexHighEntropyString : False
IbmCloudIamDetector : False
IbmCosHmacDetector : False
JwtTokenDetector : False
KeywordDetector : False
MailchimpDetector : False
PrivateKeyDetector : False
SlackDetector : False
SoftlayerDetector : False
StripeDetector : False
TwilioKeyDetector : False
However, when this quoted requirement is applied, then the entire string payload is scanned as one potential secret: DefaultEndpointsProtocol=https;AccountName=storageaccount1234;AccountKey=1OM7c6u5Ocp/zyUMWcRChowzd8czZmxPhzHZ8o45X7tAryr6JFF79+zerFFQS34KzVTK0yadoZGkvZh42A==;EndpointSuffix=core.windows.net
This doesn't get flagged because it fails the original base64 regex rules, which doesn't know how to handle ;.
>>> self.regex.findall(line)
[]
Therefore, this functionality differs, though, not immediately apparent through the invocation pattern described.
How Do I Fix This?
This is a much more challenging question, since allowing other characters would change the entropy calculation, and the probability of flagging strings. There has been some discussion around creating a plugin for all characters, but the team has not yet been able to decide on a default entropy limit for this.

Resources