I want to do two things: Convert IP Address inputs into CIDR
Here are some example inputs:
1.1.1.1
192.168.*.* #=> 192.168.0-255.0-255
192.168.1.2-20
1.1.1-10.1-100
Check if a given IP Address falls into any CIDR. This must be a very fast query, as it's a very common lookup in my web app. I'm thinking of doing something like this:
def matches?(request)
valid = #ips.select {|cidr| cidr.contains?(request.remote_ip) }
!valid.empty?
end
I think converting IP ranges into CIDR will let lookups be faster than what we're doing now, which is breaking the IP's into integer octets. We then index the first two sets of octets to partially match against IP's. Another option might be converting everything to ints and doing comparisons that way. I'd convert to ints with something like this IPAddr.new("1.1.1.1").to_i but then I'd need to store an upper and lower IP for each range instead of just a single CIDR.
Please let me know if I am overlooking any mainstream approaches, popular gems or repo's. Thanks!
Well, to get the CIDR notation of a range, you need an IP and the number of network bits (calculated from the netmask).
To enumerate the addresses of a given range, you can use the NetAddr (< 2.x) gem.
p NetAddr::CIDR.create('192.168.1.0/24').enumerate
=> ['192.168.1.0', '192.168.1.1', '192.168.1.2'... '192.168.1.255']
You can also calculate the bits from the netmask on the fly:
mask_int = NetAddr.netmask_to_i('255.255.255.0')
p NetAddr.mask_to_bits(mask_int)
=> 24
And to create a range based on two IPs:
lower = NetAddr::CIDR.create('192.168.1.1')
upper = NetAddr::CIDR.create('192.168.1.10')
p NetAddr.range(lower, upper)
=> ['192.168.1.2', '192.168.1.3'... '192.168.1.9']
So now that you can create a CIDR range, you can check to see if an IP is a part of it:
cidr = NetAddr::CIDR.create('192.168.1.0/24')
p cidr.contains?('192.168.1.10')
=> true
I suspect everything you need is in IPAddr. I use this to see if the remote IP is coming from a private network:
['127.0.0.0/8', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16', '192.168.10.0/8'
].none?{|block| IPAddr.new(block) === request.remote_ip}
Maybe I'm misunderstanding the question, but it seems that one aspect of this question has not been addressed, that is, converting a range of ip addresses to one or more CIDR entries.
I use the following approach to lookup suspicious ip activity on my firewall, and if it is in a country that I'm not interested in allowing access (you know who you are) I use whois to lookup the address range, and then calculate the merged CIDRs as follows,
whois xxx.yyy.zzz.123
# find address range for this ip
range="xxx.yyy.zzz.0-xxx.yyy.zzz.255".split(/\s*-\s*/)
lower=range[0]
upper=range[1]
ip_net_range = NetAddr.range(lower, upper, :Inclusive => true, :Objectify => true)
cidrs = NetAddr.merge(ip_net_range, :Objectify => true)
This is a example on an internal network, but it is trivial to extend to a public ip block,
whois 192.168.1.3
range="192.168.0.0 - 192.168.255.255".split(/\s*-\s*/)
upper=range[0]
lower=range[1]
ip_net_range = NetAddr.range(lower, upper, :Inclusive => true, :Objectify => true)
cidrs = NetAddr.merge(ip_net_range, :Objectify => true)
p cidrs
[192.168.0.0/16]
Then I can pass that CIDR to my firewall software (shorewall) to have it dynamically drop that cidr(s).
You can use NetAddr (> 2.x.x) gem.
Use parse class method to create a IPv4Net object from its string representation. It will default is a /32 netmask if not specified.
private_ips = NetAddr::IPv4Net.parse('192.168.0.0/16')
Now you use rel method to determine the relationship with another IPv4Net object.
For example, if we would like to know if '2.3.4.5/31' is part of private_ips
private_ips.rel(NetAddr::IPv4Net.parse('2.3.4.5/31')).eql?(1) || private_ips.rel(NetAddr::IPv4Net.parse('2.3.4.5/31')).eql?(0)
=> false
private_ips.rel(NetAddr::IPv4Net.parse('192.168.0.0/31')).eql?(1) || private_ips.rel(NetAddr::IPv4Net.parse('192.168.0.0/31')).eql?(0)
=> true
Note: rel method determines the relationship to another IPv4Net.
Returns:
1 if this IPv4Net is the supernet of other.
0 if the two are equal.
-1 if this IPv4Net is a subnet of other.
nil if the networks are unrelated.
Gem Documentation Link
Related
I'm trying to configure into Prometheus Alerting Manager an alert that appears when the status of 2 different hosts is down.
To better explain, I have these couples of hosts (host=instance):
host1.1
host1.2
host2.1
host2.2
host3.1
host3.2
host4.1
host4.2
...
and I need an alert that appears when both hosts of the SAME couple are DOWN:
expr = ( icmpping{instance=~"hostX1"}==0 and icmpping{instance=~"hostX2"}==0 )
(I know that the syntax is not correct, I just wanted to underline that X refers to the same number on both icmppingconditions)
Any hint?
The easiest way is perhaps to generate a label at ingestion time reflecting this logic, using relabel_config
relabel_configs:
- source_labels: [host]
regex: ^(.*)\.\d+$
target_label: host_group
It will generate the label you need for matching:
host=host1.1 => host_group=host1
host=host1.2 => host_group=host1
You can then use it for your alerting rules.
sum(icmpping) on(host_group) == 0
If this is not possible, you can use label_replace to achieve the same (only on instant vectors)
sum(label_replace(icmpping,"host_group","$1","host","(.*)\._\\d+")) on(host_group) == 0
I've been seeing some examples such as http://38263628 that resolves to a legitimate website https://example.com
When doing dig or nslookup on 38263628, it returns me NXDOMAIN.
How does the redirection or resolution work?
For data protection reasons, I can't give the actual numbers and domain it resolves to.
Thanks!
We usually write IP addresses as four dot-separated numbers like 12.34.56.78 but the address is just a number and it can be written in other ways. For example, 12.34.56.78 can also be written as 12.34.14414 or 12.2242638 or 203569230 or 0xc22384e or 0xc.0x22.0x38.0x4e.
The four-dot-separated form is effectively a base-256 representation of the number, so to convert from a single large number N into the four-dot-separated form A.B.C.D you would calculate these four values:
A = N / 256 / 256 / 256
B = N / 256 / 256 % 256
C = N / 256 % 256
D = N % 256
So your 38263628 is equivalent to 2.71.219.76
dig and nslookup don't work with plain numbers because they treat their arguments as strings, even when the arguments are known to represent IP addresses. (For instance, man dig says that the -x option requires an address in the four-dot form. That's because dig just wants to treat it as a string that can be split on the dots and rearranged into a in-addr.arpa query.)
However, most commands that convert their arguments into actual IP address numbers will accept any of the number formats. Try it with ping (which will helpfully show the address in four-dot format when it shows its results):
$ ping 203569230
PING 203569230 (12.34.56.78): 56 data bytes
...
That also gives you an easy way to convert any number into four-dot form -- just give the number as an argument to ping and let it do the conversion for you.
this is a three part question on the use of the Python API to Z3 (Z3Py).
I thought I knew the difference between a constant and a variable but apparently not. I was thinking I could declare a sort and instantiate a variable of that sort as follows:
Node, (a1,a2,a3) = EnumSort('Node', ['a1','a2','a3'])
n1 = Node('n1') # c.f. x = Int('x')
But python throws an exception saying that you can't "call Node". The only thing that seems to work is to declare n1 a constant
Node, (a1,a2,a3) = EnumSort('Node', ['a1','a2','a3'])
n1 = Const('n1',Node)
but I'm baffled at this since I would think that a1,a2,a3 are the constants. Perhaps n1 is a symbolic constant, but how would I declare an actual variable?
How to create a constant set? I tried starting with an empty set and adding to it but that doesn't work
Node, (a1,a2,a3) = EnumSort('Node', ['a1','a2','a3'])
n1 = Const('n1',Node)
nodes = EmptySet(Node)
SetAdd(nodes, a1) #<-- want to create a set {a1}
solve([IsMember(n1,nodes)])
But this doesn't work Z3 returns no solution. On the other hand replacing the 3rd line with
nodes = Const('nodes',SetSort(Node))
is now too permissive, allowing Z3 to interpret nodes as any set of nodes that's needed to satisfy the formula. How do I create just the set {a1}?
Is there an easy way to create pairs, other than having to go through the datatype declaration which seems a bit cumbersome? eg
Edge = Datatype('Edge')
Edge.declare('pr', ('fst', Node), ('snd',Node))
Edge.create()
edge1 = Edge.pr(a1,a2)
Declaring Enums
Const is the right way to declare as you found out. It's a bit misleading indeed, but it is actually how all symbolic variables are created. For instance, you can say:
a = Const('a', IntSort())
and that would be equivalent to saying
a = Int('a')
It's just that the latter looks nicer, but in fact it's merely a function z3 folks defined that sort of does what the former does. If you like that syntax, you can do the following:
NodeSort, (a1,a2,a3) = EnumSort('Node', ['a1','a2','a3'])
def Node(nm):
return Const(nm, NodeSort)
Now you can say:
n1 = Node ('n1')
which is what you intended I suppose.
Inserting to sets
You're on the right track; but keep in mind that the function SetAdd does not modify the set argument. It just creates a new one. So, simply give it a name and use it like this:
emptyNodes = EmptySet(Node)
myNodes = SetAdd(emptyNodes, a1)
solve([IsMember(n1,myNodes)])
Or, you can simply substitute:
mySet = SetAdd(SetAdd(EmptySet(Node), a1), a2)
which would create the set {a1, a2}.
As a rule of thumb, the API tries to be always functional, i.e., no destructive updates to existing variables, but you instead create new values out of old.
Working with pairs
That's the only way. But nothing is stopping you from defining your own functions to simplify this task, just like we did with the Node function in the first part. After all, z3py is essentially Python library and z3 folks did a lot of work to make it nicer, but you also have the entire power of Python to simplify your life. In fact, many other interfaces to z3 from other languages (Scala, Haskell, O'Caml etc.) precisely do that to provide a much easier to work with API using the features of their respective host languages.
Regarding the filters in Wireshark, what is the differencebetween !(ip.addr == 192.0.2.1) and (ip.addr != 192.0.2.1)? When check the result, it's not giving the same result and I don't know why...
Plus, when I apply the filter (ip.addr != 192.0.2.1) appears a different color as background in filters (Yellow).
image
Does anyone can help me? Thanks in advance.
In Boolean Logic, A not equals B and not A equals B are the same test.
But, the relevant part of the WireShark documentation linked by Jürgen Thelen explains that in WireShark, ip.addr covers both the source and destination field, so the test is more like:
not ((A or B) equals C)
which filters packets where source or destination match, and then hides them (correctly).
Compared to:
(A or B) not equals C
which filters packets where either the source OR the destination is not C, and that's every packet, so it shows every packet.
6.4.6. A Common Mistake
Using the != operator on combined expressions like eth.addr, ip.addr, tcp.port, and udp.port will probably not work as expected.
Often people use a filter string to display something like ip.addr == 1.2.3.4 which will display all packets containing the IP address 1.2.3.4.
Then they use ip.addr != 1.2.3.4 to see all packets not containing the IP address 1.2.3.4 in it. Unfortunately, this does not do the expected.
Instead, that expression will even be true for packets where either source or destination IP address equals 1.2.3.4. The reason for this, is that the expression ip.addr != 1.2.3.4 must be read as “the packet contains a field named ip.addr with a value different from 1.2.3.4”. As an IP datagram contains both a source and a destination address, the expression will evaluate to true whenever at least one of the two addresses differs from 1.2.3.4.
The reason it takes a yellow background is because of this potentially surprising behaviour, and there is a matching warning at the bottom of the screen, in the status bar, suggesting checking the user guide for more details:
Suppose we want to filter out any traffic to or from 10.43.54.65. We might try the following:
ip.addr != 10.43.54.65.
This translates to "pass all traffic except for traffic with a source IPv4 address of 10.43.54.65 and a destination IPv4 address of 10.43.54.65".
! ( ip.addr == 10.43.54.65 ).
This translates to "pass any traffic except with a source IPv4 address of 10.43.54.65 or a destination IPv4 address of 10.43.54.65".
You can see more on this at Wireshark: DisplayFilters.
I have a filter optimization problem in Redis.
I have a Redis SET which keeps the doc and pos pairs of a type in a corpus.
example:
smembers type_in_docs.1
result: doc.pos pairs
array (size=216627)
0 => string '2805.2339' (length=9)
1 => string '2410.14208' (length=10)
2 => string '3516.1810' (length=9)
...
Another redis set i create live according to user choices
It contains selected docs.
smembers filteredDocs
I want to filter doc.pos pairs "type_in_docs" set according to user Doc id choices.
In fact if i didnt use concat values in set it was easy with SINTER.
So i implement a php filter code as below.
It works but need an optimization.
In big doc.pairs set too much time need. (Nearly After 150000 members!)
$concordance= $this->redis->smembers('types_in_docs.'.$typeID);
$filteredDocs= $this->redis->smembers('filteredDocs');
$filtered = array_filter($concordance, function($pairs) use ($filteredDocs) {
if( in_array(substr($pairs, 0, strpos($pairs, '.')), $filteredDocs) ) return true;
});
I tried sorted set with scores as docId.
Bu couldnt find a intersect or filter option for score values.
I am thinking and searching a Redis based solution with supported keys, sets or Lua script for time optimization.
But nothing find.
How can i filter Redis sets with concat values?
Thanks for helps.
Your code is slow primarily because you're moving a lot of data from Redis to your PHP filter. The general motivation here should be perform as much filtering as possible on the server. To do that you'd need to pay some sort of price in CPU & RAM.
There are many ways to do this, here's one:
Ensure you're using Redis v2.8.9 or above.
To allow efficiently looking for doc only, keep your doc.pos pairs as is but use Sorted Sets with score = 0, your e.g.:
ZADD type_in_docs.1 0 2805.2339 0 2410.14208 0 3516.1810
This will allow you to mimic SISMEMBER for doc in the set with:
ZRANGEBYLEX type_in_docs.1 [<$typeID> (<$typeID + "\xff">
You can now just SMEMBERS on the (usually) smaller filterDocs set and then call ZRANGEBYLEX on each for immediate gains.
If you want to do better - in extreme cases (i.e. large filterDocs, small type_in_docs) you should do the reverse.
If you want to do even better, use Lua to wrap up the filtering logic - something like:
-- #usage: redis-cli --filter_doc_pos.lua <filter set keyname> <type pairs keyname>
-- #returns: list of matching doc.pos pairs
local r = {}
for _, fv in pairs(redis.call("SMEMBERS", KEYS[1])) do
local t = redis.call("ZRANGEBYLEX", KEYS[2], "[" .. fv , "(" .. fv .. "\xff")
for _, tv in pairs(t) do
r[#r+1] = tv
end
end
return r