I am using the consul exporter to ingest the health and status of my services into Prometheus. I'd like to fire alerts when the status of services and nodes in Consul is critical and then use tags extracted from Consul when routing those alerts.
I understand from this discussion that service tags are likely to be exported as a separate metric, but I'm not sure how to join one series with another so I can leverage the tags with the health status.
For example, the following query:
max(consul_health_service_status{status="critical"}) by (service_name, status,node) == 1
could return:
{node="app-server-02",service_name="app-server",status="critical"} 1
but I'd also like 'env' from this series:
consul_service_tags{node="app-server-02",service_name="app-server",env="prod"} 1
to get joined along node and service_name to pass the following to the Alertmanager as a single series:
{node="app-server-02",service_name="app-server",status="critical",env="prod"} 1
I could then match 'env' in my routing.
Is there any way to do this? It doesn't look to me like any operations or functions give me the ability to group or join like this. As far as I can see, the tags would already need to be labels on the consul_health_service_status metric.
You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)
The answer in this question is accurate. I want to also share a clearer explanation on joining two metrics preserving SAME Labels (might not be directly answering the question). In these metrics following label is there.
name (eg: aaa, bbb, ccc)
I have a metric name metric_a, and if this returns no data for some of the labels, I wish to fetch data from metric_b. i.e:
metric_a has values for {name="aaa"} and {name="bbb"}
metric_b has values for {name="ccc"}
I want the output to be for all three name labels. The solution is to use or in Prometheus.
sum by (name) (increase(metric_a[1w]))
or
sum by (name) (increase(metric_b[1w]))
The result of this will have values for {name="aaa"}, {name="bbb"} and {name="ccc"}.
It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.
The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.
For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:
consul_health_service_status
* on(service_name, node) group_left(env)
consul_service_tags
Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:
consul_health_service_status{status="critical"}
* on(service_name, node) group_left(env)
consul_service_tags
I have a problem with the implementation in cypher. My problem is this: I have a database model, which is photographed here as an overview: https://www.instpic.de/QTIhBbPgVHBHg5pKwVdk.PNG
Short for the explanation. The red nodes simulate star systems, the yellow one jump points. Each jump point has a certain size, which determines which body can pass the point. The size is stored as a property at the relation between the yellow nodes. Among the red nodes are other nodes that represent the orbital celestial bodies of a star system. (Planets, moons, stations, etc.) Now, from any point within a solar system (planet, station, moon), I would like to find the shortest path to another lying point in the same solar system or another. In addition, I can calculate the distance of two celestial bodies within a system using the plugin that I have programmed. This value should now be included in finding the path, so I have the shortest path on the database and also the smallest distance between the celestial bodies within a solar system. I already have a query, unfortunately it fails partly because of its performance. I also think that the paths here are very variable, so a change to the database model is well considered.
Here is a part of my acutal query iam using:
MATCH (origin:Marketplace)
WHERE origin.eid = 'c816c4fa501244a48292f5d881103d7f'
OPTIONAL MATCH (marketplace:Marketplace)-[:Sell]->(currentPrice:Price)-[:Content]->(product:Product)
OPTIONAL MATCH p = shortestPath((origin)-[:HasMoon|:HasStation|:HasLandingZone|:HasPlanet|:HasJumpPoint|:CanTravel*]-(marketplace))
WHERE SIZE([rel in relationships(p) WHERE EXISTS(rel.size)]) <= 3 AND ALL(rel IN [rel in relationships(p) WHERE EXISTS(rel.size)] WHERE rel.size IN ['small', 'medium', 'large'])
WITH origin, marketplace, p, currentPrice, product
CALL srt.getRoutes(origin, marketplace, p) YIELD node, jump_sizes, jump_gates, jump_distance, hops, distance
OPTIONAL MATCH (currentPrice)-[:CompletedVotes]->(:Wrapper)-[:CompletedVote]->(voteHistory:CompletedVote)
OPTIONAL MATCH (currentPrice)-[:CurrentVote]->(vote:Vote)-[:VotedPrices]->(currentVotings)
WITH node, currentPrice, product, jump_sizes, jump_gates, jump_distance, hops, distance, voteHistory, currentVotings, vote, origin
WITH {eid: product.eid, displayName: product.displayName, name: product.name, currentPrice: {eid: currentPrice.eid, price: currentPrice.price}, currentVoting: {approved: vote.approved, count: Count(currentVotings), declined: vote.declined, users: Collect(currentVotings.userId), votes: Collect(currentVotings.price), voteAvg: round(100 * avg(currentVotings.price)) / 100}, voteHistory: Collect({votings: voteHistory.votings, users: voteHistory.users, completed: voteHistory.completed,
vote: voteHistory.votes}), marketplace: {eid: node.eid, name: node.name, type: node.type, designation: node.designation}, travel: {jumpSizes: jump_sizes, jumpGate: jump_gates, jumpDistance: jump_distance, jumps: hops, totalDistance: distance}} as sellOptions, currentPrice ORDER BY currentPrice.price
WITH Collect(sellOptions) as sellOptions
For the moment, this query works pretty well, but now I want to filter (after ".... dium ',' large '])" -> line 5) the minimum total distance you need to travel to reach your destination , I would like to realize this with my written plugin, which calculates the total distance in the path (getTotalDistance (path AS PATH))
For additional: when I cut of 'big' from the possible jump sizes, I get no result, but there is still a path in my graph that leads me to the goal.
For additional 2: iam working on neo4j 3.3.1 and i have set these config:
cypher.forbid_shortestpath_common_nodes=false
which not works in 3.3.3
EIDT 1: (More detailed explanation)
I have a place where I am. Then I search for marketplaces that sell some product. For this I can specify further filters. I can e.g. say that I can travel only through jump points of the size "large". Also, I only want marketplaces that are 4 system away.
Now, looking in the database for the above restrictions, I search for the shortest path to the market places I found.
It may well be that I have several paths that meet the conditions. If this is the case, I would like to filter out of all the shortest paths, the one in which one has to overcome the smallest distance within each solar system.
Is that accurate enough? Otherwise, please just report.
The latest APOC releases may be able to help here, though the APOC path expanders work best with labels and relationship types, so a slight change to your model may be needed.
In particular, the size of your jump points. Right now this is a property on the relationships between them, but for APOC to work optimally, these might be better modeled with the size as a label on the :JumpPoint nodes themselves, so you might have :JumpPoint:Small, :JumpPoint:Medium, and :JumpPoint:Large (you can add this in addition to the rel properties if you like).
Keep in mind this approach will be more complex than shortestPath(), as the idea is we're trying to find systems within a certain number of jumps, then find :Marketplaces reachable at those star systems, then filter based on whether they sell the product we want, and we'll stitch the path together as we find the pieces.
MATCH localSystemPath = (origin:Marketplace)-[*]-(s:Solarsystem)
WHERE origin.eid = $originId
WITH origin, localSystemPath, s
LIMIT 1
WITH origin, localSystemPath, s,
CASE WHEN coalesce($maxJumps, -1) = -1
THEN -1,
ELSE 3*$maxJumps
END as maxJumps,
CASE $shipSize
WHEN 'small' THEN ''
WHEN 'medium' THEN '|-Small'
ELSE '|-Small|-Medium'
END as sizeBlacklist
CALL apoc.path.spanningTree(s,
{relationshipFilter:'HasJumpPoint|CanTravel>', maxLevel:maxJumps,
labelFilter:'>Solarsystem' + sizeBlacklist, filterStartNode:true}) YIELD path as jumpSystemPath
WITH origin, localSystemPath, jumpSystemPath, length(jumpSystemPath) / 3 as jumps, last(nodes(jumpSystemPath)) as destSystem
MATCH destSystemPath = (destSystem)-[*]-(marketplace:Market)
WHERE none(rel in relationships(destSystemPath) WHERE type(rel) = 'HasJumpPoint')
AND <insert predicate for filtering which :Markets you want>
WITH origin, apoc.path.combine(apoc.path.combine(localSystemPath, jumpSystemPath), destSystemPath) as fullPath, jumps, destSystem, marketplace
CALL srt.getRoutes(origin, marketplace, fullPath) YIELD node, jump_sizes, jump_gates, jump_distance, hops, distance
...
This assumes parameter inputs of $shipSize for the minimum size of all jump gates to pass through, $originId as the id of the origin :Marketplace (plus you DEFINITELY need an index or unique constraint on :Marketplace(eid) for fast lookups here), and $maxJumps, for the maximum number of jumps to reach a destination system.
Keep in mind the expansion procedure used, spanningTree(), will only find the single shortest path to another system. If you need all possible paths, including multiple paths to the same system, then change the procedure to expandConfig() instead.
Nodes with the Location node label have an index on Label.name
Profiling the following query gives me a smart plan, with a NodeHashJoin between the two sides of the graph on either side of Trip nodes. Very clever. Works great.
PROFILE MATCH (rosen:Location)<-[:OCCURS_AT]-(ev:Event)<-[:HAS]-(trip:Trip)-[:OPERATES_ON]->(date:Date)
WHERE rosen.name STARTS WITH "U Rosent" AND
ev.scheduled_departure_time > "07:45:00" AND
date.date = '2015-11-20'
RETURN rosen.name, ev.scheduled_departure_time, trip.headsign
ORDER BY ev.scheduled_departure_time
LIMIT 20;
However, just changing one line of the query from:
WHERE rosen.name STARTS WITH "U Rosent" AND
to
WHERE id(rosen) = 4752371 AND
seems to alter the entire behavior of the query plan, which now appears to become more "sequential", losing the parallel execution of (Trip)-[:OPERATES_ON]->(Date)
Much slower. 6x more DB hits in total.
Question
Why does changing the retrieval of one, seemingly-unrelated Location node via a different index/mechanism alter the behavior of the whole query?
(I'm not sure how best to convey more information about the graph model, but please advise, and I'd be happy to add details that are missing)
Edit:
It gets better. Changing that query line from:
WHERE rosen.name STARTS WITH "U Rosent" AND
to
WHERE rosen.name = "U Rosenthaler Platz." AND
results in the same loss of parallelism in the query plan!
Seems odd that a LIKE query is faster than an = ?
I am new to NetLogo and I am still struggling with the links between patches and agents. I am building a land-use change model, where the agents are farmers. The patches in my model have a "lotid-farmer" value (to know which patch belongs to which farmer; all of them together correspond to the farmer's farm) and a "land-use" value. I am trying to count how many "land-use = 1" patches I have in each "lotid-farmer" (farms) and assign that to a variable that the agents have called "forest-size". I have tried many different things, like this piece of code (which does not work):
(foreach lotid-farmer count patches [ land-use = 1 ] set forest-size )
I wonder if anyone could explain why this statement makes no sense and suggest something else that could work or a tutorial on how to loop in NetLogo with "foreach"? Thank you in advance.
lotid is a value. foreach requires a list and a command-task. Also, your set operator doesn't have a value associated with it.
Actually, I wouldn't use a foreach and just ask farmers to set the variable. I'm going to assume lotid-farmer is the who of the farmer.
ask farmers [
set forest-size count patches with [land-use = 1 and lotid-farmer = myself]
]
I'm currently developing some kind of a configurator using neo4j as a backend. Now I ran into a problem, I don't know how to solve best.
I've got nodes created like this:
(A:Product {name:'ProductA', minWidth:20, maxWidth:200, minHeight:10, maxHeight:400})
(B:Product {name:'ProductB', minWidth:40, maxWidth:100, minHeight:20, maxHeight:300})
...
There is an interface where the user can input a desired width & height, f.e. Width=30, Height=250. Now I'd like to check which products match the input criteria. As the input might be any long value, the approach used in http://neo4j.com/blog/modeling-a-multilevel-index-in-neoj4/ with dates doesn't seem to be suitable for me. How can I run a cypher query giving me all the nodes matching the input criteria?
I don't know if I understand well what you are asking for, but if I do, here a simple query to get this:
Assuming the user wants width = 30 and height = 50
Match (p:Product)
WHERE
p.minWidth < 30 AND p.maxWidth > 30 AND
p.minHeight < 50 AND p.maxHeight > 50
RETURN
p
If this is not what you are looking for, feel free to say it as comment.