I did some research on Jenkins primary and secondary node relationship, and it seem to be possible to have multiple secondary nodes connected to multiple primary nodes. This is done using independent workspace and have multiple agents on the secondary node.
But I want to ask if this can be accomplished using a different method.
For example, I have the following scenario
Primary Node:
- Jenkins-Primary-A # located in united states
- Jenkins-Primary-B # located in singapore
Secondary Node (for Jenkins-Primary-A):
- Jenkins-Secondary-A1 # label Alpha-1
- Jenkins-Secondary-A2 # label Alpha-1
- Jenkins-Secondary-A3 # label Alpha-2
Secondary Node (for Jenkins-Primary-B):
- Jenkins-Secondary-B1 # label Beta-1
- Jenkins-Secondary-B2 # label Beta-1
- Jenkins-Secondary-B3 # label Beta-1
- Jenkins-Secondary-B4 # label Beta-2
- Jenkins-Secondary-B5 # label Beta-2
I was wondering if is possible to configure Jenkins-Primary-B to say, I want to share Jenkins-Secondary-B4 and Jenkins-Secondary-B5 with Jenkins-Primary-A if they are not in use, but only these 2 nodes? These two nodes will also have similar labels as nodes for Jenkins-Primary-A.
So from a pipeline perspective, Jenkins-Secondary-B4 and Jenkins-Secondary-B5 might also have label of Alpha-2?
Not sure if this is feasible or there must be multiple agents set up for this?
Related
I am using the consul exporter to ingest the health and status of my services into Prometheus. I'd like to fire alerts when the status of services and nodes in Consul is critical and then use tags extracted from Consul when routing those alerts.
I understand from this discussion that service tags are likely to be exported as a separate metric, but I'm not sure how to join one series with another so I can leverage the tags with the health status.
For example, the following query:
max(consul_health_service_status{status="critical"}) by (service_name, status,node) == 1
could return:
{node="app-server-02",service_name="app-server",status="critical"} 1
but I'd also like 'env' from this series:
consul_service_tags{node="app-server-02",service_name="app-server",env="prod"} 1
to get joined along node and service_name to pass the following to the Alertmanager as a single series:
{node="app-server-02",service_name="app-server",status="critical",env="prod"} 1
I could then match 'env' in my routing.
Is there any way to do this? It doesn't look to me like any operations or functions give me the ability to group or join like this. As far as I can see, the tags would already need to be labels on the consul_health_service_status metric.
You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)
The answer in this question is accurate. I want to also share a clearer explanation on joining two metrics preserving SAME Labels (might not be directly answering the question). In these metrics following label is there.
name (eg: aaa, bbb, ccc)
I have a metric name metric_a, and if this returns no data for some of the labels, I wish to fetch data from metric_b. i.e:
metric_a has values for {name="aaa"} and {name="bbb"}
metric_b has values for {name="ccc"}
I want the output to be for all three name labels. The solution is to use or in Prometheus.
sum by (name) (increase(metric_a[1w]))
or
sum by (name) (increase(metric_b[1w]))
The result of this will have values for {name="aaa"}, {name="bbb"} and {name="ccc"}.
It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.
The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.
For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:
consul_health_service_status
* on(service_name, node) group_left(env)
consul_service_tags
Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:
consul_health_service_status{status="critical"}
* on(service_name, node) group_left(env)
consul_service_tags
I'd like to create multiple Resources for a certain node, or use a reusable type for several nodes.
In this case it is "RAM requirement", so the resource name e.g. would be 1GBRAM. alternatively 1GBRAM_Nodexy if I need to specify this on a per node basis.
In the end I'd like to limit the amount of concurrent Jobs based on the peak amount of memory a Job uses up on this node, to avoid hangs because of low memory on the server. And I can set the amount of RAM which is available for executors.
Different Nodes will have different amounts of RAM, and individual Jobs have different RAM requirements.
So I would like to configure each Job with its RAM requirements
lock(resource: '1GBRAM_Nodexy', quantity: 8)
Is this achievable with Pipelines and lockable resources?
Is there an alternative, better way to achieve this? Ideally, the locks can be checked before the slave is selected, and the best suited node is picked.
Read about resource locks and labels. I Did not find any Node specific section, also no possibility to acquire multiple items of the same resource.
lock(resource: '1GBRAM_Nodexy', quantity: 8)
I expect that each run of the Job locks the equivalent amount of RAM on the used slave node. If not enough "RAM" units are used up, a Job is not run on such a node.
I think you can't quite do what you're looking for, but perhaps you can come close.
First, what you want is to use label instead of resource. You'd define as many 1GB-representing resources (say, GB1, GB2, GB3, etc.) as you have RAM, giving them all the same label (say, GB), and then use a lock statement like this (e.g., if the job in question needed 4GB of memory):
lock(label: 'GB', quantity: 4)
This will lock 4 of the resources that have this GB label, waiting if needed until it's able to do so, and then will release them when leaving the locked scope.
The node-specific locking is the trickier part. If you were content with using a different label per node (NodeA_GB, NodeB_GB, etc.), and with "pinning" jobs to particular nodes, then the solution above would suffice, e.g.:
// Require 4GB of memory on NodeA
lock(label: 'NodeA_GB', quantity: 4)
What I'm not aware of a way to do is to have a specific node selected because it has RAM available -- i.e., your "the locks can be checked before the slave is selected, and the best suited node is picked" statement. But you could at least detect the node that was allocated by a regular agent statement, using env.NODE_NAME, then use that as part of your node-specific lock label:
agent any
stages {
stage('Build') {
steps {
// This assumes that all possible nodes have a label like this defined with their name in it
lock(label: "${NODE_NAME}_GB", quantity: 4) {
// ... build steps
}
}
}
}
Incidentally, I'm using a label+quantity approach myself but in order to achieve lock-based throttling -- restricting the total number of concurrent builds across all branches of a multibranch pipeline job -- since the Throttle Concurrent Builds plugin went through a period of not being maintained and had some significant, open issues during that time.
Addition to accepted answer(edit queue is full):
As for selecting specific node because it has RAM available -- i.e., your "the locks can be checked before the slave is selected, and the best suited node is picked" statement, a org.jenkins.plugins.lockableresources.LockableResourcesManager class may be used to check available memory on the nodes, and decide, which node to use, for example:
def nodeFreeGbThreshold = 2
def resourceManager = new org.jenkins.plugins.lockableresources.LockableResourcesManager()
def nodeAFreeGb = resourceManager.getFreeResourceAmount("NodeA_GB")
def nodeBFreeGb = resourceManager.getFreeResourceAmount("NodeB_GB")
def agentLabel = nodeAFreeGb < nodeFreeGbThreshold ? 'NodeA' : 'NodeB'
pipeline {
agent { label 'agentLabel' }
stages {
stage('Build') {
steps {
// This assumes that all possible nodes have a label like this defined with their name in it
lock(label: "${NODE_NAME}_GB", quantity: 4) {
// ... build steps
}
}
}
}
}
and for scripted pipelines:
def nodeFreeGbThreshold = 2
def resourceManager = new org.jenkins.plugins.lockableresources.LockableResourcesManager()
def nodeAFreeGb = resourceManager.getFreeResourceAmount("NodeA_GB")
def nodeBFreeGb = resourceManager.getFreeResourceAmount("NodeB_GB")
def agentLabel = nodeAFreeGb < nodeFreeGbThreshold ? 'NodeA' : 'NodeB'
node(agentLabel) {
// This assumes that all possible nodes have a label like this defined with their name in it
lock(label: "${NODE_NAME}_GB", quantity: 4) {
// ... build steps
}
}
I have a Neo4j db with different labels on the nodes such as a:Banker , b:Customer. each has an email property I want to search for an email but but not search the entire db. So I want to do something like this Match(a:Banker {email: '123#mymail.com'}) OR Match (b:Customer {email:'123#mymail.com'}). There are constraints on email for both labels but I don't want each label to have the same email so before I add a node I need to determine if the email exist in either Banker or Customer nodes. I suspect this can be done in a very efficient scalable way that would not leave the user staring at a spinner when trying to add the one millionth record.....any help would be much appreciated
How I would do it is have an addition label 'Person' on all Bankers and Customers.
CREATE CONSTRAINT ON (b:Person) ASSERT p.Email IS UNIQUE
CREATE CONSTRAINT ON (b:Banker) ASSERT p.Email IS UNIQUE
CREATE CONSTRAINT ON (b:Customer) ASSERT p.Email IS UNIQUE
CREATE (b:Person:Banker {Email: "123#mymail.com"})
CREATE (b:Person:Customer {Email: "321#mymail.com"})
CREATE (c:Person:Customer {Email: "123#mymail.com"})
The last one will fail as a Person/Banker already has the same email. You can then also search MATCH (p:Person {Email: "123#mymail.com"}) or even b:Banker, c:Customer
You can also do (p:Person:Customer:Banker) if a person is all three.
It will also allow you to do MERGE which creates an entry if it doesn't already exist.
Since you already have a database you can do:
MATCH(b:Banker)
SET b:Person
MATCH(c:Customer)
SET c:Person
A somewhat "safer" approach than #Liam's would be to just have the Person label, without the Banker and Customer labels. That way, it would be harder to accidentally create/merge a node without the Person label, since that would be the only label for a person. Also, this approach would not require 2 (or 3) uniqueness checks every time you added a person.
With this approach, you could also add isCustomer and isBanker boolean properties, as needed, and create indexes on :Person(isCustomer) and :Person(isBanker) to quickly locate customers versus bankers.
Now, having said the above, I wonder if you really need the isCustomer and isBanker properties (or the Customer and Banker labels) at all. That is, the fact that a Person node is a banker and/or a customer may be derivable from that node's relationships. It seems reasonable for your data model to contain Bank nodes with relationships between them and people. For example, in the following data model, b is a banker at "XYZ Bank", c is a customer, and bc is both:
(b:Person)-[:WORKS_AT]->(xyz:Bank {id:123, name: 'XYZ Bank'}),
(c:Person)-[:BANKS_AT]->(xyz),
(bc:Person)-[:BANKS_AT]->(xyz)<-[:WORKS_AT]-(bc)
This query would find all bankers:
MATCH (banker:Person)-[:WORKS_AT]->(:Bank)
RETURN banker;
This would find all customers:
MATCH (banker:Person)-[:BANKS_AT]->(:Bank)
RETURN banker;
This would find all bankers who are also customers at the same bank:
MATCH (both:Person)-[:WORKS_AT]->(:Bank)<-[:BANKS_AT]-(both)
RETURN both;
let's say I have two nodes with one edge as :
country ---> has ---> school
where, edge "has" has property called "since".
If I created lookup index for nodes and edges + edge property.
g.createKeyIndex('country', Vertex.class)
g.createKeyIndex('school', Vertex.class)
g.createKeyIndex('has', Edge.class)
How to create index on edge property(since).Or while creating index in edge "has". properties get indexed. Is it ?
in Neo4j.property I set as :
# Autoindexing
# Enable auto-indexing for nodes, default is false
node_auto_indexing=true
# The node property keys to be auto-indexed, if enabled
node_keys_indexable=country, school
# Enable auto-indexing for relationships, default is false
relationship_auto_indexing=true
# The relationship property keys to be auto-indexed, if enabled
relationship_keys_indexable=since
but I don't want to create auto index through this property file but need in gremlin way before adding vertex/edges.
like in Titan way :
g.makeType().name('since').dataType(Date.class).unique(OUT).makePropertyKey()
how can it possible through simple neo4j + gremlin ?
am following :
http://www.tinkerpop.com/docs/javadocs/blueprints/2.1.0/com/tinkerpop/blueprints/KeyIndexableGraph.html#createKeyIndex(java.lang.String, java.lang.Class)
You are confusing the concept of indices a bit. Using createKeyIndex on Graph in this way:
g.createKeyIndex('has', Edge.class)
Is not creating an "indexed" edge label called "has". It is creating a index on a property called "has" that will lookup an Edge. If you want an index on "since" then simply do:
g.createKeyIndex('since', Edge.class)
That said, there is nothing I know of that exists in Neo4j that is analogous to vertex centric indices in Titan, so it's not as though your creation of a key index on "since" will allow Gremlin to take advantage of that index in a traversal outside of a simple key index lookup, like:
g.E('since', new Date())
I'm new to Neo4j - just started playing with it yesterday evening.
I've notice all nodes are identified by an auto-incremented integer that is generated during node creation - is this always the case?
My dataset has natural string keys so I'd like to avoid having to map between the Neo4j assigned ids and my own. Is it possible to use string identifiers instead?
Think of the node-id as an implementation detail (like the rowid of relational databases, can be used to identify nodes but should not be relied on to be never reused).
You would add your natural keys as properties to the node and then index your nodes with the natural key (or enable auto-indexing for them).
E..g in the Java API:
Index<Node> idIndex = db.index().forNodes("identifiers");
Node n = db.createNode();
n.setProperty("id", "my-natural-key");
idIndex.add(n, "id",n.getProperty("id"));
// later
Node n = idIndex.get("id","my-natural-key").getSingle(); // node or null
With auto-indexer you would enable auto-indexing for your "id" field.
// via configuration
GraphDatabaseService db = new EmbeddedGraphDatabase("path/to/db",
MapUtils.stringMap(
Config.NODE_KEYS_INDEXABLE, "id", Config.NODE_AUTO_INDEXING, "true" ));
// programmatic (not persistent)
db.index().getNodeAutoIndexer().startAutoIndexingProperty( "id" );
// Nodes with property "id" will be automatically indexed at tx-commit
Node n = db.createNode();
n.setProperty("id", "my-natural-key");
// Usage
ReadableIndex<Node> autoIndex = db.index().getNodeAutoIndexer().getAutoIndex();
Node n = autoIndex.get("id","my-natural-key").getSingle();
See: http://docs.neo4j.org/chunked/milestone/auto-indexing.html
And: http://docs.neo4j.org/chunked/milestone/indexing.html
This should help:
Create the index to back automatic indexing during batch import We
know that if auto indexing is enabled in neo4j.properties, each node
that is created will be added to an index named node_auto_index. Now,
here’s the cool bit. If we add the original manual index (at the time
of batch import) and name it as node_auto_index and enable auto
indexing in neo4j, then the batch-inserted nodes will appear as if
auto-indexed. And from there on each time you create a node, the node
will get indexed as well.**
Source : Identifying nodes with Custom Keys
According Neo docs there should be automatic indexes in place
http://neo4j.com/docs/stable/query-schema-index.html
but there's still a lot of limitations
Beyond all answers still neo4j creates its own ids to work faster and serve better. Please make sure internal system does not conflict between ids then it will create nodes with same properties and shows in the system as empty nodes.
the ID's generated are default and cant be modified by users. user can use your string identifiers as a property for that node.