I deployed Prometheus on my cluster as well as cAdvisor and Grafana. It works tremendously well. I get all the data I need on Grafana's UI.
I started using Prometheus Java API in order to use this data. For example get the CPU usage and if it has a certain value something will be done.
What I display on Grafana is the Container CPU usage for each container. Now I would like to get that information with the Java API if possible (or something if not). But of course the PromQL queries aren't usable from a Java program (from what I tried but I may be wrong).
I thought of several ways:
Clone the cAdvisor project and directly implement what I want to do in Go
Create a bash script with the docker stat command that would get me the container and CPU usage associated
Or maybe there is actually a way to send PromQL queries.
For instance we get the metric by its name via Java or the Prometheus interface:
ex: node_cpu would get me some data.
But if I want something more precise, I need to send a request, for example irate(node_cpu{job="prometheus"}[5m]) which is not possible via Java.
Is there a way for me to get more precise metrics ?
Prometheus supports REST API requests, which are language-agnostic. You just need to send an HTTP request with your query and process the response.
see an example below, copied from their site.
the following HTTP GET request:
http://localhost:9090/api/v1/query?query=up&time=2015-07-01T20:10:51.781Z
returns something like this:
{
"status" : "success",
"data" : {
"resultType" : "vector",
"result" : [
{
"metric" : {
"__name__" : "up",
"job" : "prometheus",
"instance" : "localhost:9090"
},
"value": [ 1435781451.781, "1" ]
},
{
"metric" : {
"__name__" : "up",
"job" : "node",
"instance" : "localhost:9100"
},
"value" : [ 1435781451.781, "0" ]
}
]
}
}
lots more details, here
Related
I have a gateway fromKhomp manufacturer which delivers packages in the following format (SenML):
message: [
{
"bn": "000D6FFFFE642E09",
"bt": 1611339204
},
{
"n": "model",
"vs": "nir21z"
},
{
"n": "Geladeira Temp",
"u": "Cel",
"v": 4.0
}
When I connect to the Thingsboard platform, the internal GW/Parser breaks as an array before the Input in the Root Rule Chain, and threats as individual packets, but since the first position in this array corresponds to the device ID (MAC) I need to have the whole message to be parser in a script. Does anyone know a way to get the information before the GW parses the message?
If you're using Thingsboard CE then I think you will need to first forward the data to a middleware service to restructure the payload. If you are familiar with AWS Lambda you can do it there.
It would just be a simple script that takes an input payload, restructures, and then forwards to your Thingsboard deployment.
If you're using Thingsboard PE then you can use Integration/Data Converters to do this.
In Open Policy Agent (https://www.openpolicyagent.org/)
regarding to Kubernetes, depending which engine is used:
Gatekeeper: https://github.com/open-policy-agent/gatekeeper
OR
Plain OPA with kube-mgmt: https://www.openpolicyagent.org/docs/latest/kubernetes-introduction/#how-does-it-work-with-plain-opa-and-kube-mgmt
There are different ways to define validation rules:
In Gatekeeper the violation is used. See sample rules here: https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general
In plain OPA samples, the deny rule, see sample here:
https://www.openpolicyagent.org/docs/latest/kubernetes-introduction/#how-does-it-work-with-plain-opa-and-kube-mgmt
It seems to be the OPA constraint framework defines it as violation:
https://github.com/open-policy-agent/frameworks/tree/master/constraint#rule-schema
So what is the exact "story" behind this, why it is not consistent between the different engines?
Notes:
This doc reflects on this: https://www.openshift.com/blog/better-kubernetes-security-with-open-policy-agent-opa-part-2
Here is mentioned how to support interoperability in the script: https://github.com/open-policy-agent/gatekeeper/issues/1168#issuecomment-794759747
https://github.com/open-policy-agent/gatekeeper/issues/168 In this issue is the migration mentioned, is just because of "dry run" support?.
Plain OPA has no opinion on how you choose to name your rules. Using deny is just a convention in the tutorial. The real Kubernetes admission review response is going to look something like this:
{
"kind": "AdmissionReview",
"apiVersion": "admission.k8s.io/v1beta1",
"response": {
"allowed": false,
"status": {
"reason": "container image refers to illegal registry (must be hooli.com)"
}
}
}
So whatever you choose to name your rules the response will need to be transformed into a response like the above before it's sent back to the Kubernetes API server. If you scroll down a bit in the Detailed Admission Control Flow section of the Kubernetes primer docs, you'll see how this transformation is accomplished in the system.main rule:
package system
import data.kubernetes.admission
main = {
"apiVersion": "admission.k8s.io/v1beta1",
"kind": "AdmissionReview",
"response": response,
}
default response = {"allowed": true}
response = {
"allowed": false,
"status": {
"reason": reason,
},
} {
reason = concat(", ", admission.deny)
reason != ""
}
Note in particular how the "reason" attribute is just built by concatenating all the strings found in admission.deny:
reason = concat(", ", admission.deny)
If you'd rather use violation or some other rule name using plain OPA, this is where you would change it.
The following REST query will return parameters of the last successful build of a job:
https://localhost/job/test1/lastSuccessfulBuild/api/json
I'd be interested to retrieve one of the parameters of this build, the BUILD_VERSION:
{
"_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
"actions": [
{
"_class": "hudson.model.CauseAction",
"causes": [
{
"_class": "hudson.model.Cause$UpstreamCause",
"shortDescription": "Started by upstream project \"continuous-testing-pipeline-for-nightly\" build number 114",
"upstreamBuild": 114,
"upstreamProject": "continuous-testing-pipeline-for-nightly",
"upstreamUrl": "job/continuous-testing-pipeline-for-nightly/"
}
]
},
{ },
{
"_class": "hudson.model.ParametersAction",
"parameters": [
{
"_class": "hudson.model.StringParameterValue",
"name": "BUILD_VERSION",
"value": "1.1.15"
Is there a way to retrieve the BUILD_VERSION (1.1.15) directly using the REST Api or do I have to parse manually the json string ?
Thanks
Yeah you can get the value,But it will only work for XML API :(
The JSON API will return a simplified json object using Tree :)
So Jenkins provides you with api (XML,JSON,PYTHON) from which you can read the Jenkins related data of any project. Documentation in detail is provide in https://localhost/job/test1/lastSuccessfulBuild/api
In that it clearly states that
XML API - Use XPath to control the fragment you want.For example, ../api/xml?xpath=//[0]
JSON API - Use tree
Python API - Use st.literal_eval(urllib.urlopen("...").read())
All the above can be used to get a specific fragment/piece from the entire messy data that you get from the API.
In your case, we will use tree for obvious reasons :)
Syntax : tree=keyname[field1,field2,subkeyname[subfield1]]
In order to retrieve BUILD_VERSION i.e. value
//jenkins/job/myjob/../api/json?tree=lastSuccessfulBuild[parameters[value]]
The above should get you what you want, but a bit of trail and error is required :)
You can also refer here for a better understanding of how to use Tree in JSON API
https://www.cloudbees.com/blog/taming-jenkins-json-api-depth-and-tree
Hope it helps :)
Short answer: No.
Easiest way to programmatically access any attribute exposed via the JSON API is to take the JSON from one of Jenkins supported JSON APIs (in your case: https://localhost/job/<jobname>/lastSuccessfulBuild/api/json)
Copy the resultant JSON into http://json2csharp.com
Generate the corresponding C# code. Don't forget to create a meaningful name for top level class.
Call RestAPI programmatically from C# using RestSharp.
Deserialise the json to the C# class you defined in 2 above.
Wammo, you have access to the entire object tree and all its values.
I used this approach to write an MVC5 ASP.NET site I called "BuildDashboard" to provide all the information a development team could want and answered every question they had.
Here is an example with a public jenkins instance and one of its builds in order to get "candidate_revision" parameter for "lastSuccessfulBuild" build:
https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-vivid-i386-ci/lastSuccessfulBuild/parameters/
https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-vivid-i386-ci/lastSuccessfulBuild/api/xml?xpath=/freeStyleBuild/action/parameter[name=%22candidate_revision%22]/value
Fiware-cygnus documentation mentions that it is based on Apache Flume. However, it is not clear whether I can use native Flume sinks to persist events arriving from Orion Context Broker. Is this something I can easily do, with little (or ideally zero) coding? If not -- would be good to know why (and whether this can be supported going forward). Thanks!
You can use native Flume sinks by simply configuring them. Nothing has been changed in Cygnus in terms of configuration management, thus you can configure a Orion-like sink or a native one.
Nevertheless, there are differences between Orion-like and native Flume sinks.
The first one is the Orion-like sinks store the relevant data with certain structure, and the Flume native sinks will store the notified raw data. I mean, if you receive a Json-based notification such as:
{
"subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",
"originator" : "localhost",
"contextResponses" : [
{
"contextElement" : {
"attributes" : [
{
"name" : "speed",
"type" : "float",
"value" : "112.9",
"metadatas": []
},
{
"name" : "oil_level",
"type" : "float",
"value" : "74.6",
"metadatas": []
}
],
"type" : "car",
"isPattern" : "false",
"id" : "car1"
},
"statusCode" : {
"code" : "200",
"reasonPhrase" : "OK"
}
]
}
OrionHDFSSink will store something like:
{"recvTimeTs":"1429535775","recvTime":"2015-04-20T12:13:22.41.124Z","fiware-servicePath":"4wheels","entityId":"car1","entityType":"car","attrName":"speed","attrType":"float","attrValue":"112.9","attrMd":[]}
But a native HDFS sink (or any other one) will persist the entire notified json.
The second main difference if the handling of the notified fiware-service and fiware-servicePath. Cygnus's sinks are able to deal with these values in order to map the notified data into specific data structures (folders, databases, tables, resources, queues...). This is very important for multi-tenancy purposes.
Third, Cygnus adds sinks for storages not covered by native Flume, such as CKAN, STH, MongoDB, MySQL or DynamoDB.
There are many other differences:
The usage of the Grouping Rules.
The Management Interface.
OAuth2 authentication, which is the official FIWARE's mechanism.
...
I'm trying to get test results from a Jenkins job and was wondering if it was possible to get the json results by making a GET request (e.g. to a certain job: JENKINS_URL/job/JOB_NAME/build/api/json) while not logged into the Jenkins server.
I am having trouble understanding how you can send all the how do you pass the authentication information in the API call (e.g. API token / password, username).
I know the Jenkins CLI is an option, but I would prefer not to use it if possible.
To be clear, when logged in on my computer, calls to JENKINS_URL/job/JOB_NAME/build/api/json yield the build results, but when I log out, the link takes me to the log in page.
You have to configure your credentials and give read access for anonymous or pass your user/password in your request:
curl --user username:password jenkins_url/job/job_name/build_number/api/json
The same but using api_token:
curl --user username:api_token jenkins_url/job/job_name/build_number/api/json
If you pass also?pretty=true parameter, you should see something like this:
{
"actions" : [
{
"parameters" : [
...
}
],
...
"building" : false,
"description" : null,
"displayName" : "#36",
"duration" : 1936,
"estimatedDuration" : 2020,
"number" : 36,
"queueId" : 224,
}