I want my elasticsearch / Kibana4 to give me an overview which pages of my website are viewed most. I have a field "request" but since the URLs listed there contain parameters, I get a false list of the top requests
Example:
/search?query=123
/search?query=234
Each request is shown as a single request
but
home/foobar
home/foobar
is listed with 2 requests and is the top request in this case.
How can I tell elasticsearch to aggregate the requests that contain parameters?
There are number of ways to solve this.
One would be setting the url field as not_analyzed and then using script to parse out the unwanted value.
A sample script can be found here -
{
"aggs": {
"urls": {
"terms": {
"field": "url",
"script" : "_value.split("?")[0]"
}
}
}
}
Another approach would be to add a multifield and then remove the part after '?' using custom analyzer . You can achieve this by removing anything after '?' using pattern replace tokenizer
Related
I am trying to insert a custom dimension for an istio metric for URL path.
I am following the steps here -
https://istio.io/latest/docs/tasks/observability/metrics/classify-metrics/
Specifically, this part, where I can parse the URL and decide the value :
configuration:
"#type": type.googleapis.com/google.protobuf.StringValue
value: |
{
"attributes": [
{
"output_attribute": "istio_operationId",
"match": [
{
"value": "ListReviews",
"condition": "request.url_path == '/reviews' && request.method == 'GET'"
},
{
"value": "GetReview",
"condition": "request.url_path.matches('^/reviews/[[:alnum:]]*$') && request.method == 'GET'"
},
{
"value": "CreateReview",
"condition": "request.url_path == '/reviews/' && request.method == 'POST'"
}
]
}
]
}
I want to add a fall-back approach - i.e., if it doesn't match any URL, then make the value of istio_operationId as the original request.url_path
How can I do that?
I tried adding this as the last condition but it doesn't work
{
"value": "request.url_path", //tried without quotes as well
"condition": "request.url_path.matches('.+$')"
}
Also, is this possible in Lua?
To set a fall back value leave the condition blank. From the docs "An empty condition evaluates to true and should be used to provide a default value.".
That said, I don't think there's a way to set the attribute value to anything but a static string. What you can do instead is add the url_path as a dimension to one (or all) metrics generated by the stats filter. To say it another way, I don't think you can combine request classification with custom dimensions the way you're describing.
See this blog post for details as well as an explanation of the difference between attributes and dimensions.
Also, you may want to reconsider trying. When metrics are emitted with unbounded cardinality monitoring systems fall over. Here is a description of an Istio user crashing their Prometheus instance this way. Here is the (then) co-lead of the Istio extensions and telemetry working group discussing why they do not recommend doing this.
I want to add custom headers in the payload of a batch. I have already achived to add it on single calls but I can't figure out how to add on eacht single call inside the payload in a Batch.
I have followed the steps here
https://blogs.sap.com/2016/07/13/set-and-get-custom-http-header/
I would like to receive it here too. Is that possible?Alternatively I could also just read the header of the Batch I can't find out how.
If you add the headers in the manifest, or using setHeaders method of the model, the header is added to batch itself and also to the single requests inside the batch. At least this is the case for v2.ODataModel implementation. not absolutely sure for ui5s implementation of the v4.ODataModel, but would assume the same. Example definition in manifest.json:
"": {
"dataSource": "mainService",
"preload": true,
"settings": {
"defaultCountMode": "Inline",
"useBatch": true,
"headers": {
"foo": "bar"
}
}
}
had to use headers in a v4 Model yesterday. So quick update: it works the same with the only difference that the property is called httpHeaders rather than headers
I'm trying to use Power Automate to return a custom work item in Azure DevOps using the "workitemsearch" API (via the "Send HTTP Request" action). Part of this will require me to filter based on the value of a Custom Field, however, I have not been able to get it to work. Here is a copy of my HTTP Request Body:
{
"searchText": "ValueToSearch",
"$skip": 0,
"$top": 1,
"filters": {
"System.TeamProject": ["MyProject"],
"System.AreaPath": ["MyAreaPath"],
"System.WorkItemType": ["MyCustomWorkItem"],
"Custom.RequestNumber": ["ValueToSearch"]
},
"$orderBy": [
{
"field": "system.id",
"sortOrder": "ASC"
}
],
"includeFacets": true
}
I have been able to get it to work by removing the Custom.RequestNumber": ["ValueToSearch"] but am hesitant to use that in case my ValueToSearch is found in other places like the comments of other work items.
Any help on this would be appreciated.
Cheers!
From WorkItemSearchResponse, we can see the facets (A dictionary storing an array of Filter object against each facet) only supports the following fields:
"System.TeamProject"
"System.WorkItemType"
"System.State":
"System.AssignedTo"
If you want to filter RequestNumber, you can just set it in the searchText as the following syntax:
"searchText": "RequestNumber:ValueToSearch"
The following REST query will return parameters of the last successful build of a job:
https://localhost/job/test1/lastSuccessfulBuild/api/json
I'd be interested to retrieve one of the parameters of this build, the BUILD_VERSION:
{
"_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
"actions": [
{
"_class": "hudson.model.CauseAction",
"causes": [
{
"_class": "hudson.model.Cause$UpstreamCause",
"shortDescription": "Started by upstream project \"continuous-testing-pipeline-for-nightly\" build number 114",
"upstreamBuild": 114,
"upstreamProject": "continuous-testing-pipeline-for-nightly",
"upstreamUrl": "job/continuous-testing-pipeline-for-nightly/"
}
]
},
{ },
{
"_class": "hudson.model.ParametersAction",
"parameters": [
{
"_class": "hudson.model.StringParameterValue",
"name": "BUILD_VERSION",
"value": "1.1.15"
Is there a way to retrieve the BUILD_VERSION (1.1.15) directly using the REST Api or do I have to parse manually the json string ?
Thanks
Yeah you can get the value,But it will only work for XML API :(
The JSON API will return a simplified json object using Tree :)
So Jenkins provides you with api (XML,JSON,PYTHON) from which you can read the Jenkins related data of any project. Documentation in detail is provide in https://localhost/job/test1/lastSuccessfulBuild/api
In that it clearly states that
XML API - Use XPath to control the fragment you want.For example, ../api/xml?xpath=//[0]
JSON API - Use tree
Python API - Use st.literal_eval(urllib.urlopen("...").read())
All the above can be used to get a specific fragment/piece from the entire messy data that you get from the API.
In your case, we will use tree for obvious reasons :)
Syntax : tree=keyname[field1,field2,subkeyname[subfield1]]
In order to retrieve BUILD_VERSION i.e. value
//jenkins/job/myjob/../api/json?tree=lastSuccessfulBuild[parameters[value]]
The above should get you what you want, but a bit of trail and error is required :)
You can also refer here for a better understanding of how to use Tree in JSON API
https://www.cloudbees.com/blog/taming-jenkins-json-api-depth-and-tree
Hope it helps :)
Short answer: No.
Easiest way to programmatically access any attribute exposed via the JSON API is to take the JSON from one of Jenkins supported JSON APIs (in your case: https://localhost/job/<jobname>/lastSuccessfulBuild/api/json)
Copy the resultant JSON into http://json2csharp.com
Generate the corresponding C# code. Don't forget to create a meaningful name for top level class.
Call RestAPI programmatically from C# using RestSharp.
Deserialise the json to the C# class you defined in 2 above.
Wammo, you have access to the entire object tree and all its values.
I used this approach to write an MVC5 ASP.NET site I called "BuildDashboard" to provide all the information a development team could want and answered every question they had.
Here is an example with a public jenkins instance and one of its builds in order to get "candidate_revision" parameter for "lastSuccessfulBuild" build:
https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-vivid-i386-ci/lastSuccessfulBuild/parameters/
https://jenkins.qa.ubuntu.com/view/All/job/account-plugins-vivid-i386-ci/lastSuccessfulBuild/api/xml?xpath=/freeStyleBuild/action/parameter[name=%22candidate_revision%22]/value
I'm trying to user the contains filter on a /users query, like this for example:
https://graph.microsoft.com/v1.0/users?$filter=contains(displayName, 'Garth')
However, this results in a BadRequest response saying "An unknown function with name 'contains' was found. This may also be a key lookup on a navigation property, which is not allowed."
According to the OData 4.0 specs, the contains filter should be available though. Is there a way to use a contains filter on a list of users?
The contains function is not available for users. startsWith is available though.
e.g. https://graph.microsoft.com/v1.0/users?$filter=startswith(displayName,'Garth').
Additionally you can try the people API (this is only available on /beta). This supports $search AND will do fuzzy and phonetic matching.
https://graph.microsoft.com/beta/me/people?$search=Garth
$search can replace $contains
for example to show all the users that their names may contain the string "sa"
https://graph.microsoft.com/v1.0/users?$count=true&$search="givenName:sa"
"value": [
{....
"givenName": "Sammy",
.....
},
{
.....
"givenName": "Sabrina",
.......
},
.....