I am using Azure Logic App to fetch list of records from Azure SQL Database. I am using odata filter to solve my problem. It all works good when I pass the $filter, $select and other parameters like below
My Logic App Implementation
I would like to use Aggregation Transformation in this. But it is now working. In aggregation transformation I have used the below syntax
groupby((Name), aggregate(ID with countdistinct as Total))
can someone help me how I can use aggregation transformation in Azure Logic App?
This could be due to a query that was written incorrectly. We have GroupBy and Aggregates in the Aggregate Transformations, therefore we must mention them separately, just as we would in conventional SQL.
One of the workarounds is that when I simply mention the query in the 'Execute a SQL query(v2)' connector, I was able to extract data from a sql database applying aggregate functions as well.
Below is the query I'm using, which includes group by and sum aggregate.
SELECT
department_id, SUM(salary)
FROM
employees
GROUP BY department_id;
The screenshot of my logic app is shown below.
and here is the output that I received in logic apps
{
"statusCode": 200,
"headers": {
"Pragma": "no-cache",
"Transfer-Encoding": "chunked",
"Vary": "Accept-Encoding",
"x-ms-request-id": "8407b5c5-ba6a-49ad-8709-39890f9a7dcd",
"Strict-Transport-Security": "max-age=31536000; includeSubDomains",
"X-Content-Type-Options": "nosniff",
"X-Frame-Options": "DENY",
"Timing-Allow-Origin": "*",
"x-ms-apihub-cached-response": "true",
"Cache-Control": "no-store, no-cache",
"Date": "Thu, 21 Oct 2021 10:46:21 GMT",
"Content-Type": "application/json; charset=utf-8; odata.metadata=minimal",
"Expires": "-1",
"Content-Length": "401"
},
"body": {
"ResultSets": {
"Table1": [
{
"department_id": 1,
"": 4400
},
{
"department_id": 2,
"": 19000
},
{
"department_id": 3,
"": 24900
},
{
"department_id": 4,
"": 6500
},
{
"department_id": 5,
"": 41200
},
{
"department_id": 6,
"": 28800
},
{
"department_id": 7,
"": 10000
},
{
"department_id": 8,
"": 57700
},
{
"department_id": 9,
"": 58000
},
{
"department_id": 10,
"": 51600
},
{
"department_id": 11,
"": 20300
}
]
},
"OutputParameters": {}
}
}
REFERENCES:
SQL GROUP BY: How To Apply It Effectively (sqltutorial.org)
Azure Data Factory – Aggregate transformation in mapping data flow – Tech-Tutes
Aggregator Transformation - javatpoint
Related
Im still new to Neo4j, Im trying to figure out how to get Neo4j to return and draw only the shortest path between two nodes. im currently using APOC procedure.
basically i have two nodes (Site-1 & Site-2) with two relationships between them (relationship-1 (Link, distance:10) & (relationship-1 (Link, distance:40).
I used the below APOC procedure
CALL apoc.algo.dijkstra(site1, site2, 'Link', 'distance') YIELD path, weight
Return path, weight ```
the graph returned with a drawing of two relationship between the two nodes, however it should only draw one. any idea on how i can return one only?
Neo4j browser is a tool, that makes cypher querying easier and also helps in visualizing the graph as well. What you are seeing is not an error exactly. Neo4j browser is configured by default, to show all the relationships of a node when expanded. It doesn't necessarily represent your query output. I replicated your case on my system, I also received the same graph.
But the actual query response is under the Table tab.
This is the whole response that I got:
[
{
"path": {
"start": {
"identity": 5,
"labels": [
"Site1"
],
"properties": {}
},
"end": {
"identity": 6,
"labels": [
"Site2"
],
"properties": {}
},
"segments": [
{
"start": {
"identity": 5,
"labels": [
"Site1"
],
"properties": {}
},
"relationship": {
"identity": 0,
"start": 5,
"end": 6,
"type": "LINK",
"properties": {
"distance": 10
}
},
"end": {
"identity": 6,
"labels": [
"Site2"
],
"properties": {}
}
}
],
"length": 1.0
},
"weight": 10.0
}
]
As you can see, only one path and weight are returned. This is what your query outputs and this is what your language driver will receive be it Javascript, java, anything.
With Microsoft Graph API you can search your calendar events by issuing a request like it's documented here or like this:
POST /v1.0/search/query HTTP/1.1
Host: graph.microsoft.com
Authorization: Bearer <token>
Content-Type: application/json
Content-Length: 174
{
"requests": [
{
"entityTypes": [
"event"
],
"query": {
"queryString": "test"
},
"from": 0,
"size": 50
}
]
}
So, in my case, for the search query test, I have more than 200 events (recurring and single ones), and in the request payload we've sent to Microsoft Graph we have the size parameter which basically sets the limit of records to be returned by this call.
So, when I issue the request from above, I get something like this, but never more than 25 results. No matter how I modify the size parameter, it always returns 25 events, although there are hundreds.
HTTP/1.1 200 OK
Content-type: application/json
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#search",
"value": [
{
"#odata.type": "#microsoft.graph.searchResponse",
"searchTerms": [
"contoso"
],
"hitsContainers": [
{
"#odata.type": "#microsoft.graph.searchHitsContainer",
"hits": [
{
"#odata.type": "#microsoft.graph.searchHit",
"hitId": "AAMkADEwODY2NzllLTQ3MmEtNGRlMC05ZTUyLTE4ZDRhYmU1ZGM3NABGAAAAAAA3+iYQBnJnQabRVDelNhnzBwAejhWkAOAxQ6M4c1c9NwfrAAAAAAENAAAejhWkAOAxQ6M4c1c9NwfrAABbUZLJAAA=",
"rank": 1,
"summary": "this is a testing nothing more",
"resource": {
"#odata.type": "#microsoft.graph.event",
"end": {
"dateTime": "2020-06-16T04:15:00Z",
"timeZone": "UTC"
},
"hasAttachments": false,
"iCalUId": "040000008200E00074C5B7101A82E008000000007093FDD79B3AD60100000000000000001000000036DAA2262EB4E04DA27DA77985FB8251",
"isAllDay": false,
"sensitivity": "Normal",
"start": {
"dateTime": "2020-06-16T03:30:00Z",
"timeZone": "UTC"
},
"subject": "testing testing 123",
"type": "Single"
}
}
],
"total": 25,
"moreResultsAvailable": true
}
]
}
]
}
Furthermore, in the response you get the flag moreResultsAvailable which is in my case true, but other than that, I can't find a way to traverse between the results page, or do pagination of some sort.
What am I missing here, or in the docs?
So, when I issue the request from above, I get something like this,
but never more than 25 results. No matter how I modify the size
parameter, it always returns 25 events, although there are hundreds.
As per https://learn.microsoft.com/en-us/graph/api/resources/search-api-overview?view=graph-rest-1.0#page-search-results
The maximum results per page (size) is 25 for message and event.
For paging you just need to use the from property to specify the start of the page eg for the next page it would be
{
"requests": [
{
"entityTypes": [
"event"
],
"query": {
"queryString": "test"
},
"from": 25,
"size": 25
}
]
}
I have a list of IDs which I want to pull the corresponding contacts from Microsoft Graph. I tried filtering with https://graph.microsoft.com/v1.0/me/contacts?$filter=Id eq 'my-id-here' or Id eq 'other-id-here' but it says
ErrorInvalidProperty The property 'Id' does not support filtering.
I know than I can query by calling https://graph.microsoft.com/v1.0/me/contacts/my-id-here but I want to request multiple at once to minimize round trips.
You can use batch processing to execute multiple requests. For each request you need to provide a request id, a http method and a request url. Sample request:
URL: https://graph.microsoft.com/v1.0/$batch
Http Method: POST
Request Body:
{
"requests": [
{
"id": "1",
"method": "GET",
"url": "/me/contacts/firstId"
},
{
"id": "2",
"method": "GET",
"url": "/me/contacts/secondId"
}
]
}
Graph will send a response such as:
{
"responses": [
{
"id": "1",
"status": 200,
"headers": {
...
},
"body": {
...
}
},
{
"id": "2",
"status": 200,
"headers": {
...
},
"body": {
...
}
}
]
}
Make sure, that you check for the id of the request, since requests may not be returned in order, in which you sent them to Graph. The Microsoft documentation on batch processing provides more information: here.
I'm trying to fetch all users from a specific group via an HTTP connector, a registered app, and Microsoft Graph.
The registered app has Directory.Read.All permissions.
My idea is that I'm calling the nextLink as long as it's there while appending all of the fetched users' userPrincipalName to an array eventually filling the array with all users of the group.
My Logic App looks like this:
Unfortunately, I'm just 1 reputation short of posting images, please forgive. The 3 links should provide an overview of the structure of my app.
First, nextLink is initialized to the first Graph API endpoint. This variable is set to the current nextLink through each iteration of the until loop.
Second, For the purpose of this exercise, I only get the top 5. I know there are only 9 users:
Lastly, I call the union method on the "users" array that I initialized earlier and the "value" array from the HTTP get method, to get one single array consisting of all users:
The issue is that the HTTP action always returns the same top 5 users. I've checked that the nextLink provided in the first HTTP GET call to Graph, is correct by copying it from the Runs history and pasting it into Microsoft Graph Explorer and there the next 4 users are correctly returned.
I also made sure that, for each iteration in the until loop, I call the Graph API with the nextLink from the previous iteration as expected.
The nextLink returned inside of the Logic App is exactly the same when I test it in Graph Explorer, but the same nextLink returns 2 different results when called from Graph Explorer and inside my Logic App.
Why is the result always the same top 5 users and not the next 4 users as expected?
If not sure about the reason why you will get this issue, but based on your requirement, I did a sample below:
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "GetGroupUrl",
"type": "string",
"value": "https://graph.microsoft.com/v1.0/groups/<your group id>/members?$select=userPrincipalName&$top=5"
}
]
},
"runAfter": {},
"type": "InitializeVariable"
},
"Initialize_variable_2": {
"inputs": {
"variables": [
{
"name": "users",
"type": "array"
}
]
},
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Until": {
"actions": {
"Compose": {
"inputs": "#union(variables('users'),body('HTTP')['value'])",
"runAfter": {
"HTTP": [
"Succeeded"
]
},
"type": "Compose"
},
"HTTP": {
"inputs": {
"authentication": {
"audience": "https://graph.microsoft.com",
"clientId": "<app id>",
"secret": "<app secret>",
"tenant": "<your secret>",
"type": "ActiveDirectoryOAuth"
},
"method": "GET",
"uri": "#variables('GetGroupUrl')"
},
"runAfter": {},
"type": "Http"
},
"Set_variable": {
"inputs": {
"name": "GetGroupUrl",
"value": "#{if(equals(body('HTTP')?['#odata.nextLink'], null),null,body('HTTP')['#odata.nextLink'])}"
},
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "SetVariable"
}
},
"expression": "#equals(variables('GetGroupUrl'), '')",
"limit": {
"count": 60,
"timeout": "PT1H"
},
"runAfter": {
"Initialize_variable_2": [
"Succeeded"
]
},
"type": "Until"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"manual": {
"inputs": {
"method": "GET",
"schema": {
"properties": {
"text": {
"type": "string"
}
},
"type": "object"
}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
You can just replace the params with your own and paste it into your logic app code view and test it .
It works for me, as you can see , each request results are different :
Hope it helps .
This issue solved by OP self, this issue is due to queries in request URL , copy OP's comment as an answer :
After fiddling a bit more around with what each of you providing I
found a solution. It seems that when the query arguments are passed to
the HTTP GET outside of the endpoint itself (meaning in the "queries"
field inside of the block) it seems to keep overriding the nextLink.
When writing the endpoint URL out entirely with the odata parameters,
it works as intended.
For the Discovery REST api, the argument/parameter "return" controls which fields are returned.
So if I pass these arguments to the API
{
"query": named_sector,
"count": "10",
"filter": filter_dates,
"aggregation" : "term(docSentiment.type,count:3)"
}
my_query = discovery.query(my_disc_environment_id, my_disc_collection_id, qopts)
print(json.dumps(my_query, indent=2))
I get the following:
{
"matching_results": 14779,
"aggregations": [
{
"type": "term",
"field": "docSentiment.type",
"count": 3,
"results": [
{
"key": "positive",
"matching_results": 4212
},
{
"key": "negative",
"matching_results": 3259
},
{
"key": "neutral",
"matching_results": 152
}
]
}
],
"results": [
{
"id": "6389715fe7e7f711e0bc09d4f1236639",
"score": 1.3689895,
"yyyymm": "201704",
"url": "https://seekingalpha.com/article/4060446-valuation-dashboard-consumer-discretionary-update",
"enrichedTitle": null,
"host": "seekingalpha.com",
"text": "Valuation Dashboard: Consumer Discretionary - Update\n\nSummary\n\nValuation metrics in Consumer Discretionary.\n\nEvolution since last month.\n\nA list of stocks loo ....
and thousands of more lines. How do I restrict the output to the aggregations section? Is this an issue of me better handling the JSON structure that is returned?
thanks
If you change the count argument to 0, the returned JSON will only contain the aggregations.
Also, if you're using the Discovery web tooling, you can enter 0 for the "Number of results to return (Count)" field.
More details and an example can be found here: https://www.ibm.com/watson/developercloud/doc/discovery/using.html#building-aggregations