Related
I'm trying to convert a JSON payload to Avro to publish to a Kafka topic. However, when I do the Dataweave transformation I'm getting a "No Type" error. I'm not sure what's causing the error. I originally thought this might be due to the transformation not knowing what the MIME type on the inbound payload. So, I've made sure that it's set to application/json but that didn't make any difference.
Avro Schema
{
"compatibility" : "forward",
"name": "ContentManagerCoupons",
"type": "record",
"namespace": "com.rentpath",
"fields": [
{
"name": "clientID",
"type": "string"
},
{
"name": "outputHistoryId",
"type": "string"
},
{
"name": "categoryCoupons",
"type": {
"type": "array",
"items": {
"name": "categoryCoupons_record",
"type": "record",
"fields": [
{
"name": "applyBy",
"type": [
"string",
"int",
"null"
]
},
{
"name": "applyPeriod",
"type": [
"string",
"null"
]
},
{
"name": "cashValue",
"type": [
"int",
"null"
]
},
{
"name": "couponCategory",
"type": "string"
},
{
"name": "cashOffDesc",
"type": [
"string",
"null"
]
},
{
"name": "endDate",
"type": [
"string",
"null"
]
},
{
"name": "feeType",
"type": [
"string",
"null"
]
},
{
"name": "freeWeeks",
"type": [
"string",
"null"
]
},
{
"name": "generatedText",
"type": "string"
},
{
"name": "leaseby",
"type": [
"string",
"null"
]
},
{
"name": "leaseTerm",
"type": [
"int",
"null"
]
},
{
"name": "offerText",
"type": [
"string",
"null"
]
},
{
"name": "startDate",
"type": "string"
},
{
"name": "unitType",
"type": [
"string",
"null"
]
}
]
}
}
}
]
}
JSON Message
{
"outputHistoryId": "55324456",
"clientID": "112345",
"categoryCoupons": [
{
"unitType": null,
"startDate": "07/21/2020",
"offerText": "This would be the special offer message.",
"leaseTerm": null,
"leaseby": null,
"generatedText": "This would be the special offer message..",
"freeWeeks": null,
"feeType": null,
"endDate": "10/01/2020",
"couponCategory": "Special Offer",
"cashValue": null,
"cashOffDesc": null,
"applyPeriod": null,
"applyBy": null
}
]
}
Datawave
%dw 2.2
output application/avro schemaUrl="http://schema-registry.domain.com:8081/subjects/Coupon-value/versions/1"
---
payload
Error Message
"org.apache.avro.SchemaParseException - No type: {"subject":"ContentManager.Coupon-value","version":1,"id":342,"schema":"{"type":"record","name":"ContentManagerCoupons","namespace":"com.rentpath","fields":[{"name":"clientID","type":"string"},{"name":"outputHistoryId","type":"string"},{"name":"categoryCoupons","type":{"type":"array","items":{"type":"record","name":"categoryCoupons_record","fields":[{"name":"applyBy","type":["string","int","null"]},{"name":"applyPeriod","type":["string","null"]},{"name":"cashValue","type":["int","null"]},{"name":"couponCategory","type":"string"},{"name":"cashOffDesc","type":["string","null"]},{"name":"endDate","type":["string","null"]},{"name":"feeType","type":["string","null"]},{"name":"freeWeeks","type":["string","null"]},{"name":"generatedText","type":"string"},{"name":"leaseby","type":["string","null"]},{"name":"leaseTerm","type":["int","null"]},{"name":"offerText","type":["string","null"]},{"name":"startDate","type":"string"},{"name":"unitType","type":["string","null"]}]}}}],"compatibility":"forward"}"}
org.apache.avro.SchemaParseException: No type: {"subject":"ContentManager.Coupon-value","version":1,"id":342,"schema":"{"type":"record","name":"ContentManagerCoupons","namespace":"com.rentpath","fields":[{"name":"clientID","type":"string"},{"name":"outputHistoryId","type":"string"},{"name":"categoryCoupons","type":{"type":"array","items":{"type":"record","name":"categoryCoupons_record","fields":[{"name":"applyBy","type":["string","int","null"]},{"name":"applyPeriod","type":["string","null"]},{"name":"cashValue","type":["int","null"]},{"name":"couponCategory","type":"string"},{"name":"cashOffDesc","type":["string","null"]},{"name":"endDate","type":["string","null"]},{"name":"feeType","type":["string","null"]},{"name":"freeWeeks","type":["string","null"]},{"name":"generatedText","type":"string"},{"name":"leaseby","type":["string","null"]},{"name":"leaseTerm","type":["int","null"]},{"name":"offerText","type":["string","null"]},{"name":"startDate","type":"string"},{"name":"unitType","type":["string","null"]}]}}}],"compatibility":"forward"}"}
at org.apache.avro.Schema.getRequiredText(Schema.java:1753)
at org.apache.avro.Schema.parse(Schema.java:1604)
at org.apache.avro.Schema$Parser.parse(Schema.java:1394)
at org.apache.avro.Schema$Parser.parse(Schema.java:1365)
at org.mule.weave.v2.module.avro.AvroWriter.doWriteValue(AvroWriter.scala:195)
at org.mule.weave.v2.module.writer.Writer.writeValue(Writer.scala:41)
at org.mule.weave.v2.module.writer.Writer.writeValue$(Writer.scala:39)
at org.mule.weave.v2.module.avro.AvroWriter.writeValue(AvroWriter.scala:44)
at org.mule.weave.v2.module.writer.DeferredWriter.doWriteValue(DeferredWriter.scala:73)
at org.mule.weave.v2.module.writer.Writer.writeValue(Writer.scala:41)
at org.mule.weave.v2.module.writer.Writer.writeValue$(Writer.scala:39)
at org.mule.weave.v2.module.writer.DeferredWriter.writeValue(DeferredWriter.scala:16)
at org.mule.weave.v2.module.writer.WriterHelper$.writeValue(Writer.scala:120)
at org.mule.weave.v2.module.writer.WriterHelper$.writeAndGetResult(Writer.scala:98)
at org.mule.weave.v2.interpreted.InterpretedMappingExecutableWeave.write(InterpreterMappingCompilerPhase.scala:236)
at org.mule.weave.v2.el.WeaveExpressionLanguageSession.evaluateWithTimeout(WeaveExpressionLanguageSession.scala:243)
at org.mule.weave.v2.el.WeaveExpressionLanguageSession.evaluate(WeaveExpressionLanguageSession.scala:108)
at org.mule.runtime.core.internal.el.dataweave.DataWeaveExpressionLanguageAdaptor$1.evaluate(DataWeaveExpressionLanguageAdaptor.java:308)
at org.mule.runtime.core.internal.el.DefaultExpressionManagerSession.evaluate(DefaultExpressionManagerSession.java:105)
at com.mulesoft.mule.runtime.core.internal.processor.SetPayloadTransformationTarget.process(SetPayloadTransformationTarget.java:32)
at com.mulesoft.mule.runtime.core.internal.processor.TransformMessageProcessor.lambda$0(TransformMessageProcessor.java:92)
at java.util.Optional.ifPresent(Optional.java:159)
at com.mulesoft.mule.runtime.core.internal.processor.TransformMessageProcessor.process(TransformMessageProcessor.java:92)
at org.mule.runtime.core.api.util.func.CheckedFunction.apply(CheckedFunction.java:25)
at org.mule.runtime.core.api.rx.Exceptions.lambda$checkedFunction$2(Exceptions.java:84)
at org.mule.runtime.core.internal.util.rx.Operators.lambda$nullSafeMap$0(Operators.java:47)
at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:165)
at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain$2.onNext(AbstractMessageProcessorChain.java:425)
at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain$2.onNext(AbstractMessageProcessorChain.java:420)
at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:345)
at reactor.core.publisher.FluxSubscribeOnValue$ScheduledScalar.run(FluxSubscribeOnValue.java:178)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:111)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748), while writing Avro at payload.
It seems to work for me. Maybe there is a problem trying to access the schema. Because I don't have access to that URL I replaced it with a local file:
output application/avro schemaUrl="classpath://schema.json"
Apparently the answer is pretty simple. I just needed to add schema to the end of my URL. This removes the extraneous items like version and id that come down without it.
New Dataweave
%dw 2.2
output application/avro schemaUrl="http://schema-registry.domain.com:8081/subjects/Coupon-value/versions/1/schema"
---
payload
as per title.
Whilst dealing with my own mailbox is fine I wish to ideally process a shared mailbox in LogicApps. I can read the emails but I want to clean-up by deleting the messages.
Logic Apps doesn't seem to offer that action and unless mistaken the graph api doesnt either?
Has anyone managed this?
This has been possible since the update on 6th May 2020. A number of the actions now support an Original Mailbox Address optional parameter that you can set to access the shared mailbox:
As of May 6, 2020, shared mailbox support was added for certain operations with an optional 'Mailbox address' parameter, allowing you to specify a shared mailbox address for your operation to access.
The Delete Email (V2) action supports this parameter:
You then fill in the email address of your shared mailbox:
Which will then successfully find your message id and delete the email for you:
For this requirement, I write a logic app sample for your reference:
Before we start, provide the whole logic app:
1. In first "HTTP" action, I request for the access token.
2. Then I add a "Parse JSON" action to parse the response from the first "HTTP" action to get the access token.
The schema should be:
{
"properties": {
"access_token": {
"type": "string"
},
"expires_in": {
"type": "integer"
},
"ext_expires_in": {
"type": "integer"
},
"token_type": {
"type": "string"
}
},
"type": "object"
}
3. In the second "HTTP" action, I request for all of the messages by the access token from above steps.
4. After that, we need to use another "Parse JSON" action to parse the response from "HTTP 2".
The schema should be:
{
"properties": {
"##odata.context": {
"type": "string"
},
"value": {
"items": {
"properties": {
"##odata.etag": {
"type": "string"
},
"bccRecipients": {
"type": "array"
},
"body": {
"properties": {
"content": {
"type": "string"
},
"contentType": {
"type": "string"
}
},
"type": "object"
},
"bodyPreview": {
"type": "string"
},
"categories": {
"type": "array"
},
"ccRecipients": {
"type": "array"
},
"changeKey": {
"type": "string"
},
"conversationId": {
"type": "string"
},
"conversationIndex": {
"type": "string"
},
"createdDateTime": {
"type": "string"
},
"flag": {
"properties": {
"flagStatus": {
"type": "string"
}
},
"type": "object"
},
"from": {
"properties": {
"emailAddress": {
"properties": {
"address": {
"type": "string"
},
"name": {
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"hasAttachments": {
"type": "boolean"
},
"id": {
"type": "string"
},
"importance": {
"type": "string"
},
"inferenceClassification": {
"type": "string"
},
"internetMessageId": {
"type": "string"
},
"isDeliveryReceiptRequested": {
"type": "boolean"
},
"isDraft": {
"type": "boolean"
},
"isRead": {
"type": "boolean"
},
"isReadReceiptRequested": {
"type": "boolean"
},
"lastModifiedDateTime": {
"type": "string"
},
"parentFolderId": {
"type": "string"
},
"receivedDateTime": {
"type": "string"
},
"replyTo": {
"type": "array"
},
"sender": {
"properties": {
"emailAddress": {
"properties": {
"address": {
"type": "string"
},
"name": {
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"sentDateTime": {
"type": "string"
},
"subject": {
"type": "string"
},
"toRecipients": {
"items": {
"properties": {
"emailAddress": {
"properties": {
"address": {
"type": "string"
},
"name": {
"type": "string"
}
},
"type": "object"
}
},
"required": [
"emailAddress"
],
"type": "object"
},
"type": "array"
},
"webLink": {
"type": "string"
}
},
"required": [
"##odata.etag",
"id",
"createdDateTime",
"lastModifiedDateTime",
"changeKey",
"categories",
"receivedDateTime",
"sentDateTime",
"hasAttachments",
"internetMessageId",
"subject",
"bodyPreview",
"importance",
"parentFolderId",
"conversationId",
"conversationIndex",
"isDeliveryReceiptRequested",
"isReadReceiptRequested",
"isRead",
"isDraft",
"webLink",
"inferenceClassification",
"body",
"sender",
"from",
"toRecipients",
"ccRecipients",
"bccRecipients",
"replyTo",
"flag"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
}
5. Then use a "For each" and loop the value from "Parse JSON 2".
6. In "For each", we need to add third "HTTP" action as below screenshot:
7. Run the logic app, it will delete all of the messages in shared mailbox.
By the way:
Before run the logic app, you need to search the client_id in your azure ad app registrations to find the application and add the Mail.ReadWrite permission to it. Also don't forget grant admin consent for it.
I'm having difficulty accessing data in a collection, via PowerApps.
I create the collection with this:
Collect(coll15,mt.GetAnswers("3b....da","application/json",{question:"eco"}))
Using Developer Tools -> Network tab -> Response body - the following JSON data is returned:
{
"answers": [
{
"answer": "This is the answer",
"questions": [
"Private vehicle eco renewal"
],
"score": 82.901087775826454
}
]
}
The collection is created.
I then add a gallery control to my page - however the only options I have to bind to the labels are: ThisItem.Value
If I try to enter ThisItem.Value.answer I get the error: Invalid use of '.'
If I enter ThisItem.answers.answer I get the error: Invalid name
This is the swagger file:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "mt",
"description": "mt"
},
"host": "westus.api.cognitive.microsoft.com:443",
"basePath": "/",
"schemes": [
"https"
],
"consumes": [],
"produces": [
"application/json"
],
"paths": {
"/qnamaker/v2.0/knowledgebases/eeeee.....eeeee/generateAnswer": {
"post": {
"summary": "GetAnswers",
"description": "Get answers from qna",
"operationId": "GetAnswers",
"parameters": [
{
"name": "body",
"in": "body",
"schema": {
"type": "object",
"properties": {
"question": {
"type": "string",
"description": "question",
"x-ms-summary": "question",
"title": "question",
"x-ms-visibility": ""
}
},
"default": {
"question": "hi"
},
"required": [
"question"
]
},
"required": true
}
],
"responses": {
"default": {
"description": "default",
"schema": {
"type": "string"
}
}
}
}
}
},
"definitions": {},
"parameters": {},
"responses": {},
"securityDefinitions": {
"api_key": {
"type": "apiKey",
"in": "header",
"name": "Ocp-Apim-Subscription-Key"
}
},
"security": [
{
"oauth2_auth": [
"Offline-Access"
]
}
],
"tags": []
}
Is there any way for me to access the answer text within the collection?
Thanks for any help,
Mark
The problem is that the response type for the operation in the connector definition is string:
"responses": {
"default": {
"description": "default",
"schema": {
"type": "string"
}
}
}
But your response is an object instead. If you update your custom connector to use a typed object instead, you should be able to access the response from the operation. Something along the lines of the schema below:
"responses": {
"default": {
"description": "default",
"schema": {
"type": "object",
"properties": {
"answers": {
"type": "array",
"items": {
"type": "object",
"properties": {
"answer": {
"type": "string"
},
"questions": {
"type": "array",
"items": {
"type": "string"
}
},
"score": {
"type": "number",
"format": "float"
}
}
}
}
}
}
}
},
Notice that in the portal (web.powerapps.com), if you go to your custom connector definition, and select "Edit", you can go to the operation, and select the response you want to edit:
And then use the "Import from sample" option
With that, if you enter an example of a response from the API, it will create the schema for you (which is similar to the one I have above).
What is the best way to represent a generic response/payload object that has basic fields such as message, total elements, data, and status? Where the variability between each API is the data element. For instance, one API could be for permissions, so the data element would hold an array of permissions. But for another API, it would hold a different array of object types. My main goal is to have reuse with a payload object and to define the next "layer" of actual data.
I want to be able to define a data type that's generic - like a "response" that has basic fields but I want to be able to further define that content for each API (data containing permissions or roles or whatever).
Here are some JSON samples of what's been attempted but isn't rendering the way we would expect it to in Swagger UI (i.e. a flat structure of 4 elements with data defined per the API).
Example 1 - composition:
{
"swagger": "2.0",
"host": "localhost:8888",
"info": {
"version": "0.0.1",
"title": "API"
},
"paths": {
"/permissions": {
"get": {
"description": "Returns all permissions",
"operationId": "getPermissions",
"produces": [
"application/json"
],
"responses": {
"200": {
"description": "success",
"schema": {
"$ref": "#/definitions/permissionResponse"
}
}
}
}
}
},
"definitions": {
"response": {
"type": "object",
"properties": {
"message": {
"type": "string",
"description": "A string indicating the response from the server."
},
"totalElements": {
"type": "integer",
"format": "int64",
"description": "The number of items retrieved."
},
"status": {
"type": "string",
"description": "A string indicating the response from the server."
}
}
},
"permissionResponse": {
"allOf": [
{
"$ref": "#/definitions/response"
}, {
"type": "object",
"properties": {
"data": {
"type": "array",
"description": "The collection of items returned from the API request.",
"items": {
"$ref": "#/definitions/permission"
}
}
}
}
]
},
"permission": {
"type": "object",
"properties": {
"id": {
"type": "integer",
"format": "int64",
"description": "Unique identifier representing a specific permission."
},
"label": {
"type": "string",
"description": "The label of a permission."
},
"description": {
"type": "string",
"description": "A description of the permission."
},
"active": {
"type": "boolean",
"description": "A flag indicating whether a permission is active."
}
},
"required": [
"id",
"label",
"description",
"active"
]
}
}
}
Example 2 - composition variation:
{
"swagger": "2.0",
"host": "localhost:8888",
"info": {
"version": "0.0.1",
"title": "API"
},
"paths": {
"/permissions": {
"get": {
"description": "Returns all permissions",
"operationId": "getPermissions",
"produces": [
"application/json"
],
"responses": {
"200": {
"description": "success",
"schema": {
"$ref": "#/definitions/permissionResponse"
}
}
}
}
}
},
"definitions": {
"response": {
"type": "object",
"properties": {
"message": {
"type": "string",
"description": "A string indicating the response from the server."
},
"totalElements": {
"type": "integer",
"format": "int64",
"description": "The number of items retrieved."
},
"status": {
"type": "string",
"description": "A string indicating the response from the server."
}
}
},
"permissionResponse": {
"allOf": [
{
"$ref": "#/definitions/response"
}],
"type": "object",
"properties": {
"data": {
"type": "array",
"description": "The collection of items returned from the API request.",
"items": {
"$ref": "#/definitions/permission"
}
}
}
},
"permission": {
"type": "object",
"properties": {
"id": {
"type": "integer",
"format": "int64",
"description": "Unique identifier representing a specific permission."
},
"label": {
"type": "string",
"description": "The label of a permission."
},
"description": {
"type": "string",
"description": "A description of the permission."
},
"active": {
"type": "boolean",
"description": "A flag indicating whether a permission is active."
}
},
"required": [
"id",
"label",
"description",
"active"
]
}
}
Example 3 - additional properties:
{
"swagger": "2.0",
"host": "localhost:8888",
"info": {
"version": "0.0.1",
"title": "API"
},
"paths": {
"/permissions": {
"get": {
"description": "Returns all permissions",
"operationId": "getPermissions",
"produces": [
"application/json"
],
"responses": {
"200": {
"description": "success",
"schema": {
"$ref": "#/definitions/permissionResponse"
}
}
}
}
}
},
"definitions": {
"response": {
"type": "object",
"properties": {
"message": {
"type": "string",
"description": "A string indicating the response from the server."
},
"totalElements": {
"type": "integer",
"format": "int64",
"description": "The number of items retrieved."
},
"status": {
"type": "string",
"description": "A string indicating the response from the server."
}
}
},
"permissionResponse": {
"type": "object",
"properties": {
"data": {
"type": "array",
"description": "The collection of items returned from the API request.",
"items": {
"$ref": "#/definitions/permission"
}
}
},
"additionalProperties": {
"$ref": "#/definitions/response"
}
},
"permission": {
"type": "object",
"properties": {
"id": {
"type": "integer",
"format": "int64",
"description": "Unique identifier representing a specific permission."
},
"label": {
"type": "string",
"description": "The label of a permission."
},
"description": {
"type": "string",
"description": "A description of the permission."
},
"active": {
"type": "boolean",
"description": "A flag indicating whether a permission is active."
}
},
"required": [
"id",
"label",
"description",
"active"
]
}
}
Example 1 renders well with swagger2markup showing the consolidated view of the data with the other 3 elements and having a permission array. However, with Swagger UI, it renders it differently, separating the objects out.
Markup rendered:
Swagger UI rendered
Swagger UI rendered - expanded
Inside the Swagger API Documentation there is inside the json beside the apis array a model object entry but no documentation about it. How can I use this "models" part?
{
apiVersion: "0.2",
swaggerVersion: "1.1",
basePath: "http://petstore.swagger.wordnik.com/api",
resourcePath: "/pet.{format}"
...
apis: [...]
models: {...}
}
Models are nothing but like your POJO classes in java which have variables and properties. In models section you can define your own custom class and you can refer it as data type.
If you see below
{
"path": "/pet.{format}",
"description": "Operations about pets",
"operations": [
{
"httpMethod": "POST",
"summary": "Add a new pet to the store",
"responseClass": "void",
"nickname": "addPet",
"parameters": [
{
"description": "Pet object that needs to be added to the store",
"paramType": "body",
"required": true,
"allowMultiple": false,
"dataType": "Pet"
}
],
"errorResponses": [
{
"code": 405,
"reason": "Invalid input"
}
]
}
Here in parameter section it have one parameter who's dataType is Pet and pet is defined in models as below
{
"models": {
"Pet": {
"id": "Pet",
"properties": {
"id": {
"type": "long"
},
"status": {
"allowableValues": {
"valueType": "LIST",
"values": [
"available",
"pending",
"sold"
]
},
"description": "pet status in the store",
"type": "string"
},
"name": {
"type": "string"
},
"photoUrls": {
"items": {
"type": "string"
},
"type": "Array"
}
}
}
}}
You can have nested models , for more information see Swagger PetStore example
So models are nothing but like classes.