Twilio Autopilot - How can I receive images from Whatsapp? - twilio

Question

I'm building a chatbot with Twilio Autopilot and I want to get the images that I send to the bot, how can I get it? How can I get locations as well?
Thanks a lot.

Answer 1

Twilio developer evangelist here.
There isn't a simple way to do this, but there are a few potential work-arounds. One is to have a webhook endpoint that will get the input and if the payload contains elements of an image, then do whatever you want with it, otherwise if it is just text, then maybe send to Autopilot. That is gone over in this blog post on Autopilot enhancements in Node.js.
Another is to make a Twilio Function that would point to a Twilio Studio flow or Assets if it is media in the first message.
Another is to use Twilio Functions or a similar server. You should have an Autopilot task that redirects to that Function with JSON like this:
{
"actions": [
{
"redirect": {
"uri": "https://REPLACE-WITH-YOUR-FUNCTION-URL.twil.io/auso",
"method": "POST"
}
}
]
}
Then your Twilio Function could get the image URL with something like this in Node.js:
const bodyContent = event.MediaUrl0;
const filename = event.MessageSid + '.png';
Now in a Collect Action you can also specify the Twilio.MEDIA as the type for questions expecting Media and currently we support all media format which are supported by Twilio Messaging.
{
"question": "Please a take a picture of insurance card?",
"name": "insurance_card",
"type": "Twilio.MEDIA",
"validate": {
"allowed_types": {
"list": [
"image/jpeg",
"image/gif",
"image/png",
"image/bmp"
]
},
Lastly, you may be interested in this blog post on building an image classifier with Autopilot and TensorFlow.
Let me know if this helps at all! :D

Answer 2

Regarding images - As seen in this example of an autopilot task program, specify the input type to be an image
{
"actions": [
{
"collect": {
"name": "contact",
"questions": [
{
"question": "Please upload a cool picture of yourself",
"name": "contact_image",
"type": "Twilio.MEDIA"
}
],
"on_complete": {
"redirect": {
"method": "POST",
"uri": "https://url.twil.io/image-processing"
}
}
}
}
]
}
Then you can access the image as seen have done in the following function
exports.handler = function(context, event, callback) {
//we get the Memory from the answered questions.
let memory = JSON.parse(event.Memory);
//set up an array of object "actions" for the autopilot to continue.
let actions = [];
let responseItem;
//print the url of the image
let image_url = memory.twilio.collected_data.contact.answers.contact_image.media.url;
console.log(image_url);
responseItem = {
"redirect": {
"method": "POST",
"uri": "task://next_task"
}
};
actions.push(responseItem);
let respObj = {
"actions": actions
};
callback(null, respObj);
};

Answer 3

Autopilot Troublehooting
Unable to receive picture messages
Autopilot is currently unable to receive messages with pictures or other media types supported by Twilio on any messaging channel and will throw error with code 11200.
I haven't tried WhatsApp location data but there is a blog on the functionality that may help?
New Rich Features Support Deeper Customer Engagement on WhatsApp