Ref see: https://issuetracker.google.com/u/1/issues/128352542
We are having an issue where certain words added in the user's speech context are not being returned or prioritized.
When using phrase hints, the API will generally correctly transcribe the phrases or words supplied when uttered, however some words will not be transcribed no matter how you add them in phrase hints.
Config sent inside StreamingRecognitionConfig:
{
"config":{
"encoding":"LINEAR16",
"sampleRateHertz":8000,
"languageCode":"en-US",
"enableWordTimeOffsets":true,
"enableAutomaticPunctuation":false,
"model":"default",
"useEnhanced":true,
"speechContexts":[
{
"phrases":[
"Bill Uhma",
"Uhma",
"I got coffee with Bill Uhma"
]
}
]
}
}
Result when trying to say "I got coffee with Bill Uhma":
{
"results":{
"alternatives":[
{
"confidence":0.8440007,
"transcript":"I got coffee with Bill Uma",
"words":[
{
"confidence":0.847875,
"word":"I"
},
{
"confidence":0.9265712,
"word":"got"
},
{
"confidence":0.98762906,
"word":"coffee"
},
{
"confidence":0.98762906,
"word":"with"
},
{
"confidence":0.9239746,
"word":"Bill"
},
{
"confidence":0.23432566,
"word":"Uma"
}
]
},
{
"confidence":0.94561315,
"transcript":"I got coffee with Bill Luma"
},
{
"confidence":0.911253,
"transcript":"I got coffee with Bill Guma"
},
{
"confidence":0.91219664,
"transcript":"I got coffee with Bill Houma"
},
{
"confidence":0.94028026,
"transcript":"I got coffee with Bill looma"
},
{
"confidence":0.9403957,
"transcript":"I got coffee with Bill bouma"
},
{
"confidence":0.9403957,
"transcript":"I got coffee with Bill goomah"
},
{
"confidence":0.9403957,
"transcript":"I got coffee with Bill Wilma"
},
{
"confidence":0.938467,
"transcript":"I got coffee with Bill Boomer"
},
{
"confidence":0.9403957,
"transcript":"I got coffee with Bill buma"
},
{
"confidence":0.9403957,
"transcript":"I got coffee with Bill Ooma"
},
{
"confidence":0.9403957,
"transcript":"I got coffee with Bill Gooma"
}
],
"confidence":0.8440007,
"is_final":true,
"transcription":"I got coffee with Bill Uma"
}
}
The received transcription is "I got coffee with Bill Uma".
The expected transcription is "I got coffee with Bill Uhma".
As seen in the result, the provided hints do not appear in any of the 12 alternatives received.
Separating the phrase hints and only sending one of them has no effect on the result.
This is actually not a bug but should rather be considered as a feature request to force the recognizer to use the provided phrases/hints specially if the word/s in the phrase does not exist in its vocabulary. If you would notice, the confidence level for the word "Uma" is very low which probably suggest that the recognizer does not understand it (not in its vocabulary).
File feature request here
Related
I'm currently working on a Twitter bot that automatically reply messages, I'm doing this by using tweepy (the official python twitter library)
I need to filter messages based on the created time as I don't want to reply same message twice. Now the problem is that the API endpoint returns created_timestamp as string representation of positive integers.
Below is an example of data returned as per the doc
{
"next_cursor": "AB345dkfC",
"events": [
{ "id": "110", "created_timestamp": "1639919665615", ... },
{ "id": "109", "created_timestamp": "1639865141987", ... },
{ "id": "108", "created_timestamp": "1639827437833", ... },
{ "id": "107", "created_timestamp": "1639825389806", ... },
{ "id": "106", "created_timestamp": "1639825389796", ... },
{ "id": "105", "created_timestamp": "1639825389768", ... },
...
]
}
My question is "How do I convert the created_timestamp to a valid date using python" ?.
You might play with timestamps on this resource
And in your case could use methods like:
timestamp = int('timestamp_string')
datetime.fromtimestamp(timestamp, tz=None)
date.fromtimestamp(timestamp)
From the datetime standard library. But integers after the first line are already well comparable if the task is to distinguish differences between the timestamps.
I am using API version 2.0 and unable to fetch the user.fields results. All other parameters seem to be returning results correctly. I'm following this documentation.
url = "https://api.twitter.com/2/tweets/search/all"
query_params = {
"query": "APPL",
"max_results": "10",
"tweet.fields": "created_at,lang,text,author_id",
"user.fields": "name,username,created_at,location",
"expansions": "referenced_tweets.id.author_id",
}
response = requests.request("GET", url, headers=headers, params=query_params).json()
Sample result:
{
'author_id': '1251347502013521925',
'text': 'All conspiracy. But watch for bad news on Apple. Such a vulnerable stocktechnically for the biggest market cap # $2.1T ( Thanks Jay). This is the glue for the bulls. But, they stopped innovating when Steve died, built a fancy office and split the stock. $appl',
'lang': 'en',
'created_at': '2021-06-05T02:33:48.000Z',
'id': '1401004298738311168',
'referenced_tweets': [{
'type': 'retweeted',
'id': '1401004298738311168'
}]
}
As you can see, the following information is not returned: name, username, and location.
Any idea how to retrieve this info?
Your query does actually return the correct data. I tested this myself.
A full example response will be structured like this:
{
"data": [
{
"created_at": "2021-06-05T02:33:48.000Z",
"lang": "en",
"id": "1401004298738311168",
"text": "All conspiracy. But watch for bad news on Apple. Such a vulnerable stocktechnically for the biggest market cap # $2.1T ( Thanks Jay). This is the glue for the bulls. But, they stopped innovating when Steve died, built a fancy office and split the stock. $appl",
"author_id": "1251347502013521925",
"referenced_tweets": [
{
"type": "retweeted",
"id": "1401004298738311168"
}
]
}
],
"includes": {
"users": [
{
"name": "Gary Casper",
"id": "1251347502013521925",
"username": "Hisel1979",
"created_at": "2020-07-11T13:39:58.000Z"
}
]
}
}
The sample result you provided comes from within the data object. However, the expanded object data will be nested in the includes object (in your case name, username, and location). The corresponding user object can be referenced via the author_id field.
I'm building a chatbot with Twilio Autopilot and I want to get the images that I send to the bot, how can I get it? How can I get locations as well?
Thanks a lot.
Twilio developer evangelist here.
There isn't a simple way to do this, but there are a few potential work-arounds. One is to have a webhook endpoint that will get the input and if the payload contains elements of an image, then do whatever you want with it, otherwise if it is just text, then maybe send to Autopilot. That is gone over in this blog post on Autopilot enhancements in Node.js.
Another is to make a Twilio Function that would point to a Twilio Studio flow or Assets if it is media in the first message.
Another is to use Twilio Functions or a similar server. You should have an Autopilot task that redirects to that Function with JSON like this:
{
"actions": [
{
"redirect": {
"uri": "https://REPLACE-WITH-YOUR-FUNCTION-URL.twil.io/auso",
"method": "POST"
}
}
]
}
Then your Twilio Function could get the image URL with something like this in Node.js:
const bodyContent = event.MediaUrl0;
const filename = event.MessageSid + '.png';
Now in a Collect Action you can also specify the Twilio.MEDIA as the type for questions expecting Media and currently we support all media format which are supported by Twilio Messaging.
{
"question": "Please a take a picture of insurance card?",
"name": "insurance_card",
"type": "Twilio.MEDIA",
"validate": {
"allowed_types": {
"list": [
"image/jpeg",
"image/gif",
"image/png",
"image/bmp"
]
},
Lastly, you may be interested in this blog post on building an image classifier with Autopilot and TensorFlow.
Let me know if this helps at all! :D
Regarding images - As seen in this example of an autopilot task program, specify the input type to be an image
{
"actions": [
{
"collect": {
"name": "contact",
"questions": [
{
"question": "Please upload a cool picture of yourself",
"name": "contact_image",
"type": "Twilio.MEDIA"
}
],
"on_complete": {
"redirect": {
"method": "POST",
"uri": "https://url.twil.io/image-processing"
}
}
}
}
]
}
Then you can access the image as seen have done in the following function
exports.handler = function(context, event, callback) {
//we get the Memory from the answered questions.
let memory = JSON.parse(event.Memory);
//set up an array of object "actions" for the autopilot to continue.
let actions = [];
let responseItem;
//print the url of the image
let image_url = memory.twilio.collected_data.contact.answers.contact_image.media.url;
console.log(image_url);
responseItem = {
"redirect": {
"method": "POST",
"uri": "task://next_task"
}
};
actions.push(responseItem);
let respObj = {
"actions": actions
};
callback(null, respObj);
};
Autopilot Troublehooting
Unable to receive picture messages
Autopilot is currently unable to receive messages with pictures or other media types supported by Twilio on any messaging channel and will throw error with code 11200.
I haven't tried WhatsApp location data but there is a blog on the functionality that may help?
New Rich Features Support Deeper Customer Engagement on WhatsApp
I wanted an Autopilot bot to send its memory data to a webhook file of mine (which is in PHP) and in return (as a callback) just print a message saying "Received".
But every time, I get an error saying "It does not match with Action Schema."
So I was wondering if anyone could give me a nice example as to how to do this?
I've tried reading the ActionSchema.json file provided in the Debugger, but I couldn't understand it at all.
I have tried like this
echo '"definitions": {
"say": {
"anyOf": [
{
"type": "strdfghing"
}';
I want Twilio Autopilot to reply "Got it!" after it sends its memory events to a webhook file.
Twilio developer evangelist here.
Welcome to StackOverflow!
So you probably have your Collect flow, where you can use the Say verb to respond with "got it!" at the end in the on_complete of a Redirect. Your Autopilot task may look something like this:
"actions": [
{
"collect": {
"name": "collect_clothes_order",
"questions": [
{
"question": "What is your first name?",
"name": "first_name",
"type": "Twilio.FIRST_NAME"
},
],
"on_complete": {
"redirect": "your-web-server-ie-a-twilio-function-maybe"
}
}
}
]
Then, in your Twilio Function (or another web server in the language of your choosing), you could receive the memory data (in this case, the first name that the user responded with), and return the Say containing "got it!":
exports.handler = function(context, event, callback) {
let responseObject = {};
let memory = JSON.parse(event.Memory);
console.log(memory.twilio.collected_data);
let first_name = memory.twilio.collected_data.collect_clothes_order.answers.first_name.answer;
console.log(first_name); //collected data from memory
responseObject = {"actions":[
{ "say": { "speech": "Got it!" } }
]};
callback(null, responseObject);
};
Let me know if this helps at all!
I'm trying to do basic social app using Firebase - each user has user's he follows and on his timeline he can see their post.
I'm using fanout technique described here
After this my data structure for user1 looks like this (it contains both - his followings and timeline):
{
"timeline": {
"user2": {
"-K-zOrtjiCGe7tgRk8DG": {
"text": "I love emojis!",
"date": "2016-09-01"
},
"-K-zOrt9a8Asdczaq23G": {
"text": "Second post!",
"date": "2016-09-03"
}
},
"user3": {
"-K-zOrtjiCGe7tgRk8DG": {
"text": "Post 3",
"date": "2016-09-02"
}
}
},
"followers": {
"user2": true,
"user3": true
}
}
The problem I have is - I don't know how to read user1 timeline with all posts from all users mixed and ordered by time. I was reading about querying and filtering from Firebase doc but I am not able to figure out the proper way to do it.
I'm developing on iOS platform but I should have no problem with other Firebase solutions as long as they are supported on Apple platform.