I'm creating an app which uses Twilio to stream voice to Google STT, and then it sends transcribed text to another app. The problem is, it often takes more than a minute to transcribe the text. I'm mostly saying short phrases of 2-3 words. My code is:
const speech = require("#google-cloud/speech");
const client = new speech.SpeechClient();
const request = {
config: {
encoding: "MULAW",
sampleRateHertz: 8000,
languageCode: "pl-PL"
},
interimResults: true,
singleUtterance: false,
model: "latest_long"
};
Is there a better configuration that would give me the expected results? When the singleUtterance is true, the isFinal=true comes back within 2-3 seconds, but it stops getting any more input.
Thanks in advance!
Related
I have a twilio number. I would like to program a webhook for incoming calls to that number that handle the following scenario: the incoming call prompts the user to enter digits. I've tried using <Play> to send DTMF tones, but the subsequent transcription of the call doesn't indicate the tones had any effect. Here's the relevant part of the webhook endpoint
const vr = new VoiceResponse();
vr.say('Hello');
vr.record({
maxLength: 60,
transcribe: true,
transcribeCallback: '/voice/text',
});
vr.play({
digits: 'wwww1',
});
vr.hangup();
res.setHeader('Content-Type', 'text/xml');
return res.status(200).send(vr.toString());
The transcription reads along the lines of "Hello, for ____ press one, for ____ press two... Goodbye".
I would like to record subsequent parts of the call after sending digits. Not sure what I'm doing wrong.
Turns out the answer was with order of TwiML verbs. According to the docs:
Any TwiML verbs occurring after a <Record> are unreachable.
After modifying code to
const vr = new VoiceResponse();
vr.say('Hello');
vr.play({
digits: 'wwww1',
});
vr.record({
maxLength: 60,
transcribe: true,
transcribeCallback: '/voice/text',
});
vr.hangup();
res.setHeader('Content-Type', 'text/xml');
return res.status(200).send(vr.toString());
The IVR tree progression worked as expected.
Using express and Node.js, I'm using the twitter streaming API and the needle npm package for accessing APIs to pull tweets related to keywords. The streaming is functional and I am successfully pulling tweets using the following (simplified) code:
const needle = require('needle');
const TOKEN = // My Token
const streamURL = 'https://api.twitter.com/2/tweets/search/stream';
function streamTweets() {
const stream = needle.get(streamURL, {
headers: {
Authorization: `Bearer ${TOKEN}`
}
});
stream.on('data', (data) => {
try {
const json = JSON.parse(data); // This line appears to be causing my error
const text = json.data.text;
} catch (error) {
console.log("error");
}
});
}
However, no matter which search term I use (and the subsequent large or small volume of tweets coming through), the catch block will consistently log 1-3 errors per minute, which look like this:
SyntaxError: Unexpected end of JSON input
at JSON.parse (<anonymous>)
at PassThrough.<anonymous> (C:\Users\danie\OneDrive\Documents\Personal-Projects\twitter-program\server.js:56:31)
at PassThrough.emit (events.js:315:20)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at PassThrough.Readable.push (internal/streams/readable.js:223:10)
at PassThrough.Transform.push (internal/streams/transform.js:166:32)
at PassThrough.afterTransform (internal/streams/transform.js:101:10)
at PassThrough._transform (internal/streams/passthrough.js:46:3)
at PassThrough.Transform._read (internal/streams/transform.js:205:10).
I've seen previous advice which says that data can be fired in multiple chunks, and to push the chunks to an array i.e. something like the following:
let chunks = [];
stream.on('data', (dataChunk) => {
chunks.push(dataChunk);
}).on('end',() => {
// combine chunks to create JSON object
})
But this didn't work either (may have been my implementation but I don't think so) and now I'm wondering if it's perhaps an error with the twitter API, because most of the tweet objects do come through correctly. I should note that the streamTweets() function above is called from an async function, and I am also wondering if that is having something to do with it.
Has anyone else encountered this error? Or does anyone have any idea how I might be fix it? Ideally i'd like 100% of the tweets to stream correctly.
Thanks in advance!
For future readers, this error is triggered by Twitter's heartbeat message that is sent every 20 seconds. Per the documentation:
The endpoint provides a 20-second keep alive heartbeat (it will look like a new line character).
Adding a guard against parsing the empty string will prevent the JSON parsing error.
if (data === "")
return
An empty string is invalid JSON, hence the emitted error.
Now, acknowledging that the heartbeat exists, it may be beneficial to add read_timeout = 20 * 1000 in the needle request to avoiding a stalled program with no data, be that due to a local network outage or DNS miss, etc.
I'm trying to retrieve a list of Slack reminders, which works fine using Slack API's reminders.list method. However, reminders that are set using SlackBot (i.e. by asking Slackbot to remind me of a message) return the respective permalink of that message as text:
{
"ok": true,
"reminders": [
{
"id": "Rm012C299C1E",
"creator": "UV09YANLX",
"text": "https:\/\/team.slack.com\/archives\/DUNB811AM\/p1583441290000300",
"user": "UV09YANLX",
"recurring": false,
"time": 1586789303,
"complete_ts": 0
},
Instead of showing the permalink, I'd naturally like to show the message I wanted to be reminded of. However, I couldn't find any hints in the Slack API docs on how to retrieve a message identified by a permalink. The link is presumably generated by chat.getPermalink, but there seems to be no obvious chat.getMessageByPermalink or so.
I tried to interpet the path elements as channel and timestamp, but the timestamp (transformed from the example above: 1583441290.000300) doesn't seem to really match. At least I don't end up with the message I expected to retrieve when passing this as latest to conversations.history and limiting to 1.
After fiddling a while longer, here's how I finally managed in JS:
async function downloadSlackMsgByPermalink(permalink) {
const pathElements = permalink.substring(8).split('/');
const channel = pathElements[2];
var url;
if (permalink.includes('thread_ts')) {
// Threaded message, use conversations.replies endpoint
var ts = pathElements[3].substring(0, pathElements[3].indexOf('?'));
ts = ts.substring(0, ts.length-6) + '.' + ts.substring(ts.length-6);
var latest = pathElements[3].substring(pathElements[3].indexOf('thread_ts=')+10);
if (latest.indexOf('&') != -1) latest = latest.substring(0, latest.indexOf('&'));
url = `https://slack.com/api/conversations.replies?token=${encodeURIComponent(slackAccessToken)}&channel=${channel}&ts=${ts}&latest=${latest}&inclusive=true&limit=1`;
} else {
// Non-threaded message, use conversations.history endpoint
var latest = pathElements[3].substring(1);
if (latest.indexOf('?') != -1) latest = latest.substring(0, latest.indexOf('?'));
latest = latest.substring(0, latest.length-6) + '.' + latest.substring(latest.length-6);
url = `https://slack.com/api/conversations.history?token=${encodeURIComponent(slackAccessToken)}&channel=${channel}&latest=${latest}&inclusive=true&limit=1`;
}
const response = await fetch(url);
const result = await response.json();
if (result.ok === true) {
return result.messages[0];
}
}
It's not been tested to the latest extend, but first results look alright:
The trick with the conversations.history endpoint was to include the inclusive=true parameter
Messages might be threaded - the separate endpoint conversations.replies is required to fetch those
As the Slack API docs state: ts and thread_ts look like timestamps, but they aren't. Using them a bit like timestamps (i.e. cutting off some characters at the back and inserting a dot) seems to work, gladly, however.
Naturally, the slackAccessToken variable needs to be set beforehand
I'm aware the way to extract & transform the URL components in the code above might not the most elegant solution, but it proves the concept :-)
I see that Dialogflow has fulfiment and webhook installations to allow for further dynamic and logistic control over the bot responses. I'm trying to peg a database on top of the webhook, but the channel I'm using is Twilio Text Messaging, and I'm having a little trouble with connecting the two. When I do activate fulfillment, the twilio bot does not read it. Any way to solve this?
I already created a few webhooks using Flask, and integrated it through fulfillment briefly using ngrok, but the bot is responding via the text responses I set for it. Its for google assistance, and facebook messenger, but not with the Twilio integration.
I also tried using inlineJS to see if that held any difference to specifically define Twilio as the messaging outlet to use, however it did not peak success.
const functions = require('firebase-functions');
const {dialogflow} = require('actions-on-google');
const GOODLOCATION = 'location.good'
const NEARLOCATION = 'location.near'
const CHEAPLOCATION = 'location.cheap'
const WELCOME_INTENT = 'Default Welcome Intent'
const FALLBACK_INTENT = 'Default Fallback Intent'
const CRAVINGCULTUREINTENT = 'CravingCulture'
const CRAVINGITEM = 'CravingItem'
const app = dialogflow()
/*Supported Platforms*/
const PLATFORMS = {
UNSPECIFIED: 'PLATFORM_UNSPECIFIED',
FACEBOOK: 'FACEBOOK',
SLACK: 'SLACK',
TELEGRAM: 'TELEGRAM',
KIK: 'KIK',
SKYPE: 'SKYPE',
LINE: 'LINE',
VIBER: 'VIBER',
ACTIONS_ON_GOOGLE: 'ACTIONS_ON_GOOGLE',
TWILIO: 'TWILIO'
};
// Platforms that support Rich messaging
const SUPPORTED_RICH_MESSAGE_PLATFORMS = [
PLATFORMS.FACEBOOK,
PLATFORMS.SLACK,
PLATFORMS.TELEGRAM,
PLATFORMS.KIK,
PLATFORMS.SKYPE,
PLATFORMS.LINE,
PLATFORMS.VIBER,
PLATFORMS.ACTIONS_ON_GOOGLE,
PLATFROM.TWILIO
];
app.intent(WELCOME_INTENT, (conv)=> {
if(agent.requestSource === agent.TWILIO){
conv.ask('This is working, Congratulations!')
}
else{
conv.ask("Could not be served")
}
});
app.intent(FALLBACK_INTENT, (conv)=> {
conv.ask("I am unaware of that phrase, could you repeat that?")
});
exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app)
I want the output to be any thing that the user insert through the twilio that this bot will respond accordingly to what is passed in.
A little late, perhaps - hopefully this issue isn't still blocking/plaguing you.
I'm thinking you were intending:
if(agent.requestSource === PLATFORMS.TWILIO){
not:
if(agent.requestSource === agent.TWILIO){
However, also the value of agent.requestSource is actually lowercase "twilio".
I'm playing around with the Twilio MMS API, and I'm wondering if there's a simple way to show a Gif attachment after the body text?
So far I've only been able to send the Gif before the text.
I understand the behavior you are talking about. I'm sure you've figured it out by now, but if not, here is a hack:
Send a blank text file first, which means put it last, as the files in this Twilio SMS/MMS (twiml too) are processed in reverse order.
Twilio Functions Example (node.js)
exports.handler = function(context, event, callback) {
var client = context.getTwilioClient();
let dataMMS = {
to: ['+15556667777'],
from: '+15551112222', // your twilio num
body: 'I am the body',
mediaUrl: [
'http://s3.amazonaws.com/blog.invisionapp.com/uploads/2014/12/motion-example.gif',
'https://sepia-jaguar-8734.twil.io/assets/blank.csv'
],
contentType: [
'image/gif',
'text/csv'
]
};
client.messages.create(dataMMS).then(function(response){
console.log('Message Sent : ' + response.status);
return callback(null, 200); //response.status, queued = good (200)
}).catch(function(err){
console.log('An error is happening: ' + err);
return callback(null, err);
});
}
Extra:
This gif link was selected from a random link https://www.invisionapp.com/blog/7-tips-for-designing-awesome-gifs/. It was selected because it was nice and small, ~25 kb.
In my testing with Twilio and my carrier, I found gifs < ~100 kb worked and if I remember correctly they are not run thru a compressor like png/jpegs.
Note to Twilio employee: The Twilio public asset is in a dummy sub account, I'd be happy to replace it with an asset not related to me.
Oh yeah, make sure your urls have proper mime type encoding: https://www.twilio.com/docs/api/messaging/accepted-mime-types
I think Twilio is forgiving for .jpg and .png , as they are run thru compressors...
ps This will give you an extra blank line in your text block, you can get rid of it. Share your answer when you discover how, cause I've shared enough today.