I have implemented Twilio Voice connect with my Video Room, to let participants join using PSTN dial-in. This works alright, but muting the phone participant is a problem. There seems to be a way to mute the other participants' voice tracks, but that doesn't work for phone participants.
I'm following the implementation in the sample-app, where it sends a 'mute' message on the data track,
const handleMuteSpeaker = () => {
const [localDataTrackPublication] = [...room!.localParticipant.dataTracks.values()];
const messageString = JSON.stringify({ message_type: 'mute', to_participant_identity: speaker });
const messageBuffer = new TextEncoder().encode(messageString).buffer;
localDataTrackPublication.track.send(messageBuffer);
};
which is then handled at the speaker's end, and the track is muted:
const messageString = new TextDecoder().decode(message);
const JSONMessage = JSON.parse(messageString);
if (
JSONMessage.message_type === 'mute' &&
JSONMessage.to_participant_identity === room!.localParticipant.identity
) {
if (isAudioEnabled) {
toggleAudio(); // this disables the audio-track
// other stuff
}
}
What should I do for phone participants ?
Thanks in advance.
Related
I tried to set GaussianBlurBackgroundProcessor (I use this post as a starting point but instead of node.js i use *.min.js in a php page). On local video it works but when I connect my video in a room, remote partecipants see my video "clean".
Someone had have my same problem?
I use min version of:
twilio-video.js 2.22.1
twilio-video-processors.js 1.0.2
This is the code:
[...]
const TWVideo = Twilio.Video;
const bg = new Twilio.VideoProcessors.GaussianBlurBackgroundProcessor({
assetsPath: '',
maskBlurRadius: 5,
blurFilterRadius: 25,
});
bg.loadModel();
const localVideo = TWVideo.createLocalVideoTrack().then(track => {
let video = document.getElementById('local-media').firstElementChild;
setProcessor(bg, track);
video.appendChild(track.attach());
$('#local-media').find('video').css('width', '200px');
});
TWVideo.connect(room_token, {
name: roomName
}).then(room => {
window.room = activeRoom = room;
log('Connected to Room '+ roomName);
room.participants.forEach(participantConnected);
room.on('participantConnected', participantConnected);
room.on('participantDisconnected', participantDisconnected);
room.once('disconnected', error => room.participants.forEach(participantDisconnected));
room.on('reconnecting', error => {
assert.equal(room.state, 'reconnecting');
if (error.code === 53001) {
console.log('Reconnecting your signaling connection!', error.message);
}
else if (error.code === 53405) {
console.log('Reconnecting your media connection!', error.message);
}
});
room.on('reconnected', () => {
console.log('Reconnected your signaling and media connections!');
assert.equal(room.state, 'connected');
});
room.on('participantReconnected', remoteParticipant => {
console.log("${remoteParticipant.identity} has reconnected the signaling connection to the Room!");
assert.equals(remoteParticipant.state, 'connected');
})
});
[...]
Thanks!
The issue is that you are creating a local video track and applying the blur to it, but you’re not using that track when you connect to the room. I would create the local video track and audio track first, then apply them to the room when you connect like this:
Video.connect(roomToken, {
name: roomName,
tracks: [localVideo, localAudio]
}).then(…);
I am doing a chatbot with dialogflow-es and I connected it with twilio for whatsapp, but the problem now is that I want that dialogflow receive the audio´s user, but I don´t know how....
In that example, there is a server that receives the incoming webhook request from Twilio. It looks like this:
app.post('/', async function(req, res) {
const body = req.body;
const text = body.Body;
const id = body.From;
const dialogflowResponse = (await sessionClient.detectIntent(
text, id, body)).fulfillmentText;
const twiml = new MessagingResponse();
const message = twiml.message(dialogflowResponse);
res.send(twiml.toString());
});
You can see that the application gets the text from the message from req.body.Body and the user's WhatsApp phone number from req.body.From. When Twilio sends webhooks to this endpoint for messages it sends all the information in the body of the request. You can see all the parameters that Twilio will send in the documentation here.
In this case you are looking for media that comes with the message, so you would first look for the NumMedia parameter. If it is greater than 0 then some media has been sent with the message.
Once you know how many media items have been sent you can then access them in parameters MediaUrl{N} where N is an integer. The media is delivered as a URL, so if you need to access the file you will need to download it.
So, you might deal with this like:
app.post('/', async function(req, res) {
const body = req.body;
const text = body.Body;
const id = body.From;
const numMedia = parseInt(body.NumMedia, 10);
if (numMedia > 0) {
for (let i = 0; i < numMedia; i++) {
console.log(body[`MediaUrl${i}`]);
}
}
const dialogflowResponse = (await sessionClient.detectIntent(
text, id, body)).fulfillmentText;
const twiml = new MessagingResponse();
const message = twiml.message(dialogflowResponse);
res.send(twiml.toString());
});
I'm not sure how you would deal with the audio with respect to Dialogflow, but that's how you access it.
We currently use Twilio Conference, however we would like to allow certain participants based on their caller-IDs, rest to be rejected. Currently anyone who dials the number for conference can participate and this is not something we want due to privacy, security and compliance.
How can we do it with TwiML?
Best,
Savas
You cannot do this with TwiML Bins (static TwiML) but you can use a Twilio Function for this or even Twilio Studio with the split based on Widget, to determine if the trigger.message.From number is allowed.
Below is an example Twilio Function you can modify for your purpose. Currently, it only allows certain CallerID in as moderators and provides these numbers the ability to start a conference, others can only be participants in an active conference. You can see the steps to set-up a Twilio Function here (just apply similar steps to setting up the code below). You can extend the function and return a Twilio TwiML Reject verb in the participant path for non-moderators.
exports.handler = function(context, event, callback) {
let twiml = new Twilio.twiml.VoiceResponse();
let callerId = event.From || null;
let conferenceParams = {};
let conferenceName = "My Conference Room";
let moderators = ["+1813279xxxx", "+1813393xxxx", "+1813918xxxx"];
conferenceParams.beep = true;
if (moderators.indexOf(callerId) === -1) {
conferenceParams.startConferenceOnEnter = false;
conferenceParams.endConferenceOnExit = false;
}
else
{
conferenceParams.startConferenceOnEnter = true;
conferenceParams.endConferenceOnExit = true;
}
twiml.dial().conference(conferenceParams, conferenceName);
callback(null, twiml);
};
Here is the approach you are looking for, both Functions are viable.
exports.handler = function(context, event, callback) {
let twiml = new Twilio.twiml.VoiceResponse();
let callerId = event.From || null;
let conferenceParams = {};
let conferenceName = "My Conference Room";
let moderators = ["+1678785xxxx", "+1813393xxxx", "+1813918xxxx"];
conferenceParams.beep = true;
if (moderators.indexOf(callerId) === -1) {
twiml.reject({reason: 'busy'});
callback(null, twiml);
}
else
{
conferenceParams.startConferenceOnEnter = true;
conferenceParams.endConferenceOnExit = true;
twiml.dial().conference(conferenceParams, conferenceName);
callback(null, twiml);
}
};
We have built a web application. The application's core is to arrange the meetings/sessions on the web. So User A(Meeting co-ordinator) will arrange a meeting/session and all other participants B, C, D and etc will be joining in the meeting/session. So I have used Twilio group video call to achieve it.
I have the below use case.
We want to do the voice pitch shifting of the User A's(Meeting co-ordinator) voice. So all other participants will be receiving the pitch-shifted voice in group video. We have analyzed the AWS Polly in Twilio but it doesn’t match with our use case.
So please advice is there any services in Twilio to achieve this scenario.
(or)
will it be possible to interrupt Twilio group call and pass the pitch-shifted voice to other participants?
Sample Code Used
initAudio();
function initAudio() {
analyser1 = audioContext.createAnalyser();
analyser1.fftSize = 1024;
analyser2 = audioContext.createAnalyser();
analyser2.fftSize = 1024;
if (!navigator.getUserMedia)
navigator.getUserMedia = navigator.webkitGetUserMedia || navigator.mozGetUserMedia;
if (!navigator.getUserMedia)
return(alert("Error: getUserMedia not supported!"));
navigator.getUserMedia({ audio: true }, function(stream){
gotStream(stream);
}, function(){ console.log('Error getting Microphone stream'); });
if ((typeof MediaStreamTrack === 'undefined')||(!MediaStreamTrack.getSources)){
console.log("This browser does not support MediaStreamTrack, so doesn't support selecting sources.\n\nTry Chrome Canary.");
} else {
MediaStreamTrack.getSources(gotSources);
}
}
function gotStream (stream) {
audioInput = audioContext.createMediaStreamSource(stream);
outputMix = audioContext.createGain();
dryGain = audioContext.createGain();
wetGain = audioContext.createGain();
effectInput = audioContext.createGain();
audioInput.connect(dryGain);
audioInput.connect(effectInput);
dryGain.connect(outputMix);
wetGain.connect(outputMix);
audioOutput = audioContext.createMediaStreamDestination();
outputMix.connect(audioOutput);
outputMix.connect(analyser2);
crossfade(1.0);
changeEffect();
}
function crossfade (value) {
var gain1 = Math.cos(value * 0.5 * Math.PI);
var gain2 = Math.cos((1.0 - value) * 0.5 * Math.PI);
dryGain.gain.value = gain1;
wetGain.gain.value = gain2;
}
function createPitchShifter () {
effect = new Jungle( audioContext );
effect.output.connect( wetGain );
effect.setPitchOffset(1);
return effect.input;
}
function changeEffect () {
if (currentEffectNode)
currentEffectNode.disconnect();
if (effectInput)
effectInput.disconnect();
var effect = 'pitch';
switch (effect) {
case 'pitch':
currentEffectNode = createPitchShifter();
break;
}
audioInput.connect(currentEffectNode);
}
Facing the error while adding the Localaudiotrack to a room
var mediaStream = new Twilio.Video.LocalAudioTrack(audioOutput.stream);
room.localParticipant.publishTrack(mediaStream, {
name: 'adminaudio'
});
ERROR:
Uncaught (in promise) TypeError: Failed to execute 'addTrack' on 'MediaStream': parameter 1 is not of type 'MediaStreamTrack'.
Twilio developer evangelist here.
There is nothing within Twilio itself that pitch shifts voices.
If you are building this in a browser, then you could use the Web Audio API to take the input from the user's microphone and pitch shift it, then provide the resultant audio stream to the Video API instead of the original mic stream.
the comments in the above answer are SO helpful! I've been researching this for a couple of weeks, posted to Twilio-video.js to no avail and finally just the right phrasing pulled this up on S.O!
but to summarize and to add what I've found to work since it's hard to follow all the 27 questions/comments/code excerpts:
when connecting to Twilio:
const room = await Video.connect(twilioToken, {
name: roomName,
tracks: localTracks,
audio: false, // if you don't want to hear the normal voice at all, you can hide this and add the shifted track upon participant connections
video: true,
logLevel: "debug",
}).then((room) => {
return room;
});
upon a new (remote) participant connection:
const stream = new MediaStream([audioTrack.mediaStreamTrack]);
const audioContext = new AudioContext();
const audioInput = audioContext.createMediaStreamSource(stream);
source.disconnect(audioOutput);
console.log("using PitchShift.js");
var pitchShift = PitchShift(audioContext);
if (isFinite(pitchVal)) {
pitchShift.transpose = pitchVal;
console.log("gain is " + pitchVal);
}
pitchShift.wet.value = 1;
pitchShift.dry.value = 0.5;
try {
audioOutput.stream.getAudioTracks()[0]?.applyConstraints({
echoCancellation: true,
noiseSuppression: true,
});
} catch (e) {
console.log("tried to constrain audio track " + e);
}
var biquadFilter = audioContext.createBiquadFilter();
// Create a compressor node
var compressor = audioContext.createDynamicsCompressor();
compressor.threshold.setValueAtTime(-50, audioContext.currentTime);
compressor.knee.setValueAtTime(40, audioContext.currentTime);
compressor.ratio.setValueAtTime(12, audioContext.currentTime);
compressor.attack.setValueAtTime(0, audioContext.currentTime);
compressor.release.setValueAtTime(0.25, audioContext.currentTime);
//biquadFilter.type = "lowpass";
if (isFinite(freqVal)) {
biquadFilter.frequency.value = freqVal;
console.log("gain is " + freqVal);
}
if (isFinite(gainVal)) {
biquadFilter.gain.value = gainVal;
console.log("gain is " + gainVal);
}
source.connect(compressor);
compressor.connect(biquadFilter);
biquadFilter.connect(pitchShift);
pitchShift.connect(audioOutput);
const localAudioWarpedTracks = new Video.LocalAudioTrack(audioOutput.stream.getAudioTracks()[0]);
const audioElement2 = document.createElement("audio");
document.getElementById("audio_div").appendChild(audioElement2);
localAudioWarpedTracks.attach();
I'm trying to use a Twilio function to transfer a call to a phone number with an extension.
The Twilio function is called from a Twilio flow.
Right now, the call transfer to the phone number. However, the extension is never invoked.
I added some "w" characters for pause in the "sendDigits"...but it did not change anything.
Here is my Twilio flow
Here is my Twilio function widget with the parameter
Here is the code of the twilio function
exports.handler = function(context, event, callback) {
// set-up the variables that this Function will use to forward a phone call using TwiML
// REQUIRED - you must set this
let phoneNumber = event.PhoneNumber || "NUMBER TO FORWARD TO";
// OPTIONAL
let callerId = event.CallerId || null;
// OPTIONAL
let timeout = event.Timeout || null;
// OPTIONAL
let allowedCallers = event.allowedCallers || [];
// generate the TwiML to tell Twilio how to forward this call
let twiml = new Twilio.twiml.VoiceResponse();
let allowedThrough = true;
if (allowedCallers.length > 0) {
if (allowedCallers.indexOf(event.From) === -1) {
allowedThrough = false;
}
}
let sendDigits = event.sendDigits;
let dialParams = {};
dialParams.sendDigits = sendDigits
if (callerId) {
dialParams.callerId = callerId;
}
if (timeout) {
dialParams.timeout = timeout;
}
if (allowedThrough) {
twiml.dial(dialParams, phoneNumber);
}
else {
twiml.say('Sorry, you are calling from a restricted number. Good bye.');
}
// return the TwiML
callback(null, twiml);
};
Any idea ?
The Dial verb doesn't have a sendDigits attribute, but the REST API does.
You can use the Number noun with the Dial verb, https://www.twilio.com/docs/voice/twiml/number, and its associated URL parameter to reference a TwiML Bin with a Play, https://www.twilio.com/docs/voice/twiml/play, and its digits attribute, to play the DTMF after the dialed party answers the phone, but before the two parties communicate with one another.