I use WebAudio API and basically my setup is fairly simple.
I use 1 AudioWorkletNode as an emitter and another one as a receiver
emitter:
process(inputs, outputs) {
inputs[ 0 ].length && this.port.postMessage( inputs[ 0 ] );
return ( true );
}
receiver:
inputs = [ new Float32Array(128), new Float32Array(128) ]
constructor() {
super();
// Create a message port to receive messages from the main thread
this.port.onmessage = (event) => {
this.inputs = event.data.inputs;
};
}
process( inputs, outputs) {
const output = outputs[0];
for (let channel = 0; channel < output.length; ++channel) {
output[ channel ].set( this.inputs[ channel ] );
}
return true;
}
on client side I have
//emitter
this.inputWorklet.port.onmessage = e => this.receiverWorklet.port.postMessage( { inputs: e.data } );
and for receiving the data I have connected the nodes together
this.receiverWorklet.connect( this.gainNode );
This works but my problem is that the sound is really glitchy
One thing I though of is there might be a delay between events also WebAudio is in a DOM context
Do you have any ideas How I could achieve a fluid stream restitution?
or maybe another technique?
The reason for the glitchy audio is that your code only works if everything always happens in the exact same order.
The input worklet's process() function needs to be called. It sends an event.
The event needs to pass through the main thread.
The event needs to arrive at the receiver worklet.
Only after that the receiver worklet's process() function needs to be called.
Since there is no buffer it always has to happen in the exact same order. If for some reason the main thread is busy and it can't process the events right away the receiver will continue playing the old audio.
I think you can almost keep the current implementation by buffering a few events in your receiver worklet before you start playing. It will of course also add some latency.
Another approach would be to use a SharedArrayBuffer instead of sending events. Your input worklet would write to the SharedArrayBuffer and your receiver worklet would read from it.
Related
I tried to use hardware timer to read data from an external device periodically.
More specifically, I realized a custom driver using gpio to simulate SPI protocol, whenever an hardtimer interrupt happens, the driver is called to read gpio status. The timer is set to 2k.
When an interrupt happens, the isr shall put sample data into a buffer. When the buffer is full, the application will pause the timer and send these data out through mqtt protocol. Using signal generator and oscilloscope, I found the data was good. The whole process worked as expected.
The problem is that the sample process is not continual. When data is sending out through wifi, the timer is paused, and no data can be read into buffer.
To solve this problem, I create a special task responsible for transmitting data out. And then I use ping-pong buffers to store sample data. When one buffer is full, the sending task is notified to send these data out, meanwhile the timer isr is continually to put data into another buffer.
At first I wanted to send notify just from the isr (using xQueueSendFromISR()), which was proved not reliable. I found only a few notifies were able to be sent to the sending task. So I am obliged to using a flag. When one buffer is full, the flag is set to true, While a special task is looping this flag, whenever it find the flag is true, it will notify the sending task.
timer_isr()
{
read_data_using_gpio;
if(one buffer is full)
{
set the flag to true
}
}
task_1()
{
while(1)
{
if(the flag is true)
{
set the flag to false;
xQueueSend;
}
vTaskDelay(50ms)//it will cost 200ms to fill up the buffer
}
}
task_2()
{
while(1)
{
xStatus = xQueueReceive;
if(xStatus==pdPASS) // A message from other tasks is received.
{
transmitting data out using mqtt protocol.
}
}
}
Then I got the terrible data as below.
terroble data
I used oscilloscope to check the gpio operation in the isr.
oscilloscope1
oscilloscope2
So it seems like some isr not triggered? but what happened?
More weird thing: I added another task to get data from an audio chip through i2s. Again I used ping-pong buffers and send notify to the same sending task.
timer_isr()
{
read_data_using_gpio;
if(one buffer is full)
{
set the flag to true
}
}
task_1()
{
while(1)
{
if(the flag is true)
{
set the flag to false;
xQueueSend;
}
vTaskDelay(50ms)
}
}
task_3()
{
while(1)
{
i2s_read_to_buffer;
xQueueSend;
}
}
task_2()
{
while(1)
{
xStatus = xQueueReceive;
if(xStatus==pdPASS) // A message from other tasks is received.
{
if(data from task_1)
{
do something;
transmitting data out using mqtt protocol
}
if(data from task_2)
{
do something;
transmitting data out using mqtt protocol
}
}
}
}
And this time the data from former task turned ok!
data_ok
And what's more, after I commened task2-related code in the sending task, Again the data become bad!
So what happened? Can somebody give any hint?
task_2()
{
while(1)
{
xStatus = xQueueReceive;
if(xStatus==pdPASS) // A message from other tasks is received.
{
if(data from task_1)
{
do something;
transmitting data out using mqtt protocol
}
if(data from task_2)
{
// do something;
// transmitting data out using mqtt protocol
}
}
}
}
I have solved this problem.
If you enable power management(idf.py menuconfig-->component config-->power management), the APB(advanced peripheral bus) will low its frequency automatically, which is the clock source of hardware timer.Thus you will see the timer interrupt is not stable.
Just disable the power management.
I'm trying to develop an application which communicates with Bluetooth Low Energy Devices. I established a working "website" with the Web Bluetooth API. Everything works fine, so I used the Electron framework, to build an application.
The issue is known - if you start navigator.bluetooth.requestDevice(), you get this error message:
User cancelled the requestDevice() chooser..
This causes due to the missing device chooser in Chromium. There are several topics about workarounds I found, but no examples. This is my first Electron project. Maybe somebody solved this problem and can give me a hint :-)
thank you very much for your support. According to your suggestions and some research, I developed a working solution and like to share it with you.
These two links helped me a lot:
https://github.com/electron/electron/issues/11865
https://github.com/electron/electron/issues/10764
Especially this post from MarshallOfSound – well described:
Hook event in main process
Send message to renderer process with device list
Show UI in renderer process
Send selected device to main process
Call callback
To get more Information about the main process and renderer process, events and their API, read this:
https://www.electronjs.org/docs/tutorial/application-architecture#main-and-renderer-processes
https://www.electronjs.org/docs/api/ipc-main
https://www.electronjs.org/docs/api/web-contents#contentssendchannel-args
https://www.electronjs.org/docs/api/ipc-renderer
https://electronjs.org/docs/api/web-contents#event-select-bluetooth-device (already posted by Gerrit)
https://www.electronjs.org/docs/api/structures/bluetooth-device
For my application I wanted a device picker, as seen in Chrome. The sequence I wanted to implement is:
Start application
Search for a device
Devicepicker pops up
Select a device
Devicepicker closes
See data in application
A reference to the code for the processes of the tutorial and the code snippet:
electron application: main.js (main process) renderer.js (render process) devicepicker GUI: devicepicker.js (renderer process) devicepicker.html & layout.css (GUI)
1) Create devicepicker with a GUI (I used and two ) and a script
2) In your main.js create a select-bluetooth-device event inside the 'ready' event of your application object (docs in links above) When you start navigator.bluetooth.requestDevice() in your renderer.js, the event get’s fired and the devicelist is in the main process. With console.log(deviceList) it's visible in the shell. To handle it you need to send it to the renderer process (your application window).
3) To achieve this, we implement webContents.send of our BrowserWindow object inside the webContents.on event. Now the main process sends a devicelist every time he found new devices through the channel channelForBluetoothDeviceList
4) Create in renderer.js startDevicePicker(). devicePicker() must be started in the same function as navigator.bluetooth.requestDevice(). startDevicePicker() instantiates a new BrowserWindow() object which loads devicepicker.html
5) To get the list from the main process a ipcRenderer.on() listener must be implemented in startDevicePicker() that listens to the channelForBluetoothDeviceList channel of our main process. Now we can get the list in our electron application (renderer prcoess). To send it to the devicepicker UI, we need to forward it from our electron application (renderer process) to the devicepicker (also a renderer process)
6) To achieve this, we need the ipcRenderer.sendTo() sender in devicePicker(), which forwards messages between from a renderer process to a specific other renderer process. Additional to the channel bluetoothDeviceDiscoverList we need the BrowserWindow.id of the devicepicker. Since we just instantiated it, we can use our devicepicker object. I had a device which sended only once, the main process was faster than the build of the devicepicker and my list was never sent to the devicepicker. So I used a Promise() to wait with ipcRenderer.sendTo() until the devicepicker was ready to use.
7) To receive the devicelist at our devicepicker GUI, we need to listen to the bluetoothDeviceDiscoverList with ipcRenderer.on() (devicepicker.js). I inserted the devicelist now to the <option> of the devicepicker, you can use of course other elements (devicepicker.html). Please note: implement a query that compares the sended list to the current. Otherwise you get multiple devices and your selection gets loooong. I still need to do that, it's not finished yet :-)
8) To select a device that navigator.bluetooth.requestDevice() (renderer.js) gets resolved, we need to send back the BluetoothDevice.deviceId of our selected device to the main process where we callback ‚callback‘ with deviceId as callback parameter (main.js).
9) Now we can use ipcRenderer.sendTo() send the selected BluetoothDevice.deviceId to the mainprocess (devicepicker.js).
10) In the main process (main.js) of our electron application we listen to the channel channelForSelectingDevice with ipcMain.on() and callback with the received BluetoothDevice.deviceId. The device discovery gets stopped, navigator.bluetooth.requestDevice() gets resolved and we receive data from our device in our application (renderer.js). To cancel the discovery of devices, listen with ipcMain.on() in another channel channelForTerminationSignal just a signal to the main process (main.js), for example after a click (devicepicker.js) and call the callback with an empty string (as written in the docs)
I admit it could be done much simpler without a devicepicker. Then just send the devicelist from the main process (main.js) to your application (renderer process). But this helped me a lot to understand the processes in electron. I hope this tutorial is useful for you :-) !
Snippet:
main.js
const { ipcMain, app, BrowserWindow } = require('electron')
let win = null;
var callbackForBluetoothEvent = null;
// Create the browser window.
function createWindow () {
win = new BrowserWindow({
webPreferences: {
nodeIntegration: true //to activate require()
}
})
win.maximize()
win.show()
//This sender sends the devicelist from the main process to all renderer processes
win.webContents.on('select-bluetooth-device', (event, deviceList, callback) => {
event.preventDefault(); //important, otherwise first available device will be selected
console.log(deviceList); //if you want to see the devices in the shell
let bluetoothDeviceList = deviceList;
callbackForBluetoothEvent = callback; //to make it accessible outside createWindow()
win.webContents.send('channelForBluetoothDeviceList', bluetoothDeviceList);
});
// This method will be called when Electron has finished
// initialization and is ready to create browser windows.
// Some APIs can only be used after this event occurs.
app.on('ready', createWindow)
//cancels Discovery
ipcMain.on('channelForTerminationSignal', _ => {
callbackForBluetoothEvent(''); //reference to callback of win.webContents.on('select-bluetooth-device'...)
console.log("Discovery cancelled");
});
//resolves navigator.bluetooth.requestDevice() and stops device discovery
ipcMain.on('channelForSelectingDevice', (event, DeviceId) => {
callbackForBluetoothEvent(sentDeviceId); //reference to callback of win.webContents.on('select-bluetooth-device'...)
console.log("Device selected, discovery finished");
})
renderer.js
function discoverDevice() {
navigator.bluetooth.requestDevice()
startDevicepicker()
}
function startDevicepicker(){
let devicepicker = null;
let mainProcessDeviceList = null;
devicepicker = new BrowserWindow({
width: 350,
height: 270,
show: false, //needed to resolve promise devicepickerStarted()
webPreferences: {
nodeIntegration: true
}
})
devicepicker.loadFile('devicePicker.html');
//electron application listens for the devicelist from main process
ipcRenderer.on('channelForBluetoothDeviceList', (event, list) => {
mainProcessDeviceList = list;
devicepickerStarted.then(_=> {
console.log("Promise resolved!");
ipcRenderer.sendTo(devicepicker.webContents.id, 'bluetoothDeviceDiscoverList', mainProcessDeviceList);
})
})
//Promise that ensures that devicepicker GUI gets the list if the device only sends once
var devicepickerStarted = new Promise(
function (resolve, reject) {
console.log("Promise started");
devicepicker.once('ready-to-show', () => {
devicepicker.show();
resolve();
console.log("Devicepicker is ready!")
})
}
)
//remove listeners after closing devicepicker
devicepicker.on('closed', _ => {
devicepicker = null;
ipcRenderer.removeAllListeners('channelForBluetoothDeviceList');
ipcRenderer.removeAllListeners('currentWindowId');
ipcRenderer.removeAllListeners('receivedDeviceList');
})
}
devicepicker.js
//save received list here
var myDeviceList = new Array();
//Html elements
const devicePickerSelection = document.getElementById("devicePickerSelection");
const buttonSelect = document.getElementById("Select");
const buttonCancel = document.getElementById("Cancel");
//eventListeners for buttons
buttonSelect.addEventListener('click', selectFromDevicePicker);
buttonCancel.addEventListener('click', cancelDevicePicker);
//listens for deviceList
ipcRenderer.on('receivedDeviceList', (event, bluetoothDeviceDiscoverList) => {
console.log("list arrived!")
//code: add list to html element
});
function selectFromDevicePicker() {
let selectedDevice = devicePickerSelection.value;
let deviceId = //depends on where you save the BluetoothDevice.deviceId values
//sends deviceId to main process for callback to resolve navigator.bluetooth.requestDevice()
ipcRenderer.send('channelForSelectingDevice', deviceId);
ipcRenderer.removeAllListeners('receivedDeviceList');
closeDevicePicker();
}
function cancelDevicePicker() {
ipcRenderer.send('channelForTerminationSignal');
closeDevicePicker();
}
function closeDevicePicker() {
myDevicePicker.close();
}}
In your main.js Add this code snippet
if (process.platform === "linux"){
app.commandLine.appendSwitch("enable-experimental-web-platform-features", true);
} else {
app.commandLine.appendSwitch("enable-web-bluetooth", true);
}
This will enable the bluetooth at your Electron app.
And use this as reference
https://github.com/electron/electron/issues/11865
https://github.com/electron/electron/issues/7367
https://github.com/aalhaimi/electron-web-bluetooth
But I'd suggest you to consider your Electron version.
Here is a code sample that will just return the first device instead of having to implement a device chooser:
mainWindow.webContents.on('select-bluetooth-device', (event, deviceList, callback) => {
event.preventDefault();
console.log('Device list:', deviceList);
let result = deviceList[0];
if (!result) {
callback('');
} else {
callback(result.deviceId);
}
});
Source: https://electronjs.org/docs/api/web-contents#event-select-bluetooth-device
I built PJSIP library with PJSUA_HAS_VIDEO as 1. I want to make an option to make audio only calls. I tried
pjsua_call_setting opt;
pjsua_call_setting_default(&opt);
opt.flag = PJSUA_CALL_INCLUDE_DISABLED_MEDIA;
opt.vid_cnt = 0;
opt.aud_cnt = 1;
pj_status_t status = pjsua_call_make_call((pjsua_acc_id)[self identifier], &uri, &opt, NULL, NULL, &callIdentifier);
At the receiving end, in on_incoming_call() function, I tried
if (callInfo.rem_offerer && callInfo.rem_vid_cnt == 1)
{
call.hasVideo = YES;
} else {
call.hasVideo = NO;
}
But rem_vid_cnt is always giving 1.
How can I set the call type while making call and receive it correctly at receiving end? I want to set the setHasVideo field of CallKit also at receiving end.
Thanks in advance.
At the app end your code is correct.
You need to disable video from server side also.
This is a two way communication. You can do this from the caller side set the rem_vid_cnt = 0 when you initiate a call, and at the receiver side you will get this as "0".
Hope this will help you :)
/** Number of video streams offered by remote */
unsigned rem_vid_cnt;
I would like to use the YouTube API to make sure the video was watched till end, maybe an event or so?
You need to use onStateChange event, which will give you 0 when ended. This event fires whenever the player's state changes. The value that the API passess to your event listener function will specify an integer that corresponds to the new player state.
Here's a sample code snippet to get notified when the player's state changes:
player.addEventListener("onStateChange", function(state){
if(state === 0){
// the video is end, do something here.
}
});
Here's another way to get notified when state changes:
function onYouTubePlayerReady(playerId) {
ytplayer = document.getElementById("myytplayer");
ytplayer.addEventListener("onStateChange", "onytplayerStateChange");
}
function onytplayerStateChange(newState) {
alert("Player's new state: " + newState);
}
I'm writing a streaming Twitter client that simply throws the stream up onto a tv. I'm observing the stream with RxJava.
When the stream comes in a burst, I want to buffer it and slow it down so that each tweet is displayed for at least 6 seconds. Then during the quiet times, any buffer that's been built up will gradually empty itself out by pulling the head of the queue, one tweet every 6 seconds. If a new tweet comes in and faces an empty queue (but >6s after the last was displayed), I want it to be displayed immediately.
I imagine the stream looking like that described here:
Raw: --oooo--------------ooooo-----oo----------------ooo|
Buffered: --o--o--o--o--------o--o--o--o--o--o--o---------o--o--o|
And I understand that the question posed there has a solution. But I just can't wrap my head around its answer. Here is my solution:
myObservable
.concatMap(new Func1<Long, Observable<Long>>() {
#Override
public Observable<Long> call(Long l) {
return Observable.concat(
Observable.just(l),
Observable.<Long>empty().delay(6, TimeUnit.SECONDS)
);
}
})
.subscribe(...);
So, my question is: Is this too naïve of an approach? Where is the buffering/backpressure happening? Is there a better solution?
Looks like you want to delay a message if it came too soon relative to the previous message. You have to track the last target emission time and schedule a new emission after it:
public class SpanOutV2 {
public static void main(String[] args) {
Observable<Integer> source = Observable.just(0, 5, 13)
.concatMapEager(v -> Observable.just(v).delay(v, TimeUnit.SECONDS));
long minSpan = 6;
TimeUnit unit = TimeUnit.SECONDS;
Scheduler scheduler = Schedulers.computation();
long minSpanMillis = TimeUnit.MILLISECONDS.convert(minSpan, unit);
Observable.defer(() -> {
AtomicLong lastEmission = new AtomicLong();
return source
.concatMapEager(v -> {
long now = scheduler.now();
long emission = lastEmission.get();
if (emission + minSpanMillis > now) {
lastEmission.set(emission + minSpanMillis);
return Observable.just(v).delay(emission + minSpanMillis - now, TimeUnit.MILLISECONDS);
}
lastEmission.set(now);
return Observable.just(v);
});
})
.timeInterval()
.toBlocking()
.subscribe(System.out::println);
}
}
Here, the source is delayed by the number of seconds relative to the start of the problem. 0 should arrive immediately, 5 should arrive # T = 6 seconds and 13 should arrive # T = 13. concatMapEager makes sure the order and timing is kept. Since only standard operators are in use, backpressure and unsubscription composes naturally.