I have a task to wrap a 3rd party video decoder library in a direct show transform filter.
I believe this is going to be a problem as the 3rd party library uses an asynchronous callback when a full frame is ready i.e.
// on main thread
lib->RegisterCallback(callback function)
lib->write(raw data bytes)
void callback(frame)
{
// here is your frame (on a worker thread)
}
When I look at the pure virtual CTransformFilter.Transform function it expects the transform to be synchronous. Now I could set this up so it blocks on a event that is Set in the callback but what happens if the data coming in to the Transform function is not sufficient to generate a full new frame? I would deadlock the function.
Is my only option to go back to the library developer and ask for a synchronous decoder?
I've just been looking at the CTransformFilter Receive function. This is what calls the (overridden) Transform function. It then calls m_pOutput->m_pInputPin->Receive(pOutSample) to pass the sample onwards. Can I call this Receive function from the worker thread in the callback or do I have to keep everything on the same thread?
Thanks
You can still achieve what you want. Note that a filter is not required to output a media sample on the same thread and/or within the call it receives an input media sample. CTransformFilter is however made with this assumption in mind.
So basically your straightforward choices are:
step back from CTransformFilter and use its ancestor to inherit from to implement output media sample delivery from [a worker thread's] callback call
wait within Transform function for asynchronous completion (makes sense if the inner library still decodes 1 output frame for 1 input frame) and catch up then delivering the output; you will also have to wait for pause requests there and abort your wait in order to not block the execution
Related
I am working on exposing an audio library (C library) for Dart. To trigger the audio engine, it requires a few initializations steps (non blocking for UI), then audio processing is triggered with a perform function, which is blocking (audio processing is a heavy task). That is why I came to read about Dart isolates.
My first thought was that I only needed to call the performance method in the isolate, but it doesn't seem possible, since the perform function takes the engine state as first argument - this engine state is an opaque pointer ( Pointer in dart:ffi ). When trying to pass engine state to a new isolate with compute function, Dart VM returns an error - it cannot pass C pointers to an isolate.
I could not find a way to pass this data to the isolate, I assume this is due to the separate memory of main isolate and the one I'm creating.
So, I should probably manage the entire engine state in the isolate which means :
Create the engine state
Initialize it with some options (strings)
trigger the perform function
control audio at runtime
I couldn't find any example on how to perform this actions in the isolate, but triggered from main thread/isolate. Neither on how to manage isolate memory (keep the engine state, and use it). Of course I could do
Here is a non-isolated example of what I want to do :
Pointer<Void> engineState = createEngineState();
initEngine(engineState, parametersString);
startEngine(engineState);
perform(engineState);
And at runtime, triggered by UI actions (like slider value changed, or button clicked) :
setEngineControl(engineState, valueToSet);
double controleValue = getEngineControl(engineState);
The engine state could be encapsulated in a class, I don't think it really matters here.
Whether it is a class or an opaque datatype, I can't find how to manage and keep this state, and perform triggers from main thread (processed in isolate). Any idea ?
In advance, thanks.
PS: I notice, while writing, that my question/explaination may not be precise, I have to say I'm a bit lost here, since I never used Dart Isolates. Please tell me if some information is missing.
EDIT April 24th :
It seems to be working with creating and managing object state inside the Isolate. But the main problem isn't solved. Because the perform method is actually blocking while it is not completed, there is no way to still receive messages in the isolate.
An option I thought first was to use the performBlock method, which only performs a block of audio samples. Like this :
while(performBlock(engineState)) {
// listen messages, and do something
}
But this doesn't seem to work, process is still blocked until audio performance finishes. Even if this loop is called in an async method in the isolate, it blocks, and no message are read.
I now think about the possibility to pass the Pointer<Void> managed in main isolate to another, that would then be the worker (for perform method only), and then be able to trigger some control methods from main isolate.
The isolate Dart package provides a registry sub library to manage some shared memory. But it is still impossible to pass void pointer between isolates.
[ERROR:flutter/lib/ui/ui_dart_state.cc(157)] Unhandled Exception: Invalid argument(s): Native objects (from dart:ffi) such as Pointers and Structs cannot be passed between isolates.
Has anyone already met this kind of situation ?
It is possible to get an address which this Pointer points to as a number and construct a new Pointer from this address (see Pointer.address and Pointer.fromAddress()). Since numbers can freely be passed between isolates, this can be used to pass native pointers between them.
In your case that could be done, for example, like this (I used Flutter's compute to make the example a bit simpler but that would apparently work with explicitly using Send/ReceivePorts as well)
// Callback to be used in a backround isolate.
// Returns address of the new engine.
int initEngine(String parameters) {
Pointer<Void> engineState = createEngineState();
initEngine(engineState, parameters);
startEngine(engineState);
return engineState.address;
}
// Callback to be used in a backround isolate.
// Does whichever processing is needed using the given engine.
void processWithEngine(int engineStateAddress) {
final engineState = Pointer<Void>.fromAddress(engineStateAddress);
process(engineState);
}
void main() {
// Initialize the engine in a background isolate.
final address = compute(initEngine, "parameters");
final engineState = Pointer<Void>.fromAddress(address);
// Do some heavy computation in a background isolate using the engine.
compute(processWithEngine, engineState.address);
}
I ended up doing the processing of callbacks inside the audio loop itself.
while(performAudio())
{
tasks.forEach((String key, List<int> value) {
double val = getCallback(key);
value.forEach((int element) {
callbackPort.send([element, val]);
});
});
}
Where the 'val' is the thing you want to send to callback. The list of int 'value' is a list of callback index.
Let's say you audio loop performs with vector size of 512 samples, you will be able to pass your callbacks after every 512 audio samples are processed, which means 48000 / 512 times per second (assuming you sample rate is 48000). This method is not the best one but it works, I still have to see if it works in very intensive processing context though. Here, it has been thought for realtime audio, but it could work the same for audio rendering.
You can see the full code here : https://framagit.org/johannphilippe/csounddart/-/blob/master/lib/csoundnative.dart
For the purposes of this question, assume that I need to run some function on some object and that function will take a long time to execute (minutes). Also assume that I have no control over this function (*). How do I now cancel this function's execution?
I want to run it in a background thread to keep the main thread free and I could do that with GCD, NSOperation or NSThread. However, as far as I know, none of these support forced stopping. They can all be cancelled, but this cancellation must be implemented in the function itself - but I don't have access to that function, so I can't do that. The closest I got was using NSThread and exit(), but unfortunately it can't be applied to a instance variable (see the code example). My current plan is to try to send a notification and observe that within the object/function and kill the thread from within using Thread.exit(). I'm justing wondering if there is a "cleaner" or easier way, either built-in or 3rd party.
let someObject = Object()
// Using GCD
dispatchQueue.async { someObject.expensiveFunction() }
// Using NSOperation
operationQueue.addOperation { someObject.expensiveFunction() }
// Using NSThread
let thread = Thread { someObject.expensiveFunction() }
thread.exit() // exit is not available on an instance
(*) In this case I do have control over the function and could implement an actual cancellation, but due to the libraries I'm using, this would require a lot of refactoring.
I am working with Rime, more specifically with the runicast example. Once a message is received i store it in a linked list, then i post an event to a process which is in charge of extracting messages from the linked list and processing them. My code is something like this:
static void recv_runicast(struct runicast_conn *c,
const linkaddr_t *from, uint8_t seqno)
{
/*code to insert the message into the linked list*/
...
/*Post an event to the process which extracts messages from the linked list */
process_post(&extract_msg_from_linked_list, PROCESS_EVENT_CONTINUE, NULL);
}
My question is: Is it safe to use process_post within the callback function recv_runicast? or should i use process_poll?
Thanks in advance
Yes, it's safe. The network stack operations are done in process context, and Contiki processes are not preemptive. So pretty much any process-related operations are "safe".
The main differences between process_post and process_poll is that the former will put a new event in the process event buffer, while the latter will simply set a flag. So the second options is slightly more efficient. Also, in theory the event buffer can get full and events start to get lost, but that's very unlikely to be a problem.
I would use none of these functions at all, but do the processing directly in the callback to simplify the execution flow.
I want to know As we all know how asynchronous task are necessary for concurrency but Wanted to know why we need the synchronous tasks. while we can achieve the same with the normal usage of function.
Thanks & regards
Rohit
When you calls something synchronously, it means that 'the thread that initiated that operation will wait for the task to finish before
continuing'. Asynchronous means that it will not wait for finish the task.
synchronous calls stops your current action and returns when the call returned. with asynchronous calls you can continue.
synchronous is the opposite of asynchronous code, and therefore is ordinary code.
At the end, if asynchronous is totally out of scope then you will not emphasize the word synchronous.
It helps to synchronise threads, as the name suggests.
consider a typical usage of GCD async and sync (pseudo)
async background_thread {
//1 call webservice or other long task that would block the main thread
sync main_thread {
//2 update UI with results from 1
}
//3 do something else that relies on 2
}
now if 2 was in an async and you needed to do something at 3 that relies on the updates at 2 to have happened, then you are not guaranteed (and most likely wont) get the behaviour you are expecting. instead, you use a sync to make sure that the task is completed before continuing the execution in the background thread.
If you are asking now, why not just take out the sync/async around 2 so it executes in order anyway? the problem is, the UI must not be updated on a background thread otherwise the behaviour is undefined (which usually means the UI lags a lot). So in essence what happens is the background thread waits at 2's sync until the main thread gets round to executing that block, then it will continue with the rest of the execution on the background thread.
If you were dealing with a task that doesnt require the main thread (or some other thread) to execute properly, then yes you may as well take out the sync at 2.
This is just one example of how a sync is useful, there are others if you are doing advanced threading in your app.
Hope this helps
Typically it's because you want to do an operation on a specific different thread but you need the result of that operation. You cannot do the operation asynchronously because your code will proceed before the operation on the other thread completes.
Apple has a very nice example:
func asset() -> AVAsset? {
var theAsset : AVAsset!
self.assetQueue.sync {
theAsset = self.getAssetInternal().copy() as! AVAsset
}
return theAsset
}
Any thread might call the asset method; but to avoid problems with shared data, we require that only functions that are executed from a particular queue (self.assetQueue) may touch an AVAsset, so when we call getAssetInternal we do it on self.assetQueue. But we also need the result returned by our call to getAssetInternal; hence the call to sync rather than async.
The handler code (in this case, collecting accelerometer data) is being executed asynchronously when my accelerometer moves.
That means if I try to save data.x, data.y, data.z in a variable, even if the variable is declared outside of the handler, the variable will be nil if I attempt to print it anywhere.
How do I save this data to access in other parts of my code?
(Or does everything have to happen in my handler, best-practices wise?)
if motionManager.accelerometerAvailable{
let motionQueue = NSOperationQueue.mainQueue()
motionManager.startDeviceMotionUpdatesToQueue(motionQueue,
withHandler: gravityUpdated)
}
func gravityUpdated(motion: CMDeviceMotion!, error: NSError!) {
let grav : CMAcceleration = motion.gravity;
println(grav.x)
}
The main thing to be wary of is that these events can come in more quickly than the main thread can process them. As the documentation says:
Because the processed events might arrive at a high rate, using the main operation queue is not recommended.
Hence, you should your own background queue to handle these events.
Regarding how you then use this updated information on the main thread, there are two considerations:
To ensure your code is thread-safe, any variables that you use both from this background thread and other threads must be synchronized.
Make sure you don't just dispatch updates back to the main thread. I would generally create a dispatch source of DISPATCH_SOURCE_TYPE_DATA_OR or DISPATCH_SOURCE_TYPE_DATA_ADD, put a handler for that source on the main queue, and then your motionQueue can then then perform a dispatch_source_merge_data on this source.
GCD will then coalesce these data changes, notifying the main thread when there were updates, but not backlogging the main thread in the process.
By the way, you may also want to review Table 4-1 of the Event Handling Guide, which outlines common update intervals for acceleration events (measured in Hz), depending upon the intended usage:
10–20: Suitable for determining a device’s current orientation vector.
30–60: Suitable for games and other apps that use the accelerometer for real-time user input.
70–100: Suitable for apps that need to detect high-frequency motion. For example, you might use this interval to detect the user hitting the device or shaking it very quickly.
You might want to choose a deviceMotionUpdateInterval commensurate with your application's needs.