I have a Stream of objects, of which the first part is to be handled differently from the second part (a simple test will tell when to switch).
I've tried to create the stream and use it first with a takeWhile and then handle the rest of it:
Stream s = [...];
s.takeWhile(simplePredicate).forEach(this::handleFirstPart);
s.forEach(this::handleRest);
It doesn't work: it throws java.lang.IllegalStateException: stream has already been operated upon or closed.
Why can't I keep using the stream after takeWhile decides to stop its processing?
Is there a way to do it?
Related
I am working on exposing an audio library (C library) for Dart. To trigger the audio engine, it requires a few initializations steps (non blocking for UI), then audio processing is triggered with a perform function, which is blocking (audio processing is a heavy task). That is why I came to read about Dart isolates.
My first thought was that I only needed to call the performance method in the isolate, but it doesn't seem possible, since the perform function takes the engine state as first argument - this engine state is an opaque pointer ( Pointer in dart:ffi ). When trying to pass engine state to a new isolate with compute function, Dart VM returns an error - it cannot pass C pointers to an isolate.
I could not find a way to pass this data to the isolate, I assume this is due to the separate memory of main isolate and the one I'm creating.
So, I should probably manage the entire engine state in the isolate which means :
Create the engine state
Initialize it with some options (strings)
trigger the perform function
control audio at runtime
I couldn't find any example on how to perform this actions in the isolate, but triggered from main thread/isolate. Neither on how to manage isolate memory (keep the engine state, and use it). Of course I could do
Here is a non-isolated example of what I want to do :
Pointer<Void> engineState = createEngineState();
initEngine(engineState, parametersString);
startEngine(engineState);
perform(engineState);
And at runtime, triggered by UI actions (like slider value changed, or button clicked) :
setEngineControl(engineState, valueToSet);
double controleValue = getEngineControl(engineState);
The engine state could be encapsulated in a class, I don't think it really matters here.
Whether it is a class or an opaque datatype, I can't find how to manage and keep this state, and perform triggers from main thread (processed in isolate). Any idea ?
In advance, thanks.
PS: I notice, while writing, that my question/explaination may not be precise, I have to say I'm a bit lost here, since I never used Dart Isolates. Please tell me if some information is missing.
EDIT April 24th :
It seems to be working with creating and managing object state inside the Isolate. But the main problem isn't solved. Because the perform method is actually blocking while it is not completed, there is no way to still receive messages in the isolate.
An option I thought first was to use the performBlock method, which only performs a block of audio samples. Like this :
while(performBlock(engineState)) {
// listen messages, and do something
}
But this doesn't seem to work, process is still blocked until audio performance finishes. Even if this loop is called in an async method in the isolate, it blocks, and no message are read.
I now think about the possibility to pass the Pointer<Void> managed in main isolate to another, that would then be the worker (for perform method only), and then be able to trigger some control methods from main isolate.
The isolate Dart package provides a registry sub library to manage some shared memory. But it is still impossible to pass void pointer between isolates.
[ERROR:flutter/lib/ui/ui_dart_state.cc(157)] Unhandled Exception: Invalid argument(s): Native objects (from dart:ffi) such as Pointers and Structs cannot be passed between isolates.
Has anyone already met this kind of situation ?
It is possible to get an address which this Pointer points to as a number and construct a new Pointer from this address (see Pointer.address and Pointer.fromAddress()). Since numbers can freely be passed between isolates, this can be used to pass native pointers between them.
In your case that could be done, for example, like this (I used Flutter's compute to make the example a bit simpler but that would apparently work with explicitly using Send/ReceivePorts as well)
// Callback to be used in a backround isolate.
// Returns address of the new engine.
int initEngine(String parameters) {
Pointer<Void> engineState = createEngineState();
initEngine(engineState, parameters);
startEngine(engineState);
return engineState.address;
}
// Callback to be used in a backround isolate.
// Does whichever processing is needed using the given engine.
void processWithEngine(int engineStateAddress) {
final engineState = Pointer<Void>.fromAddress(engineStateAddress);
process(engineState);
}
void main() {
// Initialize the engine in a background isolate.
final address = compute(initEngine, "parameters");
final engineState = Pointer<Void>.fromAddress(address);
// Do some heavy computation in a background isolate using the engine.
compute(processWithEngine, engineState.address);
}
I ended up doing the processing of callbacks inside the audio loop itself.
while(performAudio())
{
tasks.forEach((String key, List<int> value) {
double val = getCallback(key);
value.forEach((int element) {
callbackPort.send([element, val]);
});
});
}
Where the 'val' is the thing you want to send to callback. The list of int 'value' is a list of callback index.
Let's say you audio loop performs with vector size of 512 samples, you will be able to pass your callbacks after every 512 audio samples are processed, which means 48000 / 512 times per second (assuming you sample rate is 48000). This method is not the best one but it works, I still have to see if it works in very intensive processing context though. Here, it has been thought for realtime audio, but it could work the same for audio rendering.
You can see the full code here : https://framagit.org/johannphilippe/csounddart/-/blob/master/lib/csoundnative.dart
I am using native NT API in my application to access files (NtCreateFile/etc). In order to avoid dealing with STATUS_PENDING I am using FILE_SYNCHRONOUS_IO_NONALERT flag when opening related file. So, opening file looks like this:
UNICODE_STRING fname = toNtUnicode(ntpath);
OBJECT_ATTRIBUTES oa;
InitializeObjectAttributes(&oa, &fname, 0, at.handle(), NULL);
HANDLE h;
IO_STATUS_BLOCK io_status;
NTSTATUS r = NtOpenFile(&h, GENERIC_READ|SYNCHRONIZE, &oa, &io_status,
FILE_SHARE_READ, FILE_SYNCHRONOUS_IO_NONALERT|FILE_DIRECTORY_FILE);
if (r != STATUS_SUCCESS)
...; // error handling
Unfortunately, it causes kernel to serialize all operations on given handle. I.e. if I try to execute multiple reads in parallel (using multiple threads) -- only one request will be processed at any point in time.
I could get rid of serialization:
HANDLE h;
IO_STATUS_BLOCK io_status;
NTSTATUS r = NtOpenFile(&h, GENERIC_READ|SYNCHRONIZE, &oa, &io_status,
FILE_SHARE_READ, FILE_DIRECTORY_FILE);
if (r == STATUS_PENDING)
...; // what to do here???
but how exactly should I wait for completion -- WaitForSingleObject() on file handle? As far as I know it can change to signaled state due to many reasons -- is there any way to tell that my open file (or dir) operation completed?
Similarly, if I submit multiple reads (from multiple threads) -- how can I tell which one (if any) has finished?
NtOpenFile is synchronous api. it never return STATUS_PENDING to you. even if driver return STATUS_PENDING for IRP_MJ_CREATE i/o sub-system will be wait for IRP complete
https://github.com/Zer0Mem0ry/ntoskrnl/blob/master/Io/iomgr/parse.c#L1404
so you never need check for STATUS_PENDING after NtOpenFile and never need wait (and in principle we can not wait here - we yet not have file handle -so can not wait on it or bind it to say IOCP. we not pass any event or another callback mechanism for NtOpenFile)
E.g. suppose I have a module that implements gen_server behavior, and it has
handle_call({foo, Foo}, _From, State) ->
{reply, result(Foo), State}
;
I can reach this handler by doing gen_server:call(Server, {foo, Foo}) from some other process (I guess if a gen_server tries to gen_server:call itself, it will deadlock). But gen_server:call blocks on response (or timeout). What if I don't want to block on the response?
Imaginary use-case: Suppose I have 5 of these gen_servers, and a response from any 2 of them is enough for me. What I want to do is something like this:
OnResponse -> fun(Response) ->
% blah
end,
lists:foreach(
fun(S) ->
gen_server:async_call(S, {foo, Foo}, OnResponse)
end,
Servers),
Result = wait_for_two_responses(Timeout),
lol_i_dunno()
I know that gen_server has cast, but cast has no way to provide any response, so I don't think that that's what I want in this case. Also, seems like it should not be the gen_server's concern whether caller wants to handle response synchronously (using gen_server:call) or async (does not seem to exist?).
Also, the server is allowed to provide response asynchronously by having handle_call return no_reply and later calling gen_server:reply. So why not also support handling response asynchronously on the other side? Or does that exist, but I'm just failing to find it??
gen_server:call is basically a sequence of
send a message to the server (with identifier)
wait for the response of that particular message
wrapped in a single function.
for your example you can decompose the behavior in 2 steps: a loop that uses gen_server:cast(Server,{Message,UniqueID,self()} with all servers, and then a receive loop that wait for a minimum of 2 answers of the form {UniqueID,Answer}. But you must take care to empty your mail box at some point in time. A better solution should be to delegate this to a separate process which will simply die when it has received the required number of answers:
[edit] make some correction in the code now it should work :o)
get_n_answers(Msg,ServerList,N) when N =< length(ServerList) ->
spawn(?MODULE,get_n_answers,[Msg,ServerList,N,[],self()]).
get_n_answers(_Msg,[],0,Rep,Pid) ->
Pid ! {Pid,Rep};
get_n_answers(_Msg,[],N,Rep,Pid) ->
NewRep = receive
Answ -> [Answ|Rep]
end,
get_n_answers(_Msg,[],N-1,NewRep,Pid);
get_n_answers(Msg,[H|T],N,Rep,Pid) ->
%gen_server:cast(H,{Msg,Pid}),
H ! {Msg,self()},
get_n_answers(Msg,T,N,Rep,Pid).
and you cane use it like this:
ID = get_n_answers(Msg,ServerList,2),
% insert some code here
Answer = receive
{ID,A} -> A % tagged with ID to do not catch another message in the mailbox
end
You can easily implement that by sending each call in a separate process and waiting for responses from as many as required (in essence this is what async is about, isn't? :-)
Have a look at this simple implementation of parallel call which is based on the async_call from rpc library in OTP.
This is how it works in plain English.
You need to make 5 calls so (in the parent process) you spawn 5 child Erlang processes.
Each process sends back to the parent process a tuple containing its PID and the result of the call.
The tuple can be only constructed and send back only when the desired call has been completed.
In the parent process you loop through responses in the receive loop.
You can wait for all responses or just 2 or 3 out of the started 5.
The parent process (which spawns the worker processes) will eventually receive all responses (I mean those you want to ignore). You need a way to discard them if you don't want the message queue to grow infinitely. There are two options:
The parent process itself can be a transient process, created only for the call to spawn the other 5 child processes. Once the desired amount of responses is collected it can send the response back to a caller and die. Messages send to the died process will be discarded.
The parent process can continue receiving messages after it has received the desired amount of responses and simply discard them.
gen_server do not have a concept of async calls on client side. It is not trivial how to implement in consistently because gen_server:call is a combination of monitor for server process, send request message and wait for either answer or monitor down or timeout. If you do something like what you mentioned you will need to deal with DOWN messages from server somehow ... so hypothetical async_call should return some key for yeld and also an internal monitor reference for a case you are processing DONW messages from other processes... and do not want to mix it with yeld errors.
Not that good but possible alternative is to use rpc:async_call(gen_server, call, [....])
But this approach have a limitation in calling process will be a short lived rex child, so if your gen server use caller pid somehow other than send it a reply logic will be broken.
gen_sever:call to the process itself would surely block until timeout. To understand the reason, one should be aware of the fact that gen_server framework actually combine your specific code together into one single module, and gen_server:call would be "translated" as "pid ! Msg" form.
So imagine how this block of code takes effect, the process actually stay in a loop keeping receiving messages, and when the control flow run into a handling function, the receiving process is temporarily interrupted, so if you call gen_server:call to the process itself, since it is a synchronous function, it waits for response, which however would never come in until the handing function returns so that the process can continue to receive messages, so the code is in a deadlock.
I have two NSInputStream and NSOutputStream between devices that are connected to each other via network. When I write something in output stream, the data is written until the NSStreamEventEndEncountered event occurs. I close the output stream but on the other side (input stream) the NSStreamEventEndEncountered event never occurs, until I exit the view controller of the output stream. So:
1. Why does not the NSStreamEventEndEncountered event occurs at input stream even after the the same occurred at the output stream ? (the output stream is even closed in this event)
2. It is my understanding that, once you open the NSOutputStream, you can only write data once. Opening the output stream again after NSStreamEventEndEncountered event (for example to write something new on any event) is not possible, right ????
I probably need more info about your connection and how you're sending your data but let me try and answer your questions:
1.
You're not encountering an end of your inputstream because you never started reading from it. The outputstream finished writing because it probably encountered an end, just like you said.
Imagine Jacob (your outputstream) delivering a envelope (your data) to his friend's house. Jacob puts the envelope on his friend's doormat and walks back to his own house. At this point Jacob's work is done so he tells himself that he's done (in your case the outputstream signals an NSStreamEventEndEncountered).
Jacob's friend George (your inputstream) could see the envelope or not, but nevertheless never looks what is in it. So unless George takes the envelope and looks what is inside it, he could never tell himself that he finished looking at it (in your case the inputstream never signals an NSStreamEventEndEncountered).
2.
This actually depends on how you plan to use your outputstream. If you plan to send data multiple times to the same device, why not leave the outputstream open? You can write data as long as the socket is open and there is space available. When you close the outputstream you need to reopen it however.
I have a task to wrap a 3rd party video decoder library in a direct show transform filter.
I believe this is going to be a problem as the 3rd party library uses an asynchronous callback when a full frame is ready i.e.
// on main thread
lib->RegisterCallback(callback function)
lib->write(raw data bytes)
void callback(frame)
{
// here is your frame (on a worker thread)
}
When I look at the pure virtual CTransformFilter.Transform function it expects the transform to be synchronous. Now I could set this up so it blocks on a event that is Set in the callback but what happens if the data coming in to the Transform function is not sufficient to generate a full new frame? I would deadlock the function.
Is my only option to go back to the library developer and ask for a synchronous decoder?
I've just been looking at the CTransformFilter Receive function. This is what calls the (overridden) Transform function. It then calls m_pOutput->m_pInputPin->Receive(pOutSample) to pass the sample onwards. Can I call this Receive function from the worker thread in the callback or do I have to keep everything on the same thread?
Thanks
You can still achieve what you want. Note that a filter is not required to output a media sample on the same thread and/or within the call it receives an input media sample. CTransformFilter is however made with this assumption in mind.
So basically your straightforward choices are:
step back from CTransformFilter and use its ancestor to inherit from to implement output media sample delivery from [a worker thread's] callback call
wait within Transform function for asynchronous completion (makes sense if the inner library still decodes 1 output frame for 1 input frame) and catch up then delivering the output; you will also have to wait for pause requests there and abort your wait in order to not block the execution