Related
I'm currently experimenting with Isolates in dart.
I'm trying to create a wrapper around an Isolate to make it more pleasant to use.
The desired interface is something along the lines:
abstract class BgIsolateInterface {
Future<Response> send<Message, Response>(Message message);
}
I want to have a method that sends a message to the background interface and then return the response to the caller.
To achieve this I figured I have to create a new RawReceivePort or ReceivePort in the send function to reliably get the correct response.
But this would mean I'm essentially creating the port and discarding it. Going against the documentations which states
Opens a long-lived port for receiving messages.
So my questions are:
what exactly are ReceivePorts and RawReceivePorts?
would my use case be valid i.e. have them be created only to read a single response?
should I look at another way of doing things?
Note: Please don't suggest the Flutter compute function as an alternative. I'm looking to do this in a long running isolate so I can share services / state between function calls. I'm just not showing this here to keep the question short.
Thank you very much!!!
Edit #1:
When providing the answer I realised there was also an underling question about how to read the Dart source, more specifically how to find external methods' implementations. That question was added to the title. The original question was just: What exactly is a ReceivePort / RawReceivePort?.
Yesterday, I've searched across the source and I think, I now have the answers. If I'm wrong, anyone more involved with the engine please correct me. This is mostly my speculation.
TLDR:
ReceivePort/RawReceivePorts are essentially int ids with a registered message handler. The SendPort knows to which id i.e. ReceivePort/RawReceivePort it should send the data to.
Yes. But for another use case there is better way.
Change the interface, so we react to states / responses coming from the isolate i.e.
abstract class BgIsolateInterface<Message, Response> {
void send(Message message);
void listen(void Function(Response) onData);
}
Long
#1
I've looked at the implementation and I'm including my findings here also to put a note for my future self on how to actually do this if I ever need to.
First, if we look at the implementation of ReceivePort (comments removed):
abstract class ReceivePort implements Stream<dynamic> {
external factory ReceivePort([String debugName = '']);
external factory ReceivePort.fromRawReceivePort(RawReceivePort rawPort);
StreamSubscription<dynamic> listen(void onData(var message)?,
{Function? onError, void onDone()?, bool? cancelOnError});
void close();
SendPort get sendPort;
}
We can see the external keyword. Now, this means implementation is defined somewhere else. Great! Where?
Let's open the SDK source and look. We are looking for a class definition of the same name i.e. ReceivePort with a #patch annotation. Also it seems the Dart team follows the convention of naming the implementation files for these external methods with the suffix _patch.dart.
We then find the three of these patch files. Two for the js runtime, one for development and one for production, and one file for the native? runtime. Since, I'm not using Dart for the web, the latter is the one I'm interested in.
In the file: sdk/lib/_internal/vm/lib/isolate_patch.dart we see:
#patch
class ReceivePort {
#patch
factory ReceivePort([String debugName = '']) =>
new _ReceivePortImpl(debugName);
#patch
factory ReceivePort.fromRawReceivePort(RawReceivePort rawPort) {
return new _ReceivePortImpl.fromRawReceivePort(rawPort);
}
}
Ok, so the implementation for ReceivePort is actually a library private _ReceivePortImpl class.
Note: As you can see factory methods don't have to return the same class the method is defined in. You just have to return an object that implements or extends it. i.e., has the same contract.
class _ReceivePortImpl extends Stream implements ReceivePort {
_ReceivePortImpl([String debugName = ''])
: this.fromRawReceivePort(new RawReceivePort(null, debugName));
_ReceivePortImpl.fromRawReceivePort(this._rawPort)
: _controller = new StreamController(sync: true) {
_controller.onCancel = close;
_rawPort.handler = _controller.add;
}
SendPort get sendPort {
return _rawPort.sendPort;
}
StreamSubscription listen(void onData(var message)?,
{Function? onError, void onDone()?, bool? cancelOnError}) {
return _controller.stream.listen(onData,
onError: onError, onDone: onDone, cancelOnError: cancelOnError);
}
close() {
_rawPort.close();
_controller.close();
}
final RawReceivePort _rawPort;
final StreamController _controller;
}
Which as we can see is really just a wrapper around a RawReceivePort where the handler is a StreamController.add method. So, what about the RawReceivePort?
If we look at initial file where ReceivePort is defined we again see. It's just one external factory method and an interface for others.
abstract class RawReceivePort {
external factory RawReceivePort([Function? handler, String debugName = '']);
void set handler(Function? newHandler);
SendPort get sendPort;
}
Luckily, its #patch version can also be found in the same place as the ReceivePorts.
#patch
class RawReceivePort {
#patch
factory RawReceivePort([Function? handler, String debugName = '']) {
_RawReceivePortImpl result = new _RawReceivePortImpl(debugName);
result.handler = handler;
return result;
}
}
Ok, again the actual implementation is _RawReceivePortImpl class.
#pragma("vm:entry-point")
class _RawReceivePortImpl implements RawReceivePort {
factory _RawReceivePortImpl(String debugName) {
final port = _RawReceivePortImpl._(debugName);
_portMap[port._get_id()] = <String, dynamic>{
'port': port,
};
return port;
}
#pragma("vm:external-name", "RawReceivePortImpl_factory")
external factory _RawReceivePortImpl._(String debugName);
close() {
_portMap.remove(this._closeInternal());
}
SendPort get sendPort {
return _get_sendport();
}
bool operator ==(var other) {
return (other is _RawReceivePortImpl) &&
(this._get_id() == other._get_id());
}
int get hashCode {
return sendPort.hashCode;
}
#pragma("vm:external-name", "RawReceivePortImpl_get_id")
external int _get_id();
#pragma("vm:external-name", "RawReceivePortImpl_get_sendport")
external SendPort _get_sendport();
#pragma("vm:entry-point", "call")
static _lookupHandler(int id) {
var result = _portMap[id]?['handler'];
return result;
}
#pragma("vm:entry-point", "call")
static _lookupOpenPorts() {
return _portMap.values.map((e) => e['port']).toList();
}
#pragma("vm:entry-point", "call")
static _handleMessage(int id, var message) {
final handler = _portMap[id]?['handler'];
if (handler == null) {
return null;
}
handler(message);
_runPendingImmediateCallback();
return handler;
}
#pragma("vm:external-name", "RawReceivePortImpl_closeInternal")
external int _closeInternal();
#pragma("vm:external-name", "RawReceivePortImpl_setActive")
external _setActive(bool active);
void set handler(Function? value) {
final int id = this._get_id();
if (!_portMap.containsKey(id)) {
_portMap[id] = <String, dynamic>{
'port': this,
};
}
_portMap[id]!['handler'] = value;
}
static final _portMap = <int, Map<String, dynamic>>{};
}
OK, now we're getting somewhere. A lot is going on.
First thing to note are the: #pragma("vm:entry-point"), #pragma("vm:entry-point", "call") and #pragma("vm:external-name", "...") annotations. Docs can be found here.
Oversimplified:
vm:entry-point tells the compiler this class / method will be used from native code.
vm:external-name tells the compiler to invoke a native function which is registered to the name provided by the annotation.
For instance to know the implementation of:
#pragma("vm:external-name", "RawReceivePortImpl_factory")
external factory _RawReceivePortImpl._(String debugName);
We have to look for DEFINE_NATIVE_ENTRY(RawReceivePortImpl_factory. And we find the entry in: runtime/lib/isolate.cc.
DEFINE_NATIVE_ENTRY(RawReceivePortImpl_factory, 0, 2) {
ASSERT(TypeArguments::CheckedHandle(zone, arguments->NativeArgAt(0)).IsNull());
GET_NON_NULL_NATIVE_ARGUMENT(String, debug_name, arguments->NativeArgAt(1));
Dart_Port port_id = PortMap::CreatePort(isolate->message_handler());
return ReceivePort::New(port_id, debug_name, false /* not control port */);
}
We see the port_id is created by PortMap::CreatePort and is of type Dart_Port. Hmmm, and what is a the type definition for Dart_Port.
runtime/include/dart_api.h
typedef int64_t Dart_Port;
OK so the actual internal representation of a RawReceivePort is a signed int stored in 64 bits, and some additional information like the type, state, debug names etc.
Most of the work is then being done in PortMap::CreatePort and other of its methods. I won't go in depth, because quite honestly I don't understand everything.
But from the looks of it the PortMap uses the port_id to point to some additional information + objects. It generates it randomly and makes sure the id is not taken. It also does a lot of different things but let's move on.
When sending a message through SendPort.send, the method essentially calls the registered entry SendPortImpl_sendInternal_ which determines which port to send the information to.
Note: SendPort essentially just points to its ReceivePort and also stores the id of the Isolate where it was created. When posting a message this id is used to determine what kind of objects can be sent through.
The a message is created and passed to PortMap::PostMessage which in turn calls MessageHandler::PostMessage.
There the message is enqueued by a call to MessageQueue::Enqueue. Then a MessageHandlerTask is ran on the ThreadPool.
The MessageHandlerTask essentially just calls the MessageHandler::TaskCallback which eventually calls MessageHandler::HandleMessages.
There the MessageHandler::HandleMessage is called, but this function is implemented by a child class of MessageHandler.
Currently there are two:
IsolateMessageHandler and
NativeMessageHandler.
We are interested in the IsolateMessageHandler.
Looking there we see IsolateMessageHandler::HandleMessage eventually calls DartLibraryCalls::HandleMessage which calls object_store->handle_message_function(). full chain: Thread::Current()->isolate_group()->object_store()->handle_message_function()
The function handle_message_function is defined by the (dynamic?) macro LAZY_ISOLATE(Function, handle_message_function) in runtime/vm/object_store.h.
The property + stores created are used in: runtime/vm/object_store.cc by the: ObjectStore::LazyInitIsolateMembers.
_RawReceivePortImpl is registered to lazily load at the isolate_lib.LookupClassAllowPrivate(Symbols::_RawReceivePortImpl()) call.
As well as, the methods marked with #pragma("vm:entry-point", "call"), including static _handleMessage(int id, var message).
Which is the handler that ->handle_message_function() returns.
Later the DartLibraryCalls::HandleMessage invokes it through DartEntry::InvokeFunction with the parameters port_id and the message.
This calls the _handleMessage function which calls the registered _RawReceivePort.handler.
#2
If we compare the Flutter's compute method implementation. It spins up an Isolate and 3 ReceivePorts for every compute call. If I used compute, I would be spending more resources and loose context between multiple message calls I can have with a long-running Isolate. So for my use case I reason, creating a new ReceivePort everytime I pass a message shouldn't be a problem.
#3
I could use a different approache. But I still wish to have a long running Isolate so I have the flexibility to share context between different calls to the Isolate.
Alternative:
Would be following a bloc / stream style interface and have a method to assign a listener and a method to send or add a message event, and have the calling code listen to the responses received and act accordingly.
i.e. an interface like:
abstract class BgIsolateInterface<Message, Response> {
void send(Message message);
void addListener(void Function(Response) onData);
void removeListener(void Function(Response) onData);
}
the down side is the Message and Response have to be determined when creating the class rather than simply when using the send method like the interface in my question. Also now some other part of the code base has to handle the Response. I prefer to handle everything at the send call site.
Note: The source code of the Dart project is put here for presentation purposes. The live source may change with time. Its distribution and use are governed by their LICENSE.
Also: I'm not C/C++ developer so any interpretation of the C/C++ code may be wrong.
While this answer is long side-steps the questions a little bit, I find it useful to include the steps to search through the Dart source. Personally, I found it difficult initially to find where external functions are defined and what some of the annotation values mean. While these steps could be extracted into a separate question, I think it's useful to keep it here where there was a use case to actually dive deep.
Thank you for reading!
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Today, I reviewed same Android code, and found a strange phenomenon.
It is the anonymous internal class directly called the external class instance method.
In my mind, directly calling a method is equivalent to adding this directly before the method, and this is an instance of an inner class.
According to this logic, an instance of an external class is invoked directly in the anonymous inner class, that will caused the compile ERROR.
But actually compile this application, and no problem. And the running log is normal.
Therefore, writing a simple Demo to verify the previous concept is wrong. code show as below:
public class InnerClass {
public static void main(String[] args) {
new InnerClass().process();
}
public void process() {
new Thread() {
#Override
public void run() {
System.out.println(toString("test"));
}
}.start();
}
public String toString(String string) {
return string;
}
}
In the Oracle:
In the OpenJDK:
So, What is the difference of the anonymous internal class directly call the external class instance method in the OpenJDK and Oracle?
Where can I find documentation to see these differences?
I worked hard, but did not get a clear answer.
Thanks.
P.S.
In According to my point of view
public class InnerClass {
public static void main(String[] args) {
new InnerClass().process();
}
public void process() {
new Thread() {
#Override
public void run() {
// In According to my point of view
// toString("test")
// <==>
// this.toString("test")
// and `this` is the instance of Thread
// what's the error of my view?
System.out.println(toString("test"));
}
}.start();
}
public String toString(String string) {
return string;
}
}
The problem is that
public void run() {
System.out.println(toString("test"));
}
is calling toString on the anonymous Thread subclass, and that is Thread::toString(). There is no Thread::toString(String), and the toString(String) method from the enclosing scope is not considered.
The JLS states that it will only check an enclosing / outer class for a method if there is no method with the required name in the inner class. See JLS 15.12.1:
If the form is MethodName, that is, just an Identifier, then:
If the Identifier appears in the scope of a visible method declaration
with that name (§6.3, §6.4.1), then:
If there is an enclosing type declaration of which that method is a member, let T be the innermost such type declaration. The class or
interface to search is T.
This search policy is called the "comb rule". It effectively looks for methods in a nested class's superclass hierarchy before looking
for methods in an enclosing class and its superclass hierarchy. See
§6.5.7.1 for an example.
As to why OpenJDK Java 7 accepts your test class .... if that is actually true, I'd call that a compiler bug. But it would be a general Java 7 bug not an OpenJDK specific one. The javac codebases are (AFAIK) identical for Oracle and OpenJDK releases of the same version.
Interestingly, I have a copy of Oracle Java 6, and javac from that version also calls this a compilation error as well.
$ /usr/java/jdk1.6.0_45/bin/javac InnerClass.java
InnerClass.java:10: cannot find symbol
symbol: method toString(java.lang.String)
System.out.println(toString("test"));
^
1 error
So ... maybe ... you should rerun your OpenJDK Java 7 test, and make sure you are compiling the same source code!
How to access the elements of a side input if I have my class extend DoFn?
For example:
Say I have a ParDo transform like:
PCollection<String> data = myData.apply("Get data",
ParDo.of(new MyClass()).withSideInputs(myDataView));
And I have a class:-
static class MyClass extends DoFn<String,String>
{
//How to access side input here
}
c.sideInput() isn't working in this case.
Thanks.
In this case, the problem is that the processElement method in your DoFn does not have access to the PCollectionView instance in your main method.
You can pass the PCollectionView to the DoFn in the constructor:
class MyClass extends DoFn<String,String>
{
private final PCollectionView<..> mySideInput;
public MyClass(PCollectionView<..> mySideInput) {
// List, or Map or anything:
this.mySideInput = mySideInput;
}
#ProcessElement
public void processElement(ProcessContext c) throws IOException
{
// List or Map or any type you need:
List<..> sideInputList = c.sideInput(mySideInput);
}
}
You would then pass the side input to the class when you instantiate it, and indicate it as a side input like so:
p.apply(ParDo.of(new MyClass(mySideInput)).withSideInputs(mySideInput));
The explanation for this is that when you use an anonymous DoFn, the process method has a closure with access to all the objects within the scope that encloses the DoFn (among them is the PCollectionView). When you're not using an anonymous DoFn, there is no closure, and you need another way of passing the PCollectionView.
So although the answer above is correct, it is still a little incomplete.
So once you finish implementing the above answer, you need to execute your pipeline like this:
p.apply(ParDo.of(new MyClass(mySideInput)).withSideInputs(mySideInput));
In my Grails app I've installed the Quartz plugin. I want to intercept calls to every Quartz job class' execute method in order to do something before the execute method is invoked (similar to AOP before advice).
Currently, I'm trying to do this interception from the doWithDynamicMethods closure of another plugin as shown below:
def doWithDynamicMethods = { ctx ->
// get all the job classes
application.getArtefacts("Job").each { klass ->
MetaClass jobMetaClass = klass.clazz.metaClass
// intercept the methods of the job classes
jobMetaClass.invokeMethod = { String name, Object args ->
// do something before invoking the called method
if (name == "execute") {
println "this should happen before execute()"
}
// now call the method that was originally invoked
def validMethod = jobMetaClass.getMetaMethod(name, args)
if (validMethod != null) {
validMethod.invoke(delegate, args)
} else {
jobMetaClass.invokeMissingMethod(delegate, name, args)
}
}
}
}
So, given a job such as
class TestJob {
static triggers = {
simple repeatInterval: 5000l // execute job once in 5 seconds
}
def execute() {
"execute called"
}
}
It should print:
this should happen before execute()
execute called
But my attempt at method interception seems to have no effect and instead it just prints:
execute called
Perhaps the cause of the problem is this Groovy bug? Even though the Job classes don't explicitly implement the org.quartz.Job interface, I suspect that implicitly (due to some Groovy voodoo), they are instances of this interface.
If indeed this bug is the cause of my problem, is there another way that I can do "before method interception"?
Because all the job classes are Spring beans you can solve this problem using Spring AOP. Define an aspect such as the following (adjust the pointcut definition so that it matches only your job classes, I've assumed they are all in a package named org.example.job and have a class name that ends with Job).
#Aspect
class JobExecutionAspect {
#Pointcut("execution(public * org.example.job.*Job.execute(..))")
public void executeMethods() {}
#Around("executeMethods()")
def interceptJobExecuteMethod(ProceedingJoinPoint jp) {
// do your stuff that should happen before execute() here, if you need access
// to the job object call jp.getTarget()
// now call the job's execute() method
jp.proceed()
}
}
You'll need to register this aspect as a Spring bean (it doesn't matter what name you give the bean).
You can have your customized JobListener registered in the application to handle logics before execute() is triggered. You can use something like:-
public class MyJobListener implements JobListener {
public void jobToBeExecuted(JobExecutionContext context) {
println "Before calling Execute"
}
public void jobWasExecuted(JobExecutionContext context,
JobExecutionException jobException) {}
public void jobExecutionVetoed(JobExecutionContext context) {}
}
Register the customized Job Listener to Quartz Scheduler in Bootstrap:-
Scheduler scheduler = ctx.getBean("quartzScheduler") //ctx being application context
scheduler.getListenerManager().addJobListener(myJobListener, allJobs())
resources.groovy:-
beans = {
myJobListener(MyJobListener)
}
One benefit I see here using this approach is that we don't need the second plugin used for method interception any more.
Second, we can register the listener to listen all jobs, specific jobs, and jobs in a group. Refer Customize Quartz JobListener and API for JobListener, TriggerListener, ScheduleListener for more insight.
Obviously, AOP is another approach if we do want want to use Quartz API.
You are not getting the job classes like that. If you refer to the Quartz plugin, you can get them by calling jobClasses:
application.jobClasses.each {GrailsJobClass tc -> ... }
see https://github.com/nebolsin/grails-quartz/blob/master/QuartzGrailsPlugin.groovy
If you actually look, you can see that they are almost doing what you are trying to acheive without the need to use aop or anything else.
For method interception implement invokeMethod on the metaclass. In my case the class was not of third party so I can modify the implementation.
Follow this blog for more information.
I have just started out in Vala, and I tried to make a simple program that asks two inputs:
An int that specifies a cycle degree; and
A char that contains I / R for either an iterative or recursive process.
Just before compiling, I got this error:
test0.vala:8.5-8.16: error: Access to instance member `test0.test_exec' denied
test_exec(q);
^^^^^^^^^^^ //the entire statement
Compilation failed: 1 error(s), 0 warning(s)
The pastebin for the very simple program is located here.
Here's a snippet:
public static void main(string[] args)
{
stdout.printf("Greetings! How many cycles would you like? INPUT: ");
int q=0;
stdin.scanf("%d", out q);
test_exec(q);
}
public void test_exec(int q)
{
//method code here
}
Can you please enlighten me about what to do, and some tips? Thanks.
You defined test_exec as an instance (non-static) method. Unlike a static method, an instance method needs to be called on an instance of the given class. However you're trying to call it without such an instance and thus get an error.
So you either need to create an instance of the test0 class and call test_exec on that (though that would make little sense since test_exec does not depend on or change any state of the object - as a matter of fact the test0 class does not have any state) or make test_exec as well as the other methods called by test_exec static.