To lone travelers stumbling upon this: see comments for the answer.
...
Writing a Java wrapper for a native library. A device generates data samples and stores them as structs. Two native ways of accessing them: either you request one with a getSample(&sampleStruct) or you set a callback function. Now, here is what does work:
The polling method does fill the JNA Structure
The callback function is called after being set
In fact, I am currently getting the sample right from the callback function
The problem: trying to do anything with the callback argument, which should be a struct, causes an "invalid memory access". Declaring the argument as the Structure does this, so I declared it as a Pointer. Trying a Pointer.getInt(0) causes invalid memory access. So then I declared the argument as an int, and an int is delivered; in fact, it looks very much like the first field of the struct I am trying to get! So does it mean that the struct was at that address but disappeared before Java had time to access it?
This is what I am doing now:
public class SampleCallback implements Callback{
SampleStruct sample;
public int callback(Pointer refToSample) throws IOException{
lib.INSTANCE.GetSample(sample); // works no problem
adapter.handleSample(sample);
return 1;
} ...
But neither of these does:
public int callback(SampleStruct sample) throws IOException{
adapter.handleSample(sample);
return 1;
}
...
public int callback(Pointer refToSample) throws IOException{
SampleStruct sample = new SampleStruct();
sample.timestamp = refToSample.getInt(0);
...
adapter.handleSample(sample);
return 1;
}
Also, this does in fact deliver the timestamp,
public int callback(int timestamp) throws IOException{
System.out.println("It is " + timestamp + "o'clock");
return 1;
}
but I would really prefer the whole struct.
This is clearly not going to be a popular topic and I do have a working solution, so the description is not exactly full. Will copy anything else that might be helpful if requested. Gratitude prematurely extended.
Related
I am wondering about a particular statement in the Moloch v2.1 smart contract code:
https://github.com/Moloch-Mystics/Molochv2.1/blob/main/Flat_Moloch_v2.1.sol
In the Moloch contract, there is a contract level mapping called members
mapping(address => Member) public members;
As expected in almost every line of code that reads or writes to this mapping, square brackets are used. However there is one line of code involving a tuple that uses parentheses instead, wrapping a pair of them around msg.sender:
(,,,bool exists,,) = moloch.members(msg.sender);
Why isn't this line of code using square brackets after the moloch.members reference, like this?:
(,,,bool exists,,) = moloch.members[msg.sender];
I don't understand why that line is using parentheses and what kind of different behavior might result from this. Can someone explain this to me?
For reference purposes, this occurs inside the function below, which is a member of the MolochSummoner contract, which inherits from CloneFactory:
function registerDao(
address _daoAdress,
string memory _daoTitle,
string memory _http,
uint _version
) public returns (bool) {
moloch = Moloch(_daoAdress);
(,,,bool exists,,) = moloch.members(msg.sender);
require(exists == true, "must be a member");
require(daos[_daoAdress] == false, "dao metadata already registered");
daos[_daoAdress] = true;
daoIdx = daoIdx + 1;
emit Register(daoIdx, _daoAdress, _daoTitle, _http, _version);
return true;
}
You can only use brackets if you are accessing the variable inside the same contract.
Outside the contract, when accessing a public variable you are calling a getter function members(). Solidity automatically generates this function for you. There is no get prefix.
More about Solidity getter functions here.
This is a bit confusing and somewhat bad design decision, so I apologize.
So I am trying to get into app development and following this guys yt-tutorial (yt-vid from The Net Ninja). However I stumbled over the following error:
Non-nullable instance field ‘{0}’ must be initialized. (error on dart.dev explained). The whole code is shown in this image
img coding: comparison mine with yt:
The issue I am facing is that I don't understand the origin of that error. I understand that you can't use a variable that was not initialized yet and that this error is trying to tell you that before you run the code and run into an exception in runtime.
I also know that I can fix this using "late" in front of the variables, but as far as I'm concerned that's just "ignoring" the error.
So my main question is: why do I have this error while the yt doesn't even though we have (except for some names and assignments) the same code?
I appreciate every answer and hope you have a great day
the links:
error on dart.dev explained
You should change your User class to:
class User {
String username;
int age;
User(this.username, this.age);
}
By doing so, the variables will be initialized with values (since we are then able to use this values in the constructor body) as part of initialization of the object.
Dart does object creation in two steps:
Initialize object.
Run constructor body.
So when the constructor body is running, the object must be initialized with values (well, the non-nullable values at least). To do so, we have a code block which is executed before { } like:
class User {
String username;
int age;
User(String username, int age)
: this.username = username,
this.age = age;
}
But since this is kinda redundant, we have the shortcut form as I first suggested.
When initializing an object, we are not allowed to use variables from the object since the object is ongoing its creation.
I'm currently experimenting with Isolates in dart.
I'm trying to create a wrapper around an Isolate to make it more pleasant to use.
The desired interface is something along the lines:
abstract class BgIsolateInterface {
Future<Response> send<Message, Response>(Message message);
}
I want to have a method that sends a message to the background interface and then return the response to the caller.
To achieve this I figured I have to create a new RawReceivePort or ReceivePort in the send function to reliably get the correct response.
But this would mean I'm essentially creating the port and discarding it. Going against the documentations which states
Opens a long-lived port for receiving messages.
So my questions are:
what exactly are ReceivePorts and RawReceivePorts?
would my use case be valid i.e. have them be created only to read a single response?
should I look at another way of doing things?
Note: Please don't suggest the Flutter compute function as an alternative. I'm looking to do this in a long running isolate so I can share services / state between function calls. I'm just not showing this here to keep the question short.
Thank you very much!!!
Edit #1:
When providing the answer I realised there was also an underling question about how to read the Dart source, more specifically how to find external methods' implementations. That question was added to the title. The original question was just: What exactly is a ReceivePort / RawReceivePort?.
Yesterday, I've searched across the source and I think, I now have the answers. If I'm wrong, anyone more involved with the engine please correct me. This is mostly my speculation.
TLDR:
ReceivePort/RawReceivePorts are essentially int ids with a registered message handler. The SendPort knows to which id i.e. ReceivePort/RawReceivePort it should send the data to.
Yes. But for another use case there is better way.
Change the interface, so we react to states / responses coming from the isolate i.e.
abstract class BgIsolateInterface<Message, Response> {
void send(Message message);
void listen(void Function(Response) onData);
}
Long
#1
I've looked at the implementation and I'm including my findings here also to put a note for my future self on how to actually do this if I ever need to.
First, if we look at the implementation of ReceivePort (comments removed):
abstract class ReceivePort implements Stream<dynamic> {
external factory ReceivePort([String debugName = '']);
external factory ReceivePort.fromRawReceivePort(RawReceivePort rawPort);
StreamSubscription<dynamic> listen(void onData(var message)?,
{Function? onError, void onDone()?, bool? cancelOnError});
void close();
SendPort get sendPort;
}
We can see the external keyword. Now, this means implementation is defined somewhere else. Great! Where?
Let's open the SDK source and look. We are looking for a class definition of the same name i.e. ReceivePort with a #patch annotation. Also it seems the Dart team follows the convention of naming the implementation files for these external methods with the suffix _patch.dart.
We then find the three of these patch files. Two for the js runtime, one for development and one for production, and one file for the native? runtime. Since, I'm not using Dart for the web, the latter is the one I'm interested in.
In the file: sdk/lib/_internal/vm/lib/isolate_patch.dart we see:
#patch
class ReceivePort {
#patch
factory ReceivePort([String debugName = '']) =>
new _ReceivePortImpl(debugName);
#patch
factory ReceivePort.fromRawReceivePort(RawReceivePort rawPort) {
return new _ReceivePortImpl.fromRawReceivePort(rawPort);
}
}
Ok, so the implementation for ReceivePort is actually a library private _ReceivePortImpl class.
Note: As you can see factory methods don't have to return the same class the method is defined in. You just have to return an object that implements or extends it. i.e., has the same contract.
class _ReceivePortImpl extends Stream implements ReceivePort {
_ReceivePortImpl([String debugName = ''])
: this.fromRawReceivePort(new RawReceivePort(null, debugName));
_ReceivePortImpl.fromRawReceivePort(this._rawPort)
: _controller = new StreamController(sync: true) {
_controller.onCancel = close;
_rawPort.handler = _controller.add;
}
SendPort get sendPort {
return _rawPort.sendPort;
}
StreamSubscription listen(void onData(var message)?,
{Function? onError, void onDone()?, bool? cancelOnError}) {
return _controller.stream.listen(onData,
onError: onError, onDone: onDone, cancelOnError: cancelOnError);
}
close() {
_rawPort.close();
_controller.close();
}
final RawReceivePort _rawPort;
final StreamController _controller;
}
Which as we can see is really just a wrapper around a RawReceivePort where the handler is a StreamController.add method. So, what about the RawReceivePort?
If we look at initial file where ReceivePort is defined we again see. It's just one external factory method and an interface for others.
abstract class RawReceivePort {
external factory RawReceivePort([Function? handler, String debugName = '']);
void set handler(Function? newHandler);
SendPort get sendPort;
}
Luckily, its #patch version can also be found in the same place as the ReceivePorts.
#patch
class RawReceivePort {
#patch
factory RawReceivePort([Function? handler, String debugName = '']) {
_RawReceivePortImpl result = new _RawReceivePortImpl(debugName);
result.handler = handler;
return result;
}
}
Ok, again the actual implementation is _RawReceivePortImpl class.
#pragma("vm:entry-point")
class _RawReceivePortImpl implements RawReceivePort {
factory _RawReceivePortImpl(String debugName) {
final port = _RawReceivePortImpl._(debugName);
_portMap[port._get_id()] = <String, dynamic>{
'port': port,
};
return port;
}
#pragma("vm:external-name", "RawReceivePortImpl_factory")
external factory _RawReceivePortImpl._(String debugName);
close() {
_portMap.remove(this._closeInternal());
}
SendPort get sendPort {
return _get_sendport();
}
bool operator ==(var other) {
return (other is _RawReceivePortImpl) &&
(this._get_id() == other._get_id());
}
int get hashCode {
return sendPort.hashCode;
}
#pragma("vm:external-name", "RawReceivePortImpl_get_id")
external int _get_id();
#pragma("vm:external-name", "RawReceivePortImpl_get_sendport")
external SendPort _get_sendport();
#pragma("vm:entry-point", "call")
static _lookupHandler(int id) {
var result = _portMap[id]?['handler'];
return result;
}
#pragma("vm:entry-point", "call")
static _lookupOpenPorts() {
return _portMap.values.map((e) => e['port']).toList();
}
#pragma("vm:entry-point", "call")
static _handleMessage(int id, var message) {
final handler = _portMap[id]?['handler'];
if (handler == null) {
return null;
}
handler(message);
_runPendingImmediateCallback();
return handler;
}
#pragma("vm:external-name", "RawReceivePortImpl_closeInternal")
external int _closeInternal();
#pragma("vm:external-name", "RawReceivePortImpl_setActive")
external _setActive(bool active);
void set handler(Function? value) {
final int id = this._get_id();
if (!_portMap.containsKey(id)) {
_portMap[id] = <String, dynamic>{
'port': this,
};
}
_portMap[id]!['handler'] = value;
}
static final _portMap = <int, Map<String, dynamic>>{};
}
OK, now we're getting somewhere. A lot is going on.
First thing to note are the: #pragma("vm:entry-point"), #pragma("vm:entry-point", "call") and #pragma("vm:external-name", "...") annotations. Docs can be found here.
Oversimplified:
vm:entry-point tells the compiler this class / method will be used from native code.
vm:external-name tells the compiler to invoke a native function which is registered to the name provided by the annotation.
For instance to know the implementation of:
#pragma("vm:external-name", "RawReceivePortImpl_factory")
external factory _RawReceivePortImpl._(String debugName);
We have to look for DEFINE_NATIVE_ENTRY(RawReceivePortImpl_factory. And we find the entry in: runtime/lib/isolate.cc.
DEFINE_NATIVE_ENTRY(RawReceivePortImpl_factory, 0, 2) {
ASSERT(TypeArguments::CheckedHandle(zone, arguments->NativeArgAt(0)).IsNull());
GET_NON_NULL_NATIVE_ARGUMENT(String, debug_name, arguments->NativeArgAt(1));
Dart_Port port_id = PortMap::CreatePort(isolate->message_handler());
return ReceivePort::New(port_id, debug_name, false /* not control port */);
}
We see the port_id is created by PortMap::CreatePort and is of type Dart_Port. Hmmm, and what is a the type definition for Dart_Port.
runtime/include/dart_api.h
typedef int64_t Dart_Port;
OK so the actual internal representation of a RawReceivePort is a signed int stored in 64 bits, and some additional information like the type, state, debug names etc.
Most of the work is then being done in PortMap::CreatePort and other of its methods. I won't go in depth, because quite honestly I don't understand everything.
But from the looks of it the PortMap uses the port_id to point to some additional information + objects. It generates it randomly and makes sure the id is not taken. It also does a lot of different things but let's move on.
When sending a message through SendPort.send, the method essentially calls the registered entry SendPortImpl_sendInternal_ which determines which port to send the information to.
Note: SendPort essentially just points to its ReceivePort and also stores the id of the Isolate where it was created. When posting a message this id is used to determine what kind of objects can be sent through.
The a message is created and passed to PortMap::PostMessage which in turn calls MessageHandler::PostMessage.
There the message is enqueued by a call to MessageQueue::Enqueue. Then a MessageHandlerTask is ran on the ThreadPool.
The MessageHandlerTask essentially just calls the MessageHandler::TaskCallback which eventually calls MessageHandler::HandleMessages.
There the MessageHandler::HandleMessage is called, but this function is implemented by a child class of MessageHandler.
Currently there are two:
IsolateMessageHandler and
NativeMessageHandler.
We are interested in the IsolateMessageHandler.
Looking there we see IsolateMessageHandler::HandleMessage eventually calls DartLibraryCalls::HandleMessage which calls object_store->handle_message_function(). full chain: Thread::Current()->isolate_group()->object_store()->handle_message_function()
The function handle_message_function is defined by the (dynamic?) macro LAZY_ISOLATE(Function, handle_message_function) in runtime/vm/object_store.h.
The property + stores created are used in: runtime/vm/object_store.cc by the: ObjectStore::LazyInitIsolateMembers.
_RawReceivePortImpl is registered to lazily load at the isolate_lib.LookupClassAllowPrivate(Symbols::_RawReceivePortImpl()) call.
As well as, the methods marked with #pragma("vm:entry-point", "call"), including static _handleMessage(int id, var message).
Which is the handler that ->handle_message_function() returns.
Later the DartLibraryCalls::HandleMessage invokes it through DartEntry::InvokeFunction with the parameters port_id and the message.
This calls the _handleMessage function which calls the registered _RawReceivePort.handler.
#2
If we compare the Flutter's compute method implementation. It spins up an Isolate and 3 ReceivePorts for every compute call. If I used compute, I would be spending more resources and loose context between multiple message calls I can have with a long-running Isolate. So for my use case I reason, creating a new ReceivePort everytime I pass a message shouldn't be a problem.
#3
I could use a different approache. But I still wish to have a long running Isolate so I have the flexibility to share context between different calls to the Isolate.
Alternative:
Would be following a bloc / stream style interface and have a method to assign a listener and a method to send or add a message event, and have the calling code listen to the responses received and act accordingly.
i.e. an interface like:
abstract class BgIsolateInterface<Message, Response> {
void send(Message message);
void addListener(void Function(Response) onData);
void removeListener(void Function(Response) onData);
}
the down side is the Message and Response have to be determined when creating the class rather than simply when using the send method like the interface in my question. Also now some other part of the code base has to handle the Response. I prefer to handle everything at the send call site.
Note: The source code of the Dart project is put here for presentation purposes. The live source may change with time. Its distribution and use are governed by their LICENSE.
Also: I'm not C/C++ developer so any interpretation of the C/C++ code may be wrong.
While this answer is long side-steps the questions a little bit, I find it useful to include the steps to search through the Dart source. Personally, I found it difficult initially to find where external functions are defined and what some of the annotation values mean. While these steps could be extracted into a separate question, I think it's useful to keep it here where there was a use case to actually dive deep.
Thank you for reading!
Coming back to C++ after a hiatus in Java. Attempting to create an immutable object and after working in Java, a public const variable seems the most sensible (like Java final).
public:
const int A;
All well and good, but if I want to defensive check this value, how might I go about it. The code below seems strange to me, but unlike Java final members, I can't seem to set A in the constructor after defensive checks (compiler error).
MyObj::MyObj(int a) : A(a) {
if (a < 0)
throw invalid_argument("must be positive");
}
A public const variable for A seems like a clearer, cleaner solution than a getter only with a non const int behind it, but open to that or other ideas if this is bad practice.
Your example as it stands should work fine:
class MyObj {
public:
const int var;
MyObj(int var) : var(var) {
if (var < 0)
throw std::invalid_argument("must be positive");
}
};
(Live example, or with out-of-line constructor)
If you intend that MyObj will always be immutable, then a const member is
probably fine. If you want the variable to be immutable in general, but still have the possibility to overwrite the entire object with an assignment, then better to have a private variable with a getter:
class MyObj {
int var;
public:
MyObj(int var) : var(var) {
if (var < 0)
throw std::invalid_argument("must be positive");
}
int getVar() const { return var; }
};
// now allows
MyObj a(5);
MyObj b(10);
a = b;
Edit
Apparently, what you want to do is something like
MyObj(int var) {
if (var < 0)
throw std::invalid_argument("must be positive");
this->var = var;
}
This is not possible; once a const variable has a value it cannot be changed. Once the body ({} bit) of the constructor starts, const variables already have a value, though in this case the value is "undefined" since you're not setting it (and the compiler is throwing an error because of it).
Moreover, there's actually no point to this. There is no efficiency difference in setting the variable after the checks or before them, and it's not like any external observers will be able to see the difference regardless since the throw statement will unroll the stack, deconstructing the object straight away.
Generally the answer by N. Shead is the regular practice - but you can also consider:
Create domain-specific types and use them instead of general primitives. E.g., if your field is a telephone number, have a type TelephoneNumber which, in its constructor (or factory), taking a string, does all the telephone number validation you'd like (and throws on invalid). Then you write something like:
class Contact {
const TelephoneNumber phone_;
public:
Contact(string phone) : phone_(phone) { ... }
...
When you do this the constructor for TelephoneNumber taking a string argument will be called when initializing the field phone_ and the validation will happen.
Using domain-specific types this way is discussed on the web under the name "primitive obsession" as a "code smell".
(The problem with this approach IMO is that you pretty much have to use it everywhere, and from the start of your project, otherwise you start having to have explicit (or implicit) casting all over the place and your code looks like crap and you can never be sure if the value you have has been validated or not. If you're working with an existing codebase it is nearly impossible to retrofit it completely though you might just start using it for particularly important/ubiquitous types.)
Create validation methods that take and return some value, and which perform the validation necessary - throwing when invalid otherwise returning its argument. Here's an example validator:
string ValidatePhoneNumber(string v) {
<some kind of validation throwing on invalid...>
return v;
}
And use it as follows:
class Contact {
const string phone_;
public:
Contact(string phone) : phone_(ValidatePhoneNumber(phone)) { ... }
I've seen this used when an application or library is doing so much validation of domain-specific types that a small library of these domain-specific validator methods has been built up and code readers are used to them. I wouldn't really consider it idiomatic, but it does have the advantage that the validation is right out there in the open where you can see it.
So, I have 2 partitions in a step which writes into a database. I want to record the number of rows written in each partition, get the sum, and print it to the log;
I was thinking of using a static variable in the Writer and use Step Context/Job Context to get it in afterStep() of the Step Listener. However when I tried it I got null. I am able to get these values in close() of the Reader.
Is this the right way to go about it? Or should I use Partition Collector/Reducer/ Analyzer?
I am using a java batch in Websphere Liberty. And I am developing in Eclipse.
I was thinking of using a static variable in the Writer and use Step Context/Job Context to get it in afterStep() of the Step Listener. However when i tried it i got null.
The ItemWriter might already be destroyed at this point, but I'm not sure.
Is this the right way to go about it?
Yes, it should be good enough. However, you need to ensure the total row count is shared for all partitions because the batch runtime maintains a StepContext clone per partition. You should rather use JobContext.
I think using PartitionCollector and PartitionAnalyzer is a good choice, too. Interface PartitionCollector has a method collectPartitionData() to collect data coming from its partition. Once collected, batch runtime passes this data to PartitionAnalyzer to analyze the data. Notice that there're
N PartitionCollector per step (1 per partition)
N StepContext per step (1 per partition)
1 PartitionAnalyzer per step
The records written can be passed via StepContext's transientUserData. Since the StepContext is reserved for its own step-partition, the transient user data won't be overwritten by other partition.
Here's the implementation :
MyItemWriter :
#Inject
private StepContext stepContext;
#Override
public void writeItems(List<Object> items) throws Exception {
// ...
Object userData = stepContext.getTransientUserData();
stepContext.setTransientUserData(partRowCount);
}
MyPartitionCollector
#Inject
private StepContext stepContext;
#Override
public Serializable collectPartitionData() throws Exception {
// get transient user data
Object userData = stepContext.getTransientUserData();
int partRowCount = userData != null ? (int) userData : 0;
return partRowCount;
}
MyPartitionAnalyzer
private int rowCount = 0;
#Override
public void analyzeCollectorData(Serializable fromCollector) throws Exception {
rowCount += (int) fromCollector;
System.out.printf("%d rows processed (all partitions).%n", rowCount);
}
Reference : JSR352 v1.0 Final Release.pdf
Let me offer a bit of an alternative on the accepted answer and add some comments.
PartitionAnalyzer variant - Use analyzeStatus() method
Another technique would be to use analyzeStatus which only gets called at the end of each entire partition, and is passed the partition-level exit status.
public void analyzeStatus(BatchStatus batchStatus, String exitStatus)
In contrast, the above answer using analyzeCollectorData gets called at the end of each chunk on each partition.
E.g.
public class MyItemWriteListener extends AbstractItemWriteListener {
#Inject
StepContext stepCtx;
#Override
public void afterWrite(List<Object> items) throws Exception {
// update 'newCount' based on items.size()
stepCtx.setExitStatus(Integer.toString(newCount));
}
Obviously this only works if you weren't using the exit status for some other purpose. You can set the exit status from any artifact (though this freedom might be one more thing to have to keep track of).
Comments
The API is designed to facilitate an implementation dispatching individual partitions across JVMs, (e.g. in Liberty you can see this here.) But using a static ties you to a single JVM, so it's not a recommended approach.
Also note that both the JobContext and the StepContext are implemented in the "thread-local"-like fashion we see in batch.