I see from http://social.msdn.microsoft.com/Forums/en/roboticsdss/thread/3169a539-f536-4b9d-bae7-01212a857660 and also from my own experience of using and debugging DSS services that DSS makes use of cloning for intra node communication and cloning + serialization for inter node communication. I request your help in understanding this mechanism. For intra node communication, I understand that DSS makes use of only cloning because we are transferring objects in the same process space (there is no need for communicating it over the network) and we do not want any unwanted shared memory dependencies. However, during inter node communication, why do we need to clone + serialize? From my understanding of serialization, when you serialize an object, the output you get at the end of serialization is completely independent of the object being serialized and can be used in isolation to construct a new object. Also, during serializing, I do not think we are going to update the original object and we are merely going to read from it. In such a scenario, will serialization alone not suffice? Why do we need to clone before serializing?
Thanks,
Venkat
In general, DSS serializes messages using the generated proxy types, so if a message is constructed using the original type it will first be transformed to the proxy type (which is implicitly a clone operation) before serializing. On the other hand if a message is constructed using a proxy type then there is no separate cloning step prior to serialization (a good reason to use proxy types for partners).
If a message type (or more strictly the message body type) implements IDssSerializable, then the behavior depends on the specific implementation of the type (because this bypasses the usual proxy mechanisms)
Related
I was going through the article, https://learn.microsoft.com/en-us/azure/architecture/patterns/cqrs which says, "If separate read and write databases are used, they must be kept in sync". One obvious benefit I can understand from having separate read replicas is that they can be scaled horizontally. However, I have some doubts:
It says, "Updating the database and publishing the event must occur in a single transaction". My understanding is that there is no guarantee that the updated data will be available immediately on the read-only nodes because it depends on when the event will be consumed by the read-only nodes. Did I get it correctly?
Data must be first written to read-only nodes before it can be read i.e. write operations are also performed on the read-only nodes. Why are they called read-only nodes? Is it because the write operations are performed on these nodes not directly by the data producer application; but rather by some serverless function (e.g. AWS Lambda or Azure Function) that picks up the event from the topic (e.g. Kafka topic) to which the write-only node has sent the event?
Is the data sharded across the read-only nodes or does every read-only node have the complete set of data?
All of these have "it depends"-like answers...
Yes, usually, although some implementations might choose to (try to) update read models transactionally with the update. With multiple nodes you're quickly forced to learn the CAP theorem, though, and so in many CQRS contexts, eventual consistency is just accepted as a feature, as the gains from tolerating it usually significantly outweigh the losses.
I suspect the bit you quoted anyway refers to transactionally updating the write store with publishing the event. Even this can be difficult to achieve, and is one of the problems event sourcing seeks to solve.
Yes. It's trivially obvious - in this context - that data must be written before it can be read, but your apps as consumers of the data see them as read-only.
Both are valid outcomes. Usually this part is less an application concern and is more delegated to the capabilities of your chosen read-model infrastructure (Mongo, Cosmos, Dynamo, etc).
I am going through the concept of IPFS. And one of the important aspect in IPFS is Bitswapping which basically deals with how blocks of data are requested using the wantlists.
My question is with regards to once a peer gets the wantlists from other peers,
how does it actually fetch the data from the actual storage device?
What are the steps involved in it?
How does the conversion happen with respect to different storage protocols based on the bitswap requests.
Please help me with these answers.
I'm still learning, so questions like this are a good opportunity to dig deeper :)
how does it actually fetch the data from the actual storage device?
What are the steps involved in it?
Based on the Bitswap api docs, it looks like bitswap operates on a provided libp2p instance and blockstore instance.
The blockstore instance is an abstraction over actual data storage, which could be software abstraction of anything - storage service like S3, virtualized device, or real device.
Based on the configuration bits I've read though, fetching could be done over whichever transports the libp2p instance was configured with, and any connected nodes also support (on per node basis).
Assuming multiple transports are supported on both ends between two nodes, I don't know how best connection is negotiated/dictated by libp2p...
How does the conversion happen with respect to different storage protocols based on the bitswap requests.
IIUC, at the block level there would not be any conversion happening - that would happen at a higher level in the stack (IPLD).
I read through these to get a better understanding:
Bitswap spec
JS-IPFS Bitswap implementation
JS-IPFS Blockservice
I am implementing a robotic system based on ROS. I have different nodes which send data multiple times per second. However, I don't need that. I want to send the robot state only when it is at a new location. What technique of ROS do you suggest to use?
Dependent on your requirements, you can either use the ROS Services or the Parameter Server.
ROS Service: The publish / subscribe model is a very flexible
communication paradigm, but its many-to-many one-way transport is not
appropriate for RPC request / reply interactions, which are often
required in a distributed system. Request / reply is done via a
Service, which is defined by a pair of messages: one for the request
and one for the reply.
Parameter Server: A parameter server is a shared, multi-variate dictionary that is accessible via network APIs. Nodes use this server
to store and retrieve parameters at runtime. As it is not designed for
high-performance, it is best used for static, non-binary data such as
configuration parameters.
I use WM_COPYDATA to enable communication between my two processes A and B. There is no problem to exchange data with basic data types.
Now I have a problem, in some case I want to pass an Interface (IDispatch) from my process A to my process B. Is it possible?
It is not possible to directly pass an interface pointer to another process. Like any other pointer, an interface is only valid in the process address space that instantiates it at runtime. COM has its own mechanism for marshaling interfaces and data across process boundaries, even across different apartments in the same process. In the case of interfaces, that involves proxies and stubs which run in each process/apartment and communicate with each other using various IPC mechanisms, such as pipes, RPC, or TCP/IP. Have a look at these articles for how using interfaces across processes/apartments is accomplished:
Inter-Object Communication
Understanding Custom Marshaling Part 1
To do what you are asking for, without resorting to implementing custom marshaling, you would have to make one of the processes act as an out-of-process COM server, and then the other process can use CoCreateInstance() or GetActiveObject() to obtain an interface pointer to the server's object that works within its local address space, and let COM handle the marshaling details for you.
It can't be done directly, but you can use a Client-Server service framework, which may be interface based.
For instance, see the last feature of our Open Source mORMot framework: Interface based services sample code and this link.
You can execute an interface on a remote process. The feature handles all communication means of the framework, i.e. in-process call, GDI messages, named pipes and TCP/HTTP. Internally it will use WM_COPYDATA for GDI messages, then transmit the parameters and results as JSON. Use this link to download the source code (use the http://synopse.info/fossil 1.16+ version) and the documentation (there are several pages about how to implement those services).
It is an Open-Source project, working with Delphi 6 up to XE2.
You can also expose your interface with a SOAP or DataSnap Client-Server (if you have the corresponding version of Delphi), or n-Tier commercial packages (like http://www.remobjects.com/da). This is similar to the method implemented in mORMot.
COM is also a good candidate, native to Windows, but it is more difficult to initialize: you'll have to register the COM on each PC (with administrator rights), and you won't be able to make it work over a network (DCOM is deprecated, remember). COM is good if you want your service to be shared with other languages, like .Net, but only locally.
I have a software architecture problem.
I have to design an IOS application which will communicate with a Linux application to get the state of a sensor, and to publish an actuator command. The two applications run in a Local network with an Ad-Hoc WiFi connection between the IOS device and the Linux computer.
So I have to synchronize two values between two applications (as described in figure 1). In a Linux/Linux system, I resolve this kind of problem thanks to any publisher / subscriber middleware. But how can I solve this problem in an IOS / Linux world ?
Actually the Linux application embed an asynchronous TCP Server, and the IOS application is an asynchronous TCP client. Both applications communicate through the TCP Socket. I think that this method is a low level method, and I would like to migrate the communication layer to a much higher level Service based communication framework.
After some bibliographic research I found three ways to resolve my problem :
The REST Way :
I can create a RESTful Web Service which modelize the sensor state, and which is able to send command to the actuator. An implementation of a RESTful web service client exists for IOS, that is "RESTKit", and I think I can use Apache/Axis2 on the server side.
The RPC Way :
I can create on my Linux computer a RPC service provider thanks to the libmaia. On the IOS side, I can use xmlrpc (https://github.com/eczarny/xmlrpc). My two programs will communicate thanks to the service described in the figure below.
The ZeroConf way :
I didn't get into detail of this methods, but I suppose I can use Bonjour on the IOS side, and AVAHI on the linux side. And then create custom service like in RPC on both side.
Discussion about these methods :
The REST way doesn't seem to be the good way because : "The REST interface is designed to be efficient for large-grain hypermedia data transfer" (from the Chapter 5 of the Fielding dissertation). My data are very fined grain data, because my command is just a float, and my sensor state too.
I think there is no big difference between the ZeroConf way and the RPC Way. ZeroConf provide "only" the service discovering mechanism, and I don't need this kind of mechanism because my application is a rigid application. Both sides knows which services exists.
So my question are :
Does XML RPC based method are the good choice to solve my problem of variable synchronization between an iPhone and a Computer ?
Does it exist other methods ?
I actually recommend you use "tcp socket + protobuf" for your application.
Socket is very efficient in pushing messages to your ios app and protobuf can save your time to deliver a message instead of character bytes. Your other high level proposal actually introduces more complications...
I can provide no answers; just some things to consider in no particular order.
I am also assuming that your model is that the iOS device polls the server to synchronize state.
It is probably best to stay away from directly using Berkeley sockets on the iOS device. iOS used to have issues with low level sockets not connecting after a period of inactivity. At the very least I would use NSStream or CFStream objects for transport or, if possible, I'd use NSURL, NSURLConnection, NSURLRequest. NSURLConnection's asynchronous data loading capability fits well with iOS' gui update loop.
I think you will have to implement some form of data definition language independent of your implementation method (RES, XML RPC, CORBA, roll your own, etc.)
The data you send and receive over the wire would probably be XML or JSON. If you use XML you would have to write your own XML document handler as iOS implements the NSXMLParser class but not the NSXMLDocument class. I would refer JSON as the JSON parser will return an NSArray or NSDictionary hierarchy of NSObjects containing the unserialized data.
I have worked on a GSOAP implementation that used CFStreams for transport. Each request and response was handled by a request specific class to create request specific objects. Each new request required a new class definition for the returned data. Interactivity was maintained by firing the requests through an NSOperationQueue. Lots of shim here. The primary advantage of this method was that the interface was defined in a wsdl schema (all requests, responses, and data structures were defined in one place.
I have not looked at CORBA on iOS - you would have to tie in C++ libraries to your code and change the transport to use CFStreams Again, lots of shim but the advantage of having the protocol defined in the idl file. Also you would have a single connection to the server instead of making and breaking TCP connections for each request.
My $.02
XML RPC and what you refer to as "RESTful Web Service" will both get the job done. If you can use JSON instead of XML as the payload format, that would simplify things somewhat on the iOS side.
Zeroconf (aka bonjour) can be used in combination with either approach. In your case it would allow the client to locate the server dynamically, as an alternative to hard-coding an URL or other address in the client. Zeroconf doesn't play any role in actual application-level data transfer.
You probably want to avoid having the linux app call the iOS app, since that will complicate the iOS app a lot, plus it will be hard on the battery.
You seem to have cherry picked some existing technologies and seem to be trying to make them fit the problem.
I would like to migrate the communication layer to a much higher level Service based communication framework
Why?
You should be seeking the method which meets your requirements in terms of available resources (should you assume that the client can maintain a consistent connection? how secure does it need to be?) However besides functionality, availability and security, the biggest concern should be how to implement this with the least amount of effort.
I'd be leaning towards the REST aproach because:
I do a lot of web development so that's where my skills lie
it has minimal dependencies
there is well supported code implementing the protocol stack at both ends
it's trivial to replace either end of the connection to test out the implementation
it's trivial to monitor the communications (if they're not encrypted) to test the implementaiton
adding encryption / authentication does not change the data exchange
Regards your citation, no HTTP is probably not the most sensible for SCADA - but then neither is iOS.