Good evening.
I'm looking for a method to share data from my application system-wide, so that other applications could read that data and then do whatever they want with it (e.g. format it for display, use it for logging, etc). The data needs to be updated dynamically in the method itself.
WMI came to mind first, but then you've got the issue of applications pausing while reading from WMI. Additionally, i've no real idea how to setup my own namespace or classes if that's even possible in Delphi.
Using files is another idea, but that could get disk heavy, and it's a real awful method to use for realtime data.
Using a driver would probably be the best option, but that's a little too intrusive on the users end for my liking, and i've no idea on where to even start with it.
WM_COPYDATA would be great, but i'm not sure if that's dynamic enough, and whether it'll be heavy on resources or not.
Using TCP/IP would be the best choice for over the network, but obviously is of little use when run on a single system with no networking requirement.
As you can see, i'm struggling to figure out where to go with this. I don't want to go into one method only to find that it's not gonna work out in the end. Essentially, something like a service, or background process, to record data and then allow other applications to read that data. I'm just unsure on methods. I'd prefer to NOT need elevation/UAC to do this, but if needs be, i'll settle for it.
I'm running in Delphi 2010 for this exercise.
Any ideas?
You want to create some Client-Server architecture, which is also called IPC.
Using WM_COPYDATA is a very good idea. I found out it is very fast, lightweight, and efficient on a local machine. And it can be broadcasted over the system, to all applications at once (to be used with care if some application does not handle it correctly).
You can also share some memory, using memory mapped files. This is may be the fastest IPC option around for huge amount of data, but synchronization is a bit complex (if you want to share more than one buffer at once).
Named pipes are a good candidates for local. They tend to be difficult to implement/configure over a network, due to security issues on modern Windows versions (and are using TCP/IP for network communication - so you should better use directly TCP/IP instead).
My personal advice is that you shall implement your data sharing with abstract classes, able to provide several implementations. You may use WM_COPYDATA first, then switch to named pipes, TCP/IP or HTTP in order to spread your application over a network.
For our Open Source Client-Server ORM, we implemented several protocols, including WM_COPY_DATA, named pipe, HTTP, or direct in-process access. You can take a look at the source code provided for implementation patterns. Here are some benchmarks, to give you data from real implementations:
Client server access:
- Http client keep alive: 3001 assertions passed
first in 7.87ms, done in 153.37ms i.e. 6520/s, average 153us
- Http client multi connect: 3001 assertions passed
first in 151us, done in 305.98ms i.e. 3268/s, average 305us
- Named pipe access: 3003 assertions passed
first in 78.67ms, done in 187.15ms i.e. 5343/s, average 187us
- Local window messages: 3002 assertions passed
first in 148us, done in 112.90ms i.e. 8857/s, average 112us
- Direct in process access: 3001 assertions passed
first in 44us, done in 41.69ms i.e. 23981/s, average 41us
Total failed: 0 / 15014 - Client server access PASSED
As you can see, fastest is direct access, then WM_COPY_DATA, then named pipes, then HTTP (i.e. TCP/IP). Message was around 5 KB of JSON data containing 113 rows, retrieved from server, then parsed on the client 100 times (yes, our framework is fast :) ). For huge blocks of data (like 4 MB), WM_COPY_DATA is slower than named pipes or HTTP-TCP/IP.
Where are several IPC (inter-process communication) methods in Windows. Your question is rather general, I can suggest memory-mapped files to store your shared data and message broadcasting via PostMessage to inform other application that the shared data changed.
If you don't mind running another process, you could use one of the NoSQL databases.
I'm pretty sure that a lot of them won't have Delphi drivers, but some of them have REST drivers and hence can be driven from pretty much anything.
Memcached is an easy way to share data between applications. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects).
A Delphi 2010 client for Memcached can be found on google code:
http://code.google.com/p/delphimemcache/
related question:
Are there any Caching Frameworks for Delphi?
Googling for 'delphi interprocess communication' will give you lots of pointers.
I suggest you take a look at http://madshi.net/, especially MadCodeHook (http://help.madshi.net/madCodeHook.htm)
I have good experience with the product.
Related
I am building framework for realtime web applications. I started to do it in Elixir, because
it is modern way how to develop application for Erlang VM. Erlang should be good if you need concurrency, fault tolerant, scalable apps (something like web server etc.). That is exactly what i need.
Question: Realtime framework always need for instance keep information about who is interested in what. This will be accomplished by using publish/subscribe pattern. So i will have 1000 clients subscribing to topic "newest-message". I need to save those clients (pid of process representing each client) somewhere to later access them if content for topic "newest-message" appears.
This is where i am confused if Erlang is really good for my framework.
ETS is probably the only option where to store shared data, but ETS is always copying everything if you save/access records. So that means copy 1000 pids always when i need to access them (instead of just iterating over some list, if i will do it for instance in c/java/python).
This will be probably great bottleneck if still copying many and many records from ETS (many clients, many subscriptions etc), i am right?
Sharing the state may be a sign of bad design. You can for example have process for each queue/topic and it will store its own list of subscribers. You send a message to that topic process and it in turn sends the message to clients. This way, you don't copy entire subscriber list.
If you need to process them in parallel, you can split the subscriber list between more processes.
The fault tolerance of Erlang is achieved, because it doesn't let you share state and you have to put more thought to the design, that will not involve state sharing, but will be efficient. This will pay off in the long run, so Erlang/Elixir is definitely good language for this kind of apps. Just look at RabbitMQ.
In my opnion, if you plan to save states like "who is interested in what" Erlang alone may not be a good idea. Of course, sometimes it is very convenient to pass everything in signals (like you'd do in Erlang), but when there is much content to store - lack of state in Erlang starts to hinder you rather than help.
On the other hand, you can keep a broad piece of convenience of Erlang and use it with a Java application, for example. Erlangs interface for Java enables you to connect both technologies quite easily, and at the same time you can use a Java app to store information for you (and save them somewhere, when necessary) and Erlang for the whole concurrent signaling real time part. Even better than that: you can still implement OTP with architecture like that, so you can create quite a lightweight application (because real-time logic is done by Erlang for you) being able to access stored data easily (because Java helps you here).
Obviously there are multiple steps and phases of implementing such a thing.
I was thinking I would eventually have a webserver that takes http json requests from the ios app, and then queries the cassandra backend and sends results back. I could load balance and all that fancy stuff still, and also provide a logical layer on server side, and keep the client app lightweight.
I'm not sure i understand how cassandra clients fit though. It seems like the cassandra objective c client could eliminate the need for the above approach.
I saw another question and answer but it wasnt clear, perhaps because it varys on the need.
An iPhone app should not directly connect to a Cassandra backend or any other DB store.
First of all, talking to a database often requires adapting a very specific binary protocol (for Cassandra in particular, binary CQL or Thrift). Writing an adapter that would let your Objective-C app communicate in this binary protocol is a major piece of work, and could easily cost more than the rest of your app in effort. If you hide the DB behind a web-server, however, you will be able to select from a variety of existing adapters available in different server-side languages, meaning that you don't need to redo all that low-level work. You'll only be responsible for a relatively small piece of server-side code that would translate your REST queries and forward them to one of the Cassandra adapters (which expose easy-to-use interfaces).
Secondly, if you wanted to connect to a remote database from the phone, your database server would have to open its ports to the internet at large, which is a very bad security practice, even if you use SSL and user credentials. Again, if you hide behind a web server, you will be putting in a layer of technology that has evolved for decades to remain secure on the public internet.
Finally, having your phone talk to Cassandra directly is a poor architectural pattern. When you write apps that communicate on the internet, you want them to know as little as possible about each other, only how to talk to each other (preferably in a standard protocol). That way you can replace or upgrade individual components while keeping everything else the same. This may not sound like a lot, but is actually the main reason why phones, or web browsers, don't directly talk to databases. (If this setup were a good idea in principle, the first two problems could be easily solved given enough engineering effort.)
The approach you first suggested with JSON and the web server is the only correct way to go.
Use something like RESTful API, there are many reasons for that.
if your servers ip addresses change you have to update all client, if you add more nodes you will need to update all clients, if you decide to upgrade your cassandra and some functions change your clients will break and you need to update all clients.
I have a software architecture problem.
I have to design an IOS application which will communicate with a Linux application to get the state of a sensor, and to publish an actuator command. The two applications run in a Local network with an Ad-Hoc WiFi connection between the IOS device and the Linux computer.
So I have to synchronize two values between two applications (as described in figure 1). In a Linux/Linux system, I resolve this kind of problem thanks to any publisher / subscriber middleware. But how can I solve this problem in an IOS / Linux world ?
Actually the Linux application embed an asynchronous TCP Server, and the IOS application is an asynchronous TCP client. Both applications communicate through the TCP Socket. I think that this method is a low level method, and I would like to migrate the communication layer to a much higher level Service based communication framework.
After some bibliographic research I found three ways to resolve my problem :
The REST Way :
I can create a RESTful Web Service which modelize the sensor state, and which is able to send command to the actuator. An implementation of a RESTful web service client exists for IOS, that is "RESTKit", and I think I can use Apache/Axis2 on the server side.
The RPC Way :
I can create on my Linux computer a RPC service provider thanks to the libmaia. On the IOS side, I can use xmlrpc (https://github.com/eczarny/xmlrpc). My two programs will communicate thanks to the service described in the figure below.
The ZeroConf way :
I didn't get into detail of this methods, but I suppose I can use Bonjour on the IOS side, and AVAHI on the linux side. And then create custom service like in RPC on both side.
Discussion about these methods :
The REST way doesn't seem to be the good way because : "The REST interface is designed to be efficient for large-grain hypermedia data transfer" (from the Chapter 5 of the Fielding dissertation). My data are very fined grain data, because my command is just a float, and my sensor state too.
I think there is no big difference between the ZeroConf way and the RPC Way. ZeroConf provide "only" the service discovering mechanism, and I don't need this kind of mechanism because my application is a rigid application. Both sides knows which services exists.
So my question are :
Does XML RPC based method are the good choice to solve my problem of variable synchronization between an iPhone and a Computer ?
Does it exist other methods ?
I actually recommend you use "tcp socket + protobuf" for your application.
Socket is very efficient in pushing messages to your ios app and protobuf can save your time to deliver a message instead of character bytes. Your other high level proposal actually introduces more complications...
I can provide no answers; just some things to consider in no particular order.
I am also assuming that your model is that the iOS device polls the server to synchronize state.
It is probably best to stay away from directly using Berkeley sockets on the iOS device. iOS used to have issues with low level sockets not connecting after a period of inactivity. At the very least I would use NSStream or CFStream objects for transport or, if possible, I'd use NSURL, NSURLConnection, NSURLRequest. NSURLConnection's asynchronous data loading capability fits well with iOS' gui update loop.
I think you will have to implement some form of data definition language independent of your implementation method (RES, XML RPC, CORBA, roll your own, etc.)
The data you send and receive over the wire would probably be XML or JSON. If you use XML you would have to write your own XML document handler as iOS implements the NSXMLParser class but not the NSXMLDocument class. I would refer JSON as the JSON parser will return an NSArray or NSDictionary hierarchy of NSObjects containing the unserialized data.
I have worked on a GSOAP implementation that used CFStreams for transport. Each request and response was handled by a request specific class to create request specific objects. Each new request required a new class definition for the returned data. Interactivity was maintained by firing the requests through an NSOperationQueue. Lots of shim here. The primary advantage of this method was that the interface was defined in a wsdl schema (all requests, responses, and data structures were defined in one place.
I have not looked at CORBA on iOS - you would have to tie in C++ libraries to your code and change the transport to use CFStreams Again, lots of shim but the advantage of having the protocol defined in the idl file. Also you would have a single connection to the server instead of making and breaking TCP connections for each request.
My $.02
XML RPC and what you refer to as "RESTful Web Service" will both get the job done. If you can use JSON instead of XML as the payload format, that would simplify things somewhat on the iOS side.
Zeroconf (aka bonjour) can be used in combination with either approach. In your case it would allow the client to locate the server dynamically, as an alternative to hard-coding an URL or other address in the client. Zeroconf doesn't play any role in actual application-level data transfer.
You probably want to avoid having the linux app call the iOS app, since that will complicate the iOS app a lot, plus it will be hard on the battery.
You seem to have cherry picked some existing technologies and seem to be trying to make them fit the problem.
I would like to migrate the communication layer to a much higher level Service based communication framework
Why?
You should be seeking the method which meets your requirements in terms of available resources (should you assume that the client can maintain a consistent connection? how secure does it need to be?) However besides functionality, availability and security, the biggest concern should be how to implement this with the least amount of effort.
I'd be leaning towards the REST aproach because:
I do a lot of web development so that's where my skills lie
it has minimal dependencies
there is well supported code implementing the protocol stack at both ends
it's trivial to replace either end of the connection to test out the implementation
it's trivial to monitor the communications (if they're not encrypted) to test the implementaiton
adding encryption / authentication does not change the data exchange
Regards your citation, no HTTP is probably not the most sensible for SCADA - but then neither is iOS.
I am implementing some networking stuff in our project. It has been decided that the communication is very important and we want to do it synchronously. So the client sends something the server acknowledges.
Are there some general best practices for the interaction between the client and the server. For instance if there isn't an answer from the server should the client automatically retry? Should there be a timeout period before it retries? What happens if the acknowledgement fails? At what point do we break the connection and reconnect? Is there some material? I have done searches but nothing is really coming up.
I am looking for best practices in general. I am implementing this in c# (probably with sockets) so if there is anything .Net specific then please let me know too.
First rule of networking - you are sending messages, you are not calling functions.
If you approach networking that way, and don't pretend that you can call functions remotely or have "remote objects", you'll be fine. You never have an actual "thing" on the other side of the network connection - what you have is basically a picture of that thing.
Everything you get from the network is old data. You are never up to date. Because of this, you need to make sure that your messages carry the correct semantics - for instance, you may increment or decrement something by a value, you should not set its value to the current value plus or minus another (as the current value may change by the time your message gets there).
If both the client and the server are written in .NET/C# I would recommend WCF insted of raw sockets as it saves you a from a lot of plumbing code with serialization and deserialization, synchronization of messages etc.
That maybe doesn't really answer your question about best practices though ;-)
The first thing to do is to characterize your specific network in terms of speed, probability of lost messages, nominal and peak traffic, bottlenecks, client and server MTBF, ...
Then and only then you decide what you need for your protocol. In many cases you don't need sophisticated error-handling mechanisms and can reliably implement a service with plain UDP.
In few cases, you will need to build something much more robust in order to maintain a consistent global state among several machines connected through a network that you cannot trust.
The most important thing I found is that messages always should be stateless (read up on REST if this means nothing to you)
For example if your application monitors the number of shipments over a network do not send incremental updates (+x) but always the new total.
In a common think about network programming, I think you should learn about :
1. Socket (of course).
2. Fork and Threading.
3. Locking process (use mutex or semaphore or others).
Hope this help..
I have a server and some clients (about 50) in an intranet. The clients send short (about 40 character) string data to the server and the server answers with a similar string. There are up to (but not permanently) 2-3 requests per second for each client. The server has to serialize the requests to get the response strings.
The system should have as less as possible impact on the network as possible (i.e. the server may run something like a webserver already). It should be as easy to install and administer as possible.
What are the possibilities to achieve this using Delphi (Client: D7, Server up to D2010)?
I use the Synapse library for such a simple server. Its lightning fast, very light, and threads easily. The demo Echo in the main synapse install is a fantastic start for what your trying to do. If you are going to be performing database access inside each request/response thread then I strongly also suggest looking at the connection pool example by Cary Jensen to keep your database connections in check.
TCP, definitely. But I'd like to give a vote for ICS. Never liked Indy ...
What about Indy's TIdTCPServer and TIdTCPClient? They provide command handlers, which makes implementing text-based protocols very straight-forward.
There are a lot of options.
Ultimately, I agree with Smasher and like using sockets. They're quick, easy and portable. If you're dealing with a fairly simple protocol and don't need a full n-tier solution, creating a TCP or HTTP server application is dead simple, very light weight, and easy to make compatible with any client. You can even add SSL support to these stand alone applications without having to configure a web server or interfering with it, if it's already running on the same box.
I use RemObjects SDK for this sort of purpose. It takes care of all the difficult stuff, and I just ask it to connect and make function calls to pass the data.