I have some data that I need to share between multiple services on multiple machines. Stuffing the data into a database or shuffling it over http won't work in this situation and ideally the different pieces of software will need to communicate with each other directly (or through one central coordinator that can send and receive).
Is it recommended to create and implement a network protocol or use some tool to do the communication?
If I did go the route of creating a protocol myself, it wouldn't have to be very complex. Under 10 different message types, but it would have to be re-implemented in a few different languages for this project, and support unicode. I have read plenty (and done some) with handling sockets, but don't have much knowledge in handling a protocol I create. Are there any good resources on this?
There are also things like ICE and RPC that look intresting. The limit of my experience is using ICE and XMLRPC for a few days each. Is this the better route to go? If so what tools are out there?
Recently I've been using Google Protocol Buffers for encoding and shipping data between different machines running software written in different languages. It is quite easy to do, and takes away a lot of the hassle of designing a custom protocol.
Without knowing what technologies and platforms you are dealing with, it's difficult to give you a very specific answer - so I'll try to give you some general feedback.
If the system(s) you are wishing to connect span more than a single platform and/or technology you are probably better using an existing transport mechanism and protocol to maximize the chance your base platform will already have a library (or multiple) to interact over it. Also, integrating security and other features in a stack with known behaviors is more likely to be documented (with examples floating around). RPC (and ICE, though I've less familiarity with it) has some useful capabilities, but it also requires a lot of control over the environment and security can be convoluted (particularly if you are passing objects between different languages).
With regards to avoiding polling, this is a performance related issue; there are design patterns which can help you to handle such things - if you understand how you need the system to work (e.g. the observer pattern - kind of a dont-call-us-we'll-call-you approach). The network environment you are playing in will dictate which options are actually viable (e.g. a local LAN will have different considerations from something which runs over a WAN or the internet). Factors like firewall tunneling, VPN traversal, etc. should play part in your final selected technology profile.
The only other major consideration (that I can think of just now... ;-)) would be to consider the type of data you need to pass about. Is it just text, or do you need to stream binary objects? Would an encoding format (like XML or JSON or bJSON) do the trick? You mention "less than ten message types" as part of the question, but is that the only information which would ever need to be communicated by the system?
Either way, unless the overhead of existing protocols is unacceptable you're better of leveraging established work 99% of the time. Creativity is great - but commercial projects usually benefit from well-known behaviors, even if not the coolest or slickest (kind of the "as long as it works..." approach).
hth!
Related
I think about technology stack for my project and I think about using ejabberd. The project will look like classic multi-user dungeon RPG where players will move across the world from one location to another (locations looks exactly like chat rooms), and they also will figth each other as well as creatures with AI in turn-based mode.
I never used ejabberd, but I have some experience in writing server applications using erlang.
Is ejabberd an overkill for this kind of game? It has a lot of features that I won't need ever. However it is well-known to erlang developers and is also very stable and mature. Is ejabberd worth using it as kind of transport layer for my online game, or I should better invent my own wheel, something tiny and simple?
I have several years of commercial experience, using ejabberd for things like this. So, my take:
Pros:
It is certainly technically capable.
It is quite easy to grok.
It is very easy to extend and modify.
If you update it regularly, it will solve two really important aspects. A. network security (this is extremely important thing for me); and B.properly done authentication. These two alone are enough of a reason to use it.
It is surprisingly fast.
It gives you chat, presence and friends list for free.
It gives you MUC (rooms) for free. With all the things like permissions solved quite well.
Cons:
Don't really expect to find any usable documentation. Source is mostly your only friend.
Don't really expect to find a community. It's a lonely path. There is a room - ejabberd#conference.jabber.ru but it's very quiet (and almost empty). Most of the people there are not developers, but just ejabberd users. Mailing list is a bit better, but usually not enough to find you the answer you seek.
The source code per-se is not the best example of an erlang project. If you want to learn how to write big, modular, distributed erlang software, better have a look at something like Riak.
The internal APIs are not very stable (they change quite a lot with releases). Because of this, I recommend writing your software as a separate erlang application, connecting to ejabberd as an external XMPP component. Thus you will be guaranteed that you communicate over a stable protocol (XMPP). Of course, you cannot escape having to write some internal stuff as well. Authentication and Roster (friend list) modules are the first that come to mind. This combination is quite hard to maintain and update, especially if you need hot code loading, but it's yet the best solution for me. Try to keep the "in-ejabberd" code to the viable minimum.
That being said, there is only one (to my knowledge) usable XMPP erlang library. It's called exmpp and is developed by the same company as ejabberd (ProcessOne). It's not yet considered stable. I have been using it for quite some time, and for now there are no problems, but you never know. It is also mostly undocumented (or was, when I was learning it).
I understand that Apache Thrift and ZeroMQ are softwares belonging to different categories, and it is not easy to do a comparison since it is an apple to orange comparison. But I don't know why they belong to different categories. Aren't they both used to pass data between different services, which may or may not be written in different languages?
When should I use Thrift and when should I use a message queue?
They belong to different categories primarily because they are targetted at different audiences with different concerns. Therefore they are better at different things.
Apache Thrift similar to Google Protocol Buffers is intended to be high-level, reasonably well abstracted means to send data between processes on different machines, possibly in different languages. They purposefully provide an IDL-ish layer to describe the message, perhaps with automatic or semi-automatic versioning and optional sections.
ZeroMQ specifically on the other hand, not message queues in general (which would be an entirely separate question), is all about speed. They efficiently move bytes to the other end. As few stops along the way as possible. As such, you are responsible for serialization, versioning, or whatever else is important to you the developer. Of course, this can mean complexity, particularly if you are communicating between different platforms and languages, but that's part of the penalty for lack of abstraction.
Which to choose? Depends on your project. If you don't need absolute raw performance, a higher level toolkit will likely serve your purpose just fine. If you are building a high-speed low-latency application, you're going to end up closer to the metal anyways.
Good Luck
Thrift defines how to represent complex data so that it can be written and read by different languages (hence it has IDL to define types that will be transported). It also defines simple means to transport such formatted message between two end points (aka thirft transport).
On the other hand ZeroMQ shines in ways you can transport message between endpoints in order to acquire different behaviors like one to one, one to many, many to many, and different expectations of speed and reliability of such transfers. And as for message itself it is just a blob to ZeroMQ, and apps should find a way to encode decode them.
So if you have complex data structures but simple messaging patterns, you might lean on the thrift side. If you have simple data but complex messaging patterns, you might lean on the ZeroMQ or something like that (AMQP).
And if you need both, you could use THrift and ZeroMQ in pair, thrift to format the message, and ZeroMQ to transport it.
Davorin mentioned using Thrift and ZeroMQ in pair and in case you're interested in that approach checkout the Thrift codebase and look under thrift/contrib/zeromq for a demo of Thrift using ZermoMQ.
I am trying to design an event driven system where the elements of the system communicate by generating events that are responded to by other components of the system. It is intended that the components be independent of each other - or as largely independent as I can make them. The system will initially be implemented on Windows 7, and is being written in Delphi. The generated events will be generated by the Delphi code. I understand how to implement a system of the type described on a single machine.
I wish to design the system so that it can readily be deployed on different machine architectures in particular with different components running on a distributed architecture, which may well be different to Windows 7. There is no requirement for the system ever to communicate with any systems external to itself.
I have tried investigating the architecture I need to consider and have looked at the questions mentioned below. These seem to point towards utilising named pipes as a mechanism for inter-hardware communications. As a result of these investigations I have sketched out the following to describe my system - the first part of the diagram is the system as I am developing it; the second part what I have deduced I would need for possible future implementations.
This leads to the following points:
Can you pass events via named pipes?
Is this an appropriate and sensible structure to tackle this problem?
Are there better alternatives?
What have I forgotten (at this level of granularity)?
How is event driven programming implemented?
How do I send a string from one instance of my Delphi program to another?
EDIT:
I had not given the points arising from "#I give crap answers" response sufficient consideration. My initial responses to his points are:
Synchronous v Asynchronous - mostly asynchronous
Events will always be in a FIFO queue.
Connection loss - is not terribly important - I can afford to deal with this non-rigourously.
Unbounded queues are a perfectly good way of dealing with events passed (if they can be) - there is no expectation of large volume of event generation.
For maximum deployment flexibility (operating-system independent), I recommend to take a look at popular open source message brokers which run on the Java platform. Using standard protocols. they integrate well with Delphi and other programming languages, can be used with web applications, and have a large installed user base and active community.
They are quite easy to install and configure in a few minutes, and free / commercial clients for Delphi are available.
Some examples are:
Apache ActiveMQ
OpenMQ
JBoss HornetQ
I also recommend the book "Enterprise Integration Patterns" by Martin Fowler as an overview and introduction, with many simple recipes to handle specific problems.
Note that I am a developer of commercial Delphi clients for enterprise messaging systems, such as xmlBlaster, RabbitMQ, Amazon Simple Queue Service and the three brokers mentioned above.
I can only answer for your point 4 here: You have not yet decided if an event is synchronous or asynchronous. In the async case, you have to decide what to do when messages arrive. Do you have a queue? How big is the queue? Can one grab arbitrary elements in the queue or is it strictly FIFO. What happens if a message is lost (somebody axes the network cable)?
In the sync variant, the advantage is that you got delivery guarantees, but then what do you do when connections are suddenly lost?
Connection loss is going to be a problem. The more machines you have, the greater is the chance that they will occur. Decide how you will handle that.
Another trouble may be what you do if you have a large event and several small. Is the order of transfer FIFO or smallest-first? Can events be reeordered? What are the assumptions here?
The aside is that I hack Erlang a lot. In Erlang all the event-handling is already solved but it also means a specific model is chosen for you (async, unbounded queues, no guaranteed delivery, but detection of connection loss).
I suggest to look at RabbitMQ, http://www.rabbitmq.com/. It has the server and client. Just need some wrapper codes in delphi and you are ready to build your business logic
Cheers
This is probably just an application for a message queue.
http://msdn.microsoft.com/en-us/library/ms632590(v=vs.85).aspx
We've got a bunch of data that we'd like to expose to the world hosted on an asp-net.mvc website. I'd like to ensure that we deliver it using technology that is easy for end developers to implement and not tied to any particular platform, rather than using technology that is unpopular/incompatible with developers.
The kind of requests we expect are mainly to retrieve search results (not many parameters), but down the like we'd like to be able to provide catalogue lookups and the like, which may be more complex.
Bearing this in mind, what is the preferred means of doing this?
Windows Communication Foundation can be used to create both SOAP services (great if your consumers are businesses, using Visual Studio/.NET or Java) or REST services (for people on other platforms). Those are the preferred means of exposing public APIs.
If you want maximum exposure, probably best to use the REST approach, since it is easier to consume from "web" languages like JavaScript. Microsoft has extensive resources on putting together a REST API using WCF.
Honestly, for the kinds of requests you say you need to handle, which all seem to be looking up data as opposed to modifying it, the difference is almost trivial - you can switch from SOAP to REST simply by changing a few attributes/configuration options and you could technically even host both at the same time using very little additional code. As long as you stick to WCF and don't use outdated technology like ASMX/WSE then you will be fine.
Reasons to use REST:
Consumable from almost anywhere (including JavaScript, RSS readers, etc.);
It's popular (in use by Google, Twitter, etc.)
Supports many different data formats (JSON, Atom, etc.)
Reasons to use SOAP:
Standardized security protocol (encryption, non-repudiation, etc.)
Distributed transactions
Message Queuing
That's not an exhaustive list but it should give you an idea of who the target markets are for each. If you're hosting a very open, very public site designed to be consumed by anyone and everyone, go with REST. If the service is part of a business system and you need to guarantee reliability, security, and consistency of data, you'll want to go with SOAP. Choose the appropriate technology based on your target market.
Create a RESTful API. As a developer who often consumes web services, it's what I would expect and prefer.
Many popular services (digg/twitter/netflix/google) are moving to REST over SOAP, so you would be wise to follow suit.
If you do create a REST API you should also create a WADL file. It's WISDL for REST. They're not well supported yet, but they're not hard to create and they'll become more useful as support increases.
YOu will want to check out odata. Look at odata.org and live.visitmix.com/videos
This will give you REST access, metadata support like in SOAP, interoperability with the whole office stack and if you are using WCF Data Services you can implement it in a matter of hours, days at most.
Take a look at netflix.com, they have done it right (IMHO).
I am considering Erlang as a potential for my upcoming project. I need a "Highly scalable, highly reliable" (duh, what project doesn't?) web server to accept HTTP requests, but not really serve up HTML. We have thousands of distributed clients (other systems, not users) that will be submitting binary data to central cluster of servers for offline processing. Responses would be very short, success, fail, error code, minimal data. We want to use HTTP since it is our best chance of traversing firewalls.
Given this limited information about the project, can you provide any weaknesses that might pop up using a technology like Erlang? For instance, I understand Erlang's text processing capabilities might leave something to be desired.
You comments are appreciated.
Thanks.
This sounds like a perfect candidate for a language like Erlang. The scaling properties of the language are very good, but if you're worried about the data processing abilities, you shouldn't be. It's a very powerful language, with many libraries available for developers. It's an old language, and it's been heavily used/tested in the past, so everything you want to do has probably already been done to some degree.
Make sure you use erlang version R11B5 or newer! Earlier versions of erlang did not provide the ability to timeout tcp sends. This results in stalled or malicious clients being able to execute a DoS attack on your application by refusing to recv data you send them, thus locking up the sending process.
See issue OTP-6684 from R11B5's release notes.
With Erlang the scalability and reliability is there but from your project definition you don't outline what type of text processing you will need.
I think Erlang's main limitation might be finding experienced developers in your area. Do some research on the availability of Erlang architects and coders.
If you are going to teach yourself or have your developers learn it on the job keep in mind that it is a very different way of coding and that while the core documentation is good a lot of people do wish there were more examples. Of course the very active community easily makes up for that.
I understand Erlang's text processing
capabilities might leave something to
be desired.
The starling project already provides basic unicode support and there is a EEP (Erlang Enhancement Proposal) currently in draft, but going in to bring it into the mainstream of Erlang/OTP support.
I encountered some problems with Redis read performance from Erlang. Here is my question. I tend to think the reason is Erlang-written module, which has troubles while processing tons of strings during communication with Redis.