I've been looking for a good general purpose binary network protocol definition framework to provide a way to write real-time game servers and clients (think World Of Warcraft or Quake III) in multiple languages (e.g. Java backend server and iPhone front-end client written in Objective-C and Cocoa).
I want to support Java Flash clients, iPhone clients and C# clients on windows (and XNA clients on XBOX).
I'm looking for a way to efficiently send/receive messages over a TCP/IP or UDP socket stream connection. I'm not looking for something that can be sent over an HTTP Web Service, like JSON or XML marshalled Objects. Although Hessian's binary web service protocol is a very interesting solution
I want a network protocol format and client/server basic implementation that will allow a client to connect to a server and send any message in the defined protocol and receive any message in the protocol without having to bind to some kind of RPC endpoint. I want a generic stream of any message in my protocol incoming and outgoing. This is so that I can support things like the server sending all clients the positions of various entities in the game every 100 milliseconds.
The network protocol frameworks I've found are as follows:
Google's Protocol Buffer - but it lacks support for things like sending/receiving arbitrary messages from your given protocol.
Apache Thrift - an interesting option but it is geared mainly towards RPC instead of generic game client/server socket type connections where the client or server can send messages at any time and not just in response to a client RPC request.
Raknet Multiplayer - Raknet provides full multiplayer network library (it's free for indie development with revenue under $250k)
UPDATE : OculusVR Acquired RakNet and its Free/OpenSource now. U can find it on Github
Hessian Binary Web Service Protocol - is a HTTP web service binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments.
Raknet provides a good game/simulation oriented multiplayer library.
Apache Thrift and Google's protocol buffers seem to be the simplest approaches to using in a game network protocol client/server architecture.
Hessian seems like a great fit if you want to create a web based game server with a Java or flash client using some type of server push technology like COMET. Hessian might provide a really interesting way to support real-time games on the web and even be able to host them on VM web solutions like Google's App engine or Amazon's EC2.
There's some discussion about using various protocol definition frameworks for games and other uses:
Comparison of Various Serialization Frameworks
Thrift vs Protocol Buffers - Thrift is declared the better framework because it has a fully supported RPC client/server implementation
Using Protocol Buffers for client server Game API determining what type of message to decode
Bi-Directional RPC using thrift
DIS
If you do go the route of writing your own protocol, you may want to read the answer I posted here.
In summary it discusses what you should think about when writing a protocol, and list a few tricks for versioning and maintaining backwards and forward compatibility.
If you are really concerned about multiple platforms and language, be sure to take into account endian issues. A binary protocol designed for this use must use network-byte-order, so it needs custom per-data-type serialization functions; you cannot just blindly push C structs into network buffers.
A common solution for this problem at game companies is to have protocol description language or specification in a simple format like XML or python or lua, and then have code generation for each target language that generated packet classes with both data structure and serialization. This specification could use a type system that starts with basic types, then extends to include game-specific types with semantic information, enumerations or more complex structures. For example a data file could look like:
Attack = {
source = 'objectId',
target = 'objectId',
weapon = 'weapon::WEAP_MAIN',
seed = 'int'
}
This could generate code like:
#define PT_ATTACK 10002
class PacketAttack : public Packet {
public:
PacketAttack () : m_packetType(PacketAttack::s_packetType) {}
ObjectId m_source;
ObjectId m_target;
WeaponType m_weapon;
int m_seed;
bool Write(Stream* outStream) {
Packet::Write(outStream);
outStream << m_source;
outStream << m_target;
outStream << m_weapon
outStream << m_seed;
}
bool Read(Stream* inStream);
static const int s_packetType;
};
This does require some more infrastructure.. streams, packet base classes, safe serialization functions..
I want to echo Bill K's suggestion. It's not hard to roll your own protocol.
For the iPhone side, have a look at AsyncSocket which support for delimiter based TCP packets built in, and it's not hard to build a solution which uses packet headers.
If you quickly want to have a testserver to play against AsyncSocket on the iPhone, you can look at Naga (for the java server part) which has ready made stuff both for delimiter based packets and packets with headers. Naga was partially written with networked games in mind.
I disagree with "roll your with simple delimited strings approach": question is, what exactly would be the benefit? Getting to write and maintain more code?
The only reasons I could see would be lack of tool support (writing for some odd platform), or specific (very) hard performance or message size constraints.
Or, sometimes, really wanting to write a format -- that's ok, but it must be an explicit reason.
Depending on exact needs I would suggest considering JSON, since it can read and write arbitrary messages; has good object binders for Java (just like xml), is easier to read than binary formats, and is all around "good enough" for many use cases.
If message size is very important, Protobuf can work well -- while its size is not always as small as gzipped alternatives (gzip+xml, gzip+json compress very well), it's usually close.
ASN.1 fits the definition of "good general purpose binary network protocol definition framework". It's also standardized by ITU-T, so there's a lot of existing tools and libraries for various languages.
The DER encoding is suitable for efficient network communications, the XER encoding for human-readable (and writable) permanent storage.
Because you want to use different languages and also because you want something clean/small, I suggest the protocol buffers of google. You need a pre-compile part for the RPC but I really think that's the best option when you begin to mix different languages.. Here's the link: http://code.google.com/apis/protocolbuffers/docs/overview.html
Why not implement UDP directly? Your question mostly mentions what you don't want.. What further form of abstration do you want on top of UDP?
Download the Quake III sourcecode and see how they frame game updates over UDP?
The IP protocol has been designed to support multiple devices/OSes in a uniform way, isn't this what you ask for?
What protocol has implementations across a huge range of systems, hmm, IP perhaps?
Related
I'm trying to write a simple DPI (Deep Packet Inspection) tool. In my current phase, I need to develop a H.323 protocol detector. To aim this goal I need a H.225 / H.245 packet generator. I seems that almost all tool use SIP and RTP & RTCP protocol. Does anybody have any idea which tool use H323 protocols in VOIP?
There is number of h323 libraries, some of them are openh323 and openoh323/openooh323.
But ALL libraries have issues and alot of bugs. Main issue - protocol is VERY complex if compare with sip or other modern protocols and ALL vendors do "vendor-specific" addons/changes in protocol way they think "better for client".
As result it is hard to work with any tools. Easy way detect is just get signatures like 'h323', 'rtp' etc.
I would like to understand the way SWIFT messages work. I tried to understand from Google and it says that it uses packet switched network mechanism.
It would be of great favour if somebody can throw light on the following questions
Whether SWIFT by itself a communication protocol like AS2
For Trade finance system if I use IMEX does using a SWIFT message requires any Adapter kind of components with IMEX
Architecture of SWIFT
How SWIFT works is a big subject, but it's basically a proprietary protocol driven by the needs of the SWIFT community.
Since it all started in the early days of computing, the core stuff uses the UNIX philosophy of: write programs to do one thing, do it well, and use text as your inter program communication.
So, to take FIN messaging, SWIFT works using ASCII text files across a secure network.
To be able to send and receive these text files you need to join SWIFT. See http://www.swift.com/ordering/join_swift
The architecture is proprietary, but, once you join SWIFT you can send and receive specifically formatted text files, or, in the "next generation" it's XML but, to me, that is trying to re-invent the UNIX philosophy, which is bound to fail, and be very expensive in the process. But apparently this is what clients wanted...........?
So there you have it...
Basically I have a Go server which parses some XML file and creates a structure with the data which I want to send over TCP to an IOS app.
At the moment I tried to use Protocol Buffers, but I am having problems with this on the IOS side. So from your experience is this the best way I should transmit my data ? Any other suggestions ?
The first thing I am looking for it's speed.
There is no best way. There are plenty of cross-platform serialization formats out there. Just pick the one you are the most comfortable with, and which suits your needs.
An obvious choice is json (human readable and well supported on all platforms).
If you favor speed, then have a look at msgpack:
http://msgpack.org/
https://github.com/msgpack/msgpack-go
https://github.com/msgpack/msgpack-c
https://github.com/msgpack/msgpack-objectivec
If I send a packet containing address, port and other stuff, would there be any alignment problem on the other side? (using socks 5 protocol)
You don't mention which language you're programming in. If you're using C or C++, there are functions like htons, htonl, ntohs, ntohl and more to standardize network data elements into a transportable format. This guide (among many others) will tell you how to write network portable code so differences in endianness will not bite you in the #ss.
A tutorial is talking about cin and cout:
"Syntactically these streams are not used as functions: instead, data are written to streams or read from them using the operators <<, called the insertion operator and >>, called the extraction operator."
What is a 'stream'?
Consider a "Stream" as a physical hose, or pipe. At one end, someone may pour some water in. At the other end, it will come out. This is 'reading' and 'writing' to the stream.
A stream is just a place where data goes. It can be a 'socket stream' (over the internet) or a 'file stream' (to a file), or perhaps a 'memory stream', just data written to a place in-memory (ram).
A "stream" is an object that represents a source of data, or a place where data can be written.
Examples include file handles and pipes - things that you can read data from or write data to.
An important property of streams is that they share a common interface, so the same code can write to either a file or a pipe (for instance) without needing to be rewritten.
You should look at streams as abstractions on underlying 'sources' or 'sinks' of data. A source is something you read data from, and a sink is something you write data to.
The concept of streams allows you to perform I/O on various forms of media, network connections, pipes between applications, files, etc.
The stream abstraction is very valuable to us as developers as it allows us to simplify input and output, and it gives us the flexibility to arrange and reconnect the sources and destinations of these streams.
A good analogy is that of a hose. You can send and receive data through hoses, and you can connect these hoses to various things.
By allowing programs to talk through hoses, we allow all sorts of programs to talk to each other, and we increase interoperability and utility vastly.
This is at the heart of the UNIX philosophy, and supports some very powerful programming idioms.