Im slowly getting my head around asn1 notation. Encoding asn1 from an struct seems to quite easy.However decoding asn1 to my application is harder. I want to know if I have an application that receives data on say udp port 600, and this can be a number of asn1 structures, how do i tell which asn1 structure i should be decoding into?
So, when i receive asn1 packets, do we first determine it's type and the decode it depending on its type?
or does my asn1 compiler handle of this for me?
finding examples of this stuff is hard...even for google ;-) .
A tutorial that outlines the building of simple network server using asn1 would be awesome!
ASN.1 corresponds roughly to the Presentation Layer of the OSI seven-layer cake. It relies on the Application Layer to determine what types of information is being exchanged between the endpoints. So, there's no universal header or identifier that would indicate which protocol or syntax is being presented by the ASN.1 stream. That could be implied by the port number, or made explicit by an additional protocol layer (e.g., HTTP.)
EDIT (responding to your comment): The ASN.1 standard defines four "classes" of tags: UNIVERSAL, APPLICATION, PRIVATE, and CONTEXT-SPECIFIC. APPLICATION basically means that the tag is defined for use within a specific application (i.e., it's not pre-defined as part of ASN.1.) But an APPLICATION tag doesn't carry enough information to specify (or even hint at) which application is in use.
(Actually, there's very little semantic difference among "APPLICATION", "PRIVATE", and "CONTEXT-SPECIFIC"; these three classes are used mostly for historical and stylistic reasons.)
Related
Does anyone know what ZCD may refer to? It is described as a segment with a link back to PreManage for the patient!
Can anyone please provide more details?
The Z segments (segments those begin with the letter "Z") are custom segments. Those are not defined in specifications. They vary from vendor to vendor. Vendor may publish a document explaining usage of segment. Two connected parties should know in advance and decide the usage by mutual understanding.
As those are custom, and if there is no way to know what data they contain, it may be safe to neglect them hoping the sender have not put critical data in it.
Please refer to this:
Z-segments can be inserted anywhere in the HL7 message. A popular approach is to place the Z-segment within a group of segments that contain similar information, such as insurance. Z-segments are also often placed at the end of the message. The advantage of doing so is that this placement prevents systems configured to parse “standard” HL7 format from requiring any configuration modifications in order to process the message. The application simply reads the segments in the order expected and then extracts the data from the Z-segment (if needed) via parser modifications.
Working with unexpected Z-segments
Sometimes systems may send unexpected Z-segments, whether or not they were part of the original specifications. Even if you are not interested in the data in the Z-segment, you may still (depending on its location) need to take the segment into account while testing and developing your interface.
I have an erlang program which generates data. This data needs to be transferred via udp to a non-erlang program for further processing. I already have this part working - sending the data via udp and receiving it on the other non-erlang side.
Here's the problem. The data (erlang terms like tuples containing lists) doesn't seem to be able to go over "as is" (i.e. I can't just send arbitrary erlang terms). It apparently needs to be converted to either text or binary first. Converting to binary seems easy enough with a bif I found. The problem is, binary gobbledygook comes out the other side, and I don't know any easy way to decode it (the other side is non-erlang).
Barring someone telling me some easy way to decode binary gobbledygook on the other side, I'd like the data to be sent as a simplistic string representation of the terms - for instance a tuple like this:
{[1,2,3],[4,5,6]}
sent like this:
"{[1,2,3],[4,5,6]}"
I haven't seen any such bif, i.e. "convert_term_to_ascii/1" etc. I know I could scan it and send token representations of the terms, but I don't want to do that - decoding that on the other side is just a pain I don't want to deal with.
I know I'm not the first, second, or third person to have this problem. It has to be fairly common. How is it normally dealt with?
Can someone point me to some resource showing me how to either 1) convert binary gobbledygook to ascii (needed on the non-erlang side), or 2) straightforwardly convert terms to a string (needed on the erlang side)?
Or, tell me how I'm wrong and how I should really be doing this?
Thanks.
1) you can convert any term to string using
R= io_lib:format("~p",[yourtermhere]),
lists:flatten(R)
2) you might look at erlang external binary format, a lot of other languages have libraries for encode/decode that erlang binaries format. And in erlang you can encode any term by term_to_binary
I'd recommend converting the erlang terms into JSON, with either of known libraries (heard good words regarding rfc4267). It'd be a trivial task to convert JSON back with any non-erlang platform, I guess. )
My very old HL7 parser has just hit a snag as it is now getting some messages with a ZDS segment present. It was easy to fix by adding a ZDS object to my parser, but I am trying to find out what it is used for. Googling hasn't helped much. This is a sample
ZDS|PERFORM|p0001236^PATEL^ATEST^^^^^^HHB_INOP_PRSNL^^^^OTHER|20100714101800|CD:653
ZDS|TRANSCRIBE|p0001236^PATEL^ATEST^^^^^^HHB_INOP_PRSNL^^^^OTHER|20100714101800|CD:653
ZDS|SIGN|p0001236^PATEL^ATEST^^^^^^HHB_INOP_PRSNL^^^^OTHER|20100714101912|CD:653
So, I'm interested in what each field is though looking at this sample data, it seems I don't lose much by just dropping the whole segment.
In HL7, all segments that begin with the letter Z are considered to be custom and are not defined further by the HL7 standard. You will need to find out what system is responsible for generating these ZDS segments and ask the owners of that system to provide you their specification.
As Scott said, "Z" segments are custom and can vary from vendor to vendor. In the Cerner realm, however, ZDS segments are typically used for "Document Succession" purposes -- a means of document version tracking and synchronization between two supportive systems.
The ZDS segment is used to communicate document endorsement information (actions done or to be done) in Unsolicited Document Results. only a specific solution of Millennium use it, so if you don't need just ignore it.
I have an external device that spits out UDP packets of binary data and software running on an embedded system that needs to read this data stream, parse it and do somethign useful. The binary data gets logged to a file as well. I would like to write a parser that can easily take the input directly from either the UDP stream, or a file, parse the data into a specific format and then direct the output to either a file (e.g. matlab dat file) or to another process that will do some real time processing. Are there any resources that would help me with this and what is the best way to go about this? I think it might make sense to use C++ streams but I'm not familiar with creating custom output streams. Does this seem like a good approach to take or is there a better way to go about it?
Thanks.
The beauty of binary data is that its is generally of very fixed format.
A typical method of parsing it is to declare a structure that maps onto the received packets, and then to just use type-casts to read the fields as structure elements.
The beauty is that this requires no parsing.
you have to be careful about structure packing rules, and endian-ness to make the structure map exactly the same way. Use of the C "offsetof" and "sizeof" macros is useful to emit some debug info to check that your structure is indeed mapping to what you think it is mapping.
Packing rules can typically be altered either by directives (such as #pragma's) or command line options. Endian-ness you are stuck with. If its different from what your embedded system uses, declare all the fields as bytes, or use something like the "ntoh" macro to do the byte swapping.
The New Jersey Machine Code Toolkit is a scheme for decoding arbitrary binary patterns. It was originally designed for decoding instruction sets, but it ought to be just fine for decoding message formats. You provide a description of the binary format, it synthesizes code to access the fields of that format (when valid). THus you can refer to message fields using generated function calls rather than think about where the field is or how it is encoded.
I've been looking for a good general purpose binary network protocol definition framework to provide a way to write real-time game servers and clients (think World Of Warcraft or Quake III) in multiple languages (e.g. Java backend server and iPhone front-end client written in Objective-C and Cocoa).
I want to support Java Flash clients, iPhone clients and C# clients on windows (and XNA clients on XBOX).
I'm looking for a way to efficiently send/receive messages over a TCP/IP or UDP socket stream connection. I'm not looking for something that can be sent over an HTTP Web Service, like JSON or XML marshalled Objects. Although Hessian's binary web service protocol is a very interesting solution
I want a network protocol format and client/server basic implementation that will allow a client to connect to a server and send any message in the defined protocol and receive any message in the protocol without having to bind to some kind of RPC endpoint. I want a generic stream of any message in my protocol incoming and outgoing. This is so that I can support things like the server sending all clients the positions of various entities in the game every 100 milliseconds.
The network protocol frameworks I've found are as follows:
Google's Protocol Buffer - but it lacks support for things like sending/receiving arbitrary messages from your given protocol.
Apache Thrift - an interesting option but it is geared mainly towards RPC instead of generic game client/server socket type connections where the client or server can send messages at any time and not just in response to a client RPC request.
Raknet Multiplayer - Raknet provides full multiplayer network library (it's free for indie development with revenue under $250k)
UPDATE : OculusVR Acquired RakNet and its Free/OpenSource now. U can find it on Github
Hessian Binary Web Service Protocol - is a HTTP web service binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments.
Raknet provides a good game/simulation oriented multiplayer library.
Apache Thrift and Google's protocol buffers seem to be the simplest approaches to using in a game network protocol client/server architecture.
Hessian seems like a great fit if you want to create a web based game server with a Java or flash client using some type of server push technology like COMET. Hessian might provide a really interesting way to support real-time games on the web and even be able to host them on VM web solutions like Google's App engine or Amazon's EC2.
There's some discussion about using various protocol definition frameworks for games and other uses:
Comparison of Various Serialization Frameworks
Thrift vs Protocol Buffers - Thrift is declared the better framework because it has a fully supported RPC client/server implementation
Using Protocol Buffers for client server Game API determining what type of message to decode
Bi-Directional RPC using thrift
DIS
If you do go the route of writing your own protocol, you may want to read the answer I posted here.
In summary it discusses what you should think about when writing a protocol, and list a few tricks for versioning and maintaining backwards and forward compatibility.
If you are really concerned about multiple platforms and language, be sure to take into account endian issues. A binary protocol designed for this use must use network-byte-order, so it needs custom per-data-type serialization functions; you cannot just blindly push C structs into network buffers.
A common solution for this problem at game companies is to have protocol description language or specification in a simple format like XML or python or lua, and then have code generation for each target language that generated packet classes with both data structure and serialization. This specification could use a type system that starts with basic types, then extends to include game-specific types with semantic information, enumerations or more complex structures. For example a data file could look like:
Attack = {
source = 'objectId',
target = 'objectId',
weapon = 'weapon::WEAP_MAIN',
seed = 'int'
}
This could generate code like:
#define PT_ATTACK 10002
class PacketAttack : public Packet {
public:
PacketAttack () : m_packetType(PacketAttack::s_packetType) {}
ObjectId m_source;
ObjectId m_target;
WeaponType m_weapon;
int m_seed;
bool Write(Stream* outStream) {
Packet::Write(outStream);
outStream << m_source;
outStream << m_target;
outStream << m_weapon
outStream << m_seed;
}
bool Read(Stream* inStream);
static const int s_packetType;
};
This does require some more infrastructure.. streams, packet base classes, safe serialization functions..
I want to echo Bill K's suggestion. It's not hard to roll your own protocol.
For the iPhone side, have a look at AsyncSocket which support for delimiter based TCP packets built in, and it's not hard to build a solution which uses packet headers.
If you quickly want to have a testserver to play against AsyncSocket on the iPhone, you can look at Naga (for the java server part) which has ready made stuff both for delimiter based packets and packets with headers. Naga was partially written with networked games in mind.
I disagree with "roll your with simple delimited strings approach": question is, what exactly would be the benefit? Getting to write and maintain more code?
The only reasons I could see would be lack of tool support (writing for some odd platform), or specific (very) hard performance or message size constraints.
Or, sometimes, really wanting to write a format -- that's ok, but it must be an explicit reason.
Depending on exact needs I would suggest considering JSON, since it can read and write arbitrary messages; has good object binders for Java (just like xml), is easier to read than binary formats, and is all around "good enough" for many use cases.
If message size is very important, Protobuf can work well -- while its size is not always as small as gzipped alternatives (gzip+xml, gzip+json compress very well), it's usually close.
ASN.1 fits the definition of "good general purpose binary network protocol definition framework". It's also standardized by ITU-T, so there's a lot of existing tools and libraries for various languages.
The DER encoding is suitable for efficient network communications, the XER encoding for human-readable (and writable) permanent storage.
Because you want to use different languages and also because you want something clean/small, I suggest the protocol buffers of google. You need a pre-compile part for the RPC but I really think that's the best option when you begin to mix different languages.. Here's the link: http://code.google.com/apis/protocolbuffers/docs/overview.html
Why not implement UDP directly? Your question mostly mentions what you don't want.. What further form of abstration do you want on top of UDP?
Download the Quake III sourcecode and see how they frame game updates over UDP?
The IP protocol has been designed to support multiple devices/OSes in a uniform way, isn't this what you ask for?
What protocol has implementations across a huge range of systems, hmm, IP perhaps?