I have an erlang program which generates data. This data needs to be transferred via udp to a non-erlang program for further processing. I already have this part working - sending the data via udp and receiving it on the other non-erlang side.
Here's the problem. The data (erlang terms like tuples containing lists) doesn't seem to be able to go over "as is" (i.e. I can't just send arbitrary erlang terms). It apparently needs to be converted to either text or binary first. Converting to binary seems easy enough with a bif I found. The problem is, binary gobbledygook comes out the other side, and I don't know any easy way to decode it (the other side is non-erlang).
Barring someone telling me some easy way to decode binary gobbledygook on the other side, I'd like the data to be sent as a simplistic string representation of the terms - for instance a tuple like this:
{[1,2,3],[4,5,6]}
sent like this:
"{[1,2,3],[4,5,6]}"
I haven't seen any such bif, i.e. "convert_term_to_ascii/1" etc. I know I could scan it and send token representations of the terms, but I don't want to do that - decoding that on the other side is just a pain I don't want to deal with.
I know I'm not the first, second, or third person to have this problem. It has to be fairly common. How is it normally dealt with?
Can someone point me to some resource showing me how to either 1) convert binary gobbledygook to ascii (needed on the non-erlang side), or 2) straightforwardly convert terms to a string (needed on the erlang side)?
Or, tell me how I'm wrong and how I should really be doing this?
Thanks.
1) you can convert any term to string using
R= io_lib:format("~p",[yourtermhere]),
lists:flatten(R)
2) you might look at erlang external binary format, a lot of other languages have libraries for encode/decode that erlang binaries format. And in erlang you can encode any term by term_to_binary
I'd recommend converting the erlang terms into JSON, with either of known libraries (heard good words regarding rfc4267). It'd be a trivial task to convert JSON back with any non-erlang platform, I guess. )
Related
I googled but mostly found links to 3rd part libraries for encryption/decryption works. However, I saw Security articles on the Apple site, though without examples.
Can you please show me an example of a simple encrypt/decrypt a string with a key function?
Security and CommonCrypto are low level frameworks. They only provide security primitives, not a full encrypted data format. It is challenging to build a secure format out of the primitives, and most examples you'll find online are insecure. Either the author did not know how to build a secure format, or the author assumes you know how to take what they've written and finish building a secure format.
There is no such thing as "decrypting a string" in the way that you likely mean. All encryption functions generate raw bytes. If you want a string, convert it to base64 or hex or whatever. Some libraries automatically add this, but it often leads to strange artifacts like double-base64-encoded data.
If you want a cross-platform "out of the box" encryption format, see RNCryptor or libSodium. Both of these convert data-to-data. If you want strings, just encode and decode the data as you like (usually as base64 or hex).
What I have found you can have a look at this url : Swift Default Encryption
I am looking to speed up the reading of a data file which has been converted from binary (it is my understanding that "binary" can mean a lot of different things - I do not know what type of binary file I have, just that it's a binary file) to plaintext. I looked into reading files quickly awhile ago, and was informed that reading/parsing a binary file is faster than text. So, I would like to parse/read the binary file (that was converted to plaintext) in an effort to speed up the program.
I'm using Matlab for this project (I have a Matlab "program" that needs the data in the file). I guess I need some information on the different "types" of binary, but I really want information on how to read/parse said binary file (I know what I'm looking for in plaintext, so I imagine I'll need to convert that to binary, search the file, then pull the result out into plaintext). The file is a logfile, if that helps in any way.
Thanks.
There are several issues in what you are asking -- however, you need to know the format of the file you are reading. If you can say "At position xx, I can expect to find data yy", that's what you need to know. In you question/comments you talk about searching for strings. You can also do it (much like a text file) "when I find xxxx in the file, give me the following data up to nth character, or up to the next yyyy".
You want to look at the documentation for fread. In the documentation there are snippets of code that will get you started, but as I (and others) said you need to know the format of your binary files. You can use a hex editor to ascertain some information if you are desperate, but what should be quicker is the documentation for the program that outputs these files.
Regarding different "binary files", well, there is least significant byte first or LSB last. You really don't need to know about that for this work. There are also other platform-dependent issues which I am almost certain you don't need to know about (unless you are moving the binary files from Mac to PC to unix machines). If you read to almost the bottom of the fread documentation, there is a section entitled "Reading Files Created on Other Systems" which talks about the issues and how to deal with them.
Another comment that I have to make, you say that "reading/parsing a binary file is faster than text". This is not true (or even if it is, odds are you won't notice the performance gain). In terms of development time, however, reading/parsing a textfile will save you huge amounts of time.
The simple way to store data in a binary file is to use the 'save' command.
If you load from a saved variable it should be significantly faster than if you load from a text file.
So, we are working on iOS push notifications for vendors that make apps and I think i am seeing two different ways that it can be sent to us. I am just wanting to know if this is a true or am i going down the wrong path.
I see that they can either A take the DeviceToken for push (NSObject) and Base64 Encode it and passes it to us to send. OR they could take the NSObject and HEX string it and pass it to us and not do the Base64 Encoding.
is this true? is it separate logic for handling when trying to send to Apple?
Thanks ahead of time guys!
Basically, you are correct. A hexadecimal number is definitely not the same as base64-encoded data. Theoretically, there is an infinite number of other possible encodings you could use to send the token to your server, but those two are probably among the most popular.
Base64 uses more characters for encoding so it needs less space, and it is kind of a de-facto standard for sending binary data over the network, so that's what I would opt for.
Im slowly getting my head around asn1 notation. Encoding asn1 from an struct seems to quite easy.However decoding asn1 to my application is harder. I want to know if I have an application that receives data on say udp port 600, and this can be a number of asn1 structures, how do i tell which asn1 structure i should be decoding into?
So, when i receive asn1 packets, do we first determine it's type and the decode it depending on its type?
or does my asn1 compiler handle of this for me?
finding examples of this stuff is hard...even for google ;-) .
A tutorial that outlines the building of simple network server using asn1 would be awesome!
ASN.1 corresponds roughly to the Presentation Layer of the OSI seven-layer cake. It relies on the Application Layer to determine what types of information is being exchanged between the endpoints. So, there's no universal header or identifier that would indicate which protocol or syntax is being presented by the ASN.1 stream. That could be implied by the port number, or made explicit by an additional protocol layer (e.g., HTTP.)
EDIT (responding to your comment): The ASN.1 standard defines four "classes" of tags: UNIVERSAL, APPLICATION, PRIVATE, and CONTEXT-SPECIFIC. APPLICATION basically means that the tag is defined for use within a specific application (i.e., it's not pre-defined as part of ASN.1.) But an APPLICATION tag doesn't carry enough information to specify (or even hint at) which application is in use.
(Actually, there's very little semantic difference among "APPLICATION", "PRIVATE", and "CONTEXT-SPECIFIC"; these three classes are used mostly for historical and stylistic reasons.)
I have an external device that spits out UDP packets of binary data and software running on an embedded system that needs to read this data stream, parse it and do somethign useful. The binary data gets logged to a file as well. I would like to write a parser that can easily take the input directly from either the UDP stream, or a file, parse the data into a specific format and then direct the output to either a file (e.g. matlab dat file) or to another process that will do some real time processing. Are there any resources that would help me with this and what is the best way to go about this? I think it might make sense to use C++ streams but I'm not familiar with creating custom output streams. Does this seem like a good approach to take or is there a better way to go about it?
Thanks.
The beauty of binary data is that its is generally of very fixed format.
A typical method of parsing it is to declare a structure that maps onto the received packets, and then to just use type-casts to read the fields as structure elements.
The beauty is that this requires no parsing.
you have to be careful about structure packing rules, and endian-ness to make the structure map exactly the same way. Use of the C "offsetof" and "sizeof" macros is useful to emit some debug info to check that your structure is indeed mapping to what you think it is mapping.
Packing rules can typically be altered either by directives (such as #pragma's) or command line options. Endian-ness you are stuck with. If its different from what your embedded system uses, declare all the fields as bytes, or use something like the "ntoh" macro to do the byte swapping.
The New Jersey Machine Code Toolkit is a scheme for decoding arbitrary binary patterns. It was originally designed for decoding instruction sets, but it ought to be just fine for decoding message formats. You provide a description of the binary format, it synthesizes code to access the fields of that format (when valid). THus you can refer to message fields using generated function calls rather than think about where the field is or how it is encoded.