I am writing a distributed Erlang application where several nodes are connected via a network with limited bandwidth. Thus, I would like to be able to minimize the size of the packets sent across the network when processes on different nodes send each other messages.
From http://www.erlang.org/doc/apps/erts/erl_ext_dist.html, I understand that the Erlang distribution mechanism uses erlang:term_to_binary/1,2 internally to convert Erlang messages to the external binary format that is sent over the network. Now, term_to_binary/2 supports several options that are useful for reducing the size of the binaries (http://www.erlang.org/doc/man/erlang.html#term_to_binary-1), including a compression option as well as the ability to choose a minor version with more efficient encoding of floats.
I would like to be able to tell the distribution mechanism to use both of these options every time it sends a message over the network. In other words, I would like to be able to specify the Options list that the distribution mechanism calls term_to_binary with. However, I have not been able to find any documentation on this subject. Is this possible to do?
Thanks for your help! :)
If I understand the code correctly, message encoding hardcoded around the line 1565 of dist.c/dsig_send() so you can't change the way messages are encoded without patching and recompiling the emulator.
However you can change the carrier for message distribution as described here. There is an example of use SSL for Erlang distribution. So you can create a connection which compress all transmission messages (maybe it's even possible with tweaked SSL example).
There are few examples of standard distribution modules:
inet_tcp_dist.erl
inet6_tcp_dist.erl
inet_ssl_dist.erl
uds_dist example
Are you using rpc from node to node? Or OTP behaviours? if so try to compress with zlib the binary before it is sent
Related
Is there a simple way to implement communication between two computers running GNUradio using the standard blocks set?
What I am have now is this:
On a Linux computer, GNUradio is running and receiving input from a Radio peripheral. On that computer I can see the received waveform on a WX scope. I can also use sliders and input boxes to change things like the receiver frequency.
What I'd like to do is this:
On a Windows computer, I have the WX scope and sliders. When I move the a slider or change an input box, that data gets sent to the Linux, which is still running the radio receiver on Gnuradio. The received signal goes through a stream back to the windows, and gets displayed on the WX scope on Windows.
Someone elsewhere suggested using the ZMQ blocks, however, when I tried setting up a PUSH/PULL to transmit a sine wave from the Linux to the Windows, nothing went through. The guy who recommended that approach tried the same and also could not get it working, so I think that block might be broken?
So is there any alternative blocks that can do what I'm trying to do? Preferably something well documented, and available on GNUradio-companion.
Depending on the data rate from the receiver, it's possible to encounter performance issues attempting to send raw waveform data using e.g. the UDP blocks, where the sender may print an error similar to the following:
gr::log :WARN: udp_source0 - Too much data; dropping packet.
Because the scope widgets usually only display a portion of the input data, a more ideal way of remotely visualizing the waveform might be to only send the rendered scope widget (e.g. using a remote desktop such as VNC or X2Go). Although this solution reaches beyond your original problem, it is probably easier to use in the long run for cases involving two-way GUI interaction.
For the scope widget data, the UDP sink and source blocks seem to be native to GNU Radio, and are either sufficiently documented solution or simple enough for this problem, again taking firewall configuration into consideration as #Zephyr mentioned.
From GRC, specify in the UDP blocks:
the hostname or IP address of the display computer, and
a choose port number that isn't already in use (and were you using Linux, OS X, or anything UNIX-like, not any port below 1024).
For setting variables over the network, you might try the XMLRPC blocks, as described in another answer. These were recently deprecated, however.
See my other answer for discussion of alternative if performance issues arise.
Both Linux and Windows should have firewalls which might be blocking the connections.
You need to post the error messages displayed in gnuradio-companion.
We used the library SimplePing, but multiple challenges of our network scanner sometimes some devices were not found. ie This scanner does not work reliably.
Prompt another library or a more reliable algorithm for IOS?
Perhaps I need to write a ping using sockets:
socket(AF_INET,SOCK_DGRAM,IPPROTO_ICMP)
But what further code?
You can see the implementation of ping utility in FreeBSD source code: https://svnweb.freebsd.org/base/release/10.1.0/sbin/ping/ping.c?view=markup
Don't be afraid to read source codes of existing implementations for any platforms, because they all use the same basis, and your app will not be an exception.
Btw, you can compile C-code for your app written in Objective-C.
Update.
And here is a good explanation about how to use sockets: http://www.linuxhowtos.org/C_C++/socket.htm
Not all devices respond to ICMP PING (which is what you're using). Are there specific devices that are never discovered? If so, do they respond to the "ping" command?
If it is more erratic, then you need to start with network traces (most commonly using wireshark). You need to determine if the pinged machine received the ping, and separately whether your device received the response. You will generally want to build a small, private network for this so you can control the hardware.
Generally speaking, it is easier to initially develop this kind of code in the simulator rather than a device. Does it work more reliably in the simulator?
Are you receiving errors? Are there firewalls involved? (Firewalls often swallow ICMP without generating errors.)
You're going to need a lot more specific diagnostic information beyond "does not work reliably." You need to know exactly which pieces do and don't work. Did you send the packet? Did they receive the packet? Did they send the response? Did you receive the response? Without those basic data, it's very hard to troubleshoot networking.
I have already developed a Clinic management application for Allergy Control Clinics which stores patients' medical files and test results in a database and generates reports for analysis.
there's a section for storing spirometry results in the database. currently i get results
from an Excel file which is exported by WinspiroPro (the application that comes with spirolab devices) and store them in the database.
few days ago i came across the word "HL7" which seems to be a Standard protocol for communicating with these medical devices, so i can directly get the results from the device using Delphi.
also in spirolab device user manual it is mentioned that the device is compatible with this system.
now my question is, how can I implement this system (HL7) in delphi?
Thanks
As is usual with these kind of inter-professional standards, you need to pay to get them, at least on http://www.hl7.org in this case.
If I search around on the net, there may be existing tools that you can use, or have a look how they work internally:
http://code.ohloh.net/search?s=HL7
https://code.google.com/hosting/search?q=HL7&sa=Search
http://sourceforge.net/directory/?q=HL7
HL7 is not bound to a specific transport layer. It is a protocol on the application level, the seventh layer of the ISO 7-layer-model, hence Level 7. It describes messages and the events, when this messages should be send.
It just gives some recommendations how to do message transfer on the subjacent layers, e.g. MLLP with tcp socket communication. But in principle you are free to use any transport layers you want, may it be direct socket communication, file transfer or what ever.
Although most systems now can use tcp, it is also possible to use HL7 with different underlying transport protocols as RS232. If I remember right, there was also an example about message transfer / coupling with RS232 in the implementation guides of the documentation. And yes, the documentation and protocol standard documetation is free after registering.
Did you ask your provider for the WinspiroPRO version with HL7 ability? Maybe it supports already socket communication with tcp.
Otherwise you either need access to the sourcecode of ldTCPCClient and replace the tcp part with an RS232 part or you have to use a software just for parsing/building (unmarshalling/marshalling) of HL7 messages together with a software, that handles the transportation level.
By the way, just from the name, I guess that ldTCPclient is not apt for your need, as you will probably need a host and not a client component.
We intend to design a system with three "tiers".
HQ, with a single server
lots of "nodes" on a regional basis
users, with iPads.
HQ communicates 2-way with the nodes which communciate 2-way with the users. Users never communicate with HQ nor vice-versa.
The powers that be decree a Windows app from HQ (using Delphi) and a native desktop app for the users' iPads. They have no opinion on the nodes.
If there are compelling technical arguments, I might be able to beat them down from "decree" to "prefer" on the Windows program (and, for isntance, make it browser based). The nodes have no GUI, they just sit there playing middle-man.
What's the best way for these things to communicate (SOAP/HTTP/AJAX/jQuery/home-brewed-protocol-on-top-of-TCP/something-else?) Is it best to use the same protocol end to end, or different protocols for hq<-->node and node<-->iPad?
Both ends of each of those two interfaces might wish to initiate a transaction (which I can easily do if I roll my own protocol), so should I use push/pull/long-poll or what?
I hope that this description makes sense. Please ask questions if it does not. Thanks.
Update:
File size is typcially below 1MB with nothing likely to be above 10MB or even 5MB. No second file will be sent before a first file is acknowledged.
Files flow "downhill" from HQ to node to iPad. Files will never flow "uphill", but there will be some small packets of data (in addition to acks) which are initiated by user action on the iPad. These will go to the local node and then to the HQ. We are probably talking <128 bytes.
I suppose there will also be general control & maintenance traffic at a low rate, in all directions.
For push / pull (publish / subscribe or peer to peer communication), cross-platform message brokers could be used. I am not sure if there are (iOS) client libraries for Microsoft Message Queue (MSMQ), but I would also evaluate open source solutions like HornetQ, Apache ActiveMQ, Apollo, OpenMQ, Apache QPid or RabbitMQ.
All these solutions provide a reliable foundation for distributed messaging, like failover, clustering, persistence, with high performance and many clients attached. On this infrastructure message with any content type (JSON, binary, plain text) can be exchanged, and on top messages can contain routing and priority information. They also support transacted messaging.
There are Delphi and Free Pascal client libraries available for many enterprise quality open source messaging products. (I am am the author of some of them, supporting ActiveMQ, Apollo, HornetQ, OpenMQ and RabbitMQ)
Check out MessagePack: http://msgpack.org/
Also, here's more RPC discussion on SO:
RPC frameworks available?
MessagePack: fast cross-platform serializer and RPC - please share experience
ICE might be of interest to you: http://zeroc.com/index.html
They have an iOS layer: http://zeroc.com/icetouch/index.html
IMHO there are too little requisites to decide what technology to use. What data are exchanged, how often, what size? Are there request/response time constraints? etc. etc. Never start selecting a technology before you understand your needs deeply.
I need to use one logical PGM based multicast address in application while enable such application "seamlessly" running across several different geo-locations (i.e. think US/Europe/Australia).
Application is quite throughput (several million biz. messages a day) and latency demanding whith a lot of small but very frequently send messages. Classical Atom pub will not work here due some external limits of latencies.
I have come up with several options to connect those datacenters but can’t find the best one.
Options which I have considered are:
1) Forward multicast messages via VPN’s (can VPN handle such big load).
2) Translate all multicast messages to “wrapper messages” and forward them via AMQP.
3) Write specialized in-house gate which tunnels multicast messages via TCP to other two locations.
4) Any other solution
I would prefer option 1 as it does not need additional code writes from devs. but I’m afraid it will not be reliable connection.
Are there any rules to apply for such connectivity?
What the best network configuration with regard to the geographical configuration is for above constrains.
Just wanted to say hello :)
As for the topic, we have not much experience with multicasting over WAN, however, my feeling is that PGM + WAN + high volume of data would lead to retransmission storms. VPN won't make this problem disappear as all the Australian receivers would, when confronted with missing packets, send NACKS to Europe etc.
PGM specification does allow for tree structure of nodes for message delivery, so in theory you could place a single node on the receiving side that would in its turn re-multicast the data locally. However, I am not sure whether this kind of functionality is available with MS implementation of PGM. Optionally, you can place a Cisco router with PGM support on the receiving side that would handle this for you.
In any case, my preference would be to convert the data to TCP stream, pass it over the WAN and then convert it back to PGM on the other side. Some code has to be written, but no nasty surprises are to be expected.
Martin S.
at CohesiveFT we ran into a very similar problem when we designed our "VPN-Cubed" product for connecting multiple clouds up to servers behind our own firewall, in one VPN. We wanted to be able to run apps that talked to each other using multicast, but for example Amazon EC2 does not support multicast for reasons that should be fairly obvious if you consider the potential for network storms across a whole data center. We also wanted to route traffic across a wide area federation of nodes using the internet.
Without going into too much detail, the solution involved combining tunneling with standard routing protocols like BGP, and open technologies for VPNs. We used RabbitMQ AMQP to deliver messages in a pubsub style without needing physical multicast. This means you can fake multicast over wide area subnets, even across domains and firewalls, provided you are in the VPN-Cubed safe harbour. It works because it is a 'network overlay' as described in technical note here: http://blog.elasticserver.com/2008/12/vpn-cubed-technical-overview.html
I don't intend to actually offer you a specific solution, but I do hope this answer gives you confidence to try some of these approaches.
Cheers, alexis