I have a Go program that starts a C/Lua process. Now I'd like to communicate between those two. For example, in the middle of the run of the child (the c/lua process), I'd like to ask the parent (the Go program) to do some calculations and wait for the result. I am not keen to use stdin/stdout for communication, as this is already used for regular output. Now I am thinking of using sockets for the communication, but I don't want to reinvent the wheel.
What are the obvious choices for this kind of communication?
Is there a (more or less) simple standard way to pass objects between Lua and Go? If not, text blobs would suffice.
Are Protocol Buffers suitable for this? Looks like overkill, but I have no experience here.
Besides all the usual IPC methods you've mentioned (yeah, a unix socket with protobuf should do it, and stdin/stdout as well), if you run the C/Lua code embedded in your program, and not start it as a process, you can actually communicate between the languages directly.
Using the cgo module, Go code can call C functions, and embedded C code can call Go functions. See here: http://golang.org/cmd/cgo/#hdr-C_references_to_Go
Also, you have a couple embeddable Lua binding libraries for Go which you can try, that let you call Lua code and let your Lua code call Go. see https://github.com/aarzilli/golua and https://github.com/stevedonovan/luar/
I now talk to the subprocess via regular tcp sockets. The child process (Lua) has luasocket built in and this seems to work on Windows, Mac and Linux without problems.
Also, I have (for now) defined my own very simple protocol which looks OK in the first steps.
Just in case anybody is interested: https://github.com/speedata/publisher/commit/ea253382c1096274bca2d4124867c39cd0d512e5 and the child commits implement the tcp connection.
Related
I am working with two tcl files. One is a program that connects to a thermal scanner to get information from it via ethernet. I have integrated the other tcl GUI that can reflash the scanner via ethernet. How do I use the connection from the first one to use the flash utility on the second GUI? At first, the two GUI's were separate but I had to combine them for customers.
-let me know if you need any code, the programs are about 4000 lines long so I didnt want to put them on here.
A single TCP connection can only really be used by one process (at each end) at a time; trying to do otherwise causes confusion. Can you just get each program to talk to the thermal scanner directly? At least theoretically, it ought be able to support multiple simultaneous connections, and that'd be a simple way to work.
But if the device won't work that way, the easiest way to multiplex the communications is to have a single process that actually talks to the device and for the other programs to talk to that multiplexer process (probably via a local socket). The multiplexer doesn't need to have a GUI, but would be told to do things like “flash this file to the scanner” or “stream values out of the scanner”. The details of how to do all this will depend very much on what's actually going on with the communications, but the fcopy command is highly likely to be useful in this, as it can be used to efficiently stream data from one Tcl channel to another.
NGINX and other servers offer the option to use named pipes (mkfifo).
Can erlang use these instead of ports for nif interaction. What if I wanted to make 70,000 connections to my NIF (don't judge).
In short, no.
This is covered in the Erlang FAQ on opening device files. It boils down to it being hard/impossible to write the Erlang runtime in a portable way across Unices (not to mention Windows) so that it can access things like device files and named pipes without blocking on at least some of them. That blocking would screw up the "soft realtime" nature of the Erlang runtime.
What is possible is to write a C program that communicates with the Erlang runtime as a "port process", and that program can communicate over the named pipe (and block or not or whatever without screwing up the Erlang runtime).
I have a C++ code that implement a special protocol over the serial port. The code is multi-threaded and internally polls the serial port and do its own cyclic processing. I would like to call this driver from erlang and also receive events from this driver. My concern is that this C++ code is multi-threaded and also statefull meaning that when I call a certain function on the driver, it caches things internally which will be used/required on the subsequent calls of the driver. My questions are
1.Does NIF run in the same os process as the rest of my erlang proceses or NIF is launched in a separate os process?
2.Does it make sense to warp this multi-threaded stateful C++ code with NIF?
4.If NIF is not the right approach, what is the better way for me to make Elrang talk back and forth with this C++ code. I also prefer my C++ code to be inside the same OS process as the rest of my Erlang processes and as it looks like linked-in drivers are an option but not sure if the multi-threaded nature of my C++ code will be ok to that model. Plus I hear they can mess up elrang scheduler?
Unlike ports, NIFs are run within Erlang VM process, similar to drivers. Because of that, any NIF crashes will bring VM down as well. And, answering in advance, to your last question, NIFs, like drivers, may block your scheduler.
That depends on the functionality you are implementing by this C++ code. Due to the answer 1), you probably want to avoid concurrency in the C++ part, since it's a potential source of errors. It's not always possible, of course. But if you are implementing, say, some workers pool, go ahead and implement 1-threaded code, spawning it as many times as you need.
Drivers can be multi-threaded too, with same potential problems and quite similar performance (well, still slightly faster than NIFs). If you are not completely sure about your C++ code stability, use it as an Erlang port.
Speaking of the difference between NIFs and drivers, the former is synchronous natively, and the latter can be asynchronous (which can be really a huge advantage if you don't want to receive any answers for most of the commands). Drivers are easier to mess up and harder to implement (but once you grasp the main patterns and problems, they seem okay, actually).
Here's a good start for drivers:
http://www.erlang.org/doc/apps/erts/driver.html
And something similar (behold the difference in complexity) for NIFs:
http://www.erlang.org/doc/tutorial/nif.html
I've grown to like erlang, and it's a great (cough) architectural fit to my problem. Meanwhile I still like to imagine that I can kludge erlang processes & asynchronous message passing in python (I am currently in therapy to rid myself of this obsession).
During a recent binge I came across 0MQ & I like its messaging features. These may be self-evident to an erlang/OTP expert, but I'm just a humble python programmer (my shrink will no doubt get to read this clever argument). The 0MQ user-guide states that it uses native OS threads, and not virtual "green" threads.
Is there a way to make 0MQ work with say eventlet/gevent?
Or, should I avoid the green-eyed monster and stick to a single Python app thread, with non-blocking I/O handled by 0MQ's message queuing & its own (skilled) use of native threads?
Or, check out of rehab & go back to erlang?
Responding to a stale thread because I am kind of in the same boat. Thought I would share my thoughts.
1: It looks like all the heavy lifting has already been done: https://github.com/traviscline/gevent-zeromq has integrated the gevent loop with a nonblocking zmq socket and even some Cpython speedups. It also seems to be (at the time of this writing), reasonably well maintainted.
2: It depends; if you are writing something that can use zmq without a ton of external event logic, then you should just use zmq. If OTOH you need to integrate with other protocols, you may want to use gevent (or twisted perhaps, although it has no workable zmq now at all). My projects generally require multiple protocols (ie: private queue manager, public http, public https, private memcache, etc), so I am investigating switching to gevent for quicker project turnaround than my current favorite: twisted.
3: You may want to skip zmq entirely and integrate with an existing erlang based solution like rabbitMQ; the performance advantages of zmq may not be as important as you think, and then you have an erlang message queue that easily integrates with python with existing libraries.
Also see: Messsage Queue comparison at second life wiki
Zero MQ now works with Eventlet:
https://lists.secondlife.com/pipermail/eventletdev/2010-October/000907.html
How might you call a Lisp program from a Rails application?... For example, allow the end user to enter a block of text in the Rails web app, have the text processed by the Lisp program and return results to the Rails app?
There are a couple ways that come to mind:
Execute the lisp program with Process. Communicate with the Lisp program via standard in, and have the Lisp program output its result over stdout.
Do the same thing as above, but communicate via named pipes instead. Have your Ruby code write data into a named pipe, then have the Lisp program read from that pipe, and write data out over another named pipe which you then read with your Ruby app. The Lisp program can either run in the background as a daemon that checks for data on its incoming pipe, or you can fire it up as-needed using Ruby's command-line utilities (as above).
Find a Ruby-Lisp bridge. I have no experience with such a bridge (nor do I know off-hand if one even exists) and I think the above 2 mechanisms are easier, but your mileage may vary.
Another simple way is to have Lisp running a HTTP server and contact Lisp from the outside via HTTP requests.
CL-JSON supports JSON-RPC. It's very easy to set up with a web server such as Hunchentoot to have a Lisp-based web service that anything that speaks JSON-RPC (e.g. this) can use.
It would depend on how often it's going to happen.
If it's once in a blue moon, then just run a backquote command that starts the lisp interpreter, or popen it and write to it.
If it happens all the time, you will need to have Lisp already running, so the question then is how to communicate. Any of the interprocessor mechanisms will work, but I would suggest a TCP socket for development, testing, and production flexibility.
If it happens a million times a day, but a toy Lisp would be good enough, it is a simple matter to implement Lisp with Ruby classes. This was done as chapter 8 of Practical Ruby Projects.