I have a java client which is sending some message to an erlang server process listening on TCP.The java client sends the data using outputstream.On the server side i am using following call to uncompress the data after initialising zlib
zlib:inflate(ZStream, Data),
where Data is binary.I am getting data_error on this call.
Under what conditions do I get data_error with zlib.
Try setting a 0 or -15 WindowBits, would help if you paste more code like the zlib:inflateInit call, the binary dump of Data variable, and the Java side zlib init.
If you are streaming the data in relatively small chunks, you can use my ezlib on Github.
Performance wise it's around 69 % faster than erlang driver and also works better when you have concurrent sessions.
To integrate, use rebar as you would do for any other erlang app. To run a small example:
StringBin = <<"this is a string compressed with zlib nif library">>,
{ok, DeflateRef} = ezlib:new(?Z_DEFLATE),
{ok, InflateRef} = ezlib:new(?Z_INFLATE),
CompressedBin = ezlib:process(DeflateRef, StringBin),
DecompressedBin = ezlib:process(InflateRef, CompressedBin).
Do not use it to compress large blocks, because you can block the erlang scheduler. I will change this in the subsequent versions.
Related
I'm trying to read a 100gb file through stdin one line at a time using
Port = open_port({fd, 0, 1}, [in, binary, {line, 4096}]),
but this floods my system with messages until I run out of ram. Is there a away to make it like {active, once} with ports? There is also io:get_line() but I was wondering if this could work.
No, there is not flow control over ports so if you can't process fast enough you should use another method of processing. You can set binary mode on STDIN using
ok = io:setopts(standard_io, [binary]),
and then you can read it using file:read_line(standard_io) if you are using version 17 or newer (there was performance impacting bug).
I posted this to the Squeak Beginners list too - I'll be sure to make sure any answers from there get here :)
I'm using Squeak 4.2 and working on the smalltalk end of a named pipe connection, which sends a message to the named pipe server with:
msg := 'Here''s Johnny!!!!'.
pipe nextPutAll: msg; flush.
It should then receive an acknowledgement, which will be a 32-byte md5 hash of the received message (which the smalltalk app can then verify). It's possible the named pipe server may have gone away or otherwise been unable to deal with the request, and so I'd like to set a timeout on reading the acknowledgement. I've tried using this:
ack := [ pipe next: 32 ] valueWithin: (Duration seconds: 3) onTimeout: [ 'timeout'. ].
and then made the pipe server pause artificially to test the code. But the smalltalk thread blocks on the read and doesn't carry on (even after the timeout), although if I then get the pipe server to send the correct response (after a 5 second delay, for example), the value of 'ack' is 'timeout'. Obviously the timeout did what it's supposed to do, but couldn't 'unblock' the blocking read on the pipe.
Is there a way to accomplish this even with a blocking FileStream read? I'd rather avoid a busy wait on there being 32 characters available if at all possible.
This one may come in handy but not on Windows I am afraid.
http://www.samadhiweb.com/blog/2013.07.27.unixdomainsockets.html
i am making a program that sends data between a server program and its clients. They all use the server and client socket components found in Delphi! I have looked on the Internet and cannot find a way on how to measure how much data has been transferred through a socket!
Any help, especially some code (pascal/Delphi), would be very much appreciated!
here you have a full example http://delphi.about.com/od/fullcodeprojects/l/aa112903a.htm
If you use win/*nix API sockets (not a special libraries) you have to count returns from recv and send functions.
total_data = 0;
...
...
get_data = recv(...)
total_data = total_data + get_data
...
send_data = send(...)
total_data = total_data + send_data
Magenta Systems has a free set of components that can monitor network traffic using either raw sockets or WinPcap.
Update:
From your comment to RBA's response: The Magenta components let you identify and differentiate between different IP addresses and services (ports).
I keep stats on all my TCP traffic to the byte. Every time I send data I update the stats and every time I receieve data I update the stats. This is accurate to one byte and requires only a few lines of code. And it does not rely on any particular TCP components. Why is doing it this way so hard?
Suppose I create new local process in an Erlang application, and i want to send to it a big message.
-module(chain_hello).
start(N, Some_big_data)->
Pid1 = spawn(chain_hello, some_fun, [N]),
Pid1 ! Some_big_data,
io:format("done \n").
despite Some_big_data is a reference to really big data (eg. content of a file) - is it copied when sending? Are there big penalties for performance?
Normally I would use some thread safe shared object (and/or mutex). Is there any solution in Erlang to avoid copying message content?
ADDED:
Interesting case is when Some_big_data is structured content - to be specific: map, on which I can perform some operations.
ADDED2
Ok, I see there is no such solution for Erlang (sharing some structured data like map in worker process) - because of Erlang design. But I think it is justified by solid work, and easily concurrence management.
From the Erlang Efficiency Guide:
All data in messages between Erlang
processes is copied, with the
exception of refc binaries on the same
Erlang node.
Yes, you should avoid sending big terms between processes. If you need to send a lot of data, send it as a binary.
As a suggestion, you can send only the pid of the current process (self()) to the process that you want to handle that some_big_data. That way, when you want to use that some_big_data, you can reference it back to it from the 2nd process.
For instance:
-module(proc1).
send_big_data() ->
BigData = get_some_big_data(),
Pid = spawn(fun proc2:start/0),
Pid ! {process_big_data, self()},
loop(Pid, BigData).
loop(Pid,BigData) ->
receive
get_first_element ->
[H|T] = BigData,
Pid ! {response, H};
_ ->
Pid ! {nothing}
end,
loop(Pid).
(Sorry for the eventual syntax mistakes).
I'm making a application in Erlang, with a GUI in Java.
I've managed to establish a connection between the to languages, but now i need to (i guess) send a message from Java to Erlang, every time I e.g press a button.
Is that the right way to go?
How would such a message look?
I've found a few good sites about this form of integration, but I feel like im not getting everything.
http://www.trapexit.org/How_to_communicate_java_and_erlang
Besides classic Java-Erlang communication via OTP jinterface you can research such methods like:
- thrift
- ice from zeroC (no official erlang binding)
- maybe two http servers on both sides (I like this approach)
- protocol buffers (rather not, it is better for larger data transfers)
You need to learn the shape of your traffic and choose the best solution.
Jinterface is not so bad, tho.. (here is official doc: http://www.erlang.org/doc/apps/jinterface/jinterface_users_guide.html)
If jinterface is too complicated you might just use the packet option on open_port and use
byte[] in_buf = new byte[256];
byte[] out_buf = new byte[256];
int in_count = System.in.read ();
int offset = 0;
do
{
int c = System.in.read (in_buf, offset, in_count-offset);
offset += c;
}
while (offset < in_count);
To read packets from erlang and to write use:
System.out.write(out_count);
System.out.write(out_buf, 0, out_count);
On the erlang side this would match with
open_port({spawn, "<path-to-java> -cp <classpath> your-java-prog",
[{packet, 1}]).
If you need larger packets use {packet, 2} or {packet, 4} and adapt the java.
Inside the packets you can run whatever protocol you like on both sides.
I am working on an application similar to yours: C++ GUI and Erlang server. I use TCP sockets to exchange messages between the GUI and server, and Erlang server patterns for handling requests (I may have more than one GUI hooked up to the server at the same time).