DataOutputStream equivalent for Dart - dart

I am looking for a Java DataOutputStream equivalent for Dart where I can write arbitrary types (int, string, float, byte array etc). There is RandomAccessFile but it does not provide byte array or float-double values. ByteArray seems to have some necessary functions but I am not sure how to write it to a file or an OutputStream.

Here is some simple code showing how to write a ByteArray into an OutputStream:
#import('dart:io');
#import('dart:scalarlist');
main() {
File file = new File("c:\\temp\\foo.txt");
OutputStream os = file.openOutputStream();
os.onNoPendingWrites = () {
print('Finished writing. Closing.');
os.flush();
os.close();
};
Uint8List byteList = new Uint8List(64);
ByteArray byteArray = byteList.asByteArray();
int offset = 0;
offset = byteArray.setUint8(offset, 72);
offset = byteArray.setUint8(offset, 101);
offset = byteArray.setUint8(offset, 108);
offset = byteArray.setUint8(offset, 108);
offset = byteArray.setUint8(offset, 111);
offset = byteArray.setUint8(offset, 0);
byteArray.setFloat32(offset, 1.0);
os.write(byteList);
}

This has been around for a while, but I searched and didn't find good DataInput/OutputStream interoperability classes. I wanted a version that works with streams, so I could process files that don't comfortably fit in RAM. So I wrote one.
It's published over at https://pub.dev/packages/jovial_misc in io_streams, or if you prefer, https://github.com/zathras/misc/tree/master/dart/jovial_misc. I made it so it interoperates with java.io.DataInputStream and java.io.DataOutputStream. Code using it looks a little like this:
import 'package:convert/convert.dart';
import 'package:jovial_misc/io_utils.dart';
void main() async {
final acc = ByteAccumulatorSink();
final out = DataOutputSink(acc);
out.writeUTF8('Hello, world.');
out.close();
final stream = Stream<List<int>>.fromIterable([acc.bytes]);
final dis = DataInputStream(stream);
print(await dis.readUTF8());
await dis.close();
}
The Stream<List<int>> would of course typically come from a socket, or File.openRead(), etc. There's also a DataInputStream variant that is synchronous and takes an Iterable, if you do have all the byte data available up front.
DataInputStream and DataOutputSink are pretty much the obvious mapping of the java.io classes. The tricky part is the buffer management, since a stream shoves data at you in List<int> instances that probably aren't lined up with the data you want. And, of course, it's necessary to do everything asynchronously.
HTH.

You are essentially asking for arbitrary object serialization. And while the Dart VM has one, it isn't exposed to programmers (it is only used for snapshotting and message passing). I'd say that it would be a mistake to expose it -- in different situations, we have different requirements for serialization and "one true solution" isn't gonna work (Java showed us that already).
For example, I'm working on a MsgPack implementation for Dart, I know that Protobuf port is also in the works, maybe someone will start a Thrift port... the possibilities are endless.

The closest thing I could find is this package: https://github.com/TomCaserta/dart_io/ . Unfortunately there is a bug when reading to the end of the byte array - see my pull request in GitHub.

You could use this class:
https://github.com/TomCaserta/dart_io/blob/master/lib/data_output.dart
Unfortunately (a) it doesn't handle streams; (b) writeLong doesn't take a single integer. I have raised an issue for the Dart SDK: https://github.com/dart-lang/sdk/issues/31166
Edit: I have forked the dart_io package and fixed the two problems described above. My new package is published as dart_data_io:
https://github.com/markmclaren2/dart_data_io

Related

Why doesn't File provide an interface to read the contents to return Uint8List data in dart?

I want to read the contents of a file piece by piece through an interface (instead of reading the whole file at once with readAsBytes()). openRead() seems to do the trick, but it returns a List<int> type. And I expect it to be Uint8List, because I want to do block operations on some of the contents.
If you convert the returned List<int> to Uint8List, it seems to make a copy of the contents, which is a big loss in efficiency.
Is this how it was designed?
Historically Dart used List<int> for sequences of bytes before a more specific Uint8List class was added. A Uint8List is a subtype of List<int>, and in most cases where a Dart SDK function returns a List<int> for a list of bytes, it's actually a Uint8List object. You therefore usually can just cast the result:
var file = File('/path/to/some/file');
var stream = file.openRead();
await for (var chunk in stream) {
var bytes = chunk as Uint8List;
}
If you are uncomfortable relying on the cast, you can create a helper function that falls back to creating a copy if and only if necessary.
There have been efforts to change the Dart SDK function signatures to use Uint8List types explicitly, and that has happened in some cases (e.g. File.readAsBytes). Such changes would be breaking API changes, so they cannot be done lightly. I don't know why File.openRead was not changed, but it's quite likely that the amount of breakage was deemed to be not worth the effort. (At a minimum, the SDK documentation should be updated to indicate whether it is guaranteed to return a Uint8List object. Also see https://github.com/dart-lang/sdk/issues/39947)
Alternatively, instead of using File.openRead, you could use File.open and then use RandomAccessFile.read, which is declared to return a Uint8List.

Z3 API: Crash when parsing fixed point SMTLib string

I am trying to use the C/C++ API of Z3 to parse fixed point constraints in the SMTLib2 format (specifically files produced by SeaHorn). However, my application crashes when parsing the string (I am using the Z3_fixedpoint_from_string method). The Z3 version I'm working with is version 4.5.1 64 bit.
The SMTLib file I try to parse works find with the Z3 binary, which I have compiled from the sources, but it runs into a segmentation fault when calling Z3_fixedpoint_from_string. I narrowed the problem down to the point that I think the issue is related to adding relations to the fixed point context. A simple example that produces a seg fault on my machine is the following:
#include "z3.h"
int main()
{
Z3_context c = Z3_mk_context(Z3_mk_config());
Z3_fixedpoint f = Z3_mk_fixedpoint(c);
Z3_fixedpoint_from_string (c, f, "(declare-rel R ())");
Z3_del_context(c);
}
Running this code with valgrind reports a lot of invalid reads and writes. So, either this is not how the API is supposed to be used, or there is a problem somewhere. Unfortunately, I could not find any examples on how to use the fixed point engine programmatically. However, calling Z3_fixedpoint_from_string (c, f, "(declare-var x Int)"); for instance works just fine.
BTW, where is Z3_del_fixedpoint()?
The fixedpoint object "f" is reference counted. the caller is responsible for taking a reference count immediately after it is created. It is easier to use C++ smart pointers to control this, similar to how we control it for other objects. The C++ API does not have a wrapper for fixedpoint objects so you would have to create your own in the style of other wrappers.
Instead of del_fixedpoint one uses reference counters.
class fixedpoint : public object {
Z3_fixedpoint m_fp;
public:
fixedpoint(context& c):object(c) { mfp = Z3_mk_fixedpoint(c); Z3_fixedpoint_inc_ref(c, m_fp); }
~fixedpoint() { Z3_fixedpoint_dec_ref(ctx(), m_fp); }
operator Z3_fixedpoint() const { return m_fp; }
void from_string(char const* s) {
Z3_fixedpoint_from_string (ctx(), m_fp, s);
}
};
int main()
{
context c;
fixedpoint f(c);
f.from_string("....");
}

JNA pointer to pointer mapping

I am working on a Java binding for the excellent libvips
Using this function all is fine:
VipsImage *in;
in = vips_image_new_from_file( test.jpg, NULL )
vips_image_write_to_file( in, "out.jpg", NULL )
So mapped in Java:
Pointer vips_image_new_from_file(String filename,String params);
But I have a problem when the parameter like this:
VipsImage *in;
VipsImage *out;
vips_invert( in, &out, NULL )
vips_image_write_to_file( out, "out.jpg", NULL )
I have tried:
int vips_resize(Pointer in, PointerByReference out, Double scale, String params);
Pointer in = vips_image_new_from_file("file.png",null);
PointerByReference ptr1 = new PointerByReference();
vips_invert(in, ptr1, null);
vips_image_write_to_file( ptr1.getValue(), "fileout.png", null);
But doesn't work. The ptr1.getValue() does not contains the expected result.
How can I do it?
Thanks
I'm the libvips maintainer, a Java binding would be great!
But I think you might be taking the wrong approach. I think you are trying a straight wrap of the C API, but that's going to be tricky to do well, since it makes use of a lot of C-isms that don't map well to Java. For example, in C you can write:
VipsImage *image;
if (!(image = vips_image_new_from_file("somefile.jpg",
"shrink", 2,
"autorotate", TRUE,
NULL)))
error ...;
ie. the final NULL marks the end of a varargs name / value list. Here I'm asking the jpeg loader to do a x2 shrink during load, and to apply any Orientation tags it finds in the EXIF.
libvips has a lower-level API based on GObject which is much easier to bind to. There's some discussion and example code in this issue, where someone is making a C# binding using p/invoke.
https://github.com/jcupitt/libvips/issues/558
The code for the C++ and PHP bindings might be a useful reference:
https://github.com/jcupitt/libvips/tree/master/cplusplus
https://github.com/jcupitt/php-vips-ext
That's a PHP binding for the entire library in 1800 lines of C.
I'd be very happy to help if I can. Open an issue on the libvips tracker:
https://github.com/jcupitt/libvips/issues

Where are the parameters for Indy's TIdCompressorZLib.CompressStream Method documented?

The TIdComproessorZLib component is used for compression and decompression in the Delphi/C++ Builder Indy library. The CompressStream Method has the following definition:
public: virtual __fastcall CompressStream(TStream AInStream, TStream AOutStream, const TIdCompressionLevel ALevel, const int AWindowBits, const int AMemLevel, const int AStrategy);
The complete description of those parameters in the help file is:
CompressStream is a public overridden procedure. that implements the
abstract the virtual method declared in the ancestor class.
AInStream is the stream containing the uncompressed contents used in
the compression operation.
AOutStream is the stream used to store the compressed contents from
the compression operation. AOutStream is cleared prior to outputting
the compressed contents from the operation. When AOutStream is
omitted, the stream in AInStream is cleared and reused for the output
from the compression operation.
Use ALevel to indicate the desired compression level for the
operation.
Use AWindowsBits and AMemLevel to control the memory footprint
required to perform in-memory compression using the ZLib library.
Use AStrategy to control the RLE-encoding strategy used in the
compression operation.
ALevel's values defined on the help page for TIdCompressionLevel, but I cannot find any indication of what values should be used for AWindowBits, AMemLevel, or AStrategy, which are just integers.
I looked in the source code, but CompressStream just delegates to IndyCompressStream, which is listed in the help file as:
IndyCompressStream(TStream InStream, TStream OutStream, const int level = Z_DEFAULT_COMPRESSION, const int WinBits = MAX_WBITS, const int MemLevel = MAX_MEM_LEVEL, const int Stratagy = Z_DEFAULT_STRATEGY);
The help for IndyCompressStream doesn't even list the minimal description of the parameters that CompressStream does.
I tracked down the file where (I think) those default constants mentioned in IndyCompressStream live, source\Indy10\Protocols\IdZLibHeaders.pas, and they are
Z_DEFAULT_STRATEGY = 0;
Z_DEFAULT_COMPRESSION = -1;
MAX_WBITS = 15; { 32K LZ77 window }
MAX_MEM_LEVEL = 9;
However, the value given for Z_DEFAULT_COMPRESSION is not even a legal value for that parameter according to the documentation for TIdCompressionLevel
Is there some documentation somewhere about what AWindowBits, AMemLevel, and AStrategy mean to this component, and what values are reasonable to use for them? Are the values listed above the actual recommended defaults? Also, the source files include "indy", "Indy10", and "indyimpl" directories. Which of those should we be using to find the source for the current Indy components?
Thanks!
You will need to look to the zlib documentation in zlib.h. In particular, the parameters to deflateInit2().
In nearly all cases, the only ones you should mess with are the compression level and the window bits. For window bits, you would normally leave the window size at 32K (15), but either add 16 for the gzip format (31), or negate (-15) to get the raw deflate format with no header or trailer. For some special kinds of data, you may get an improvement with a different compression strategy, e.g. image or other numerical arrays of data.
Thank you for the comments and answers, especially Remy and Mark. I had not realized that the Indy units were wrappers around zlib, and that the parameters were defined in the zlib library.
I was trying to create a gzip format stream for uploading to a server that was expecting gzip.
Here is the working code for gzip compression and decompression:
void __fastcall TForm1::Button1Click(TObject *Sender)
{
TStringStream* streamIn = new TStringStream(String("This is some data to compress"));
TMemoryStream* streamCompressed = new TMemoryStream;
TStringStream* streamOut = new TStringStream;
/* this also works to compress to gzip format, but you must #include <IdZlib.hpp>
CompressStreamEx(streamIn, streamCompressed, Idzlib::clDefault, zsGZip); */
// NOTE: according to docs, you can leave outstream null, and instream
// will be replaced and reused, but I could not get that to work
IdCompressorZLib1->CompressStream(
streamIn, // System::Classes::TStream* AInStream,
streamCompressed, // System::Classes::TStream* AOutStream,
1, // const Idzlibcompressorbase::TIdCompressionLevel ALevel,
15 + 16, // const int AWindowBits, -- add 16 to get gzip format
8, // const int AMemLevel, -- see note below
0); // const int AStrategy);
streamCompressed->Position = 0;
IdCompressorZLib1->DecompressGZipStream(streamCompressed, streamOut);
String out = streamOut->DataString;
ShowMessage(out);
}
In particular, note that passing -1 for ALevel produces ZLib Error -2, Z_STREAM_ERROR which means invalid parameter, in spite of the defaults I had found. Also, AWindowBits normally ranges from 8 to 15, but adding 16 gives you a gzip format, and negative numbers give you a raw format, as described in the zlib documentation referenced by Mark Adler, one of the authors of the zlib library. I changed AMemLevel from Indy's default per Mark Adler's comment.
Also, as noted the CompressStreamEx function will produce gzip compression using the parameters included in the comments above.
The above was tested in RAD Studio XE3. Thanks again for your help!

BlackBerry J2ME Efficient Coding GuideLines? Could Somebody elaborate this?

I found the folliwing code sample in BlackBerry Java Development, Best Practices. Could somebody explain what the below same code means? What is the this in the code sample poining to?
Avoiding StringBuffer.append (StringBuffer)
To append a String buffer to another, a BlackBerry® Java Application should use net.rim.device.api.util.StringUtilities.append( StringBuffer dst, StringBuffer src[, int offset, int length ] ).
Code sample
public synchronized StringBuffer append(Object obj) {
if (obj instanceof StringBuffer) {
StringBuffer sb = (StringBuffer)obj;
net.rim.device.api.util.StringUtilities.append( this, sb, 0, sb )
return this;
}
return append(String.valueOf(obj));
}
StringBuffer does not offer an overload for the append() method that takes another StringBuffer. This means developers are likely to use StringBuffer.append(String str) and call .toString() on the second StringBuffer. This requires the second buffer to be turned into a string, which is immutable, and then the characters from the string are appended to the first StringBuffer. Thus every character in the second buffer is touched twice, and there is the unnecessary allocation of the String just to transfer the characters to the first StringBuffer.
The efficient way of doing this would copy each character from the second buffer onto the end of the first. However, StringBuffer does not provide any easy way of doing this. Thus the recommendation is to use StringUtilities.append(StringBuffer, StringBuffer) which is able to directly read the characters from the second buffer without copying them into an intermediate collection.
This saves the runtime of the extra copying, the runtime needed to allocate a temporary String, and the memory needed to allocate a temporary string.
It means that the StringBuffer class is not implemented efficiently. Java Strings are supposed to be immutable, that's what StringBuffer is used for. However, the StringBuffer class you're using is not efficient when using StringBuffer.append() so you need to use net.rim.device.api.util.StringUtilities. That's what the code is doing, encapsulating the use of that class in a new append() method.

Resources