The TIdComproessorZLib component is used for compression and decompression in the Delphi/C++ Builder Indy library. The CompressStream Method has the following definition:
public: virtual __fastcall CompressStream(TStream AInStream, TStream AOutStream, const TIdCompressionLevel ALevel, const int AWindowBits, const int AMemLevel, const int AStrategy);
The complete description of those parameters in the help file is:
CompressStream is a public overridden procedure. that implements the
abstract the virtual method declared in the ancestor class.
AInStream is the stream containing the uncompressed contents used in
the compression operation.
AOutStream is the stream used to store the compressed contents from
the compression operation. AOutStream is cleared prior to outputting
the compressed contents from the operation. When AOutStream is
omitted, the stream in AInStream is cleared and reused for the output
from the compression operation.
Use ALevel to indicate the desired compression level for the
operation.
Use AWindowsBits and AMemLevel to control the memory footprint
required to perform in-memory compression using the ZLib library.
Use AStrategy to control the RLE-encoding strategy used in the
compression operation.
ALevel's values defined on the help page for TIdCompressionLevel, but I cannot find any indication of what values should be used for AWindowBits, AMemLevel, or AStrategy, which are just integers.
I looked in the source code, but CompressStream just delegates to IndyCompressStream, which is listed in the help file as:
IndyCompressStream(TStream InStream, TStream OutStream, const int level = Z_DEFAULT_COMPRESSION, const int WinBits = MAX_WBITS, const int MemLevel = MAX_MEM_LEVEL, const int Stratagy = Z_DEFAULT_STRATEGY);
The help for IndyCompressStream doesn't even list the minimal description of the parameters that CompressStream does.
I tracked down the file where (I think) those default constants mentioned in IndyCompressStream live, source\Indy10\Protocols\IdZLibHeaders.pas, and they are
Z_DEFAULT_STRATEGY = 0;
Z_DEFAULT_COMPRESSION = -1;
MAX_WBITS = 15; { 32K LZ77 window }
MAX_MEM_LEVEL = 9;
However, the value given for Z_DEFAULT_COMPRESSION is not even a legal value for that parameter according to the documentation for TIdCompressionLevel
Is there some documentation somewhere about what AWindowBits, AMemLevel, and AStrategy mean to this component, and what values are reasonable to use for them? Are the values listed above the actual recommended defaults? Also, the source files include "indy", "Indy10", and "indyimpl" directories. Which of those should we be using to find the source for the current Indy components?
Thanks!
You will need to look to the zlib documentation in zlib.h. In particular, the parameters to deflateInit2().
In nearly all cases, the only ones you should mess with are the compression level and the window bits. For window bits, you would normally leave the window size at 32K (15), but either add 16 for the gzip format (31), or negate (-15) to get the raw deflate format with no header or trailer. For some special kinds of data, you may get an improvement with a different compression strategy, e.g. image or other numerical arrays of data.
Thank you for the comments and answers, especially Remy and Mark. I had not realized that the Indy units were wrappers around zlib, and that the parameters were defined in the zlib library.
I was trying to create a gzip format stream for uploading to a server that was expecting gzip.
Here is the working code for gzip compression and decompression:
void __fastcall TForm1::Button1Click(TObject *Sender)
{
TStringStream* streamIn = new TStringStream(String("This is some data to compress"));
TMemoryStream* streamCompressed = new TMemoryStream;
TStringStream* streamOut = new TStringStream;
/* this also works to compress to gzip format, but you must #include <IdZlib.hpp>
CompressStreamEx(streamIn, streamCompressed, Idzlib::clDefault, zsGZip); */
// NOTE: according to docs, you can leave outstream null, and instream
// will be replaced and reused, but I could not get that to work
IdCompressorZLib1->CompressStream(
streamIn, // System::Classes::TStream* AInStream,
streamCompressed, // System::Classes::TStream* AOutStream,
1, // const Idzlibcompressorbase::TIdCompressionLevel ALevel,
15 + 16, // const int AWindowBits, -- add 16 to get gzip format
8, // const int AMemLevel, -- see note below
0); // const int AStrategy);
streamCompressed->Position = 0;
IdCompressorZLib1->DecompressGZipStream(streamCompressed, streamOut);
String out = streamOut->DataString;
ShowMessage(out);
}
In particular, note that passing -1 for ALevel produces ZLib Error -2, Z_STREAM_ERROR which means invalid parameter, in spite of the defaults I had found. Also, AWindowBits normally ranges from 8 to 15, but adding 16 gives you a gzip format, and negative numbers give you a raw format, as described in the zlib documentation referenced by Mark Adler, one of the authors of the zlib library. I changed AMemLevel from Indy's default per Mark Adler's comment.
Also, as noted the CompressStreamEx function will produce gzip compression using the parameters included in the comments above.
The above was tested in RAD Studio XE3. Thanks again for your help!
Related
Using Embarcadero C++ Builder 10.3.
I have a DynamicArray<uint8_t> myData object. I want to send/write its raw binary content (bytes) to a server using the TIdTcpClient component. I'm going about it like this:
TIdTcpClient tcpClient1;
// Bla Bla Bla
tcpClient1->IOHandler->Write(rawData);
Where rawData should be of type TIdBytes or TIdStream
So basically, it boils down to the following: How to convert myData object to a rawData type of either TIdBytes or TIdStream?
First off, TIdStream has not been part of Indy in a VERY VERY LONG time, which makes me wonder if you are using a very old version of Indy, not the one that shipped with C++Builder 10.3. Indy has supported the RTL's standard TStream class for a very long time.
That being said...
TIdBytes is an alias for System::DynamicArray<System::Byte>, where System::Byte is an alias for unsigned char, which is the same size and sign-ness as uint8_t (depending on compiler, uint8_t might even just be an alias for unsigned char).
So, the simplest solution, without having to make a separate copy of your data, is to simply type-cast it, eg:
tcpClient1->IOHandler->Write(reinterpret_cast<TIdBytes&>(myData));
This is technically undefined behavior, since DynamicArray<uint8_t> and DynamicArray<Byte> are unrelated types (unless uint8_t and Byte are both aliases for unsigned char), but it will work in your case since it is the same underlying code behind both arrays, and uint8_t and Byte have the same underlying memory layout.
Alternatively, the next simplest solution, without copying data or invoking undefined behavior, is to use Indy's TIdReadOnlyMemoryBufferStream class in IdGlobal.hpp, eg:
TIdReadOnlyMemoryBufferStream *ms = new TIdReadOnlyMemoryBufferStream(&myData[0], myData.Length);
try {
tcpClient1->IOHandler->Write(ms);
}
__finally {
delete ms;
}
Or:
{
auto ms = std::make_unique<TIdReadOnlyMemoryBufferStream>(&myData[0], myData.Length);
tcpClient1->IOHandler->Write(ms.get());
}
Otherwise, the final solution is to just copy the data into a TIdBytes, eg:
{
TIdBytes bytes;
bytes.Length = myData.Length;
memcpy(&bytes[0], &myData[0], myData.Length);
or:
std::copy(myData.begin(), myData.end(), bytes.begin());
tcpClient1->IOHandler->Write(bytes);
}
I want to read the contents of a file piece by piece through an interface (instead of reading the whole file at once with readAsBytes()). openRead() seems to do the trick, but it returns a List<int> type. And I expect it to be Uint8List, because I want to do block operations on some of the contents.
If you convert the returned List<int> to Uint8List, it seems to make a copy of the contents, which is a big loss in efficiency.
Is this how it was designed?
Historically Dart used List<int> for sequences of bytes before a more specific Uint8List class was added. A Uint8List is a subtype of List<int>, and in most cases where a Dart SDK function returns a List<int> for a list of bytes, it's actually a Uint8List object. You therefore usually can just cast the result:
var file = File('/path/to/some/file');
var stream = file.openRead();
await for (var chunk in stream) {
var bytes = chunk as Uint8List;
}
If you are uncomfortable relying on the cast, you can create a helper function that falls back to creating a copy if and only if necessary.
There have been efforts to change the Dart SDK function signatures to use Uint8List types explicitly, and that has happened in some cases (e.g. File.readAsBytes). Such changes would be breaking API changes, so they cannot be done lightly. I don't know why File.openRead was not changed, but it's quite likely that the amount of breakage was deemed to be not worth the effort. (At a minimum, the SDK documentation should be updated to indicate whether it is guaranteed to return a Uint8List object. Also see https://github.com/dart-lang/sdk/issues/39947)
Alternatively, instead of using File.openRead, you could use File.open and then use RandomAccessFile.read, which is declared to return a Uint8List.
I have my own class (nBuffer) like wxMemoryBuffer and I use it to load/save custom data, it's more convenient than using streams because I have a lot of overloaded methods for different data types based on these:
class nBuffer
{ // ...
bool wr(void* buf, long unsigned int length);// write
bool rd(void* buf, long unsigned int length);// read
}
I'm trying to implemets methods to load/save wxString from/to this buffer.
With wxWidgets 2.8 I've used the next code (simplified):
bool nBuffer::wrString(wxString s)
{ // save string:
int32 lng=s.Length()*4;
wr(&lng,4);// length
wr(s.GetData(),lng);// string itself
return true;
}
bool nBuffer::rdString(wxString &s)
{ // load string:
uint32 lng;
rd(&lng,4);// length
s.Alloc(lng);
rd(s.GetWriteBuf(lng),lng);// string itself
s.UngetWriteBuf();
s=s.Left(lng/4);
return true;
}
This code is not good because:
Is assumes there are 4 bytes of data for each string character (it might be less),
With wxWidgets 3.0, wxString.GetData() returns wxCStrData instead of *void, so the compiler fails on wr(s.GetData(),lng); and I have no idea of how to convert it to a simple byte buffer.
Strange, but I found nothing googling that for hours... Also I've found nothing useful in wxWidgets docs.
The questions are:
That is the preferred, correct and safe way to convert wxString to byte buffer,
The same about converting the byte buffer back to wxString.
For arbitrary wxStrings you need to serialize them in either UTF-8 or UTF-16 format. The former is a de facto standard for data exchange, so I advise to use it, but you could prefer UTF-16 if you know that your data is biased to the sort of characters that take less space in it than in UTF-8 and if space saving is important for you.
Assuming you use UTF-8, serializing is done using utf8_str() method:
wxScopedCharBuffer const utf8 = s.utf8_str();
wr(utf8.data(), utf8.length());
Deserializing is as simple as using wxString::FromUTF8(data, length).
For UTF-16 you would use general mb_str(wxMBConvUTF16) and wxString(data, wxMBConvUTF16, length) methods, which could also be used with wxMBConvUTF8, but the UTF-8-specific methods above are more convenient and, in some build configurations, more efficient.
I am looking for a Java DataOutputStream equivalent for Dart where I can write arbitrary types (int, string, float, byte array etc). There is RandomAccessFile but it does not provide byte array or float-double values. ByteArray seems to have some necessary functions but I am not sure how to write it to a file or an OutputStream.
Here is some simple code showing how to write a ByteArray into an OutputStream:
#import('dart:io');
#import('dart:scalarlist');
main() {
File file = new File("c:\\temp\\foo.txt");
OutputStream os = file.openOutputStream();
os.onNoPendingWrites = () {
print('Finished writing. Closing.');
os.flush();
os.close();
};
Uint8List byteList = new Uint8List(64);
ByteArray byteArray = byteList.asByteArray();
int offset = 0;
offset = byteArray.setUint8(offset, 72);
offset = byteArray.setUint8(offset, 101);
offset = byteArray.setUint8(offset, 108);
offset = byteArray.setUint8(offset, 108);
offset = byteArray.setUint8(offset, 111);
offset = byteArray.setUint8(offset, 0);
byteArray.setFloat32(offset, 1.0);
os.write(byteList);
}
This has been around for a while, but I searched and didn't find good DataInput/OutputStream interoperability classes. I wanted a version that works with streams, so I could process files that don't comfortably fit in RAM. So I wrote one.
It's published over at https://pub.dev/packages/jovial_misc in io_streams, or if you prefer, https://github.com/zathras/misc/tree/master/dart/jovial_misc. I made it so it interoperates with java.io.DataInputStream and java.io.DataOutputStream. Code using it looks a little like this:
import 'package:convert/convert.dart';
import 'package:jovial_misc/io_utils.dart';
void main() async {
final acc = ByteAccumulatorSink();
final out = DataOutputSink(acc);
out.writeUTF8('Hello, world.');
out.close();
final stream = Stream<List<int>>.fromIterable([acc.bytes]);
final dis = DataInputStream(stream);
print(await dis.readUTF8());
await dis.close();
}
The Stream<List<int>> would of course typically come from a socket, or File.openRead(), etc. There's also a DataInputStream variant that is synchronous and takes an Iterable, if you do have all the byte data available up front.
DataInputStream and DataOutputSink are pretty much the obvious mapping of the java.io classes. The tricky part is the buffer management, since a stream shoves data at you in List<int> instances that probably aren't lined up with the data you want. And, of course, it's necessary to do everything asynchronously.
HTH.
You are essentially asking for arbitrary object serialization. And while the Dart VM has one, it isn't exposed to programmers (it is only used for snapshotting and message passing). I'd say that it would be a mistake to expose it -- in different situations, we have different requirements for serialization and "one true solution" isn't gonna work (Java showed us that already).
For example, I'm working on a MsgPack implementation for Dart, I know that Protobuf port is also in the works, maybe someone will start a Thrift port... the possibilities are endless.
The closest thing I could find is this package: https://github.com/TomCaserta/dart_io/ . Unfortunately there is a bug when reading to the end of the byte array - see my pull request in GitHub.
You could use this class:
https://github.com/TomCaserta/dart_io/blob/master/lib/data_output.dart
Unfortunately (a) it doesn't handle streams; (b) writeLong doesn't take a single integer. I have raised an issue for the Dart SDK: https://github.com/dart-lang/sdk/issues/31166
Edit: I have forked the dart_io package and fixed the two problems described above. My new package is published as dart_data_io:
https://github.com/markmclaren2/dart_data_io
I have a site where I allow members to upload photos. In the MVC Controller I take the FormCollection as the parameter to the Action. I then read the first file as type HttpPostedFileBase. I use this to generate thumbnails. This all works fine.
In addition to allowing members to upload their own photos, I would like to use the System.Net.WebClient to import photos myself.
I am trying to generalize the method that processes the uploaded photo (file) so that it can take a general Stream object instead of the specific HttpPostedFileBase.
I am trying to base everything off of Stream since the HttpPostedFileBase has an InputStream property that contains the stream of the file and the WebClient has an OpenRead method that returns Stream.
However, by going with Stream over HttpPostedFileBase, it looks like I am loosing ContentType and ContentLength properties which I use for validating the file.
Not having worked with binary stream before, is there a way to get the ContentType and ContentLength from a Stream? Or is there a way to create a HttpPostedFileBase object using the Stream?
You're right to look at it from a raw stream perspective because then you can create one method that handles streams and therefore many scenarios from which they come.
In the file upload scenario, the stream you're acquiring is on a separate property from the content-type. Sometimes magic numbers (also a great source here) can be used to detect the data type by the stream header bytes but this might be overkill since the data is already available to you through other means (i.e. the Content-Type header, or the .ext file extension, etc).
You can measure the byte length of the stream just by virtue of reading it so you don't really need the Content-Length header: the browser just finds it useful to know what size of file to expect in advance.
If your WebClient is accessing a resource URI on the Internet, it will know the file extension like http://www.example.com/image.gif and that can be a good file type identifier.
Since the file info is already available to you, why not open up one more argument on your custom processing method to accept a content type string identifier like:
public static class Custom {
// Works with a stream from any source and a content type string indentifier.
static public void SavePicture(Stream inStream, string contentIdentifer) {
// Parse and recognize contentIdentifer to know the kind of file.
// Read the bytes of the file in the stream (while counting them).
// Write the bytes to wherever the destination is (e.g. disk)
// Example:
long totalBytesSeen = 0L;
byte[] bytes = new byte[1024]; //1K buffer to store bytes.
// Read one chunk of bytes at a time.
do
{
int num = inStream.Read(bytes, 0, 1024); // read up to 1024 bytes
// No bytes read means end of file.
if (num == 0)
break; // good bye
totalBytesSeen += num; //Actual length is accumulating.
/* Can check for "magic number" here, while reading this stream
* in the case the file extension or content-type cannot be trusted.
*/
/* Write logic here to write the byte buffer to
* disk or do what you want with them.
*/
} while (true);
}
}
Some useful filename parsing features are in the IO namespace:
using System.IO;
Use your custom method in the scenarios you mentioned like so:
From an HttpPostedFileBase instance named myPostedFile
Custom.SavePicture(myPostedFile.InputStream, myPostedFile.ContentType);
When using a WebClient instance named webClient1:
var imageFilename = "pic.gif";
var stream = webClient1.DownloadFile("http://www.example.com/images/", imageFilename)
//...
Custom.SavePicture(stream, Path.GetExtension(imageFilename));
Or even when processing a file from disk:
Custom.SavePicture(File.Open(pathToFile), Path.GetExtension(pathToFile));
Call the same custom method for any stream with a content identifer that you can parse and recognize.