I have many float data which is generated from an image. I want to store it to a file, like XX.dat ( general in C). and I will read it again to do further processing.
I have method to represent float by nsstring and write it in to .txt file. but it is too slow. Is there some function which is same as fwrite( *data , *pfile) and fread(*buf, *pfile) in c? or some new idea?
many thanks!
In iOS you can still make use of the standard low-level file (and socket, among other things) API's. So you can use fopen(), fwrite(), fread(), etc. just as you would in any other C program.
This question has some examples of using the low-level file API on iOS: iPhone Unzip code
Another option to consider is writing your floats into something like an NSMutableData instance, and then writing that to file. That will be faster than converting everything to strings (you'll get a binary file instead of a text one), though probably still not as fast as using the low-level API's. And you'd probably have to use something like this to convert between floats and byte-arrays.
If you are familiar with lower level access, you could mmap your file, and then access the data directly just as you would any allocated memory.
Related
I have a simple question.
I want to use AnsiString as a container for binary data. I mostly load such data from TMemoryStream or TFileStream and I save it back from AnsiString after some processing. Works fine, haven't found a problem with that.
But from what I've seen using it like that sparcles debates to use Sysutils::TBytes instead. Why? Sysutils::TBytes has much fewer useful methods which I can use to manipulate data stored inside for example AnsiString. It is clearly half-finished container, compared to AnsiString.
Is the only problem I should care about conversion to regular string or is there something else why I should really use the less-than-adequate TBytes instead? I do not make conversions of AnsiString to other string types - that is what is quoted as a possible problem elsewhere.
An example of how I load data:
AnsiString data;
boost::scoped_ptr<TFileStream> fs(new TFileStream(FileName, fmOpenRead | fmShareDenyWrite));
data.SetLength(fs->Size);
fs->Read(data.c_str(), fs->Size);
An example how I save data:
// fs wants void * so I have to use data.data() instead of data.c_str() here
fs->Write(data.data(), data.Length());
So it should be safe to store binary data correct?
I want to use AnsiString as a container for binary data.
One word - DON'T! It will bite you someday. Use a more appropriate container, such as TBytes, TMemoryStream, std::vector<byte>, etc.
Works fine, haven't found a problem with that.
Consider yourself lucky. From C++Builder 2009 onwards, AnsiString is codepage-aware, and it WILL cause data conversions if you are not VERY careful when passing AnsiString around. Sooner or later, you are likely to slip up and it will risk corrupting your binary data.
But from what I've seen using it like that sparcles debates to use Sysutils::TBytes instead. Why?
Because it is an actual raw binary container meant specifically for raw bytes.
Sysutils::TBytes has much fewer useful methods which I can use to manipulate data stored inside for example AnsiString.
You should not be manipulating binary data as text to begin with. And since you are using things like Boost and STL, you should consider using their binary containers instead. They have more functions available.
That being said, XE7 does introduce some new functions for manipulating Delphi-style dynamic arrays (like TBytes) including inserts, deletes, and concatenations:
String-Like Operations Supported on Dynamic Arrays
It does not look like those new functions made it into C++Builder's DynamicArray class (which TBytes is a typedef of), though.
It is clearly half-finished container, compared to AnsiString.
AnsiString is a container of text characters. Period. Always has been, always will be. People ABUSE it by taking advantage of the fact that sizeof(char)==sizeof(byte). That worked up to a point, but it has become dangerous in recent years to continue abusing it.
Is the only problem I should care about conversion to regular string or is there something else why I should really use the less-than-adequate TBytes instead?
That, and the fact that Embarcadero has been phasing out AnsiString since 2009. 8bit strings are disabled in the mobile compilers, it is only a matter of time before the desktop compilers follow suit.
Why are you wanting to manipulate raw bytes as strings to begin with? Can you provide an example of something you can do with AnsiString that you cannot do with TBytes?
So it should be safe to store binary data correct?
In your specific example, yes (and yes, you can use c_str() instead of data() when calling fs->Write()).
I have a large text resource in my bundle. It's a CSV containing lines like
0,1,100,2.2345
It's over 9MB. What's the best way to open and sequentially read it in so I can do something like:
myObject->initData(col0, col1, col2, col3);
(Which just stuffs the float value into one of a number of multidimensional arrays indexed by the integers in the file.)
I tried reading it into a string using [NSString stringWithContentsOfFile:] and using an NSScanner to loop over it, but I don't want to double up my memory usage, even temporarily. It was also quite slow.
What's the best way to do this?
Thanks
You can use NSFileHandle or C APIs to read incrementally, or you can use mmap with NSData.
The remainder is basic cstring buffer handling, or you could use NSString line-by-line.
Your life may be easier if you can export it as a sequence of binary floats, rather than CSV.
I am looking to create a navigation based reference app for the iOS.
I have considered the following ways to store the data:
hard coding
plist file
some kind of comma delimited file
The data structure that I will be using has a bunch of strings, an array, and a reference to a picture.
What do you think the best way to store this data is without getting into CoreData?
Thanks
Also I dont think it would be more than 500 entries.
Well, this is not a "best-practice" problem for any case.
The data structure that I will be using has a bunch of strings, an array, and a reference to a picture.
What do you think the best way to store this data is without getting into CoreData?
For your needs, I suggest you look into NSKeyedArchiver.
NSString, NSArray, and UIImage all know how to encode and decode themselves. Just use an NSKeyedArchiver. Note that the objects in your collections (e.g. NSArray) must adopt #protocol NSCoding.
If you need to open this on a mac, then convert the UIImage to NSData using a proper image file format representation (e.g. PNG or JPEG) because UIImage is not available.
In detail:
hard coding
That could mean a number of things.
plist file
You're working with large non-plist types. That would mean you would need to convert to and from UIImage<->NSData unnecessarily, which would add a lot of overhead -- memory, CPU, and potentially file size. All these types can encode themselves better than (or as good as) a plist representation.
some kind of comma delimited file
Your image will not allow that to happen (reliably).
I recorded a radio signal into a .wav, I can open it in audacity and see that there is binary data encoded using a certain algorithm. Does anyone know of a way to process the signal that is contained within the .wav? so that i can extract the binary data from it?
I know that I need to know the encoding algorithm for it to work properly, anyone know of any program that does something like that?
Thanks
sox will convert most audio formats to most other audio formats - including raw binary.
The .wav format is generally very simple and wav files usually don't have compressed data. It's quite feasible to parse it yourself, but much easier to use something already made. So the short answer is to find something that can read wav files in your language of choice.
Here's an example in Python, using the wave module:
import wave
w = wave.open("myfile.wav", "rb")
binary_data = w.readframes(w.getnframes())
w.close()
Now where you go depends on what else you want to do. binary_data is now a python string of the raw bytes. If you just want to chop this and repackage it, it's probably easiest to leave it in this form. If you want to manipulated the data, such as scale it, interpolate, filter, etc, you would probably want to convert this into a sequence of numbers, and for this, in Python, you'd want to convert it to a numpy array. You could do this yourself using the struct module, which is for interpreting strings as packed binary data, or you could just have read in the data using scipy.io.wave module which does this for you. As you can see, most of this becomes fairly language dependent quickly.
I have an application for storing many strings in a TStringList. The strings will be largely similar to one another and it occurs to me that one could compress them on the fly - i.e. store a given string in terms of a mixture of unique text fragments plus references to previously stored fragments. StringLists such as lists of fully-qualified path and filenames should be able to be compressed greatly.
Does anyone know of a TStringlist descendant that implement this - i.e. provides read and write access to the uncompressed strings but stores them internally compressed, so that a TStringList.SaveToFile produces a compressed file?
While you could implement this by uncompressing the entire stringlist before each access and re-compressing it afterwards, it would be unnecessarily slow. I'm after something that is efficient for incremental operations and random "seeks" and reads.
TIA
Ross
I don't think there's any freely available implementation around for this (not that I know of anyway, although I've written at least 3 similar constructs in commercial code), so you'd have to roll your own.
The remark Marcelo made about adding items in order is very relevant, as I suppose you'll probably want to compress the data at addition time - having quick access to entries already similar to the one being added, gives a much better performance than having to look up a 'best fit entry' (needed for similarity-compression) over the entire set.
Another thing you might want to read up about, are 'ropes' - a conceptually different type than strings, which I already suggested to Marco Cantu a while back. At the cost of a next-pointer per 'twine' (for lack of a better word) you can concatenate parts of a string without keeping any duplicate data around. The main problem is how to retrieve the parts that can be combined into a new 'rope', representing your original string. Once that problem is solved, you can reconstruct the data as a string at any time, while still having compact storage.
If you don't want to go the 'rope' route, you could also try something called 'prefix reduction', which is a simple form of compression - just start out each string with an index of a previous string and the number of characters that should be treated as a prefix for the new string. Be aware that you should not recurse this too far back, or access-speed will suffer greatly. In one simple implementation, I did a mod 16 on the index, to establish the entry at which prefix-reduction started, which gave me on average about 40% memory savings (this number is completely data-dependant of course).
You could try to wrap a Delphi or COM API around Judy arrays. The JudySL type would do the trick, and has a fairly simple interface.
EDIT: I assume you are storing unique strings and want to (or are happy to) store them in lexicographical order. If these constraints aren't acceptable, then Judy arrays are not for you. Mind you, any compression system will suffer if you don't sort your strings.
I suppose you expect general flexibility from the list (including delete operation), in this case I don't know about any out of the box solution, but I'd suggest one of the two approaches:
You split your string into words and
keep separated growning dictionary
to reference the words and save list of indexes internally
You implement something related to
zlib stream available in Delphi, but operating by the block that
for example can contains 10-100
strings. In this case you still have
to recompress/compress the complete
block, but the "price" you pay is lower.
I dont think you really want to compress TStrings items in memory, because it terribly ineffecient. I suggest you to look at TStream implementation in Zlib unit. Just wrap regular stream into TDecompressionStream on load and TCompressionStream on save (you can even emit gzip header there).
Hint: you will want to override LoadFromStream/SaveToStream instead of LoadFromFile/SaveToFile