I am just updating one of my applications and I have an error with my decompression method.
This is the warning I am experiencing
Implicit conversion loses integer precision: 'unsigned long' to 'unit' (aka 'unsigned int')
this is the line of code its happening on
stream.avail_in = len - stream.total_in;
And this is what the whole method looks like
#pragma mark - ZLib Compression Methods
// Returns the decompressed version if the zlib compressed input data or nil if there was an error
- (NSData*) dataByDecompressingData:(NSData*)data {
NSLog(#"%lu", (unsigned long)data.length);
Byte* bytes = (Byte*)[data bytes];
NSInteger len = [data length];
NSMutableData *decompressedData = [[NSMutableData alloc] initWithCapacity:COMPRESSION_BLOCK];
Byte* decompressedBytes = (Byte*) malloc(COMPRESSION_BLOCK);
z_stream stream;
int err;
stream.zalloc = (alloc_func)0;
stream.zfree = (free_func)0;
stream.opaque = (voidpf)0;
stream.next_in = bytes;
err = inflateInit(&stream);
CHECK_ERR(err, #"inflateInit");
while (true) {
stream.avail_in = len - stream.total_in;
stream.next_out = decompressedBytes;
stream.avail_out = COMPRESSION_BLOCK;
err = inflate(&stream, Z_NO_FLUSH);
[decompressedData appendBytes:decompressedBytes length:(stream.total_out-[decompressedData length])];
if(err == Z_STREAM_END)
break;
CHECK_ERR(err, #"inflate");
}
err = inflateEnd(&stream);
CHECK_ERR(err, #"inflateEnd");
free(decompressedBytes);
return decompressedData;
}
First off, you should not use stream.total_in. It may or may not be a large enough type for your application. It is always unsigned long. Use your own total input counter sized for your application and ignore stream.total_in.
Second, I'm guessing that your CHECK_ERR() aborts somehow. You should not abort in the event of a Z_BUF_ERROR. In that case, you can continue by simply providing more input and/or more output space.
Third, the problem here is that you need to pick a stream.avail_in that is assured to fit in unsigned. You should be comparing the amount of remaining input to the largest value of unsigned, e.g. UINT_MAX or (unsigned)0 - 1. If the remaining data is larger, use the max value and deduct that from the remaining input. If smaller or equal, use all of it and set the remaining input to zero.
Related
I am decrypting a video file which works perfectly for small sizes files but for files above 300mb, there is memory crash. The code is as below :
I checked the start byte value it goes uptill 315mb and then crashes, my file is sized at 350mb.
It works for few iphones and crashes for few, The best solution was do it in chunks to avoid memory issue but it crashes doing that too.
#define kChunkSizeBytes (1024*1024) // 1 MB
#implementation NSMutableData (Crypto)
-(BOOL) doCrypto:(NSString *)key operation: (CCOperation) operation
{
//Keeping it 32 as per our key
char keyPtr[512 + 1]; // room for terminator (unused)
bzero(keyPtr, sizeof(keyPtr)); // fill with zeroes (for padding)
// Fetch key data
if (![key getCString:keyPtr maxLength:sizeof(keyPtr) encoding:NSUTF8StringEncoding]) {return FALSE;} // Length of 'key' is bigger than keyPtr
CCCryptorRef cryptor;
CCCryptorStatus cryptStatus = CCCryptorCreate(operation, kCCAlgorithmRC4, 0,
keyPtr, key.length,
NULL, // IV - needed?
&cryptor);
if (cryptStatus != kCCSuccess) { // Handle error here
return FALSE;
}
size_t dataOutMoved;
size_t dataInLength = kChunkSizeBytes; // #define kChunkSizeBytes (16)
size_t dataOutLength = CCCryptorGetOutputLength(cryptor, dataInLength, FALSE);
size_t totalLength = 0; // Keeps track of the total length of the output buffer
size_t filePtr = 0; // Maintains the file pointer for the output buffer
NSInteger startByte; // Maintains the file pointer for the input buffer
char *dataIn = malloc(dataInLength);
char *dataOut = malloc(dataOutLength);
NSRange bytesRange = NSMakeRange((NSUInteger) 0, (NSUInteger) 0);
for (startByte = 0; startByte <= [self length]; startByte += kChunkSizeBytes) {
if ((startByte + kChunkSizeBytes) > [self length]) {
dataInLength = [self length] - startByte;
}
else {
dataInLength = kChunkSizeBytes;
}
// Get the chunk to be ciphered from the input buffer
bytesRange = NSMakeRange((NSUInteger) startByte, (NSUInteger) dataInLength);
[self getBytes:dataIn range:bytesRange];
cryptStatus = CCCryptorUpdate(cryptor, dataIn, dataInLength, dataOut, dataOutLength, &dataOutMoved);
if (startByte >= 203728200) {
NSLog(#"%ld",(long)startByte);
}
if (dataOutMoved != dataOutLength) {
NSLog(#"dataOutMoved (%d) != dataOutLength (%d)", dataOutMoved, dataOutLength);
}
if ( cryptStatus != kCCSuccess)
{
NSLog(#"Failed CCCryptorUpdate: %d", cryptStatus);
}
// Write the ciphered buffer into the output buffer
bytesRange = NSMakeRange(filePtr, (NSUInteger) dataOutMoved);
[self replaceBytesInRange:bytesRange withBytes:dataOut];
totalLength += dataOutMoved;
filePtr += dataOutMoved;
}
// Finalize encryption/decryption.
cryptStatus = CCCryptorFinal(cryptor, dataOut, dataOutLength, &dataOutMoved);
totalLength += dataOutMoved;
if ( cryptStatus != kCCSuccess)
{
NSLog(#"Failed CCCryptorFinal: %d", cryptStatus);
}
// In the case of encryption, expand the buffer if it required some padding (an encrypted buffer will always be a multiple of 16).
// In the case of decryption, truncate our buffer in case the encrypted buffer contained some padding
[self setLength:totalLength];
// Finalize the buffer with data from the CCCryptorFinal call
NSRange bytesNewRange = NSMakeRange(filePtr, (NSUInteger) dataOutMoved);
[self replaceBytesInRange:bytesNewRange withBytes:dataOut];
CCCryptorRelease(cryptor);
free(dataIn);
free(dataOut);
return 1;
}
#end
If replaceBytesInRange:bytesRange is causing the crash, then my first suggestion for how to avoid the crash is to add error checking for the preceding function calls that it depends upon.
For example, in those cases where it crashes, maybe bytesRange is not a valid / useable value. Maybe its dataOut that is not valid / usable. Your code in your question sets those values from function calls, but does not check the return value for error conditions / error indicators / invalid values.
It might be a related dependent function call. e.g., cryptStatus is set with a call to CCCryptorUpdate(), which has dataOut as an input parameter. I don't know Objective-C, and I'm not familiar with the function CCCryptorUpdate(), but it looks like it would affect / populate dataOut. If it's actually returning an error, then dataOut is probably not in a usable state by the time you use it on the replaceBytesInRange line. Checking the return value of cryptStatus might flag conditions when you should not proceed to use dataOut in later calls.
Which brings me to another thing I noticed : you do check a couple things, but only log them. The checks for if (dataOutMoved != dataOutLength) and for (cryptStatus != kCCSuccess) look like things that should stop execution, or break out of the loop, or something like that, not just log the occurrence.
Another thing I see is that dataOut is malloc()'d once, not cleared, and used repeatedly. This can be entirely valid in some circumstances, but it can also cause exactly the kind of error you are seeing. Does anything in your code assume anything about the content of dataOut? I'm thinking along the lines of C string operations that assume null-termination. If one is not careful, null terminators that are assumed to be there (or maybe are in fact there at first) can be overwritten if, say, the entire buffer is filled. Again, I don't know Objective-C and these functions so well, so this is not meant as a specific statement, but more of an analogy of the kind of thing that might be happening.
TL; DR : Add error checking to each of your function calls, and respond accordingly (break, exit, retry, whatever) so that no later function call attempts to use values that indicate errors or are otherwise invalid.
I'll wager that with added error checking you will (1) stop the crashes, and (2) learn something specific about why it is that files larger than a certain amount yield these crashes.
I am working on a bluetooth iOS project and have managed to get some data from the bluetooth device.
However, I am struggling to convert this data into something useful, such as an NSString. Whenever I try to NSLog the NSString that was converted from the NSData received, it is a bunch of gibberish. The output is:
ēဥ၆䄀
The bluetooth device is a heart monitor from a manufacturer in Asia and they have provided the protocol reference on how to make calls to the device. This one thing they mention in the protocol reference:
The PC send 16-byte packets to the device, then the device sent back the 16-byte packets. Except for some special commands, all others can use this communication mode.
Can anyone tell me what I am doing wrong? I have tried everything I know, including every single encoding in the apple docs as well as both initWithData and initWithBytes. Thanks!
-(void)peripheral:(CBPeripheral *)peripheral didUpdateValueForCharacteristic:(CBCharacteristic *)characteristic
error:(NSError *)error {
if (error)
{
NSLog(#"erorr in read is %#", error.description);
return;
}
NSData *data= characteristic.value;
NSString *myString = [[NSString alloc] initWithBytes:[data bytes] length:[data length] encoding:NSUTF16StringEncoding];
NSLog(#"Value from device is %#", myString); //OUTPUT IS ēဥ၆䄀
}
What you have here is a string of raw data that can't be directly converted into a human readable string - unless you consider hex-representation to be human readable :)
To make sense of this data you need to either have a protocol specification at hand or prepare for hours (sometimes) days of reverse-engineering.
This byte-sequence can be composed of multiple values formatted in standard (float IEEE 754, uint8_t, uint16_t...) or even proprietary formats.
One important thing to consider when communicating with the outside world is also endianness (ie: does the 'biggest' byte in multi-byte format come first or last).
There are many ways to manipulate this data. To get the raw array of bytes you could do:
NSData *rxData = ...
uint8_t *bytes = (uint8_t *)[rxData bytes];
And then if (for example) first byte tells you what type of payload the string holds you can switch like:
switch (bytes[0])
{
case 0x00:
//first byte 0x00: do the parsing
break;
case 0x01:
//first byte 0x01: do the parsing
break;
// ...
default:
break;
}
Here would be an example of parsing data that consists of:
byte 0: byte holding some bit-coded flags
bytes 1,2,3,4: 32-bit float
bytes 5,6: uint16_t
bool bitFlag0;
bool bitFlag1;
bool bitFlag2;
bool bitFlag3;
uint8_t firstByte;
float theFloat;
uint16_t theInteger;
NSData *rxData = ...
uint8_t *bytes = (uint8_t *)[rxData bytes];
// getting the flags
firstByte = bytes[0];
bitFlag0 = firstByte & 0x01;
bitFlag1 = firstByte & 0x02;
bitFlag2 = firstByte & 0x04;
bitFlag3 = firstByte & 0x08;
//getting the float
[[rxData subdataWithRange:NSMakeRange(1, 4)] getBytes:&theFloat length:sizeof(float)];
NSLog (#"the float is &.2f",theFloat);
//getting the unsigned integer
[[data subdataWithRange:NSMakeRange(6, 2)] getBytes:&theInteger length:sizeof(uint16_t)];
NSLog (#"the integer is %u",theInteger);
One note: depending on the endianness you might need to reverse the 4-float or the 2-uint16_t bytes before converting them. Converting this byte arrays can also be done with unions.
union bytesToFloat
{
uint8_t b[4];
float f;
};
and then:
bytesToFloat conv;
//float would be written on bytes b1b2b3b4 in protocol
conv.b[0] = bytes[1]; //or bytes[4] .. endianness!
conv.b[1] = bytes[2]; //or bytes[3] .. endianness!
conv.b[2] = bytes[3]; //or bytes[2] .. endianness!
conv.b[3] = bytes[4]; //or bytes[1] .. endianness!
theFloat = conv.f,
If for example you know that byte6 and byte7 represent an uint16_t value you can calculate it from raw bytes:
value = uint16_t((bytes[6]<<8)+bytes[7]);
or (again - endianness):
value = uint16_t((bytes[7]<<8)+bytes[6]);
One more note: using simply sizeof(float) is a bit risky since float can be 32-bit on one platform and 64-bit on another.
I'm developing an app which pass bytes through the network, the server declares its bytes order is Big-Endian. In my app, I wrap my data with a header which takes 2 bytes, I assign the byte as following :
int length = [self.dataLengthHeader length];
if (length <= 255) {
high = 0;
low = length;
}else if (length == 256)
{
high = 1;
low = 0;
}else {
high = length/256;
low = length%256;
}
Byte byte[] = {high, low};
NSLog(#"%hhu %hhu", high, low);
NSMutableData *dataToSend = [NSMutableData dataWithBytes:byte length:2];
For example
The first byte is 00(8 bits), the second is 05(8 bits)
Where another app receives the header, it parse the header which takes 2 bytes into two int(two NSInteger would be better) to get the information of the real message.
NSData *twoBytes = [NSData dataWithBytes:payloadptr length:2];
NSData *low = [twoBytes subdataWithRange:NSMakeRange(1, 1)];
int lowP;
[low getBytes:&lowP length:sizeof(lowP)];
NSData *high = [twoBytes subdataWithRange:NSMakeRange(0, 1)];
int highP;
[high getBytes:&highP length:sizeof(highP)];
Memory shortcut:
When I log out the bytes, it turns out to be something like this:highP = 70074112 lowP = 365573
I can never get the correct result, anybody could help me?
Any help would be appreciated!
Read about serialization.
You could make yourself using e.g. htonl(3), or endian(3).
You could use XDR with RPCGEN, or ASN.1.
You could use libs11n (in C++). You could also consider protocol buffers, etc...
Unless you have a lot of data, or bandwidth, you may consider using textual serialization formats like JSON (there are somehow flexible, easier to debug, etc...) or binary counterparts like BSON. Notice that sending data on a network is much slower than your CPU, so the overhead of textual serialization is generally lost in the noise (even if you compress it).
I have a fairly complex issue regarding the interpretation of packets in an app that I am making. A host app sends a packet to client apps with the following structure:
[Header of 10 bytes][peerID of selected client of variable byte length][empty byte][peerID of a client of variable byte length][empty byte][int of 4 bytes][peerID of client of variable byte length][empty byte][int of 4 bytes]
Here is a sample packet that is produced under this structure:
434e4c50 00000000 006a3134 31303837 34393634 00313233 38313638 35383900 000003e8 31343130 38373439 36340000 0003e8
Converted it looks like this:
CNLP j1410874964 1238168589 Ë1410874964 Ë
"CNLP j" is the packet header of 10 bytes. "1410874964" is the peerID of the selected client. "1238168589" is the peerID of another client. " Ë" has an int value of 1000. "1410874964" is the peerID of the other client (in this case, the selected client). " Ë" also has an int value of 1000. Basically, in this packet I am communicating 2 things - who the selected client is and the int value associated with each client.
My problem exists on the interpretation side (client side). To interpret this particular type of packet, I use the following method:
+ (NSMutableDictionary *)infoFromData:(NSData *)data atOffset:(size_t) offset
{
size_t count;
NSMutableDictionary *info = [NSMutableDictionary dictionaryWithCapacity:8];
while (offset < [data length])
{
NSString *peerID = [data cnl_stringAtOffset:offset bytesRead:&count];
offset += count;
NSNumber *number = [NSNumber numberWithInteger:[data cnl_int32AtOffset:offset]];
offset += 4;
[info setObject:number forKey:peerID];
}
return info;
}
Typically, each of these packets range between 49 and 51 bytes. "offset" is set in a previous method to reflect the byte number after the packet header plus the empty byte after the selected player (in the case of the above packet, 21). "count" is initialized with a value of 1. In the case of this particular example, length is 51. The following method is passed the above arguments:
- (NSString *)cnl_stringAtOffset:(size_t)offset bytesRead:(size_t *)amount
{
const char *charBytes = (const char *)[self bytes];
NSString *string = [NSString stringWithUTF8String:charBytes + offset];
*amount = strlen(charBytes + offset) + 1;
return string;
}
This method is supposed to read through a variable length string in the packet, set the offset to the byte immediately after the empty byte pad behind the peerID string, and return the string that was read. "amount" is then set to the number of bytes the method read through for the string (this is becomes the new value of count after returning to the first method). "offset" and "count" are then added together to become the new "offset" - where interpretation of the int portion of the packet will begin. The above arguments are passed to the following method:
- (int)cnl_int32AtOffset:(size_t)offset
{
const int *intBytes = (const int *)[self bytes];
return ntohl(intBytes[offset / 4]);
}
This method is intended to return the 32 bit (4 byte) int value read at the current offset value of the packet. I believe that the problem exists in this method when the offset is a number that is not divisible by 4. In this case, the first int value of 1000 was correctly interpreted, and 32 was returned as the offset during the first iteration of the while loop. However, during the second iteration, the int value interpreted was 909377536 (obtained from reading bytes 36340000 in the packet instead of bytes 000003E8) This was likely due to the fact that the offset during this iteration was set to 47 (not divisible by 4). After interpreting the 32 bit int in the category above, 4 is added to the offset in the first method to account for a 4 byte (32 bit int). If my intuition about an offset not divisible by zero is correct, any suggestions to get around this problem are greatly appreciated. I have been looking for a way to solve this problem for quite some time and perhaps fresh eyes may help. Thanks for any help!!!
The unportable version (undefined behaviour for many reasons):
return ntohl(*(const int *)([self bytes]+offset));
A semi-portable version is somewhat trickier, but in C99 it appears that you can assume int32_t is "the usual" two's complement representation (no trap representations, no padding bits), thus:
// The cast is necessary to prevent arithmetic on void* which is nonstandard.
const uint8_t * p = (const uint8_t *)[self bytes]+offset;
// The casts ensure the result type is big enough to hold the shifted value.
// We use uint32_t to prevent UB when shifting into the sign bit.
uint32_t n = ((uint32_t)p[0]<<24) | ((uint32_t)p[1]<<16) | ((uint32_t)p[2]<<8) | ((uint32_t)p[3]);
// Jump through some hoops to prevent UB on "negative" numbers.
// An equivalent to the third expression is -(int32_t)~n-1.
// A good compiler should be able to optimize this into nothing.
return (n <= INT32_MAX) ? (int32_t)n : -(int32_t)(UINT32_MAX-n)-1;
This won't work on architectures without 8-bit bytes, but such architectures probably have different conventions for how things are passed over the network.
A good compiler should be able to optimize this into a single (possibly byte-swapped) load on suitable architectures.
I have a binary file I've loaded using an NSData object. Is there a way to locate a sequence of characters, 'abcd' for example, within that binary data and return the offset without converting the entire file to a string? Seems like it should be a simple answer, but I'm not sure how to do it. Any ideas?
I'm doing this on iOS 3 so I don't have -rangeOfData:options:range: available.
I'm going to award this one to Sixteen Otto for suggesting strstr. I went and found the source code for the C function strstr and rewrote it to work on a fixed length Byte array--which incidentally is different from a char array as it is not null terminated. Here is the code I ended up with:
- (Byte*)offsetOfBytes:(Byte*)bytes inBuffer:(const Byte*)buffer ofLength:(int)len;
{
Byte *cp = bytes;
Byte *s1, *s2;
if ( !*buffer )
return bytes;
int i = 0;
for (i=0; i < len; ++i)
{
s1 = cp;
s2 = (Byte*)buffer;
while ( *s1 && *s2 && !(*s1-*s2) )
s1++, s2++;
if (!*s2)
return cp;
cp++;
}
return NULL;
}
This returns a pointer to the first occurrence of bytes, the thing I'm looking for, in buffer, the byte array that should contain bytes.
I call it like this:
// data is the NSData object
const Byte *bytes = [data bytes];
Byte* index = [self offsetOfBytes:tag inBuffer:bytes ofLength:[data length]];
Convert your substring to an NSData object, and search for those bytes in the larger NSData using rangeOfData:options:range:. Make sure that the string encodings match!
On iPhone, where that isn't available, you may have to do this yourself. The C function strstr() will give you a pointer to the first occurrence of a pattern within the buffer (as long as neither contain nulls!), but not the index. Here's a function that should do the job (but no promises, since I haven't tried actually running it...):
- (NSUInteger)indexOfData:(NSData*)needle inData:(NSData*)haystack
{
const void* needleBytes = [needle bytes];
const void* haystackBytes = [haystack bytes];
// walk the length of the buffer, looking for a byte that matches the start
// of the pattern; we can skip (|needle|-1) bytes at the end, since we can't
// have a match that's shorter than needle itself
for (NSUInteger i=0; i < [haystack length]-[needle length]+1; i++)
{
// walk needle's bytes while they still match the bytes of haystack
// starting at i; if we walk off the end of needle, we found a match
NSUInteger j=0;
while (j < [needle length] && needleBytes[j] == haystackBytes[i+j])
{
j++;
}
if (j == [needle length])
{
return i;
}
}
return NSNotFound;
}
This runs in something like O(nm), where n is the buffer length, and m is the size of the substring. It's written to work with NSData for two reasons: 1) that's what you seem to have in hand, and 2) those objects already encapsulate both the actual bytes, and the length of the buffer.
If you're using Snow Leopard, a convenient way is the new -rangeOfData:options:range: method in NSData that returns the range of the first occurrence of a piece of data. Otherwise, you can access the NSData's contents yourself using its -bytes method to perform your own search.
I had the same problem.
I solved it doing the other way round, compared to the suggestions.
first, I reformat the data (assume your NSData is stored in var rawFile) with:
NSString *ascii = [[NSString alloc] initWithData:rawFile encoding:NSAsciiStringEncoding];
Now, you can easily do string searches like 'abcd' or whatever you want using the NSScanner class and passing the ascii string to the scanner. Maybe this is not really efficient, but it works until the -rangeOfData method will be available for iPhone also.