There is a strange problem I've run in using RIM compression API, I can't make it work as it's described in documentation.
If I gzip plain text file using win gzip tool, add gz to resources of blackberry project and in app try to decompress it, there will be infinite loop, gzis.read() never return -1...
try
{
InputStream inputStream = getClass().getResourceAsStream("test.gz");
GZIPInputStream gzis = new GZIPInputStream(inputStream);
StringBuffer sb = new StringBuffer();
char c;
while ((c = (char)gzis.read()) != -1)
{
sb.append(c);
}
String data = sb.toString();
add(new RichTextField(data));
gzis.close();
}
catch(IOException ioe)
{
}
After the compressed content there is repetition of 65535 value in gzis.read(). The only workaround I've found is dumb
while ((c = (char)gzis.read()) != -1 && c != 65535)
But I'm curious what is the reason, what I'm doing wrong, and why 65535?
char is an unsigned, 16-bit data type. -1 cast to a char is 65535.
Change to:
int i;
while ((i = gzis.read()) != -1)
{
sb.append((char)i);
}
And it should work. The example on RIM's API can't possibly work, as no char will ever equal -1.
Related
I have a GUI written in R that utilizes Tcl/TK package as well a C .dll that also uses Tcl library. I have done some research on this issue, and it seems to be memory related. I am an inexperienced programmer, so I am not sure where I should be looking for this memory issue. Each call of malloc() has a matching free(), and same with the analogous Tcl_Alloc() and Tcl_Free(). This error is very hard to reproduce as well, thus I am afraid I cannot provide a reproducible example as it is seemingly random in nature. One pattern is however that it seems to only happen upon closure of the program, though this is very inconsistent.
By making this post, I am hoping to gain a logical process that one should take in an attempt to debug this problem in a general context under Tcl/Tk - C - R applications. I am not looking for a solution specific to my code, but rather what an individual should think about when encountering this problem.
The message comes from the function Ptr2Block() in tclThreadAlloc.c (or there's something else about which produces the same error message; possible but unlikely) which is Tcl's thread-specific memory allocator (which is used widely inside Tcl to reduce the number of times global locks are hit). Specifically, it's this bit:
if (blockPtr->magicNum1 != MAGIC || blockPtr->magicNum2 != MAGIC) {
Tcl_Panic("alloc: invalid block: %p: %x %x",
blockPtr, blockPtr->magicNum1, blockPtr->magicNum2);
}
The problem? Those zeroes should be MAGIC (which is equal to 0xEF). This indicates that something has overwritten the memory block's metadata — which also should include the size of the block, but that is now likely hot garbage — and program memory integrity can no longer be trusted. Alas, at this point we're now dealing with a program in a broken state where the breakage happened some time previously; the place where the panic happened is merely where detection of the bug happened, not the actual location of the bug.
Debugging further is usually done by building a version of everything with fancy memory allocators turned off (in Tcl's code, this is done by defining the PURIFY symbol when building) and then running the resulting code — which hopefully still has the bug — with a tool like electricfence or purify (hence the special symbol name) to see what sort of out-of-bounds errors are found; they're very good at hunting down this sort of issue.
I would advise you to start by having a closer look to the sizeof() values provided to your Tcl_Alloc() calls in this C .dll.
I'm writing myself a Tcl binding for a C library and I faced recently exactly the same problem and therefore I'm assuming you may have the same error than me in your code.
Here below a minimal example that reproduces the problem:
#include <tcl.h>
#include <stdlib.h> // malloc
static unsigned int dataCtr;
struct tDataWrapper {
const char *str; // Tcl_GetCommandName(interp, cmd)
unsigned int n; // dataCtr value
void *data; // pointer to wrapped object
};
static void wrapDelCmd(ClientData clientData)
{
struct tDataWrapper *wrap = (struct tDataWrapper *) clientData;
if (wrap != NULL) {
/* with false sizeof value provided while creating the wrapper
* (see above), this data pointer would overwrite the
* overhead section of the allocated tcl memory block
* from what I understood and this is what can be causing
* the panic with message like following one when the
* memory is freed with ckfree (here after calling unload)
* alloc: invalid block: 0000018F2624E760: 0 0 */
printf("DEBUG: #%s(%s) &wrap->data #%p\n",
__func__, wrap->str, &wrap->data);
if (wrap->data != NULL) {
// call your wrapped API to deinstantiate the object
}
ckfree(wrap);
}
}
static int wrapCmd(ClientData clientData, Tcl_Interp *interp,
int objc, Tcl_Obj *const objv[])
{
struct tDataWrapper *wrap = (struct tDataWrapper *) clientData;
if (wrap == NULL)
return TCL_ERROR;
else if (wrap->data != NULL) {
// call your wrapped API to do something with instantiated object
return TCL_OK;
} else {
Tcl_Obj *obj = Tcl_ObjPrintf("wrap: {str=\"%s\", n=%u, data=%llx}",
wrap->str, wrap->n, (unsigned long long) wrap->data);
if (obj != NULL) {
Tcl_SetObjResult(interp, obj);
return TCL_OK;
} else
return TCL_ERROR;
}
}
static int newCmd(ClientData clientData, Tcl_Interp *interp,
int objc, Tcl_Obj *const objv[])
{
struct tDataWrapper *wrap;
Tcl_Obj *obj;
Tcl_Command cmd;
// 3) this is correct
// if ((wrap = attemptckalloc(sizeof(struct tDataWrapper))) == NULL)
// 2) still incorrect but GCC gives more warning regarding the inconsistent pointer handling
// if ((wrap = malloc(sizeof(struct tDataWrapper *))) == NULL)
// 1) this is incorrect
if ((wrap = attemptckalloc(sizeof(struct tDataWrapper *))) == NULL)
Tcl_Panic("%s:%u: attemptckalloc failed\n", __func__, __LINE__);
else if ((obj = Tcl_ObjPrintf("data%u", dataCtr+1)) == NULL)
Tcl_Panic("%s:%u: Tcl_ObjPrintf failed\n", __func__, __LINE__);
else if ((cmd = Tcl_CreateObjCommand(interp, Tcl_GetString(obj),
wrapCmd, (ClientData) wrap, wrapDelCmd)) == NULL)
Tcl_Panic("%s:%u: Tcl_CreateObjCommand failed\n", __func__, __LINE__);
else {
wrap->str = Tcl_GetCommandName(interp, cmd);
wrap->n = dataCtr;
wrap->data = NULL; // call your wrapped API to instantiate an object
dataCtr++;
Tcl_SetObjResult(interp, obj);
}
return TCL_OK;
}
int Allocinvalidblock_Init(Tcl_Interp *interp)
{
dataCtr = 0;
return (Tcl_CreateObjCommand(interp, "new",
newCmd, (ClientData) NULL, NULL)
== NULL) ? TCL_ERROR : TCL_OK;
}
int Allocinvalidblock_Unload(Tcl_Interp *interp, int flags)
{
Tcl_Namespace *ns = Tcl_GetGlobalNamespace(interp);
Tcl_Obj *obj;
Tcl_Command cmd;
unsigned int i;
for(i=0; i<dataCtr; i++) {
if ((obj = Tcl_ObjPrintf("data%u", i+1)) != NULL) {
if ((cmd = Tcl_FindCommand(interp,
Tcl_GetString(obj), ns, TCL_GLOBAL_ONLY)) != NULL)
Tcl_DeleteCommandFromToken(interp, cmd);
Tcl_DecrRefCount(obj);
}
}
return TCL_OK;
}
Once built (for example with Code::Blocks as shared library project linking against C:/msys64/mingw64/lib/libtcl.dll.a), the error can be triggered when more than a data object is created and the library immediately unloaded:
load bin/Release/libAllocInvalidBlock.dll
new
new
unload bin/Release/libAllocInvalidBlock.dll
If used otherwise the crash may even be not triggered... Anyway, such an error in the C code is not particularly obvious to identify (although easy to fix) because the compilation is running without any warning (although -Wall compiler flag is set).
I am working on an Arduino project which is required an authorized authentication based on OAuth 1.0 to connects to the cloud. This is alike [Authorizing a request to Twitter API][1], and I am stuck in the step of [Creating a signature][2]. The whole process of creating a signature requires algorithms like encodeURL, base64encode, and hmac-sha1. On my Arduino project, I use Cryptosuite(link 3) library for hmac-sha1 and arduino-base64(link 4) library for base64encode. Both of them are working fine separately. However, I need to get a base64-formatted output of hmac-sha1. So I have tried this:
#include <avr/pgmspace.h>
#include <sha1.h>
#include <Base64.h>
uint8_t *in, out, i;
char b64[29];
static const char PROGMEM b64chars[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
char key[] = "testKey";
char basestring[] = "testing";
void printHash(uint8_t* hash) {
int i;
for (i=0; i<20; i++) {
Serial.print("0123456789abcdef"[hash[i]>>4]);
Serial.print("0123456789abcdef"[hash[i]&0xf]);
}
Serial.println();
}
void setup() {
Serial.begin(115200);
Serial.print("Result:");
Sha1.initHmac((uint8_t*)key, strlen(key));
Sha1.print(basestring);
printHash(Sha1.resultHmac());
Serial.println();
// encoding
char* input;
input = (char*)(Sha1.resultHmac());
int inputLen = strlen(input);
int encodedLen = base64_enc_len(inputLen);
char encoded[encodedLen];
// note input is consumed in this step: it will be empty afterwards
base64_encode(encoded, input, inputLen);
Serial.print("base64 result: ");
Serial.println(encoded);
}
void loop() {
}
The output of printHash that I got is 60d41271d43b875b791e2d54c34bf3f018a29763, which is exactly same with the online verification tool(link 5).
However, I supposed to get YNQScdQ7h1t5Hi1Uw0vz8Biil2M= for the base64 result. But I got L18B0HicKRhuxmB6SIFpZP+DpHxU which seems wrong. I have also tried to write a JAVA program and a python program, which also said that the output of the base64 result should be YNQScdQ7h1t5Hi1Uw0vz8Biil2M=
I also found this post: Issues talking between Arduino SHA1-HMAC and base64 encoding and Python(link 6). I have also tried the tidy function it mentioned from Adafruit-Tweet-Receipt(link 7).
// base64-encode SHA-1 hash output. This is NOT a general-purpose base64
// encoder! It's stripped down for the fixed-length hash -- always 20
// bytes input, always 27 chars output + '='.
for(in = Sha1.resultHmac(), out=0; ; in += 3) { // octets to sextets
b64[out++] = in[0] >> 2;
b64[out++] = ((in[0] & 0x03) << 4) | (in[1] >> 4);
if(out >= 26) break;
b64[out++] = ((in[1] & 0x0f) << 2) | (in[2] >> 6);
b64[out++] = in[2] & 0x3f;
}
b64[out] = (in[1] & 0x0f) << 2;
// Remap sextets to base64 ASCII chars
for(i=0; i<=out; i++) b64[i] = pgm_read_byte(&b64chars[b64[i]]);
b64[i++] = '=';
b64[i++] = 0;
Is there any mistake I've made in here?
Thanks!
So full example will be:
#include <avr/pgmspace.h>
#include <sha1.h>
#include <Base64.h>
char key[] = "testKey";
char basestring[] = "testing";
void printHash(uint8_t* hash) {
for (int i=0; i<20; i++) {
Serial.print("0123456789abcdef"[hash[i]>>4]);
Serial.print("0123456789abcdef"[hash[i]&0xf]);
}
Serial.println();
}
void setup() {
Serial.begin(115200);
Serial.print("Input: ");
Serial.println(basestring);
Serial.print("Key: ");
Serial.println(key);
Serial.print("Hmac-sha1 (hex): ");
Sha1.initHmac((uint8_t*)key, strlen(key));
Sha1.print(basestring);
uint8_t *hash;
hash = Sha1.resultHmac();
printHash(hash);
// base64 encoding
char* input = (char*) hash;
int inputLen = strlen(input) - 1; // skip null termination
int encodedLen = base64_enc_len(inputLen);
char encoded[encodedLen];
// note input is consumed in this step: it will be empty afterwards
base64_encode(encoded, input, inputLen);
Serial.print("Hmac-sha1 (base64): ");
Serial.println(encoded);
}
void loop() { }
which outputs:
Input: testing
Key: testKey
Hmac-sha1 (hex): 60d41271d43b875b791e2d54c34bf3f018a29763
Hmac-sha1 (base64): YNQScdQ7h1t5Hi1Uw0vz8Biil2M=
I am new in communications programming. Basically, I need to get the hex equivalent of the CRC output. I have a hex string which is the parameter -
EE0000000015202020202020202020202020323134373030353935
This is concatenation of two strings. The output I need is E6EB in hex or 59115 in ushort. I tried different approaches based on what I found in the web but to no avail. The polynomial that I should be using is 0x8408, which is [CRC-16-CCITT][1], http://en.wikipedia.org/wiki/Polynomial_representations_of_cyclic_redundancy_checks.
I tried this approach, CRC_CCITT Kermit 16 in C#, but the output is incorrect. I also tried the bitwise ~ operator as some suggested for reverse computation, but still failed.
Any help is very much appreciated.
RevEng reports:
% ./reveng -s -w 16 EE0000000015202020202020202020202020323134373030353935e6eb
width=16 poly=0x1021 init=0xffff refin=true refout=true xorout=0xffff check=0x906e name="X-25"
So there's your CRC. Note that the CRC is reflected, where 0x8408 is 0x1021 reflected.
I found a solution and I'll post them in case someone will encounter the same.
private ushort CCITT_CRC16(string strInput)
{
ushort data;
ushort crc = 0xFFFF;
byte[] bytes = GetBytesFromHexString(strInput);
for (int j = 0; j < bytes.Length; j++)
{
crc = (ushort)(crc ^ bytes[j]);
for (int i = 0; i < 8; i++)
{
if ((crc & 0x0001) == 1)
crc = (ushort)((crc >> 1) ^ 0x8408);
else
crc >>= 1;
}
}
crc = (ushort)~crc;
data = crc;
crc = (ushort)((crc << 8) ^ (data >> 8 & 0xFF));
return crc;
}
private byte[] GetBytesFromHexString(string strInput)
{
Byte[] bytArOutput = new Byte[] { };
if (!string.IsNullOrEmpty(strInput) && strInput.Length % 2 == 0)
{
SoapHexBinary hexBinary = null;
try
{
hexBinary = SoapHexBinary.Parse(strInput);
if (hexBinary != null)
{
bytArOutput = hexBinary.Value;
}
}
catch (Exception ex)
{
throw ex;
}
}
return bytArOutput;
}
import System.Runtime.Remoting.Metadata.W3cXsd2001 for SoapHexBinary.
I saw the sample APIas below
public static byte[] compress( byte[] data )
{
try
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gzipStream = new GZIPOutputStream( baos, 6, GZIPOutputStream.MAX_LOG2_WINDOW_LENGTH );
gzipStream.write( data );
gzipStream.close();
}
catch(IOException ioe)
{
return null;
}
return baos.toByteArray();
}
But when I tried to compress with a large file with Curve 8900 OS 4.6, I got a "OutOfMemoryError" so I would like to know that how to compress as a chunk small data?
I already tried with this code as below but it doesn't work, compressed file cannot decompress...
file = (FileConnection)Connector.open(_fileOutputPath, Connector.READ_WRITE);
if (!file.exists()) {
file.create();
}
os = file.openOutputStream();
is = FileUtil.getInputStream(_fileInputPath, 0);
int tmpSize = 1024;
byte[] tmp = new byte[tmpSize];
int len = -1;
gzipStream = new GZIPOutputStream( os, 6, GZIPOutputStream.MAX_LOG2_WINDOW_LENGTH );
while((len = is.read(tmp, 0, tmpSize)) != -1) {
gzipStream.write(tmp, 0, len);
}
GZIPOutputStream does not produce a file suitable for use with the gzip command line tool. This is because it doesn't produce the necessary file headers. How did you test decompressing it? You should write a similar Java program that makes use of GZIPInputStream to test, as 'gunzip' is not going to recognize the input.
The problem of the first code sample is that the ByteArrayOutputStream is getting too big for the limited memory of a mobile device.
An option could be to first write to a file (for instance) on SD card.
The second code sample seems fine, but see Michael's answer.
From the simulator, this all works.
I'm using wifi on the device as i'm assuming it's the most stable.
The problem occurs when i try to post more than 1.5K of urlencoded data.
If i send less then it's fine.
It seems to hang the .flush command();
It works on a physical 9700, so i'm presuming that it's possibly device specific
In the example below i'm using form variables, but i've also tried posting the content type json, but still had the same issue
I've written a small testapp, and using the main thread so i know that it's not threads getting confused
If anyone has any ideas that would be great.
private String PostEventsTest()
{
String returnValue = "Error";
HttpConnection hc = null;
DataInputStream dis = null;
DataOutputStream dos = null;
StringBuffer messagebuffer = new StringBuffer();
URLEncodedPostData postValuePairs;
try
{
postValuePairs = new URLEncodedPostData(null, false);
postValuePairs.append("DATA",postData);// postData);
hc = (HttpConnection) Connector.open(postURL, Connector.READ_WRITE);
hc.setRequestMethod(HttpConnection.POST);
hc.setRequestProperty("User-Agent", "BlackBerry");
hc.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
hc.setRequestProperty("Content-Length", Integer.toString(postValuePairs.getBytes().length));
//hc.setRequestProperty("Content-Length", Integer.toString(postData.length()));
dos = hc.openDataOutputStream();
dos.write(postValuePairs.getBytes());
dos.flush();
dos.close();
// Retrieve the response back from the servlet
dis = new DataInputStream(hc.openInputStream());
int ch;
// Check the Content-Length first
long len = hc.getLength();
if (len != -1)
{
for (int i = 0; i < len; i++)
if ((ch = dis.read()) != -1)
messagebuffer.append((char) ch);
}
else
{ // if the content-length is not available
while ((ch = dis.read()) != -1)
messagebuffer.append((char) ch);
}
dis.close();
returnValue = "Yahoo";
}
catch (Exception ex)
{
returnValue = ex.toString();
ex.printStackTrace();
}
return returnValue;
}
Instead of data streams you should just use the regular input and output streams. So instead of hc.openDataOutputStream() use hc.openOutputStream(). Data streams are for serializing Java objects to a stream, but you just want to write the raw bytes to the stream -- so a regular outputstream is what you want. Same for reading the response - just use the inputstream returned by hc.openInputStream()