func Encode(p SomeStruct) (buf []byte, err error) {
var bs bytes.Buffer
if err = json.NewEncoder(&bs).Encode(p); err != nil {
return
}
in := C.CString(bs.String())
defer C.free(unsafe.Pointer(in)) //Is this necessary?
res := make([]byte, 1024)
out := C.CString(string(res))
defer C.free(unsafe.Pointer(out)) //and this one
res := C.encode(in, out, 1024) //int encode(char *, void *, int)
//... ...
}
Question: shall I use C.free() to ensure that the code does not leak memory? I feel that this is not required as the variable is allocated by Go, not C, right?
My question is, will CString always allocate memory or, it will do that only if necessary?
// Go string to C string
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func C.CString(string) *C.char
As stated in the documentation, you need release the memory manually. The CString function (_Cfunc_CString in cmd/cgo/out.go) actually calls malloc from C standard library and the programmer is supposed to manage this memory. The returned string can be deallocated with C.free in the Go side or with free inside the C code.
Related
I have a GUI written in R that utilizes Tcl/TK package as well a C .dll that also uses Tcl library. I have done some research on this issue, and it seems to be memory related. I am an inexperienced programmer, so I am not sure where I should be looking for this memory issue. Each call of malloc() has a matching free(), and same with the analogous Tcl_Alloc() and Tcl_Free(). This error is very hard to reproduce as well, thus I am afraid I cannot provide a reproducible example as it is seemingly random in nature. One pattern is however that it seems to only happen upon closure of the program, though this is very inconsistent.
By making this post, I am hoping to gain a logical process that one should take in an attempt to debug this problem in a general context under Tcl/Tk - C - R applications. I am not looking for a solution specific to my code, but rather what an individual should think about when encountering this problem.
The message comes from the function Ptr2Block() in tclThreadAlloc.c (or there's something else about which produces the same error message; possible but unlikely) which is Tcl's thread-specific memory allocator (which is used widely inside Tcl to reduce the number of times global locks are hit). Specifically, it's this bit:
if (blockPtr->magicNum1 != MAGIC || blockPtr->magicNum2 != MAGIC) {
Tcl_Panic("alloc: invalid block: %p: %x %x",
blockPtr, blockPtr->magicNum1, blockPtr->magicNum2);
}
The problem? Those zeroes should be MAGIC (which is equal to 0xEF). This indicates that something has overwritten the memory block's metadata — which also should include the size of the block, but that is now likely hot garbage — and program memory integrity can no longer be trusted. Alas, at this point we're now dealing with a program in a broken state where the breakage happened some time previously; the place where the panic happened is merely where detection of the bug happened, not the actual location of the bug.
Debugging further is usually done by building a version of everything with fancy memory allocators turned off (in Tcl's code, this is done by defining the PURIFY symbol when building) and then running the resulting code — which hopefully still has the bug — with a tool like electricfence or purify (hence the special symbol name) to see what sort of out-of-bounds errors are found; they're very good at hunting down this sort of issue.
I would advise you to start by having a closer look to the sizeof() values provided to your Tcl_Alloc() calls in this C .dll.
I'm writing myself a Tcl binding for a C library and I faced recently exactly the same problem and therefore I'm assuming you may have the same error than me in your code.
Here below a minimal example that reproduces the problem:
#include <tcl.h>
#include <stdlib.h> // malloc
static unsigned int dataCtr;
struct tDataWrapper {
const char *str; // Tcl_GetCommandName(interp, cmd)
unsigned int n; // dataCtr value
void *data; // pointer to wrapped object
};
static void wrapDelCmd(ClientData clientData)
{
struct tDataWrapper *wrap = (struct tDataWrapper *) clientData;
if (wrap != NULL) {
/* with false sizeof value provided while creating the wrapper
* (see above), this data pointer would overwrite the
* overhead section of the allocated tcl memory block
* from what I understood and this is what can be causing
* the panic with message like following one when the
* memory is freed with ckfree (here after calling unload)
* alloc: invalid block: 0000018F2624E760: 0 0 */
printf("DEBUG: #%s(%s) &wrap->data #%p\n",
__func__, wrap->str, &wrap->data);
if (wrap->data != NULL) {
// call your wrapped API to deinstantiate the object
}
ckfree(wrap);
}
}
static int wrapCmd(ClientData clientData, Tcl_Interp *interp,
int objc, Tcl_Obj *const objv[])
{
struct tDataWrapper *wrap = (struct tDataWrapper *) clientData;
if (wrap == NULL)
return TCL_ERROR;
else if (wrap->data != NULL) {
// call your wrapped API to do something with instantiated object
return TCL_OK;
} else {
Tcl_Obj *obj = Tcl_ObjPrintf("wrap: {str=\"%s\", n=%u, data=%llx}",
wrap->str, wrap->n, (unsigned long long) wrap->data);
if (obj != NULL) {
Tcl_SetObjResult(interp, obj);
return TCL_OK;
} else
return TCL_ERROR;
}
}
static int newCmd(ClientData clientData, Tcl_Interp *interp,
int objc, Tcl_Obj *const objv[])
{
struct tDataWrapper *wrap;
Tcl_Obj *obj;
Tcl_Command cmd;
// 3) this is correct
// if ((wrap = attemptckalloc(sizeof(struct tDataWrapper))) == NULL)
// 2) still incorrect but GCC gives more warning regarding the inconsistent pointer handling
// if ((wrap = malloc(sizeof(struct tDataWrapper *))) == NULL)
// 1) this is incorrect
if ((wrap = attemptckalloc(sizeof(struct tDataWrapper *))) == NULL)
Tcl_Panic("%s:%u: attemptckalloc failed\n", __func__, __LINE__);
else if ((obj = Tcl_ObjPrintf("data%u", dataCtr+1)) == NULL)
Tcl_Panic("%s:%u: Tcl_ObjPrintf failed\n", __func__, __LINE__);
else if ((cmd = Tcl_CreateObjCommand(interp, Tcl_GetString(obj),
wrapCmd, (ClientData) wrap, wrapDelCmd)) == NULL)
Tcl_Panic("%s:%u: Tcl_CreateObjCommand failed\n", __func__, __LINE__);
else {
wrap->str = Tcl_GetCommandName(interp, cmd);
wrap->n = dataCtr;
wrap->data = NULL; // call your wrapped API to instantiate an object
dataCtr++;
Tcl_SetObjResult(interp, obj);
}
return TCL_OK;
}
int Allocinvalidblock_Init(Tcl_Interp *interp)
{
dataCtr = 0;
return (Tcl_CreateObjCommand(interp, "new",
newCmd, (ClientData) NULL, NULL)
== NULL) ? TCL_ERROR : TCL_OK;
}
int Allocinvalidblock_Unload(Tcl_Interp *interp, int flags)
{
Tcl_Namespace *ns = Tcl_GetGlobalNamespace(interp);
Tcl_Obj *obj;
Tcl_Command cmd;
unsigned int i;
for(i=0; i<dataCtr; i++) {
if ((obj = Tcl_ObjPrintf("data%u", i+1)) != NULL) {
if ((cmd = Tcl_FindCommand(interp,
Tcl_GetString(obj), ns, TCL_GLOBAL_ONLY)) != NULL)
Tcl_DeleteCommandFromToken(interp, cmd);
Tcl_DecrRefCount(obj);
}
}
return TCL_OK;
}
Once built (for example with Code::Blocks as shared library project linking against C:/msys64/mingw64/lib/libtcl.dll.a), the error can be triggered when more than a data object is created and the library immediately unloaded:
load bin/Release/libAllocInvalidBlock.dll
new
new
unload bin/Release/libAllocInvalidBlock.dll
If used otherwise the crash may even be not triggered... Anyway, such an error in the C code is not particularly obvious to identify (although easy to fix) because the compilation is running without any warning (although -Wall compiler flag is set).
I'm trying to use TeamSpeak3 SDK with my delphi my have come accross a few problems, the code compiles and appears to work, most of the code is example code from example projects, that's except the attempt to read the returned data.
1. Do I free the memory correct?
2. Do I read the returned data from the SDK correct or can it be done in a better way?
I have asked a question about this SDK in another thread, but I was obviously too quick to mark the thread as answered. :/
SDK Documentation:
To get a list of all currently visible clients on the specified virtual server:
unsigned int ts3client_getClientList(serverConnectionHandlerID, result);
uint64 serverConnectionHandlerID;
anyID** result;
Parameters
• serverConnectionHandlerID
ID of the server connection handler for which the list of clients is requested.
• result
Address of a variable that receives a NULL-termianted array of client IDs.
Unless an error occurs, the array must be released using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. If an error has occured, the result array is uninitialized and must not be released.
A list of all channels on the specified virtual server can be queried with:
unsigned int ts3client_getChannelList(serverConnectionHandlerID, result);
uint64 serverConnectionHandlerID;
uint64** result;
Parameters
• serverConnectionHandlerID
ID of the server connection handler for which the list of channels is requested.
• result
Address of a variable that receives a NULL-termianted array of channel IDs. Unless an error occurs, the array must be released using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. If an error has occured, the result array is uninitialized and must not be released.
unsigned int ts3client_getCaptureDeviceList (modeID, result); const char* modeID; char**** result;
Parameters
• modeID
Defines the playback/capture mode to use. For different modes there might be different device lists. Valid modes are returned ts3client_getDefaultPlayBackMode/ts3client_getDefaultCaptureMode and ts3client_getPlaybackModeList/ts3client_getCaptureModeList.
• result
Address of a variable that receives a NULL-terminated array { { char* deviceName, char* deviceID }, { char* deviceName, char* deviceID }, ... , NULL }.
Unless the function returns an error, the elements of the array and the array itself need to be freed using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. In case of an error, the result array is uninitialized and must not be released.
Playback and capture devices available for the given mode can be listed, as well as the current operating systems default. The returned device values can be used to initialize the devices.
To query the default playback and capture device, call
To get a list of all available playback and capture devices for the specified mode, call
unsigned int ts3client_getPlaybackDeviceList(modeID, result);
const char* modeID;
char**** result;
unsigned int ts3client_getCaptureDeviceList(modeID, result);
const char* modeID;
char**** result;
Parameters
• modeID
Defines the playback/capture mode to use. For different modes there might be different device lists. Valid modes are returned by
ts3client_getDefaultPlayBackMode / s3client_getDefaultCaptureMode and ts3client_getPlaybackModeList / ts3client_getCaptureModeList.
• result
Address of a variable that receives a NULL-terminated array { { char* deviceName, char* deviceID }, { char* deviceName, char* deviceID }, ... , NULL }.
Unless the function returns an error, the elements of the array and the array itself need to be freed using ts3client_freeMemory.
Returns ERROR_ok on success, otherwise an error code as defined in public_errors.h. In case of an error, the result array is uninitialized and must not be released.
unsigned int ts3client_startConnection(serverConnectionHandlerID,identity,ip,port,nickname,defaultChannelArray,defaultChannelPassword,serverPassword);
uint64 serverConnectionHandlerID; const char* identity; const
char* ip; unsigned int port; const char* nickname; const char**
defaultChannelArray; // This the thingy I dont get const char*
defaultChannelPassword; const char* serverPassword;
Parameters
• serverConnectionHandlerID
Unique identifier for this server connection. Created with ts3client_spawnNewServerConnectionHandler
• identity
The clients identity. This string has to be created by calling ts3client_createIdentity.
Please note an application should create the identity only once, store the string locally and reuse it for future connections.
• ip
Hostname or IP of the TeamSpeak 3 server.
If you pass a hostname instead of an IP, the Client Lib will try to resolve it to an IP, but the function may block for an unusually long period of time while resolving is taking place. If you are relying on the function to return quickly, we recommend to resolve the hostname yourself (e.g. asynchronously) and then call ts3client_startConnection with the IP instead of the hostname.
• port
UDP port of the TeamSpeak 3 server, by default 9987. TeamSpeak 3 uses UDP. Support for TCP might be added in the future.
• nickname
On login, the client attempts to take this nickname on the connected server. Note this is not necessarily the actually assigned nickname, as the server can modifiy the nickname ("gandalf_1" instead the requested "gandalf") or refuse blocked names.
• defaultChannelArray
String array defining the path to a channel on the TeamSpeak 3 server. If the channel exists and the user has sufficient rights and supplies the correct password if required, the channel will be joined on login.
To define the path to a subchannel of arbitrary level, create an array of channel names detailing the position of the default channel (e.g. "grandparent", "parent", "mydefault", ""). The array is terminated with a empty string.
Pass NULL to join the servers default channel.
• defaultChannelPassword
Password for the default channel. Pass an empty string if no password is required or no default channel is specified.
• serverPassword
Password for the server. Pass an empty string if the server does not require a password.
All strings need to be encoded in UTF-8 format
Important
Client Lib functions returning C-strings or arrays dynamically allocate memory which has to be freed by the caller using ts3client_freeMemory. It is important to only access and release the memory if the function returned ERROR_ok.
Should the function return an error, the result variable is uninitialized, so freeing or accessing it
could crash the application.
See the section Calling Client Lib functions for additional notes and examples.
A printable error string for a specific error code can be queried with
unsigned int ts3client_getErrorMessage(errorCode, error);
unsigned int errorCode;
char** error;
Parameters
• errorCode
The error code returned from all Client Lib functions.
• error
Address of a variable that receives the error message string, encoded in UTF-8 format. Unless the return value of the function is not ERROR_ok, the string should be released with ts3client_freeMemory.
Example:
unsigned int error;
anyID myID;
error = ts3client_getClientID(scHandlerID, &myID); /* Calling some Client Lib function */
if(error != ERROR_ok) {
char* errorMsg;
if(ts3client_getErrorMessage(error, &errorMsg) == ERROR_ok)
{ /* Query printable error */
printf("Error querying client ID: %s\n", errorMsg);
ts3client_freeMemory(errorMsg); /* Release memory */
}
}
type
PPanyID = ^PAnyID;
PanyID = ^anyID;
anyID = word;
var
error: longword;
errormsg: PAnsiChar;
procedure TfrmMain.RequestOnlineClients;
var
ids : PanyID;
pids : PanyID;
aid : anyID;
begin
error := ts3client_getClientList(FTSServerHandlerID, #ids);
if (error <> ERROR_ok) then
begin
if (ts3client_getErrorMessage(error, #errormsg) = ERROR_ok) then
begin
LogMsg(Format('Error requesting online clients: %s', [errormsg]));
ts3client_freeMemory(errormsg);
end;
end else
begin
pids := ids;
while (pids^ <> 0) do
begin
aid := pids^;
LogMsg(format('userid %u',[aid, getUserNickNameById(aid)]));
inc(pids);
end;
ts3client_freeMemory(#pids^); // here's potiential problem
end;
end;
procedure TfrmMain.RequestChannels;
var
ids : PUint64;
pids : PUint64;
aid : uint64;
channelname : PAnsiChar;
begin
error := ts3client_getChannelList(FTSServerHandlerID, #ids);
if (error <> ERROR_ok) then
begin
if (ts3client_getErrorMessage(error, #errormsg) = ERROR_ok) then
begin
LogMsg(Format('Error requesting channels: %s', [errormsg]));
ts3client_freeMemory(errormsg);
end;
end else
begin
pids := ids;
while (pids^ <> 0) do
begin
aid := pids^;
LogMsg(format('channelid %u %s',[aid, getChannelNameById(aid)]));
inc(pids);
end;
ts3client_freeMemory(#pids^);
end;
end;
**// Added details 25-11-2014**
char* defaultMode;
if(ts3client_getDefaultPlayBackMode(&defaultMode) == ERROR_ok) {
char*** array;
if(ts3client_getPlaybackDeviceList(defaultMode, &array) == ERROR_ok) {
for(int i=0; array[i] != NULL; ++i) {
printf("Playback device name: %s\n", array[i][0]); /* First element: Device name */
printf("Playback device ID: %s\n", array[i][1]); /* Second element: Device ID */
/* Free element */
ts3client_freeMemory(array[i][0]);
ts3client_freeMemory(array[i][1]);
ts3client_freeMemory(array[i]);
}
ts3client_freeMemory(array); /* Free complete array */
} else {
printf("Error getting playback device list\n");
}
} else {
printf("Error getting default playback mode\n");
}
Example to query all available playback devices:
char* defaultMode;
if(ts3client_getDefaultPlayBackMode(&defaultMode) == ERROR_ok) {
char*** array;
if(ts3client_getPlaybackDeviceList(defaultMode, &array) == ERROR_ok) {
for(int i=0; array[i] != NULL; ++i) {
printf("Playback device name: %s\n", array[i][0]); /* First element: Device name */
printf("Playback device ID: %s\n", array[i][1]); /* Second element: Device ID */
/* Free element */
ts3client_freeMemory(array[i][0]);
ts3client_freeMemory(array[i][1]);
ts3client_freeMemory(array[i]);
}
ts3client_freeMemory(array); /* Free complete array */
} else {
printf("Error getting playback device list\n");
}
} else {
printf("Error getting default playback mode\n");
}
procedure TfrmMain.ConnectServer2;
var
version : PAnsiChar;
DefaultChannelsArr : PPAnsiChar;
begin
if Connected then Exit;
if not ClientInitialized then
InitializeClient;
// Dbl Check if we can connect
if ClientInitialized then
try
// Connect to server on localhost:9987 with nickname "client", no default channel, no default channel password and server password "secret"
// error := ts3client_startConnection(FTSServerHandlerID, identity, '127.0.0.1', 9987, 'Delphi Client', nil, '', 'secret'); // example connection setup
ts3check(ts3client_startConnection(FTSServerHandlerID, PAnsiChar(FSetup.ClientIdentity), PAnsiChar(FSetup.ServerAddress), FSetup.FServerPort, PAnsiChar(FSetup.NickName), nil, '', PAnsiChar(FSetup.ServerPassword)));
{ TODO -oMe -cImportant : Need to check how to convert ansistrings to UTF8 } // UnicodeToUtf8() // AnsiToUtf8()...
// Query and print client lib version
ts3check(ts3client_getClientLibVersion(#version));
LogMsg(Format('Client lib version: %s', [version]));
ts3client_freeMemory(version); // Release dynamically allocated memory
// Do not set connected here, wait for the callback connected state
except
on e: exception do
begin
UnInitializeClient; // clear the hole thing and start over
LogMsg(Format('Error connecting: %s',[e.Message]));
end;
end;
end;
I'd translate ts3client_getClientList like this:
function ts3client_getClientList(serverConnectionHandlerID: UInt64;
out result: PAnyID): Cardinal; cdecl; external '...';
I think that an out parameter is better than a double pointer. It makes the intent clearer.
Then to call the function I'd write it like this:
var
ids: PAnyID;
idarr: TArray<anyID>;
....
ts3check(ts3client_getClientList(serverConnectionHandlerID, ids));
try
idarr := GetIDs(ids);
finally
ts3check(ts3client_freeMemory(ids));
end;
Here, ts3check is a function that raises an exception if it is passed a return value other than ERROR_ok.
function ts3client_getErrorMessage(error: Cardinal;
out errormsg: PAnsiChar): Cardinal; cdecl; external '...';
....
procedure ts3check(error: Cardinal);
var
errormsg: PAnsiChar;
errorstr: string;
begin
if error = ERROR_ok then
exit;
if ts3client_getErrorMessage(error, #errormsg) <> ERROR_ok then
raise Ets3Error.CreateFmt('Error code %d', [error]);
errorstr := UTF8ToUnicodeString(errormsg);
ts3client_freeMemory(errormsg);
raise Ets3Error.CreateFmt('Error code %d (%s)', [error, errorstr]);
end;
And you can implement GetIDs like this:
function GetIDs(const ids: PAnyID): TArray<anyID>;
var
Count: Integer;
p: PAnyID;
begin
Count := 0;
p := ids;
while p^ <> 0 do
begin
inc(Count);
inc(p);
end;
SetLength(Result, Count);
Count := 0;
p := ids;
while p^ <> 0 do
begin
Result[Count] := p^;
inc(Count);
inc(p);
end;
end;
Now, I don't imagine that you really want an array of IDs. You'd probably be happy to process the IDs inline. I don't want to get into how to do that though because that leads me into code of yours that I cannot see. You won't write the code exactly as I have done above, but you can hopefully use the above as a source of ideas.
The main point in all of this is to try to encapsulate as much of the messy boiler plate code as possible. Wrapping the call to ts3client_getErrorMessage makes the higher level code so much easier to read. Use things like OleCheck and Win32Check as inspiration.
One point I would make is that it feels wrong for this code to live inside a form. Normally it is cleaner to keep such code removed from your UI. Make a wrapper to this library that can be consumed by your UI code. Keep that wrapper in a dedicated unit and so hide away the gnarly details.
I working with Keil, MDK-ARM Pro 4.71 for a Cortex-M3 target(STM32F107).
I have compiled the Lua interpreter and a Lua "timer" module that interfaces the chip's timers. I'd like to call a lua function when the timer is elapsed.
Here is a sample use :
t = timer.open()
t.event = function() print("Bing !") end
t:start()
Until here, everything works fine :-) ! I see the "Bing !" message being printed each time the timer elapses.
Now if I use a closure :
t = timer.open()
i = 0
t.event = function() i = i + 1; print(i); end
t:start()
I'm having a bad memory access in the GC after some amount of timer's updates. Since it's an embedded context with very few memory, I may be running out of memory quite fast if there is a leak.
Here is the "t.event" setter (ELIB_TIMER is a C structure representing my timer) :
static int ElibTimerSetEvent(lua_State* L)
{
ELIB_TIMER* pTimer_X = ElibCheckTimer(L, 1, TRUE);
if (pTimer_X->LuaFuncKey_i != LUA_REFNIL)
{
luaL_unref(L, LUA_REGISTRYINDEX, pTimer_X->LuaFuncKey_i);
pTimer_X->LuaFuncKey_i = LUA_REFNIL;
}
if (!lua_isnil(L, 2))
{
pTimer_X->LuaFuncKey_i = luaL_ref(L, LUA_REGISTRYINDEX);
}
return 0;
}
And here is the native callback implementation :
static void ElibTimerEventHandler(SYSEVT_HANDLE Event_H)
{
ELIB_TIMER* pTimer_X = (ELIB_TIMER*)SWLIB_SYSEVT_GetSideData(Event_H);
lua_State* L = pTimer_X->L;
int i = lua_gettop(L);
if (pTimer_X->LuaFuncKey_i != LUA_REFNIL)
{
lua_rawgeti(L, LUA_REGISTRYINDEX, pTimer_X->LuaFuncKey_i);
lua_call(L, 0, 0);
lua_settop(L, i);
}
}
This is synchronized externally, so this isn't a synchronization issue.
Am I doing something wrong ?
EDIT
Here is the callstack (with a lua_pcall instead of lua_call, but it is the same). The first line is my hard fault handler.
I have found the problem ! I ran out of stack (native stack, not Lua) space :p.
I guess this specific script was causing a particularly long call stack. After increasing the allocated memory for my native stack, the problem is gone. On the opposite, if I reduce it, I can't even initialize the interpreter.
Many thanks to those who tried to help here.
Found a bug in your C code. You broke the lua stack in static int ElibTimerSetEvent(lua_State* L)
luaL_ref will pop the value on the top of Lua stack: http://www.lua.org/manual/5.2/manual.html#luaL_ref
So you need to copy the value to be refed, before the call to luaL_ref:
lua_pushvalue(L, 2); // push the callback to stack top, and then it will be consumed by luaL_ref()
Please fix this and try again.
My pthread_detach calls fail with a "Bad file descriptor" error. The calls are in the destructor for my class and look like this -
if(pthread_detach(get_sensors) != 0)
printf("\ndetach on get_sensors failed with error %m", errno);
if(pthread_detach(get_real_velocity) != 0)
printf("\ndetach on get_real_velocity failed with error %m", errno);
I have only ever dealt with this error when using sockets. What could be causing this to happen in a pthread_detach call that I should look for? Or is it likely something in the thread callback that could be causing it? Just in case, the callbacks look like this -
void* Robot::get_real_velocity_thread(void* threadid) {
Robot* r = (Robot*)threadid;
r->get_real_velocity_thread_i();
}
inline void Robot::get_real_velocity_thread_i() {
while(1) {
usleep(14500);
sensor_packet temp = get_sensor_value(REQUESTED_VELOCITY);
real_velocity = temp.values[0];
if(temp.values[1] != -1)
real_velocity += temp.values[1];
} //end while
}
/*Callback for get sensors thread*/
void* Robot::get_sensors_thread(void* threadid) {
Robot* r = (Robot*)threadid;
r->get_sensors_thread_i();
} //END GETSENSORS_THREAD
inline void Robot::get_sensors_thread_i() {
while(1) {
usleep(14500);
if(sensorsstreaming) {
unsigned char receive;
int read = 0;
read = connection.PollComport(port, &receive, sizeof(unsigned char));
if((int)receive == 19) {
read = connection.PollComport(port, &receive, sizeof(unsigned char));
unsigned char rest[54];
read = connection.PollComport(port, rest, 54);
/* ***SET SENSOR VALUES*** */
//bump + wheel drop
sensor_values[0] = (int)rest[1];
sensor_values[1] = -1;
//wall
sensor_values[2] = (int)rest[2];
sensor_values[3] = -1;
...
...
lots more setting just like the two above
} //end if header == 19
} //end if sensors streaming
} //end while
} //END GET_SENSORS_THREAD_I
Thank you for any help.
The pthread_* functions return an error code; they do not set errno. (Well, they may of course, but not in any way that is documented.)
Your code should print the value returned by pthread_detach and print that.
Single Unix Spec documents two return values for this function: ESRCH (no thread by that ID was found) and EINVAL (the thread is not joinable).
Detaching threads in the destructor of an object seems silly. Firstly, if they are going to be detached eventually, why not just create them that way?
If there is any risk that the threads can use the object that is being destroyed, they need to be stopped, not detached. I.e. you somehow indicate to the threads that they should shut down, and then wait for them to reach some safe place after which they will not touch the object any more. pthread_join is useful for this.
Also, it is a little late to be doing that from the destructor. A destructor should only be run when the thread executing it is the only thread with a reference to that object. If threads are still using the object, then you're destroying it from under them.
I'm attempting to write a bit of JS that will read a file and write it out to a stream. The deal is that the file is extremely large, and so I have to read it bit by bit. It seems that I shouldn't be running out of memory, but I do. Here's the code:
var size = fs.statSync("tmpfile.tmp").size;
var fp = fs.openSync("tmpfile.tmp", "r");
for(var pos = 0; pos < size; pos += 50000){
var buf = new Buffer(50000),
len = fs.readSync(fp, buf, 0, 50000, (function(){
console.log(pos);
return pos;
})());
data_output.write(buf.toString("utf8", 0, len));
delete buf;
}
data_output.end();
For some reason it hits 264900000 and then throws FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory. I'd figure that the data_output.write() call would force it to write the data out to data_output, and then discard it from memory, but I could be wrong. Something is causing the data to stay in memory, and I've no idea what it would be. Any help would be greatly appreciated.
I had a very similar problem. I was reading in a very large csv file with 10M lines, and writing out its json equivalent. I saw in the windows task manager that my process was using > 2GB of memory. Eventually I figured out that the output stream was probably slower than the input stream, and that the outstream was buffering a huge amount of data. I was able to fix this by pausing the instream every 100 writes to the outstream, and waiting for the outstream to empty. This gives time for the outstream to catch up with the instream. I don't think it matters for the sake of this discussion, but I was using 'readline' to process the csv file one line at a time.
I also figured out along the way that if, instead of writing every line to the outstream, I concatenate 100 or so lines together, then write them together, this also improved the memory situation and made for faster operation.
In the end, I found that I could do the file transfer (csv -> json) using just 70M of memory.
Here's a code snippet for my write function:
var write_counter = 0;
var out_string = "";
function myWrite(inStream, outStream, string, finalWrite) {
out_string += string;
write_counter++;
if ((write_counter === 100) || (finalWrite)) {
// pause the instream until the outstream clears
inStream.pause();
outStream.write(out_string, function () {
inStream.resume();
});
write_counter = 0;
out_string = "";
}
}
You should be using pipes, such as:
var fp = fs.createReadStream("tmpfile.tmp");
fp.pipe(data_output);
For more information, check out: http://nodejs.org/docs/v0.5.10/api/streams.html#stream.pipe
EDIT: the problem in your implementation, btw, is that by doing it in chunks like that, the write buffer isn't going to get flushed, and you're going to read in the entire file before writing much of it back out.
According to the documentation, data_output.write(...) will return true if the string has been flushed, and false if it has not (due to the kernel buffer being full). What kind of stream is this?
Also, I'm (fairly) sure this isn't the problem, but: how come you allocate a new Buffer on each loop iteration? Wouldn't it make more sense to initialize buf before the loop?
I don't know how the synchronous file functions are implemented, but have you considered using the asynch ones? That would be more likely to allow garbage collection and i/o flushing to happen. So instead of a for loop, you would trigger the next read in the callback function of the previous read.
Something along these lines (note also that, per other comments, I'm reusing the Buffer):
var buf = new Buffer(50000),
var pos = 0, bytesRead;
function readNextChunk () {
fs.read(fp, buf, 0, 50000, pos,
function(err, bytesRead){
if (err) {
// handle error
}
else {
data_output.write(buf.toString("utf8", 0, bytesRead));
pos += bytesRead;
if (pos<size)
readNextChunk();
}
});
}
readNextChunk();