The documentation says:
void g_ptr_array_insert (GPtrArray *array,
gint index_,
gpointer data);
Inserts an element into the pointer array at the given index. The
array will grow in size automatically if necessary.
Same question for g_ptr_array_add().
Documentation is silent about thread safety of these functions. The following functions are described expressly to be thread-safe:
g_ptr_array_free ()
g_ptr_array_ref()
g_ptr_array_unref ()
What if multiple-threads execute g_ptr_array_insert() at the same time on the same array of pointers? Do I have to provide thread safety myself?
No, it's not thread-safe (just as almost all GLib data types; source, see “you must coordinate accesses…”). Chances are that
two threads enter marked line simultaneously.
static void
g_ptr_array_maybe_expand (GRealPtrArray *array,
gint len)
{
if ((array->len + len) > array->alloc)
{
guint old_alloc = array->alloc;
array->alloc = g_nearest_pow (array->len + len);
array->alloc = MAX (array->alloc, MIN_ARRAY_SIZE);
array->pdata = g_realloc (array->pdata, sizeof (gpointer) * array->alloc); // here
if (G_UNLIKELY (g_mem_gc_friendly))
for ( ; old_alloc < array->alloc; old_alloc++)
array->pdata [old_alloc] = NULL;
}
}
void
g_ptr_array_add (GPtrArray *array,
gpointer data)
{
GRealPtrArray *rarray = (GRealPtrArray *)array;
g_return_if_fail (rarray);
g_ptr_array_maybe_expand (rarray, 1);
rarray->pdata[rarray->len++] = data;
}
Provide your own locking using, for example, GMutex.
Related
I have a GUI written in R that utilizes Tcl/TK package as well a C .dll that also uses Tcl library. I have done some research on this issue, and it seems to be memory related. I am an inexperienced programmer, so I am not sure where I should be looking for this memory issue. Each call of malloc() has a matching free(), and same with the analogous Tcl_Alloc() and Tcl_Free(). This error is very hard to reproduce as well, thus I am afraid I cannot provide a reproducible example as it is seemingly random in nature. One pattern is however that it seems to only happen upon closure of the program, though this is very inconsistent.
By making this post, I am hoping to gain a logical process that one should take in an attempt to debug this problem in a general context under Tcl/Tk - C - R applications. I am not looking for a solution specific to my code, but rather what an individual should think about when encountering this problem.
The message comes from the function Ptr2Block() in tclThreadAlloc.c (or there's something else about which produces the same error message; possible but unlikely) which is Tcl's thread-specific memory allocator (which is used widely inside Tcl to reduce the number of times global locks are hit). Specifically, it's this bit:
if (blockPtr->magicNum1 != MAGIC || blockPtr->magicNum2 != MAGIC) {
Tcl_Panic("alloc: invalid block: %p: %x %x",
blockPtr, blockPtr->magicNum1, blockPtr->magicNum2);
}
The problem? Those zeroes should be MAGIC (which is equal to 0xEF). This indicates that something has overwritten the memory block's metadata — which also should include the size of the block, but that is now likely hot garbage — and program memory integrity can no longer be trusted. Alas, at this point we're now dealing with a program in a broken state where the breakage happened some time previously; the place where the panic happened is merely where detection of the bug happened, not the actual location of the bug.
Debugging further is usually done by building a version of everything with fancy memory allocators turned off (in Tcl's code, this is done by defining the PURIFY symbol when building) and then running the resulting code — which hopefully still has the bug — with a tool like electricfence or purify (hence the special symbol name) to see what sort of out-of-bounds errors are found; they're very good at hunting down this sort of issue.
I would advise you to start by having a closer look to the sizeof() values provided to your Tcl_Alloc() calls in this C .dll.
I'm writing myself a Tcl binding for a C library and I faced recently exactly the same problem and therefore I'm assuming you may have the same error than me in your code.
Here below a minimal example that reproduces the problem:
#include <tcl.h>
#include <stdlib.h> // malloc
static unsigned int dataCtr;
struct tDataWrapper {
const char *str; // Tcl_GetCommandName(interp, cmd)
unsigned int n; // dataCtr value
void *data; // pointer to wrapped object
};
static void wrapDelCmd(ClientData clientData)
{
struct tDataWrapper *wrap = (struct tDataWrapper *) clientData;
if (wrap != NULL) {
/* with false sizeof value provided while creating the wrapper
* (see above), this data pointer would overwrite the
* overhead section of the allocated tcl memory block
* from what I understood and this is what can be causing
* the panic with message like following one when the
* memory is freed with ckfree (here after calling unload)
* alloc: invalid block: 0000018F2624E760: 0 0 */
printf("DEBUG: #%s(%s) &wrap->data #%p\n",
__func__, wrap->str, &wrap->data);
if (wrap->data != NULL) {
// call your wrapped API to deinstantiate the object
}
ckfree(wrap);
}
}
static int wrapCmd(ClientData clientData, Tcl_Interp *interp,
int objc, Tcl_Obj *const objv[])
{
struct tDataWrapper *wrap = (struct tDataWrapper *) clientData;
if (wrap == NULL)
return TCL_ERROR;
else if (wrap->data != NULL) {
// call your wrapped API to do something with instantiated object
return TCL_OK;
} else {
Tcl_Obj *obj = Tcl_ObjPrintf("wrap: {str=\"%s\", n=%u, data=%llx}",
wrap->str, wrap->n, (unsigned long long) wrap->data);
if (obj != NULL) {
Tcl_SetObjResult(interp, obj);
return TCL_OK;
} else
return TCL_ERROR;
}
}
static int newCmd(ClientData clientData, Tcl_Interp *interp,
int objc, Tcl_Obj *const objv[])
{
struct tDataWrapper *wrap;
Tcl_Obj *obj;
Tcl_Command cmd;
// 3) this is correct
// if ((wrap = attemptckalloc(sizeof(struct tDataWrapper))) == NULL)
// 2) still incorrect but GCC gives more warning regarding the inconsistent pointer handling
// if ((wrap = malloc(sizeof(struct tDataWrapper *))) == NULL)
// 1) this is incorrect
if ((wrap = attemptckalloc(sizeof(struct tDataWrapper *))) == NULL)
Tcl_Panic("%s:%u: attemptckalloc failed\n", __func__, __LINE__);
else if ((obj = Tcl_ObjPrintf("data%u", dataCtr+1)) == NULL)
Tcl_Panic("%s:%u: Tcl_ObjPrintf failed\n", __func__, __LINE__);
else if ((cmd = Tcl_CreateObjCommand(interp, Tcl_GetString(obj),
wrapCmd, (ClientData) wrap, wrapDelCmd)) == NULL)
Tcl_Panic("%s:%u: Tcl_CreateObjCommand failed\n", __func__, __LINE__);
else {
wrap->str = Tcl_GetCommandName(interp, cmd);
wrap->n = dataCtr;
wrap->data = NULL; // call your wrapped API to instantiate an object
dataCtr++;
Tcl_SetObjResult(interp, obj);
}
return TCL_OK;
}
int Allocinvalidblock_Init(Tcl_Interp *interp)
{
dataCtr = 0;
return (Tcl_CreateObjCommand(interp, "new",
newCmd, (ClientData) NULL, NULL)
== NULL) ? TCL_ERROR : TCL_OK;
}
int Allocinvalidblock_Unload(Tcl_Interp *interp, int flags)
{
Tcl_Namespace *ns = Tcl_GetGlobalNamespace(interp);
Tcl_Obj *obj;
Tcl_Command cmd;
unsigned int i;
for(i=0; i<dataCtr; i++) {
if ((obj = Tcl_ObjPrintf("data%u", i+1)) != NULL) {
if ((cmd = Tcl_FindCommand(interp,
Tcl_GetString(obj), ns, TCL_GLOBAL_ONLY)) != NULL)
Tcl_DeleteCommandFromToken(interp, cmd);
Tcl_DecrRefCount(obj);
}
}
return TCL_OK;
}
Once built (for example with Code::Blocks as shared library project linking against C:/msys64/mingw64/lib/libtcl.dll.a), the error can be triggered when more than a data object is created and the library immediately unloaded:
load bin/Release/libAllocInvalidBlock.dll
new
new
unload bin/Release/libAllocInvalidBlock.dll
If used otherwise the crash may even be not triggered... Anyway, such an error in the C code is not particularly obvious to identify (although easy to fix) because the compilation is running without any warning (although -Wall compiler flag is set).
I'm facing some problem with gpu resize using opencv.
Here is my code:
#define MX 500
#define ASYNC 0
class job {
public:
cv::cuda::GpuMat gpuImage;
cv::cuda::Stream stream;
cv::Mat cpuImage;
~job() {
printf("job deleted\n");
}
};
void onComplete(int status, void* uData) {
job* _job = (job*) uData;
delete _job;
}
void resize(job* _job, vector<uchar> buffer) {
_job->cpuImage = cv::imdecode(buffer, cv::IMREAD_COLOR);
if (ASYNC) {
_job->gpuImage.upload(_job->cpuImage, _job->stream);
cv::cuda::resize(_job->gpuImage, _job->gpuImage, cv::Size(100, 100), 0, 0, cv::INTER_NEAREST, _job->stream);
_job->gpuImage.download(_job->cpuImage, _job->stream);
_job->stream.enqueueHostCallback(onComplete, _job);
// _job->stream.waitForCompletion();
} else {
_job->gpuImage.upload(_job->cpuImage);
cv::cuda::resize(_job->gpuImage, _job->gpuImage, cv::Size(100, 100), 0, 0, cv::INTER_NEAREST);
_job->gpuImage.download(_job->cpuImage);
delete _job;
}
}
vector<uchar> readFile(string filename) {
std::ifstream input(filename, std::ios::binary);
std::vector<unsigned char> buffer(std::istreambuf_iterator<char>(input),{});
return buffer;
}
int main() {
for (int i = 0; i < MX; i++) {
vector<uchar> buf = readFile("input.jpg");
job* _job = new job();
resize(_job, buf);
printFreeGPUMemory();
}
while (true) {
// wait
}
return 0;
}
When I run resize synchronously (ASYNC = 0), the code works perfectly fine. But when I run it asynchronously (ASYNC = 1), it seems that some gpu memory is lost somewhere despite the fact that I have deleted all created GpuMats and Streams. The more loop I run, the less free memory I have. is there a bug or part of my code is wrong?
problem solved.
here is the note of the callback from OpenCV docs:
Callbacks must not make any CUDA API calls. Callbacks must not perform
any synchronization that may depend on outstanding device work or
other callbacks that are not mandated to run earlier. Callbacks
without a mandated order (in independent streams) execute in undefined
order and may be serialized.
I had read the note but didn't actually notice that even deleting a cv::cuda::* still causes problems. So the solution is to avoid "touching" any cv::cuda::* in the callback, even deleting or releasing.
I used Eclispse Indigo + CDT 8.0.2 + cygwin to develope a multi-thread systerm, the code is below:
pthread_mutex_t mutexCmd = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t signalCmd = PTHREAD_COND_INITIALIZER;
void * Func(void * arg)
{
int iStatus;
while (1)
{
int a = 1;
pthread_cleanup_push(pthread_mutex_unlock, &mutexCmd);
pthread_mutex_lock(&mutexCmd);
iStatus = pthread_cond_wait(&signalCmd, &mutexCmd);
if (iStatus) {
err_abort(iStatus, "signalCmd status error");
}
if(arg->Cmd != navi_Go) //Just a command tag;
{
pthread_mutex_unlock(&(pNaviCtrl->mutexCmd));
continue;
}
//do some work
//.....
pthread_mutex_unlock(&mutexCmd);
pthread_cleanup_pop(1);
}
//pthread_detach(pthread_self());
return NULL;
}
int main()
{
int iStatus = 0;
pthread = tid;
iStatus = pthread_create(&tid;NULL, Func, NULL);
if(iStatus)
{
err_abort(iStatus, "Start pthread error");
}
// do some work
...
//Cancel thread
void * retval;
iStatus = pthread_cancel(tid)
iStatus = pthread_join(tid; &retval);
if(iStatus){
err_abort(iStatus,"Stop thread error");
}
return iStatus;
}
where program run, it stop at "iStatus = pthread_join(tid1; &retval);" couldn't go forward anymore, I think the thread could be happed to deadlock, but can't find the reason. I supposed after call pthread_cancel(), the thread will exit and return to the pthread_join(),
who can tell me what's wrong with my code?
Don't put cleanup_push and _pop inside the while loop. Don't call them more than once. If you look at them, they are macros that wrap the code between them in { }. They setup a longjump that is used when you call pthread_cancel.
pthread_cleanup_pop(1) tells the pthread library to not only pop the cleanup entry off the stack, but to also execute it. So that call will also implicitly call:
pthread_mutex_unlock(&mutexCmd);
Since you've already unlocked the mutex, that call has undefined behavior (assuming the mutex type is PTHREAD_MUTEX_NORMAL). I imagine that call is just never returning or something.
Note that your code has other problems handing the cleanup - if you execute the continue for the loop, you'll call pthread_cleanup_push() a second time (or more), which will add another cleanup context.
There may be other problems (I'm not very familiar with pthread_cancel()).
My pthread_detach calls fail with a "Bad file descriptor" error. The calls are in the destructor for my class and look like this -
if(pthread_detach(get_sensors) != 0)
printf("\ndetach on get_sensors failed with error %m", errno);
if(pthread_detach(get_real_velocity) != 0)
printf("\ndetach on get_real_velocity failed with error %m", errno);
I have only ever dealt with this error when using sockets. What could be causing this to happen in a pthread_detach call that I should look for? Or is it likely something in the thread callback that could be causing it? Just in case, the callbacks look like this -
void* Robot::get_real_velocity_thread(void* threadid) {
Robot* r = (Robot*)threadid;
r->get_real_velocity_thread_i();
}
inline void Robot::get_real_velocity_thread_i() {
while(1) {
usleep(14500);
sensor_packet temp = get_sensor_value(REQUESTED_VELOCITY);
real_velocity = temp.values[0];
if(temp.values[1] != -1)
real_velocity += temp.values[1];
} //end while
}
/*Callback for get sensors thread*/
void* Robot::get_sensors_thread(void* threadid) {
Robot* r = (Robot*)threadid;
r->get_sensors_thread_i();
} //END GETSENSORS_THREAD
inline void Robot::get_sensors_thread_i() {
while(1) {
usleep(14500);
if(sensorsstreaming) {
unsigned char receive;
int read = 0;
read = connection.PollComport(port, &receive, sizeof(unsigned char));
if((int)receive == 19) {
read = connection.PollComport(port, &receive, sizeof(unsigned char));
unsigned char rest[54];
read = connection.PollComport(port, rest, 54);
/* ***SET SENSOR VALUES*** */
//bump + wheel drop
sensor_values[0] = (int)rest[1];
sensor_values[1] = -1;
//wall
sensor_values[2] = (int)rest[2];
sensor_values[3] = -1;
...
...
lots more setting just like the two above
} //end if header == 19
} //end if sensors streaming
} //end while
} //END GET_SENSORS_THREAD_I
Thank you for any help.
The pthread_* functions return an error code; they do not set errno. (Well, they may of course, but not in any way that is documented.)
Your code should print the value returned by pthread_detach and print that.
Single Unix Spec documents two return values for this function: ESRCH (no thread by that ID was found) and EINVAL (the thread is not joinable).
Detaching threads in the destructor of an object seems silly. Firstly, if they are going to be detached eventually, why not just create them that way?
If there is any risk that the threads can use the object that is being destroyed, they need to be stopped, not detached. I.e. you somehow indicate to the threads that they should shut down, and then wait for them to reach some safe place after which they will not touch the object any more. pthread_join is useful for this.
Also, it is a little late to be doing that from the destructor. A destructor should only be run when the thread executing it is the only thread with a reference to that object. If threads are still using the object, then you're destroying it from under them.
I've seen it said multiple times that there is no way to limit a Lua script's memory usage, including people jumping through hoops to prevent Lua scripts from creating functions and tables. But given that lua_newstate allows you to pass a custom allocator, couldn't one just use that to limit memory consumption? At worst, one could use an arena-based allocator and put a hard limit even on the amount of memory that could be used by fragmentation.
Am I missing something here?
static void *l_alloc_restricted (void *ud, void *ptr, size_t osize, size_t nsize)
{
const int MAX_SIZE = 1024; /* set limit here */
int *used = (int *)ud;
if(ptr == NULL) {
/*
* <http://www.lua.org/manual/5.2/manual.html#lua_Alloc>:
* When ptr is NULL, osize encodes the kind of object that Lua is
* allocating.
*
* Since we don’t care about that, just mark it as 0.
*/
osize = 0;
}
if (nsize == 0)
{
free(ptr);
*used -= osize; /* substract old size from used memory */
return NULL;
}
else
{
if (*used + (nsize - osize) > MAX_SIZE) /* too much memory in use */
return NULL;
ptr = realloc(ptr, nsize);
if (ptr) /* reallocation successful? */
*used += (nsize - osize);
return ptr;
}
}
To make Lua use your allocator, you can use
int *ud = malloc(sizeof(int)); *ud = 0;
lua_State *L = lua_State *lua_newstate (l_alloc_restricted, ud);
Note: I haven't tested the source, but it should work.