I am using vfork() in glibc and according to vfork()'s man page:
Fork handlers established using pthread_atfork(3) are not called
when a multithreaded program employing the NPTL threading library
calls vfork(). Fork handlers are called in this case in a
program using the LinuxThreads threading library.
On NPTL fork handlers are not being called.
In my specific case I need this protection to be engaged so fork handlers will be called the same way as when calling fork().
Is there a way to cause pthread library to call registered handlers or even call them manually?
I thought of using clone() as it gives more precise control of the cloned process but it also avoids fork handlers:
Handlers registered using pthread_atfork(3) are not executed
during a clone call.
Also read how to reset handlers registered by pthread_atfork - in my case I don't want to remove handlers but only call them.
Thanks.
Is there a way to cause pthread library to call registered handlers or even call them manually?
If you succeeded in doing this, your program would become corrupt. There is a reason pthread_atfork handlers aren't called on vfork.
A typical usage of pthread_atfork handlers is to guarantee that any locks held in the parent process are in a consistent state in both the child and the parent after the fork.
Without handlers, suppose thread T1 is holding lock L, and thread T2 calls fork. Since only T2 will be replicated into the child, L will remain locked forever. If T2 needs that lock (say it was the malloc lock), T2 will deadlock (with T1 which doesn't exist in the child).
The solution: pthread_atfork handlers. In the parent process prepare() will acquire L, and both parent() and child() handlers will unlock their own instance of L in the parent and child processes respectively. Everything is happy1.
Now consider what happens if you do the same after vfork. prepare() acquires L as expected, but when you call parent() and child(), they unlock the same instance of L (because parent and child processes share memory). Thus L gets unlocked twice, corrupting it!
1 In practice calling either fork or vfork in a multithreaded program is fraught with peril, and doing this safely in the presence of arbitrary libraries is near impossible.
Related
Good morning,
I would like to use a TThread object to process numeric values.
On a recurring basis (through the TTimer object), different/updated values are always presented in processing.
Basically:
The first time I create and launch the TThread which is actually executed but not deleted.
Subsequently (by TTimer), I get new values for TThread to process.
Request:
Is there a way to "restart" the TThread with the new values, without creating a new TThread object every time? (TThread already exists)
This would save time, since the values would always use the space allocated in the first TThread creation.
A TThread (or any other kind of thread wrapper, for that matter) cannot be "restarted" once it has finished running. All you can do is free it.
So, to do what you are asking for, you would need to make the thread's Execute() method run a loop that processes your numeric values on each iteration as needed, until you are ready to signal the thread to terminate itself.
You will have to implement your own thread-safe system to push new numerics into the thread when needed, and to make the thread wait for new numerics to arrive between each loop iteration. For instance, you could push values into a TThreadList or TThreadedQueue, signaling a waitable TEvent or TMonitor on each push, and then the thread loop can wait on that signal before pulling values on each iteration.
Otherwise, consider using TTask instead of TThread. Tasks can utilize thread pooling internally, so you can just create a new TTask each time you have a new numeric to process, and let it pull out an available thread from the pool, and put the thread back in the pool when finished.
I'm allocating my pthread thread-specific data from a fixed-size global pool that's controlled by a mutex. (The code in question is not permitted to allocate memory dynamically; all the memory it's allowed to use is provided by the caller as a single buffer. pthreads might allocate memory, I couldn't say, but this doesn't mean that my code is allowed to.)
This is easy to handle when creating the data, because the function can check the result of pthread_getspecific: if it returns NULL, the global pool's mutex can be taken there and then, the pool entry acquired, and the value set using pthread_setspecific.
When the thread is destroyed, the destructor function (as per pthread_key_create) is called, but the pthreads manual is a bit vague about any restrictions that might be in place.
(I can't impose any requirements on the thread code, such as needing it to call a destructor manually before it exits. So, I could leave the data allocated, and maybe treat the pool as some kind of cache, reusing entries on an LRU basis once it becomes full -- and this is probably the approach I'd take on Windows when using the native API -- but it would be neatest to have the per-thread data correctly freed when each thread is destroyed.)
Can I just take the mutex in the destructor? There's no problem with thread destruction being delayed a bit, should some other thread have the mutex taken at that point. But is this guaranteed to work? My worry is that the thread may "no longer exist" at that point. I use quotes, because of course it certainly exists if it's still running code! -- but will it exist enough to permit a mutex to be acquired? Is this documented anywhere?
The pthread_key_create() rationale seems to justify doing whatever you want from a destructor, provided you keep signal handlers from calling pthread_exit():
There is no notion of a destructor-safe function. If an application does not call pthread_exit() from a signal handler, or if it blocks any signal whose handler may call pthread_exit() while calling async-unsafe functions, all functions may be safely called from destructors.
Do note, however, that this section is informative, not normative.
The thread's existence or non-existence will most likely not affect the mutex in the least, unless the mutex is error-checking. Even then, the kernel is still scheduling whatever thread your destructor is being run on, so there should definitely be enough thread to go around.
Can an OTP event manager process (e.g. a logger) have some state of its own (e.g. logging level) and filter/transform events based on it?
I also have a need to put some state into the gen_event itself, and my best idea at the moment is to use the process dictionary (get/put). Handlers are invoked in the context of the gen_event process, so the same process dictionary will be there for all handler calls.
Yes, process dictionaries are evil, but in this case they seem less evil than alternatives (ets table, state server).
The gen_event implementation as contained in the OTP does no provide means for adding state.
You could extend the implementation to achieve this and use your implementation instead of gen_event. However I would advise against it.
The kind of state you want to add to the event manager belongs really in the event handler for several reasons:
You might want to use different levels in different handlers, e.g. only show errors on the console but write everything to the disk.
If the event level would be changed in the manager event handlers depending on getting all unfiltered events might cease to function (events have more uses than just logging). This might lead to hard to debug problems.
If you want a event manager for multiple handlers that all only get filtered events you can easily achieve this by having two managers: one for unfiltered messages and one for e.g. level filtered messages. Then install a handler to the unfiltered one, filter in the handler by level (easy) and pass on the filtered events to the other manager. All handlers that only want to get filtered messages can be registered to the second manager.
The handlers can have their own state that gets passed on every callback like:
Module:handle_event(Event, State) -> Result
Filtering might look like this (assuming e.g. {level N, Content} events):
handle_event({level, Lvl, Content}, State#state{max_level=Max}) when Lvl >= Max ->
gen_event:notify(filtered_man, Content);
The State can be changed either by special events, by gen_event:call\3,4 (preferably) or by messages handled by handle_info.
For details see Gen_Event Behaviour and gen_event(3)
When you start_link a gen_event process - thing that you should always do via a supervisor -, you can merely specify a name for the new process, if you need/want it to be registered.
As far as I can see, there's no way to initiate a state of some sort using that behaviour.
Of course, you can write your own behaviour, on the top of a gen_event or of a simple gen_server.
As an alternative, you might use a separate gen_event process for each debugging level.
Or you can just filter the messages in the handlers.
The documentation of delphi says that the WaitFor function for TMutex and others sychronization objects wait until a handle of object is signaled.But this function also guarantee the ownership of the object for the caller?
Yes, the calling thread of a TMutex owns the mutex; the class is just a wrapper for the OS mutex object. See for yourself by inspecting SyncObjs.pas.
The same is not true for other synchronization objects, such as TCriticalSection. Any thread my call the Release method on such an object, not just the thread that called Acquire.
TMutex.Acquire is a wrapper around THandleObjects.WaitFor, which will call WaitForSingleObject OR CoWaitForMultipleHandles depending on the UseCOMWait contructor argument.
This may be very important, if you use STA COM objects in your application (you may do so without knowing, dbGO/ADO is COM, for instance) and you don't want to deadlock.
It's still a dangerous idea to enter a long/infinite wait in the main thread, 'cause the only method which correctly handles calls made via TThread.Synchronize is TThread.WaitFor and you may stall (or deadlock) your worker threads if you use the SyncObjs objects or WinAPI wait functions.
In commercial projects, I use a custom wait method, built upon the ideas from both THandleObjects.WaitFor AND TThread.WaitFor with optional alertable waiting (good for asynchronous IO but irreplaceable for the possibility to abort long waits).
Edit: further clarification regarding COM/OLE:
COM/OLE model (e.g. ADO) can use different threading models: STA (single-threaded) and MTA (multi or free-threaded).
By definition, the main GUI thread is initialized as STA, which means, the COM objects can use window messages for their asynchronous messaging (particulary when invoked from other threads, to safely synchronize). AFAIK, they may also use APC procedures.
There is a good reason for the CoWaitForMultipleHandles function to exist - see its' use in SyncObjs.pas THandleObject.WaitFor - depending on the threading model, it can process internal COM messages, while blocking on the wait handle.
I've found several implementations for single producer-single consumer, but none for multiple producer-single consumer.
Does a lock-free queue for "multiple producers-single consumer" exist for Delphi?
Lock-free queue from the OmniThreadLibrary supports multiple producers. You can use it separately from the threading library (i.e. you can use OtlContainers unit in any other framework).
As the Daniele pointed below, there are two queues in the OmniThreadLibrary. The one in the OtlContainers supports multiple producers and multiple consumers while the "smarter" version in OtlComm (which is just a wrapper for the simpler version) is only single producer/single consumer.
Documentation is still a big problem of the OmniThreadLibrary project :(. Some information on the queue can be found here .
May be that could be helpful: Interlocked SList functions.
http://svn.berlios.de/svnroot/repos/dzchart/utilities/dzLib/trunk/lockfree/
#Daniele Teti:
The reader must wait for all writers who still have access to the old queue to exit the Enqueue method. Since the first thing the reader does in the Dequeue method is providing a new queue for new writers which enter Enqueue it should not take long for all writers that have a reference to the old queue to exit Enqueue. But you are right: It is lock free only for the writers but might still require the reader thread to wait for some writers to exit Enqueue.
For a multiple-producer / single-consumer Queue/FIFO, you can easily make one LockFree using SLIST or a trivial Lock Free LIFO stack. What you do is have a second "private" stack for the consumer (which can also be done as a SLIST for simplicity or any other stack model you choose). The consumer pops items off the private stack. Whenever the private LIFO is exhasted, you do a Flush rather than Pop off the shared concurrent SLIST (grabbing the entire SLIST chain) and then walk the Flushed list in-order pushing items onto the private stack.
That works for single-producer / single-consumer and for multiple-producer / single-consumer.
However, it does not work for multiple-producer / multiple-consumer cases.