TWaitResult.wrIOCompletion is undocumented. Does anyone know when and how it's used?
It is used only on Windows, by THandleObject (and its descendants TEvent, TSimpleEvent, TMutex, and TSemaphore) in the following methods:
THandleObject.WaitFor(). When the object is created with UseCOMWait set to True, the wait is handled by the Win32 API CoWaitForMultipleHandles() function, where wrIOCompletion is returned if RPC_S_CALLPENDING is reported. When UseCOMWait is False instead, the wait is handled by the Win32 API WaitForMultipleObjectsEx() function 1.
THandleObject.WaitForMultiple(). wrIOCompletion is returned if the UseCOMWait parameter is True and CoWaitForMultipleHandles() reports RPC_S_CALLPENDING, or when UseCOMWait is False and WaitForMultipleObjectsEx() reports WAIT_IO_COMPLETION.
Either way, the meaning is the same 2:
The wait was ended by one or more user-mode asynchronous procedure calls (APC) queued to the thread
Refer to MSDN for more details about APC queues:
Asynchronous Procedure Calls
Alertable I/O
In a nutshell, an Alertable I/O or APC operation allows a user-defined operation/function to be queued in a thread so it will be called by the thread when in a safe state to make such a call. wrIOCompletion indicates that the thread that is calling WaitFor/Multiple() had to stop waiting before the timeout elapsed so it could execute one or more queued Alertable/APC functions. The thread will have to call WaitFor/Multiple() again to finish waiting on its desired object(s) 3.
1: THandleObject.WaitFor() does not currently handle WAIT_IO_COMPLETION (bug?) when UseCOMWait is false. It will return wrError instead (and the value of the THandleObject.LastError property will not be assigned!)
2: the CoWaitForMultipleHandles() documentation describes RPC_S_CALLPENDING as "The timeout period elapsed before the required handle or handles were signaled", but that would be a more appropriate description for RPC_E_TIMEOUT instead. (documentation error?)
3: in practice wrIOCompletion should never happen, because CoWaitForMultipleHandles() is never called with the COWAIT_ALERTABLE flag:
If the COWAIT_ALERTABLE flag is set in dwFlags, a value of WAIT_IO_COMPLETION indicates the wait was ended by one or more user-mode asynchronous procedure calls (APC) queued to the thread.
And WaitForMultipleObjectsEx() is never called with its bAlertable parameter set to True:
bAlertable [in]
If this parameter is TRUE and the thread is in the waiting state, the function returns when the system queues an I/O completion routine or APC, and the thread runs the routine or function. Otherwise, the function does not return and the completion routine or APC function is not executed.
These conditions are needed to trigger the wrIOCompletion result. I have opened a bug report for this in Quality Portal:
RSP-14047 THandleObject never returns wrIOCompletion.
Related
With the actual implementation of TMultiReadExclusiveWriteSynchronizer, you must do in the same Thread MyMultiReadExclusiveWriteSynchronizer.beginRead and MyMultiReadExclusiveWriteSynchronizer.endRead
However me I need to call MyMultiReadExclusiveWriteSynchronizer.beginRead in one thread and call MyMultiReadExclusiveWriteSynchronizer.endRead in another thread (this because I use winapi IO completion block)
Is there any way ?
I am using IO completion ports and AcceptEx() whilst learning about servers, and am studying Len Holgate's free server framework to do this. He has the following code:
// Basically calls AcceptEx() via a previously obtained function pointer
if (!CMSWinSock::AcceptEx(
m_listeningSocket,
pSocket->m_socket,
reinterpret_cast<void*>(const_cast<BYTE*>(pBuffer->GetBuffer())),
bufferSize,
sizeOfAddress,
sizeOfAddress,
&bytesReceived,
pBuffer))
{
const DWORD lastError = ::WSAGetLastError();
if (ERROR_IO_PENDING != lastError)
{
Output(_T("CSocketServerEx::Accept() - AcceptEx: ") + GetLastErrorMessage(lastError));
pSocket->Release();
pBuffer->Release();
}
}
else
{
// Accept completed synchronously. We need to marshal the data recieved over to the
// worker thread ourselves...
m_iocp.PostStatus((ULONG_PTR)m_listeningSocket, bytesReceived, pBuffer);
}
I am confused about the "Accept completed synchronously" else-case. I have tried many times to get this code path to be hit (by pausing the code before I issue the AcceptEx, connecting, then resuming the code), but whenever I try the call always fails with ERROR_IO_PENDING and I get my notification packet. Furthermore, I have read this MS knowledgebase article (which I may have misinterpreted) which states
Additionally, if a Winsock2 I/O call returns SUCCESS or IO_PENDING, it
is guaranteed that a completion packet will be queued to the IOCP when
the I/O completes
However, I am thinking this doesn't apply to AcceptEx() because the dox for AcceptEx() states of the parameter lpdwBytesReceived
This parameter is set only if the operation completes synchronously.
So it seems it can complete synchronously...can someone tell me how AcceptEx() can complete synchronously (i.e. how I can replicate it in my server?)
Additionally, if a Winsock2 I/O call returns SUCCESS or ERROR_IO_PENDING, it
is guaranteed that a completion packet will be queued to the IOCP when
the I/O completes
this is apply for any I/O request if completion port is associated with the file. but begin from windows vista this also depend from notification mode set for a file handle.
but need first begin look from native view.
by default, if FILE_SKIP_COMPLETION_PORT_ON_SUCCESS not set, exist 3 case by returned NTSTATUS status :
NT_SUCCESS(status) or status >= 0 - will be completion
NT_ERROR(status) or status >= 0xc0000000 - will be no completion
NT_WARNING(status) or status < 0xc0000000 - unclear - if this
error from I/O manager (say - STATUS_DATATYPE_MISALIGNMENT - will
be no completion). if this error from driver (say
STATUS_NO_MORE_FILES - will be completion).
the win32 layer usually separate check for STATUS_PENDING and return ERROR_IO_PENDING in this case (but exist and exceptions, like ReadDirectoryChangesW). otherwise in case NT_ERROR(status) api return fail and set error code. otherwise return success. visible that case NT_WARNING(status) considered as success, but in this case, if error from I/O manager, will be no completion. I/O usually return errors from NT_ERROR(status) range, if parameters is incorrect. only case which i know (for asynchronous api) - STATUS_DATATYPE_MISALIGNMENT can be returned in case wrong aligned buffers, when I/O manager have special knowledge about buffer align. in NtNotifyChangeDirectoryFile
(ReadDirectoryChangesW for win32) or NtQueryDirectoryFile (no corresponded win32 api). so only case which i know when will be no completion, when win32 return success - call ReadDirectoryChangesW with unaligned lpBuffer (it must be DWORD-aligned ) - in this case I/O manager just return STATUS_DATATYPE_MISALIGNMENT but win32 layer interpret this as success code and return true. but will be no completion in this case. however this is rarely case and you probably need use wrong align structures for this. so in general yes:
by default if I/O call returns SUCCESS or ERROR_IO_PENDING will be queued a completion entry to the port. (with special exception case which i try describe)
if we set FILE_SKIP_COMPLETION_PORT_ON_SUCCESS on file object (note this is per file object but not per file handle - documentation not exactly here) all become much more simply and efficient - completion entry will be queue to the port - when and only when I/O request return STATUS_PENDING. ERROR_IO_PENDING from win32 view (except ReadDirectoryChangesW (maybe some else api ?) where win32 layer simply lost return code information)
However, I am thinking this doesn't apply to AcceptEx()
you mistake. this, how i say, apply to any io request. "This parameter is set only if the operation completes synchronously." - and so what ?
if look to code snippet, clear visible that code assume - in case AcceptEx completed synchronous and no error occurs - will be no io completion. or SetFileCompletionNotificationModes(m_listeningSocket, FILE_SKIP_COMPLETION_PORT_ON_SUCCESS) called or code is wrong - will be io completion in this case and not need m_iocp.PostStatus - this is fatal error will be. however i doubt that code used FILE_SKIP_COMPLETION_PORT_ON_SUCCESS - so it wrong. but error never raised because driver side implementation of AcceptEx (underlining ioctl) never return STATUS_SUCCESS: it check parameters - if it wrong - just return some error, otherwise always return STATUS_PENDING. as result, for asynchronous sockets AcceptEx never return true and code never jump to error else case. but anyway code is wrong. also i think design not the best - in case we determinate will be no completion - better just direct call completion routine with returned error code instead Release() (this will be done in completion routine) or PostStatus - for what post ?! - call direct.
how AcceptEx() can complete synchronously
very easy - if m_listeningSocket is handle to synchronous file object. however in this case you can not bind IOCP to file (it can be bind only in case asynchronous file object).
about lpdwBytesReceived parameter - system copy Information member of IO_STATUS_BLOCK or if want OVERLAPPED.InternalHigh, in case operation is completed just. in case pending returned - this data simply not ready and not filled. you got actual number of bytes returned by io in completion
I read somewhere that we should lock the mutex before calling pthread_cond_signal and unlock the mutex after calling it:
The pthread_cond_signal() routine is
used to signal (or wake up) another
thread which is waiting on the
condition variable. It should be
called after mutex is locked, and must
unlock mutex in order for
pthread_cond_wait() routine to
complete.
My question is: isn't it OK to call pthread_cond_signal or pthread_cond_broadcast methods without locking the mutex?
If you do not lock the mutex in the codepath that changes the condition and signals, you can lose wakeups. Consider this pair of processes:
Process A:
pthread_mutex_lock(&mutex);
while (condition == FALSE)
pthread_cond_wait(&cond, &mutex);
pthread_mutex_unlock(&mutex);
Process B (incorrect):
condition = TRUE;
pthread_cond_signal(&cond);
Then consider this possible interleaving of instructions, where condition starts out as FALSE:
Process A Process B
pthread_mutex_lock(&mutex);
while (condition == FALSE)
condition = TRUE;
pthread_cond_signal(&cond);
pthread_cond_wait(&cond, &mutex);
The condition is now TRUE, but Process A is stuck waiting on the condition variable - it missed the wakeup signal. If we alter Process B to lock the mutex:
Process B (correct):
pthread_mutex_lock(&mutex);
condition = TRUE;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&mutex);
...then the above cannot occur; the wakeup will never be missed.
(Note that you can actually move the pthread_cond_signal() itself after the pthread_mutex_unlock(), but this can result in less optimal scheduling of threads, and you've necessarily locked the mutex already in this code path due to changing the condition itself).
According to this manual :
The pthread_cond_broadcast() or
pthread_cond_signal() functions
may be called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait()
or pthread_cond_timedwait() have
associated with the condition variable
during their waits; however, if
predictable scheduling behavior is
required, then that mutex shall be
locked by the thread calling
pthread_cond_broadcast() or
pthread_cond_signal().
The meaning of the predictable scheduling behavior statement was explained by Dave Butenhof (author of Programming with POSIX Threads) on comp.programming.threads and is available here.
caf, in your sample code, Process B modifies condition without locking the mutex first. If Process B simply locked the mutex during that modification, and then still unlocked the mutex before calling pthread_cond_signal, there would be no problem --- am I right about that?
I believe intuitively that caf's position is correct: calling pthread_cond_signal without owning the mutex lock is a Bad Idea. But caf's example is not actually evidence in support of this position; it's simply evidence in support of the much weaker (practically self-evident) position that it is a Bad Idea to modify shared state protected by a mutex unless you have locked that mutex first.
Can anyone provide some sample code in which calling pthread_cond_signal followed by pthread_mutex_unlock yields correct behavior, but calling pthread_mutex_unlock followed by pthread_cond_signal yields incorrect behavior?
Hello I have an question about cancelling a thread that uses mutexes and conditional variables. The thread has cancel type deferred. When I use only the functions pthread_mutex_lock/unlock and pthread_cond_wait, and a cancel request arrives, the thread's cancelation point is only pthread_cond_wait. Will it lock the mutex or not? I am not sure, if thread always leaves the mutex unlock. Or are the pthread_mutex_lock/unlock functions also cancellation points? Thank you.
I doubt that I can phrase this better than the documentation:
A condition wait (whether timed or not) is a cancellation point. When
the cancelability type of a thread is set to PTHREAD_CANCEL_DEFERRED,
a side effect of acting upon a cancellation request while in a
condition wait is that the mutex is (in effect) re-acquired before
calling the first cancellation cleanup handler. The effect is as if
the thread were unblocked, allowed to execute up to the point of
returning from the call to pthread_cond_timedwait() or
pthread_cond_wait(), but at that point notices the cancellation
request and instead of returning to the caller of
pthread_cond_timedwait() or pthread_cond_wait(), starts the thread
cancellation activities, which includes calling cancellation cleanup
handlers.
Also just be sure you are aware that other functions are cancellation points as well.
If you have multiple threads waiting on pthread_cond_wait and you call pthread_cancel before sending out some of kind of broadcast using pthread_cond_broadcast or pthread_cond_signal, then it may lead to a deadlock where the pthread_cond_wait is waiting on thread to cancel and thread is waiting on the pthread_cond_wait to come out.
Per "The Linux Programming Interface" book pages 667-678
the mutex will be locked. So you can use
pthread_mutex_lock(&mutex);
pthread_cleanup_push((void (*)(void *))pthread_mutex_unlock, &mutex);
while (var == 0)) {
pthread_cond_wait(&cond, &mutex);
}
pthread_cleanup_pop(1);
When pthread_exit(PTHREAD_CANCELED) is called I have expected behavior (stack unwinding, destructors calls) but the call to pthread_cancel(pthread_self()) just terminated the thread.
Why pthread_exit(PTHREAD_CANCELED) and pthread_cancel(pthread_self()) differ significantly and the thread memory is not released in the later case?
The background is as follows:
The calls are made from a signal handler and reasoning behind this strange approach is to cancel a thread waiting for the external library semop() to complete (looping around on EINTR I suppose)
I have noticed that calling pthread_cancel from other thread does not work (as if semop was not a cancellation point) but signalling the thread and then calling pthread_exit works but calls the destructor within a signal handler.
pthread_cancel could postpone the action to the next cancellation point.
In terms of thread specific clean-up behaviour there should be no difference between cancelling a thread via pthread_cancel() and exiting a thread via pthread_exit().
POSIX says:
[...] When the cancellation is acted on, the cancellation clean-up handlers for thread shall be called. When the last cancellation clean-up handler returns, the thread-specific data destructor functions shall be called for thread. When the last destructor function returns, thread shall be terminated.
From Linux's man pthread_cancel:
When a cancellation requested is acted on, the following steps occur for thread (in this order):
Cancellation clean-up handlers are popped (in the reverse of the order in which they were pushed) and called. (See pthread_cleanup_push(3).)
Thread-specific data destructors are called, in an unspecified order. (See pthread_key_create(3).)
The thread is terminated. (See pthread_exit(3).)
Referring the strategy to introduce a cancellation point by signalling a thread, I have my doubts this were the cleanest way.
As many system calls return on receiving a signal while setting errno to EINTR, it would be easy to catch this case and simply let the thread end itself cleanly under this condition via pthread_exit().
Some pseudo code:
while (some condition)
{
if (-1 == semop(...))
{ /* getting here on error or signal reception */
if (EINTR == errno)
{ /* getting here on signal reception */
pthread_exit(...);
}
}
}
Turned out that there is no difference.
However some interesting side effects took place.
Operations on std::iostream especially cerr/cout include cancellation points. When the underlying operation is canceled the stream is marked as not good. So you will get no output from any other thread if only one has discovered cancellation on an attempt to print.
So play with pthread_setcancelstate() and pthread_testcancel() or just call cerr.clear() when needed.
Applies to C++ streams only, stderr,stdin seems not be affected.
First of all, there are two things associated to thread which will tell what to do when you call pthread_cancel().
1. pthread_setcancelstate
2. pthread_setcanceltype
first function will tell whether that particular thread can be cancelled or not, and the second function tells when and how that thread should be cancelled, for example, should that thread be terminated as soon as you send cancellation request or it need to wait till that thread reaches some milestone before getting terminated.
when you call pthread_cancel(), thread wont be terminated directly, above two actions will be performed, i.e., checking whether that thread can be cancelled or not, and if yes, when to cancel.
if you disable cancel state, then pthread_cancel() can't terminate that thread, but the cancellation request will stay in a queue waiting for that thread to become cancellable, i.e., at some point of time if you are enabling cancel state, then your cancel request will start working on terminating that thread
whereas if you use pthread_exit(), then the thread will be terminated irrespective to the cancel state and cancel type of that particular thread.
*this is one of the differences between pthread_exit() and pthread_cancel(), there can be few more.