splWOW64 hanging on print server - printing

Question

So I have a print server running Windows Server 2008 64 bit. It server up crystal reports to a variety of printers, some old, some new. That means there's a few different drivers on there. Recently we started having issues where the splWOW64 process will hang and all printing will back up. If we kill that process the queue prints normally. Looking at the what appears to be the hung print job every time we can see what printer and what report were printing, however it's never the same report or printer. We have full dumps of the splwow64 process and were told that the HP universal print driver PCL5 was causing the issue. It had been working for most of our printers for years before with no problems. So we removed that driving and started using individual drivers for each model of printer, all PCL6 if we could find them on the microsoft driver database. None of this has worked to fix the issue. It still happens 2-3 times a day depending on how busy it is. I've never used windbg to debug anything, I have below the results of !analyze -v -hang for a recent dump. It's gibberish to me at the moment. Maybe someone out there can see something obviously wrong?
FAULTING_IP:
+0
00000000`00000000 ?? ???
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
CONTEXT: 0000000000000000 -- (.cxr 0x0;r)
rax=0000000000000000 rbx=0000000000000000 rcx=00000000004486f8
rdx=00000000ffffffff rsi=00000000ffffffff rdi=0000000000000088
rip=0000000076d812fa rsp=000000000028f708 rbp=0000000000000001
r8=000000000028f7d8 r9=0000000000000001 r10=0000000000000000
r11=0000000000000202 r12=0000000000000000 r13=00000000ff963440
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!NtWaitForSingleObject+0xa:
00000000`76d812fa c3 ret
FAULTING_THREAD: 0000000000000000
BUGCHECK_STR: HANG
DEFAULT_BUCKET_ID: APPLICATION_HANG
PROCESS_NAME: splwow64.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: splwow64.exe
ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 b68.19bc Unknown
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00000000000019bc
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG
LAST_CONTROL_TRANSFER: from 000007fefcfa10dc to 0000000076d812fa
STACK_TEXT:
00000000`0028f708 000007fe`fcfa10dc : 00000000`0044d000 00000000`00400000 00000000`0044cff0 00000000`76d840fd : ntdll!NtWaitForSingleObject+0xa
00000000`0028f710 000007fe`fd2ed95d : 00000000`004485f0 00000000`0000000a 00000000`00000000 00000000`00000088 : KERNELBASE!WaitForSingleObjectEx+0x79
00000000`0028f7b0 000007fe`fd36f42c : 00000000`00000000 00000000`00000000 00000000`004485f0 000007fe`fd2ff74e : rpcrt4!EVENT::Wait+0xd
00000000`0028f7e0 000007fe`fd33a879 : 00000000`004485f0 00000000`004485f0 00000000`00000000 00000000`00000001 : rpcrt4!RPC_SERVER::WaitForStopServerListening+0x1c
00000000`0028f810 000007fe`fd2ffa49 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000001 : rpcrt4!Invoke+0x13e46
00000000`0028f850 00000000`ff966b98 : 00000000`00000000 00000000`0000000a 00000000`0000000a 00000000`000004d2 : rpcrt4!RpcServerListen+0x49
00000000`0028f880 00000000`ff9671f1 : 00000000`00000000 00000000`0028fa20 00000000`00187c90 00000000`00003000 : splwow64!TLoad64BitDllsMgr::StartLdrRPCServer+0x19c
00000000`0028f9d0 00000000`ff967fb2 : 00000000`00187c90 00000000`00003000 00000000`00001a20 00000000`00003000 : splwow64!TLoad64BitDllsMgr::Run+0x4d
00000000`0028fa10 00000000`ff96d095 : 00000000`00000000 00000000`00000000 00000000`00187d20 00000000`00000000 : splwow64!wmain+0x1ae
00000000`0028fa50 00000000`76b2652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : splwow64!ConvertStringSecurityDescriptorToSecurityDescriptorW+0x19b
00000000`0028fa90 00000000`76d5c541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0028fac0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
FOLLOWUP_IP:
splwow64!TLoad64BitDllsMgr::StartLdrRPCServer+19c
00000000`ff966b98 8bd8 mov ebx,eax
SYMBOL_STACK_INDEX: 6
SYMBOL_NAME: splwow64!TLoad64BitDllsMgr::StartLdrRPCServer+19c
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: splwow64
IMAGE_NAME: splwow64.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 4f35fbfe
STACK_COMMAND: ~0s ; kb
BUCKET_ID: X64_HANG_splwow64!TLoad64BitDllsMgr::StartLdrRPCServer+19c
FAILURE_BUCKET_ID: APPLICATION_HANG_cfffffff_splwow64.exe!TLoad64BitDllsMgr::StartLdrRPCServer
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:application_hang_cfffffff_splwow64.exe!tload64bitdllsmgr::startldrrpcserver
FAILURE_ID_HASH: {369fae16-3854-e2c0-c756-fdab044a0958}
Followup: MachineOwner

Answer 1

You should make a kernel dump ( see: http://support.microsoft.com/kb/244139 )
Then you should do:
search your process !process 0 0 splwow64
switch to the found process
.process /p addr
list all thread for found process !process addr 17
find your thread
find ALPC handle in the stack and find a kernel object: !handle handle
print ALPC port object !alpc ob_addr
find a print corresponding server port
If you have done these step you must know a RPC server process which is hang your RPC request