call to postmessage returns "access denied" - delphi

Question

Using Delphi 2007...
I have an application which uses a mutex to enforce one running instance only. In the dpr unit, if the mutex already exists, I get a handle to the running instance with FindWindow, no problem so far. A second instance is commonly started by a virtual printer driver with reference to a filename on the command line. If there is a command line file reference then I want to simply post a message to the running instance and halt the new instance.
I'm using this...
PostMessage(hwnd,WM_STARTUP_MESSAGE,0,0); //hwnd as returned by FindWindow
WM_STARTUP_MESSAGE is defined as WM_APP + 6057
I have one user with the problem that the WM_STARTUP_MESSAGE is not processed in the main thread. From logging startup information in the dpr unit it was shown that the PostMessage returned false and SysErrorMessage(GetLastError) is:
Zugriff verweigert (his windows german translation is Access Denied).
I have many, many users of this application and I have only had 2 reports of this problem and cannot reproduce here. On Windows 10 here and so is at least 1 of the problem users, the other one I'm not sure of.
I am using ChangeWindowMessageFilterEx in the main form's OnCreate to allow WM_COPYDATA. I had the idea of simply including WM_STARTUP_MESSAGE there as well but that leads to a crash as that function does not like that message index value, so I presume it is reserved for a specific range of message values.
Has anyone seen this before and can offer some guidance?

Answer 1

Per the PostMessage() documentation:
When a message is blocked by UIPI the last error, retrieved with GetLastError, is set to 5 (access denied).
What is User Interface Privilege Isolation (UIPI) on Vista
This is also known as UI Privilege Level Isolation (UIPI).
As part of the secure initiatuve in Vista, applications with UI will run in three different levels of privilege. Application windows can interact with others windows of the same or lower levels, but cannot interact with applications at higher level/permission.
Lower privilege modes can send messages to higher privileged applications only if explicitly allowed by the higher privilege application with a message calling ChangeWindowMessageFilter(). Also lower privileged applications can only read a HWND owned by a higher privileged application.
Internet Explorer is an example process that runs at the lowest privilege level.
Windows Integrity Mechanism Design
User Interface Privilege Isolation (UIPI) implements restrictions in the windows subsystem that prevents lower-privilege applications from sending window messages or installing hooks in higher-privilege processes. Higher-privilege applications are permitted to send window messages to lower-privilege processes. The restrictions are implemented in the SendMessage and related window message functions. Not all window messages that are sent from a lower-privilege process to a higher-privilege process are blocked. Generally, “read” type messages, for example WM_GETTEXT, can be sent from a lower-privilege to a higher-privilege window. However, write type messages, such as WM_SETTEXT, are blocked.
...
UIPI does not interfere with or change the behavior of window messaging between applications at the same privilege (or integrity) level. UIPI prevents lower-privilege processes from accessing higher-privilege processes by blocking the following behavior. A lower-privilege process cannot:
Perform a window handle validation of a process running with higher rights.
Use SendMessage or PostMessage to application windows running with higher rights. These APIs return success but silently drop the window message.
Use thread hooks to attach to a process running with higher rights.
Use journal hooks to monitor a process running with higher rights.
Perform dynamic link library (DLL) injection to a process running with higher rights.
So clearly, when the error occurs, the HWND you are posting to belongs to a process that is running at a higher integrity/privilege level than the process that is posting the message. You can use SysInternals' Process Explorer to check that.
You said the second instance of your app is being run by a virtual printer driver, so the driver is likely running at a lower integrity level than an instance of the app that is started by the user.
ChangeWidowMessageFilter/Ex() does not have any restrictions on non-system message IDs (some system messages cannot be filtered). It will certainly not crash on a user-defined message ID. If you do not have permission to change the message filter for a given HWND/MsgID, the function will simply return FALSE instead, and GetLastError() will tell you why.
If you are experiencing a crash, it is related to something else in your code.
Also, the Form's OnCreate event is not the best place to call ChangeWindowMessageFilterEx(). During the course of the program's lifetime, the Form may have to recreate its HWND dynamically, maybe even more than once. Every time a new HWND is created, you have to call ChangeWindowMessageFilterEx() again. The best way to account for that is to override the Form's virtual CreateWnd() method, eg:
type
TMyForm = class(TForm)
protected
procedure CreateWnd; override;
end;
procedure TMyForm.CreateWnd;
begin
inherited;
ChangeWindowMessageFilterEx(Handle, WM_STARTUP_MESSAGE, MSGFLT_ALLOW, nil);
end;