VCL Style from DLL is affecting TMenuItem in Application - delphi

Question

I am using Delphi XE6 and VCL styles. I have main application and dlls. My main application has enabled runtime themes and I am using vcl style files. I did quite similar thing to my DLLs. I enabled runtime themes and added VCL.Themes, VCL.Styles under uses and resource file with VCL style file within it. When DLL is loaded I load VCL style from resources and set it for DLL gui. Main app and DLL are not built with runtime packages.
Now I have main app GUI styled with own style and DLL gui styled with own style too. This seems to work fine until...
When I click on button in my main app which event opens TPopupMenu it's styled with same style as DLL GUI instead of main app style. If I navigate through menu I get AV too and program crashes. Take a look at the attached image.
What am I doing wrong? The only workaround I currently see would be to make my own customized TPopupMenu derived from some other control.
As I promised I prepared simple demo program which is similar to my application. It consists of host application with own style and DLL with style added to resource. Run it and click on button Popup then try select something from popup. It will crash and stop in some StdWindowProc or something like that. Also if you go to window system menu (left top corner) when you try to select something from that menu you will notice that system menu is styled as DLL gui and crashes too. Link to rar file: dropbox.com/sh/f2jmbsmw18akpyg/AAA6SWdBmVhf6n6K-mvYLLmua?dl=0
Thanks for your help.

Answer 1

This is a fundamental problem with VCL styles and the way that they style menus. The styling is implemented with a process wide hook. Specifically a CBT hook installed by a call to SetWindowsHookEx from TCustomStyleEngine.CreateSysHook in the Vcl.Themes unit. In fact, the hook applies just to the GUI thread, but that is process wide in the sense that there is exactly one GUI thread in the process.
Since you have multiple instances of the VCL in your application (one in the DLL and one in the application), two hooks are installed. That is one too many. The hook installed most recently (the DLL as it happens) wins, and that's why the DLL menu styling infects your executable. And why you encounter an access violation. The DLL is trying to operate on a menu that belongs to the executable. And so, in spite of your best efforts, you've ended up with the DLL code accessing VCL objects from the host executable.
There's no simple way to work around this and support styles fully in both modules. What we have here is a fundamental consequence of the design. The system was not designed to support multiple VCL instances. If you wish to use VCL styles in multiple modules, then the designers expect you to use runtime packages.
I suppose that you might be able to get some traction by operating the DLL out of a completely different thread. That would involve loading the DLL from that different thread so that the VCL is initialized in the thread. And all calls to the DLL would have to be from that thread. And you'd need to run a message loop in that thread. It's possible that you might be able to make that work, but I doubt it. Even with all the provisos mentioned you still have to handle the fact that you have two GUI threads which presents all sorts of issues with the input queue handling.
Perhaps another approach would be to uninstall the hook from the DLL. So long as your DLL is not showing menus then you may well be able to get away with uninstalling that hook. It would disable styling for menus shown by the DLL, but perhaps that's acceptable.
This version of your DLL (after I simplified it somewhat also) uninstalls the hook.
library VCLStyleDLL;
{$R 'Style.res' 'Style.rc'}
uses
VCL.Styles,
VCL.Themes,
VCL.SysStyles; // to gain access to TSysPopupStyleHook
{$R *.res}
begin
TStyleManager.TrySetStyle('Glossy', false);
TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook);
end.
With this version of the DLL, the host executable does not suffer the problems your describe in your question.

Answer 2

As David says this is caused because each VCL instance install a hook to detect when a popup menu (#32768) is created. So there is two hook instances working at the same time.
As workaround you can disable the popupmenu style hook in the dll (or in the app) using the UnRegisterSysStyleHook function defined in the Vcl.SysStyles unit.
TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook);