Yes, I've read the warning label, and I know that dynamically loading assemblies is somewhat discouraged. That said, I have an application that loads assemblies - that's just how it works. It works fine on Windows. Works fine on Windows CE. I need it to "work fine" on Android, even if it takes some massaging.
Basically the app is an engine that loads up plug-in DLLs (we'll call it an Adapter) that meet specific interfaces at run time. Under Windows, it even detects the appearance of a DLL at any point and goes and loads it - I'm fine if that's not going to work under Android.
What I'm having trouble getting working is having the Engine load an Adapter that it knew about at design/compile time but without hard coding the name of that Adapter into the Engine code. I'm fine with adding a reference to the Adapter to get it to not get linked out, but I really, really don't want to have to add in the DLL name every time, as the DLLs change with different deployments, and that would lead to a huge headache.
So I figured that if it's referenced, it would get into the APK, and I could use reflection to load it like this:
var asm = Assembly.Load("TheAdapterName.dll");
Initial tests show that this works for the Adapter if I just hard code in the name, but again, I really, really want to avoid that.
So I thought that maybe I could reflect through the references and extract the name, but oddly, not all references actually show up when I do that. So I do this:
var refs = asm.GetReferencedAssemblies().Select(a => a.Name).ToArray();
And I get back an array of 14 assembly names. But the assembly (asm) has 16 references, one of which is the Adapter plug-in I need to load. The Adapter is definitely there - heck I used Assembly.Load with the full name two lines above and it resolved.
I thought, ok, maybe I can figure out the "path" to the folder from which I'm running, and then look for DLLs there and load that way. Ha. After several hours of trying to figure out a way to get the path that would work under Debug and Release, I came up with nothing but more grey hair.
Sooooo...... any thoughts on how I might get the name of a DLL that I know is in my APK, but that I don't "know" the name of at build time (I'm loading them and looking for interfaces via reflection to detect their "Adapterness").
If those methods aren't working for you, then the only suggestion I can think of is to add a prebuild step which updates either a C# or an Assets file in order to provide the list you need.
Obviously this is extra work, but should be fully automated and is guaranteed to work no matter what platform changes get thrown at you.
As an aside, I also just looked at one of my mvx projects using reflector - it shows the same asm.GetReferencedAssemblies() list as your investigations report - runtime-loaded plugins are not listed. I guess that the GetReferencedAssemblies method is reporting only on assemblies actually used to import Type references at the IL level - so if you reference an assembly in the csproj but don't import any types then it doesn't list them as references in the compiled code.
Related
I've got a group project built in Delphi XE2 that has 3 projects that always build to the wrong folder for one option set. (I've got 4 configurations under Release and Debug, one for our software configurations and one for FastMM and it's only the debug one that I want to use for debugging that always goes in to the wrong folder. Compiling the project even says it's building to the correct folder, but the DLL always winds up in a different one which I only used once when I was unit testing the code outside of the main project.
I've deleted every associated file, .identcache, .res, .tvsproj (whatever that was) and nothing changed. One very strange thing I noticed is that I copied one of the projects to configure the second one and mimics the behavior of the one it was copied from and I never even unit tested that one, so it never had that output path configured for it.
Obviously this makes it pretty annoying to debug, I have to copy files in to the correct folder just to do that (I was kind of astonished when it actually worked, because I thought Delphi might expect to find the files in it's output path, but oh well, those things are magic)
Let me know if I can post anything to help, I don't really know what's necessary, I checked the registry for the output path that it is getting built do and found nothing that I thought was of any consequence (nothing related to these projects).
One thing I did notice was, because I copied the original project into another project (they're plugins to the same part of the main program) it has the same and when I try using it in the "Build Group" it automatically selects both projects. That's one mystery solved, but is probably a red herring?
OK so as usually happens, after 3 years of suffering with this when I finally ask the questions I'm lead straight to the answer it appears as if RAD Studio is lying to us. The configuration shows this:
but the dproj had this:
in it.
there were two conditions for cfg_3 and only the last one showed up in RAD Studio, well for some odd reason the build path was taken from the first one (even though it's specified in both). So, removing the wrong one (the first one) fixed the problem and things are now building to the correct folder.
I had imported the Utils option set when I was testing the library, but when I incorporated the program in to the main program, I removed it. Somehow it didn't find it's way completely out of the dproj and I guess (not sure why) but it seems like the other library got messed up because it shared a GUID.
I've just written the first version of a workflow activity that will run Resharper's Code Issues on the projects and parse the output to display the issues as build warnings and errors.
At first, I was going to just call Resharper's command line and parse the resulting xml manually. After fiddling with the dlls in Resharper's SDK (through disassembly mostly), I found a way to parse the results using it's own public classes, which I figured was a much more elegant and safe way to do this.
The first problem I have is that that nuget package is absolutely huge. There is 140mb of files in there, which to me is absurd for a single, unpartitioned package. There seems to be such heavy coupling between them that by using just a few model classes and the parser class, I have to drag a dozen or so of those dlls along, some of them which seemingly have nothing to do with the main dlls I need. This is not a show stopper though, I'm struggling with something else now:
In the end, I managed to track down the dependencies I needed to 41 assemblies (which is, again, insane, but alas). Initially, I tried removing everything and adding the missing references one by one, but this turned out to be unreliable, still missing some indirect references, even after compiling successfully. Then, I decided to code a small console application to find all referenced assemblies in the main Resharper assemblies I used, which gave me the 41 references I mentioned. This is the code I used to find every dependency.
Since these are custom activities we are talking about, I decided to create a unit test project to validate them. Using these 41 references only, everything works correctly.
When I added the activity to the build workflow though, and pointed the build controller to the source control folder containing the required assemblies, every time I schedule a build, the process fails stating that I need one extra dll from Resharper's SDK. For example, this is the first one it asks:
Could not load file or assembly 'AsyncBridge.Net35, PublicKeyToken=b3b1c0202c0d6a87' or one of its dependencies. The system cannot find the file specified. (type FileNotFoundException)
When I add this specific assembly to the TFS folder, I get another similar error for another dll, and this keeps going on and on.
What I wanted to know is how can I know exactly which assemblies a workflow XAML will need in order to run correctly? My custom activity dll has two specific CodeActivities and a XAML only activity that uses these two. This XAML acticity is what I'm directly using in the modified workflow template.
I see that besides the references in my project, the XAML activity also contains a TextExpression.ReferencesForImplementation section, with some assembly names. I've run my dependency finder program on those dependencies too, and the results are the same 41 assemblies already at the TFS folder.
Meanwhile I'll go with having the whole SDK into the custom assemblies folder, but I would really like to avoid this in the future since it has such an enormous amount of unneeded and big dlls in there.
First, we have request for our command line tool to support workflow activity and we decided to implement just plain MsBuild task which is universal and works in TFS too. Task and targets files are included in ReSharper CLT 8.2.
Second, if you still want to implement workflow activity it's pretty easy to do with new API in CLT, designed specially for custom processing of found issues - http://confluence.jetbrains.com/display/NETCOM/Custom+InspectCode+Issue+Logger.
And last, but not least, you do not need to put in VCS binaries of ReSharper SDK package.
Use NuGet's restore package functionality.
If you have any other questions I'll be glad to answer them.
A custom activity is being load and run by .NET CLR like any other .NET program. If the stack trace reports a missing file, then it's required by the CLR and you can't change this fact without refactoring your code.
Having an entire SDK references in the custom assembly folder doesn't make sense. I would prefer GAC deployment over huge binaries folder in the source control. Or maybe consider having these activities running an pre\post build scripts in MSBuild or PowerShell.
tldr; at bottom.
Ok, so once again an interesting problem and I'm looking for a fun and interesting solution.
My current project involves being very modular, meaning the program functionality will be easily changed based on different modules and the program would adapt.
So I started out with the typical route, which is using DLL plugins. Now this is just way to normal, I want to think outside the box a bit.
The modules included in my program are long running campaigns that may take weeks to finish, and there will be many running at a time. So stability is a big issue, so I thought about what Google Chrome does. Processes, not DLLs or threads.
So I have a framework going and I need a way to get some information about each module (which are now EXEs). Now for my DLL framework I was exporting a "Register" function that would fill in some information.
So I thought to myself, hey EXEs can export functions, let's see if that actually works...It doesn't. I did some research into how Windows handles theses things and I don't feel like hacking the PE headers on the fly (but it's the out of the box kind of thinking I'm going for).
I'm planning on using named pipes and CLI parameters to transfer data between the main program and the module exe's. I could use that in a register fashion, but I want to here other peoples thoughts.
tldr: I'm using EXE's instead of DLL's for plugins. Looking for a way to easily export one time information like a exported "Register" function would on a DLL. Thoughts?
You might still consider having the modules written as DLLs with defined entrypoints (e.g., the Register function). Then you write the executable that loads the specified DLL. Your main application would fire off the driver executable and give it a name of a plugin DLL.
That way it is still easy to define and export the set of APIs that must be provided yet still run it as a separate process. The one executable that you write can load the specified DLL and then handle the necessary IPC with the main app.
You could define a protocol via the stdin/stdout, named pipes, sockets, etc.
I have successfully used 'plain' COM for several projects, and objects inheriting from TAutoObject. The bonusses here are IDL; the interopability with .Net, VBA and other non-Delphi things; and the fact that implementors still can choose wether to supply a DLL, an exe, an NT-service, and optionally run hosted over the network (COM+/DCOM). There may be several considerations you should handle about multi-threading and locking, but I found all that I needed to know online.
You can, of course, not use symbols exported by a (running) exe since it is running in another boundary. But, you can load an exe as an image (as you would do with a library) using LoadLibrary(Ex) and then, use the functions exported by the exe. I have tested (just for fun) when debugging PeStudio. See the snapshot below of chrome.exe loaded in the process space of PeStudio.exe using LoadLibrary.
I'm using TJvPluginManager in the JVCL to create and load BPL-based plugins for my program. Problem is, one of the plugins isn't loading properly, and I can't debug it. Every time I try to trace into the loading sequence, it gets as far as the LoadLibrary API call, and then the debugger seems to forget what it's there for. It completely loses the ability to associate program code with source lines, give meaningful data in a call stack, or display local variables. It will still stop at breakpoints, but it breaks to the CPU window, with all the inline source code stripped out.
This happens on Delphi 2007 and 2009, and it's driving me nuts. Does anyone know how to load a plugin without it breaking the debugger? Does anyone even know why it's breaking it in the first place?
NOTE: I'm not looking for alternative methods of debugging. I know all about tracing and logging and all the rest. What I want is to understand what's going wrong and how to fix it. Surely I'm not the only person who's ever used TJvPluginManager?
Not quite the answer to your question: Have you tried to debug the package project, by setting the host application and putting a breakpoint into the package's startup code?
I've found Ray Kanopka's (Raize) CodeSite to be invaluable for debugging in situations where the integrated debugger is acting up. Thinking about the things I want to monitor using CodeSite actually helps me focus on what's important - it enforces good habits.
Another alternative to Codesite is Overseer which is part of the nexus project, but stands alone so does not require you to use their framework. Codesite is by far the better option, but in a pinch Overseer would work just as well.
I found that using packages for plugins can be problematic and many years ago switched to a completely COM based implementation for plugins and never had any problems. The other advantage to COM based plugins, they don't require Delphi to write, do not need to be recompiled when the main app switches to a new version of the compiler (my plugins compiled with Delphi 5 still run fine against the main application compiled in Delphi 2009!) and they are easier to write test applications to assist in debugging.
The only side effect I notice, is that shared code ends up in both executables and the plugins require registration into the registry.
Hmmmm... This is a stupid question, but I have to ask: the initialization function have the EXACT declaration syntax like the other plugins that work ?(from your question, I deducted you made some others that work)
Check your dependencies. Make sure each unit is compiled into one package only. Whenever a package needs to reference a unit from another package, use the requires clause to do so. Watch for compiler warnings about implicitly linked units.
Our team had been using Delphi 6 for many years, then switched to Delphi 2006 years ago. With both versions we have the following problem: frequently the compiler complains about a unit which is supposedly used recursively. This unit is a 40k LOC unit which is at the core of a project with almost 1 million LOC (third party included).
The error message is incorrect: a full build on the project always works. Unfortunately, the error message does not tell us where the supposed circular reference is, just the name of that unit. Sometimes it even happens that valid error messages are listed 2-4 times until that circular reference problem is "found". Clearly the compiler is running in a circle here. Because of the size of that project it is hard to find the problem manually. Therefore I made a tool which proves that there really is no circular reference (the tool creates a directed dependency graph of the units and determines the coherence components in that graph - there are none except if I deliberately put some in).
This is not only affecting F9 compilation but also code completion / insight which is not working most of the time. Sometimes it works when I press ctrl-space a second time...
Any ideas how we can isolate or even fix the problem? Note that it will be very hard to split the 40k LOC unit into smaller ones because it contains about 15 large classes which depend on each other in the interface section (I know it's bad but should work anyway).
Update
We are constantly refactoring but this is one tough unit to refactor because everything depends on everything, almost. Have been trying to get around it via interfaces but we are talking about some classes with 100s of methods and properties. And it would be slower.
Upgrading to D2009 may be an option down the road but right now we're stuck with D2006 (the unicode stuff and the price tag are two of the stoppers here). Question is anyway if it would help since the problem is in there since D6 at least.
About trimming the uses clauses, we have been doing this frequently with Icarus. But that did not help so far. We are down to 90 custom units in the interface section now. However, with a true circular reference the problem could be in any unit. Also tried to add all units to the dpr.
The project shares a lot of code with other projects, and there are some IFDEFs. However, the defines are not set up in project options but via a common include file. Therefore all modules should see the same defines. Also, the problem reoccurs shortly after a full rebuild without switching to another project.
I will probably be downvoted for this. In D2005 I had a 10k loc unit (datamodule) that flat out stopped compiling. Had to separate out some datasets/code to another datamodule. That 10k unit was and is a mess. You really should consider refactoring out some code to other units. My module has since D2005 / separation grown even worse, but it still compiles in D2007. So my answer is a) refactor and b) upgrade to D2009.
It seems clear that this is due to a slight difference between the background compiler and the real thing. You could look around (QualityCentral) what's known on that topic.
Also, since you didn't explicitly state this, you should remove unnecessary units and move uses statements down to implementation if possible. Maybe your tool could help with this.
And just to be sure you should check the unit aliases and Path settings.
You write that a full build does always succeed, but shortly after the incremental build fails with this error. Assuming that you experience this in the IDE, have you tried to use the command line compiler dcc32 to do incremental builds?
If you don't feed it the "-Q" switch (which probably most Makefiles or scripts for command line builds do) it will output a lot of information what files it compiles in what order. You could either try to do an incremental build after the error appeared in the IDE, or you could keep a command line open next to the IDE and Alt+Tab to it for compilation, skipping compilation in the IDE completely.
I simply assume you have a way to build using dcc32, one way or another - with the size of your project I can't imagine otherwise.
We regularly fall in similar problems, and we never managed (or bothered long enough) to find the precise cause. There seems to be a problem in the order which Delphi chooses to compile the units when hitting Ctrl-F9, which is incompatible with the actual dependency order of the units.
Have you tried deleting "MyBigFatUnit.dcu" before hitting Ctrl-F9?
Have you tried to re-order the declaration of your units in your dpr/dpk files, so that units appear in a correct compilation order? (i.e.: if unit B depends on unit A, unit A should appear first in the dpr/dpk)
Do you have any other projects that use part of the same codebase? If you compile one of them under different compiler settings or IFDEFs, it might change certain things in some of the DCUs which would lead to a circular dependency. A full build rebuilds all DCUs and then the problem goes away.
Try Icarus (free) from Peganza. If that does not tell you what the problem is, try their Pascal Analyzer.
We have this problem as well, also with a fairly large codebase.
We are currently using D2009, but have had this problem with all previous versions of Delphi.
It most frequently happens immediately after doing an update from source control, so I suspect there is some timestamp issue within the Delphi build process.
In our case, if Ctrl-F9 fails and reports the circular reference, a second Ctrl-F9 will generally work
A way I have been told to deal with this is to open another arbitrary file in the project, change that file, save it, and then try running the incremental compile again. Surprisingly enough, this usually works.
We have a 4 MLOC project where this comes up from time to time and this "solution" works for me.
I've fought this before, in my experience the error is quasi-legitimate. It's been a quite a while (the error has nothing to do with the version) but my memory of the situation is that it involves a loop in which part of the loop is in the implementation.
Unit A uses B in the implementation. Unit B uses A in the interface. If you compile B first it calls for A but since the call for B is in the implementation it succeeds. If you compile A first it calls for B, B turns around and calls for A in the interface, boom. Such loops are only safe if both cross references are in the implementation.
The solution is to design things so there is a minimum of stuff used in the interface and to make certain there's nothing resembling a loop in those units. So long as you keep your type definitions separate from units with code this is pretty easy to do.
The error coming and going depending on what you are doing is a hallmark of this issue as it comes down to how you enter the loop. When you do a full build the order is consistent and you either get it 100% or 0%, it's not random.