Complicated dynamic and static linking with duplicates (Firefox Add-on, WebRTC and VP8) - firefox-addon

Question

I've tried to formulate the problem in an abstract way, but anyway I give details about the actual libraries in the end.
Dynamic library Addon is statically linked against other library WebRTC which has some code in assembly and this code is linked into WebRTC as object files together with WebRTC's own object files. Lets call this assembly code VP8. Functions of VP8 are marked extern inside WebRTC. Some function Encode() from Addon calls functions of WebRTC which eventually calls functions from VP8.
Now, the application Firefox which is going to load library Addon is quite complex and has its own version (means statically linked) of library WebRTC(let's call it WebRTC2), but an older one.
So, here is a problem: if a call of Encode() is made from the application Firefox, WebRTC functions get called (not WebRTC2, which is correct) BUT when WebRTC tries to call VP8 functions, they get called from the WebRTC2 version (means application's version of WebRTC), but not from WebRTC.
Is there are way to force WebRTC make calls only from local copy of VP8?
Application Firefox is a Firefox browser, WebRTC is a WebRTC library, VP8 is a VP8 codec library (inside WebRTC) and Addon is my Firefox C++ add-on.
UPDATE - DETAILED DESCRIPTION
Here is "unabstract" description of the problem:
So there is a C++ XPCOM add-on which is statically linked against latest version of WebRTC library.
At some point inside add-on a call for encoding a frame is made (method Encode of VP8Encoder class) and it crashes in Firefox all the time, while continue to work well on test programs using gtest framework.
The problem is that at some point inside WebRTC there is VP8 assembly code which is get called for encoding and functions of this assembly code are declared as extern in implementation files. Actually, it crashes on vp8_intra_pred_y_ve_sse2 function.
I've compared three assembly codes of this function: one is from my version of WebRTC (used in add-on), second - where debugger crashed and the third one - from source code of Mozilla's WebRTC.
It turned out that for some weird reason, Mozilla's code get called instead of add-on's WebRTC (they both have same names of course) and as Mozilla's WebRTC code is outdated, it crashes with EXC_BAD_ACCESS.

Answer 1

This probably won't be much help to you, but since no one else has responded, here I go ...
You didn't mention if you're running on Linux, Windows, or something else. My answer is for Linux. I believe Windows loads dynamic libraries differently.
I think what's happening is that you've statically linked against the WebRTC library stub interfaces and those stub interfaces are dynamically linking to the actual implementation, and then getting the first instance of the WebRTC library loaded into firefox instead of the second. Be sure that you're really linking against a statically compiled version of the WebRTC library.
The linux dlopen(3) man page has an interesting flag listed that seems like it would help if your library was the one loading the WebRTC library:
void *dlopen(const char *filename, int flag);
dlopen()
The function dlopen() loads the dynamic library file named by the null-
terminated string filename and returns an opaque "handle" for the
dynamic library. [...]
RTLD_DEEPBIND (since glibc 2.3.4)
Place the lookup scope of the symbols in this library ahead of
the global scope. This means that a self-contained library will
use its own symbols in preference to global symbols with the
same name contained in libraries that have already been loaded.
This flag is not specified in POSIX.1-2001.
Unfortunately, firefox is the one loading your library.
If your library was dynamically linked against WebRTC (which it seems to be) and you explicitly loaded the WebRTC library you want using dl_open() with the RTLD_DEEPBIND flag, than that would solve your problem.
This won't help you much since its not your code that's binding to vp8_intra_pred_y_ve_sse2, but its worth pointing out that gcc also has the dlsym() function which can take a couple of special flags:
void *dlsym(void *handle, const char *symbol);
dlsym()
The function dlsym() takes a "handle" of a dynamic library returned by
dlopen() and the null-terminated symbol name, returning the address
where that symbol is loaded into memory. [...]
There are two special pseudo-handles, RTLD_DEFAULT and RTLD_NEXT. The
former will find the first occurrence of the desired symbol using the
default library search order. The latter will find the next occurrence
of a function in the search order after the current library. This
allows one to provide a wrapper around a function in another shared
library.
You could use that in your code to at least debug what's going on by having it print out the values it gets for a lookup of vp8_intra_pred_y_ve_sse2.
Finally, the same man page notes that linux has a dlvsm() that takes a version string argument as well, allowing the XPCOM code to specify which version of the function it wants:
#define _GNU_SOURCE /* See feature_test_macros(7) */
#include <dlfcn.h>
void *dlvsym(void *handle, char *symbol, char *version);
If it were me, and I was forced to dynamically link the stuff, I'd go with the brute force approach. Go into the libraries and change the function names (in both libraries). Its not elegant, and it will be a headache whenever a new version of either library comes out, but its simple and direct.