This question is a follow on after such a great answer Is there a way to upload jars for a dataflow job so we don't have to serialize everything?
This made me realize 'ok, what I want is injection with no serialization so that I can mock and test'.
Our current method requires our apis/mocks to be serialiable BUT THEN, I have to put static fields in the mock because it gets serialized and deserialized creating a new instance that dataflow uses.
My colleague pointed out that perhaps this needs to be a sink and that is treated differently? <- We may try that later and update but we are not sure right now.
My desire is from the top to replace the apis with mocks during testing. Does someone have an example for this?
Here is our bootstrap code that does not know if it is in production or inside a feature test. We test end to end results with no apache beam imports in our tests meaning we swap to any tech if we want to pivot and keep all our tests. Not only that, we catch way more integration bugs and can refactor without rewriting tests since the contracts we test are customer ones we can't easily change.
public class App {
private Pipeline pipeline;
private RosterFileTransform transform;
#Inject
public App(Pipeline pipeline, RosterFileTransform transform) {
this.pipeline = pipeline;
this.transform = transform;
}
public void start() {
pipeline.apply(transform);
pipeline.run();
}
}
Notice that everything we do is Guice Injection based so the Pipeline may be direct runner or not. I may need to modify this class to pass things through :( but anything that works for now would be great.
The function I am trying to get our api(and mock and impl to) with no serialization is thus
private class ValidRecordPublisher extends DoFn<Validated<PractitionerDataRecord>, String> {
#ProcessElement
public void processElement(#Element Validated<PractitionerDataRecord>element) {
microServiceApi.writeRecord(element.getValue);
}
}
I am not sure how to pass in microServiceApi in a way that avoid serialization. I would be ok with delayed creation as well after deserialization using guice Provider provider; with provider.get() if there is a solution there too.
Solved in such a way that mocks no longer need static or serialization anymore by one since glass bridging the world of dataflow(in prod and in test) like so
NOTE: There is additional magic-ness we have in our company that passes through headers from service to service and through dataflow and that is some of it in there which you can ignore(ie. the RouterRequest request = Current.request();). so for anyone else, they will have to pass in projectId into getInstance each time.
public abstract class DataflowClientFactory implements Serializable {
private static final Logger log = LoggerFactory.getLogger(DataflowClientFactory.class);
public static final String PROJECT_KEY = "projectKey";
private transient static Injector injector;
private transient static Module overrides;
private static int counter = 0;
public DataflowClientFactory() {
counter++;
log.info("creating again(usually due to deserialization). counter="+counter);
}
public static void injectOverrides(Module dfOverrides) {
overrides = dfOverrides;
}
private synchronized void initialize(String project) {
if(injector != null)
return;
/********************************************
* The hardest part is this piece since this is specific to each Dataflow
* so each project subclasses DataflowClientFactory
* This solution is the best ONLY in the fact of time crunch and it works
* decently for end to end testing without developers needing fancy
* wrappers around mocks anymore.
***/
Module module = loadProjectModule();
Module modules = Modules.combine(module, new OrderlyDataflowModule(project));
if(overrides != null) {
modules = Modules.override(modules).with(overrides);
}
injector = Guice.createInjector(modules);
}
protected abstract Module loadProjectModule();
public <T> T getInstance(Class<T> clazz) {
if(!Current.isContextSet()) {
throw new IllegalStateException("Someone on the stack is extending DoFn instead of OrderlyDoFn so you need to fix that first");
}
RouterRequest request = Current.request();
String project = (String)request.requestState.get(PROJECT_KEY);
initialize(project);
return injector.getInstance(clazz);
}
}
I suppose this may not be what you're looking for, but your use case makes me think of using factory objects. They may depend on the pipeline options that you pass (i.e. your PipelineOptions object), or on some other configuration object.
Perhaps something like this:
class MicroserviceApiClientFactory implements Serializable {
MicroserviceApiClientFactory(PipelineOptions options) {
this.options = options;
}
public static MicroserviceApiClient getClient() {
MyPipelineOptions specialOpts = options.as(MySpecialOptions.class);
if (specialOpts.getMockMicroserviceApi()) {
return new MockedMicroserviceApiClient(...); // Or whatever
} else {
return new MicroserviceApiClient(specialOpts.getMicroserviceEndpoint()); // Or whatever parameters it needs
}
}
}
And for your DoFns and any other execution-time objects that need it, you would pass the factory:
private class ValidRecordPublisher extends DoFn<Validated<PractitionerDataRecord>, String> {
ValidRecordPublisher(MicroserviceApiClientFactory msFactory) {
this.msFactory = msFactory;
}
#ProcessElement
public void processElement(#Element Validated<PractitionerDataRecord>element) {
if (microServiceapi == null) microServiceApi = msFactory.getClient();
microServiceApi.writeRecord(element.getValue);
}
}
This should allow you to encapsulate the mocking functionality into a single class that lazily creates your mock or your client at pipeline execution time.
Let me know if this matches what you want somewhat, or if we should try to iterate further.
I have no experience with Guice, so I don't know if Guice configurations can easily pass the boundary between pipeline construction and pipeline execution (serialization / submittin JARs / etc).
Should this be a sink? Maybe, if you have an external service, and you're writing to it, you can write a PTransform that takes care of it - but the question of how you inject various dependencies will remain.
Related
I'm relatively new to Guice, and some things still give me a pretty hard time.
My particular question is, how do you handle nested injections in Guice.
Example:
Class A uses Class B via #Inject, and Class B uses Class C.
Explicitly:
My Module where I bind to Providers.
public class ModuleBinder extends AbstractModule {
#Override
protected void configure() {
bind(DatabaseControllerInterface.class)
.toProvider(DatabaseControllerProvider.class).asEagerSingleton();
bind(AnalyzerInterface.class)
.toProvider(AnalyzerProvider.class).asEagerSingleton();
bind(SystemAdministrationInterface.class)
.toProvider(SystemAdministrationProvider.class).asEagerSingleton();
bind(LogInServiceInterface.class)
.toProvider(LogInServiceProvider.class);
}
}
The DatabaseControllerProvider:
public class DatabaseControllerProvider implements Provider<DatabaseControllerInterface> {
#Override
public DatabaseControllerInterface get() {
return new DatabaseControllerImpl();
}
}
The LogInServiceProvider:
public class LogInServiceProvider implements Provider<LogInServiceInterface> {
#Override
public LogInServiceInterface get() {
return new LogInServiceImpl();
}
}
And finally, the LogInService uses:
public class LogInServiceImpl implements LogInServiceInterface{
#Inject
private DatabaseControllerProvider databaseControllerProvider;
private final DatabaseControllerInterface databaseController;
public LogInServiceImpl() {
this.databaseController = databaseControllerProvider.get();
}
#Override
public User register(final String mail, final String userName, final String password) {
databaseController.registerUser(userName, mail, password, UserRole.ADMIN);
}
}
The call is then:
public class Test() {
public static test() {
final Injector injector = Guice.createInjector(new ModuleBinder());
logInService = injector.getInstance(LogInServiceInterface.class);
logInService.registerUser("test", "test", "test");
}
}
I know most of you guys will get sick with that code, but hey, I'm a beginner with Guice, so please be gentle with me.
I want to use Constructor injection, I already realized that field injection is considered "evil". Do you have any idea how to get that working by keeping the providers (I need them)?
Using the injections in the example does nothing on the "second" level, the DatabaseControllerImpl in LogInServiceImpl is null.
Did I configure something wrong? Did I misunderstand the usage of Provides and/or Modules?
I hope somebody can and wants to help me. If you need more informations, post a comment.
With best regards,
JosefRucksack
Your direct answer: You're calling new T(); in your Providers, which doesn't support field injection.
First, a real timesaver: Don't keep your explicit Providers. If you have bound a T, Guice allows you to inject a Provider or call Injector.getProvider for that T, even if you haven't explicitly created a Provider yourself. See the Built-In Bindings page on the wiki,
or the Injector docs (emphasis mine):
Contains several default bindings:
This Injector instance itself
A Provider<T> for each binding of type T
The Logger for the class being injected
The Stage in which the Injector was created
Instead, do it this way:
public class ModuleBinder extends AbstractModule {
#Override
protected void configure() {
bind(DatabaseControllerInterface.class)
.to(DatabaseControllerImpl.class).asEagerSingleton();
bind(AnalyzerInterface.class)
.to(AnalyzerImpl.class).asEagerSingleton();
bind(SystemAdministrationInterface.class)
.to(SystemAdministrationImpl.class).asEagerSingleton();
bind(LogInServiceInterface.class)
.to(LogInServiceImpl.class);
}
}
You then have the same choice you do now, to inject T or Provider<T> and call getInstance or getProvider as needed.
If your Providers are absolutely necessary, especially if they actually receive an instance from some other system or service locator, one other option is to add your #Inject fields into them as in the Provider bindings wiki page and pass them into your constructor, or to just inject a MembersInjector<T>:
public class LogInServiceProvider implements Provider<LogInServiceInterface> {
#Inject MembersInjector<LogInServiceImpl> logInServiceImplInjector;
#Override
public LogInServiceInterface get() {
LogInServiceImpl logInServiceImpl = YourExternalDep.getLogInService();
logInServiceImplInjector.injectMembers(logInServiceImpl);
return logInServiceImpl;
}
}
However, this explicit-Provider solution is not idiomatic Guice, and should only be used with external or legacy code. Guice's whole reason for existence is to automate away boilerplate and let your systems come together clearly and flexibly. Providers are an implementation detail; let Guice create them for you.
My question is really a follow up question to
RabbitMQ Integration Test and Threading
There it states to wrap "your listeners" and pass in a CountDownLatch and eventually all the threads will merge. This answer works if we were manually creating and injecting the message listener but for #RabbitListener annotations... i'm not sure how to pass in a CountDownLatch. The framework is auto magically creating the message listener behind the scenes.
Are there any other approaches?
With the help of #Gary Russell I was able to get an answer and used the following solution.
Conclusion: I must admit i'm indifferent about this solution (feels like a hack) but this is the only thing I could get to work and once you get over the initial one time setup and actually understand the 'work flow' it is not so painful. Basically comes down to defining ( 2 ) #Beans and adding them to your Integration Test config.
Example solution posted below with explanations. Please feel free to suggest improvements to this solution.
1. Define a ProxyListenerBPP that during spring initialization will listen for a specified clazz (i.e our test class that contains #RabbitListener) and
inject our custom CountDownLatchListenerInterceptor advice defined in the next step.
import org.aopalliance.aop.Advice;
import org.springframework.aop.framework.ProxyFactoryBean;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.BeanFactoryAware;
import org.springframework.beans.factory.config.BeanPostProcessor;
import org.springframework.core.Ordered;
import org.springframework.core.PriorityOrdered;
/**
* Implements BeanPostProcessor bean... during spring initialization we will
* listen for a specified clazz
* (i.e our #RabbitListener annotated class) and
* inject our custom CountDownLatchListenerInterceptor advice
* #author sjacobs
*
*/
public class ProxyListenerBPP implements BeanPostProcessor, BeanFactoryAware, Ordered, PriorityOrdered{
private BeanFactory beanFactory;
private Class<?> clazz;
public static final String ADVICE_BEAN_NAME = "wasCalled";
public ProxyListenerBPP(Class<?> clazz) {
this.clazz = clazz;
}
#Override
public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
this.beanFactory = beanFactory;
}
#Override
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
if (clazz.isAssignableFrom(bean.getClass())) {
ProxyFactoryBean pfb = new ProxyFactoryBean();
pfb.setProxyTargetClass(true); // CGLIB, false for JDK proxy (interface needed)
pfb.setTarget(bean);
pfb.addAdvice(this.beanFactory.getBean(ADVICE_BEAN_NAME, Advice.class));
return pfb.getObject();
}
else {
return bean;
}
}
#Override
public int getOrder() {
return Ordered.LOWEST_PRECEDENCE - 1000; // Just before #RabbitListener post processor
}
2. Create the MethodInterceptor advice impl that will hold the reference to the CountDownLatch. The CountDownLatch needs to be referenced in both in the Integration test thread and inside the async worker thread in the #RabbitListener. So we can later release back to the Integration Test thread as soon as the #RabbitListener async thread has completed execution. No need for polling.
import java.util.concurrent.CountDownLatch;
import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
/**
* AOP MethodInterceptor that maps a <b>Single</b> CountDownLatch to one method and invokes
* CountDownLatch.countDown() after the method has completed execution. The motivation behind this
* is for integration testing purposes of Spring RabbitMq Async Worker threads to be able to merge
* the Integration Test thread after an Async 'worker' thread completed its task.
* #author sjacobs
*
*/
public class CountDownLatchListenerInterceptor implements MethodInterceptor {
private CountDownLatch countDownLatch = new CountDownLatch(1);
private final String methodNameToInvokeCDL ;
public CountDownLatchListenerInterceptor(String methodName) {
this.methodNameToInvokeCDL = methodName;
}
#Override
public Object invoke(MethodInvocation invocation) throws Throwable {
String methodName = invocation.getMethod().getName();
if (this.methodNameToInvokeCDL.equals(methodName) ) {
//invoke async work
Object result = invocation.proceed();
//returns us back to the 'awaiting' thread inside the integration test
this.countDownLatch.countDown();
//"reset" CountDownLatch for next #Test (if testing for more async worker)
this.countDownLatch = new CountDownLatch(1);
return result;
} else
return invocation.proceed();
}
public CountDownLatch getCountDownLatch() {
return countDownLatch;
}
}
3. Next add to your Integration Test Config the following #Bean(s)
public class SomeClassThatHasRabbitListenerAnnotationsITConfig extends BaseIntegrationTestConfig {
// pass into the constructor the test Clazz that contains the #RabbitListener annotation into the constructor
#Bean
public static ProxyListenerBPP listenerProxier() { // note static
return new ProxyListenerBPP(SomeClassThatHasRabbitListenerAnnotations.class);
}
// pass the method name that will be invoked by the async thread in SomeClassThatHasRabbitListenerAnnotations.Class
// I.E the method name annotated with #RabbitListener or #RabbitHandler
// in our example 'listen' is the method name inside SomeClassThatHasRabbitListenerAnnotations.Class
#Bean(name=ProxyListenerBPP.ADVICE_BEAN_NAME)
public static Advice wasCalled() {
String methodName = "listen";
return new CountDownLatchListenerInterceptor( methodName );
}
// this is the #RabbitListener bean we are testing
#Bean
public SomeClassThatHasRabbitListenerAnnotations rabbitListener() {
return new SomeClassThatHasRabbitListenerAnnotations();
}
}
4. Finally, in the integration #Test call... after sending a message via rabbitTemplate to trigger the async thread... now call the CountDownLatch#await(...) method obtained from the interceptor and make sure to pass in a TimeUnit args so it can timeout in case of long running process or something goes wrong. Once the async the Integration Test thread is notified (awakened) and now we can finally begin to actually test/validate/verify the results of the async work.
#ContextConfiguration(classes={ SomeClassThatHasRabbitListenerAnnotationsITConfig.class } )
public class SomeClassThatHasRabbitListenerAnnotationsIT extends BaseIntegrationTest{
#Inject
private CountDownLatchListenerInterceptor interceptor;
#Inject
private RabbitTemplate rabbitTemplate;
#Test
public void shouldReturnBackAfterAsyncThreadIsFinished() throws Exception {
MyObject payload = new MyObject();
rabbitTemplate.convertAndSend("some.defined.work.queue", payload);
CountDownLatch cdl = interceptor.getCountDownLatch();
// wait for async thread to finish
cdl.await(10, TimeUnit.SECONDS); // IMPORTANT: set timeout args.
//Begin the actual testing of the results of the async work
// check the database?
// download a msg from another queue?
// verify email was sent...
// etc...
}
It's a bit more tricky with #RabbitListener but the simplest way is to advise the listener.
With the custom listener container factory just have your test case add the advice to the factory.
The advice would be a MethodInterceptor; the invocation will have 2 arguments; the channel and the (unconverted) Message. The advice has to be injected before the container(s) are created.
Alternatively, get a reference to the container using the registry and add the advice later (but you'll have to call initialize() to force the new advice to be applied).
An alternative would be a simple BeanPostProcessor to proxy your listener class before it is injected into the container. That way, you will see the method argumen(s) after any conversion; you will also be able to verify any result returned by the listener (for request/reply scenarios).
If you are not familiar with these techniques, I can try to find some time to spin up a quick example for you.
EDIT
I issued a pull request to add an example to EnableRabbitIntegrationTests. This adds a listener bean with 2 annotated listener methods, a BeanPostProcessor that proxies the listener bean before it is injected into a listener container. An Advice is added to the proxy which counts latches down when the expected messages are received.
This line in TopLevelTransaction (neo4j-kernel-2.1.2) throws a NullPointerException every time I call next() on an iterator obtained via GraphRepository#findAll():
protected void markAsRollbackOnly()
{
try
{
transactionManager.getTransaction().setRollbackOnly(); // NPE here
}
catch ( Exception e )
{
throw new TransactionFailureException(
"Failed to mark transaction as rollback only.", e );
}
}
I found some threads about similar crashes with slightly different stack traces. The accepted solution on this question is to use "proxy" transaction management, but that seems like a band-aid solution. This question also mentions "proxy" transaction management and suggests that there might be something wrong with the #Transactional annotation when using AspectJ.
Is this legitimately a bug, or have I just set up my project incorrectly? My code is essentially the same as in my standalone hello world, with a slightly more complex main class:
#Component
public class Test2 {
#Autowired
FooRepository repo;
public static void main(String[] args) {
AbstractApplicationContext context = new AnnotationConfigApplicationContext("test2");
Test2 test2 = context.getBean(Test2.class);
test2.doStuff();
}
public void doStuff() {
createFoo();
printFoos();
}
#Transactional
public Foo createFoo() {
Foo foo = new Foo();
foo.setName("Derp" + System.currentTimeMillis());
repo.save(foo);
System.out.println("saved " + foo.toString());
return foo;
}
#Transactional
public void printFoos() {
Iterable<Foo> foos = repo.findAll();
System.out.println("findAll() returned instance of " + foos.getClass().getName());
Iterator<Foo> iter = foos.iterator();
System.out.println("iterator is instance of " + iter.getClass().getName());
if(iter.hasNext()) {
iter.next(); // CRASHES HERE
}
}
}
I can post my POM if needed.
I didn't find a bug. Two or three things are required to make this work, depending on whether you want to use proxy or AspectJ transaction management.
First, transaction management must be enabled. Since I'm using annotation-based configuration, I did this by annotating my #Configuration class with #EnableTransactionManagement. Contrary to the docs, the default mode now seems to be AdviceMode.ASPECTJ, not AdviceMode.PROXY.
Next, you need to ensure that the Iterator is used within a transaction. In my example, if I use AdviceMode.PROXY the entire bean containing the #Autowired repository has to be annotated #Transactional. If I use AdviceMode.ASPECTJ I can annotate just the method. This is because the call to the method using the iterator is a self-call from within the bean, and proxy transaction management cannot intercept and manage internal calls.
Finally, if you're using AdviceMode.ASPECTJ you must set up weaving as discussed here.
I would like to have a static instance method with Guice for one of the components (non-managed bean should be able to access this class). I created something like this:
public class LookupService {
#Inject
private static Provider<Injector> injector = null;
private final ILookup<IWS> lookup;
#Inject
public LookupService(ILookup<IWS> lookup) {
this.lookup = lookup;
}
public static LookupService instance() {
return injector.get().getInstance(LookupService.class);
}
public <T extends IWS> T lookup(Class<T> localInterface) {
return lookup.lookup(localInterface);
}
}
What do you think about this design ? Any other ideas on this ? (accessing managed beans from non-managed objects)
Basically, the pattern you're looking for is called "requesting static injection" and there's a Binder method dedicated to it. Once you have that down, your code looks a lot like this example from the Guice docs.
public class MainModule extends AbstractModule {
#Override public void configure() {
requestStaticInjection(LookupService.class);
}
}
public class LookupService {
/** This will be set as soon as the injector is created. */
#Inject
static Provider<LookupService> provider = null;
private final ILookup<IWS> lookup;
#Inject
public LookupService(ILookup<IWS> lookup) {
this.lookup = lookup;
}
public static LookupService instance() {
return provider.get();
}
public <T extends IWS> T lookup(Class<T> localInterface) {
return lookup.lookup(localInterface);
}
}
A few notes:
While you can still set your field to be private, remember that this means you cannot set it in tests (or in future non-Guice usage) without Guice's private-field-access magic. When using injected fields, we often make them package-private and then put the tests in the same package.
Static injection is generally seen as something to endorse only when migrating to Guice, or when you use other code you can't change. When possible, try to avoid global state--even if this means making FooBean data-only and creating an injected FooBeanService.
Even though you can inject an Injector wherever you'd like, you might find it easier to test if you simply inject a Provider<LookupService> instead. Only inject an Injector if you don't know what type you're going to need until runtime--for example, if you implement LookupService.lookup(...) using an Injector by passing the class literal to the injector to get an instance.
In fact, it's hard to say from here, but ILookup seems to act a lot like the Service Locator pattern, which solves the exact type of problem that Guice solves with dependency injection! If that's the case, you might as well rewrite ILookup to use Guice: Just remove calls to LookupService.instance().lookup(Foo.class) and instead create a matching pair of #Inject static Provider<Foo> fooProvider and requestStaticInjection(FooUser.class).
Hope that helps!
I'd like to vary the injected implementations based on something that's not known until runtime. Specifically, I'd like my app to operate as different versions where the "version" is not determined until a request is executing. Also, the "version" could vary per request.
After reading the docs it seems that I could implement a providers in cases where I need to choose an implementation at runtime based on the "version". Additionally, I could roll my own on top of juice.
Is implementing a provider the best way to go in this scenario? I'd like to know if there is a best practice or if anyone else out there has tried to use Guice to tackle this problem.
Thanks for any help!
-Joe
I think that if the version can be known only at runtime, you must provide the versioned "services" manually with custom provider. Possibly something like this:
#Singleton
public abstract class VersionedProvider<T, V> {
private Map<V, T> objects;
T get(V version) {
if (!objects.containsKey(version)) {
objects.put(version, generateVersioned(version));
}
return objects.get(version);
}
// Here everything must be done manually or use some injected
// implementations
public abstract T generateVersioned(V version);
}
public class MyRuntimeServiceModule extends AbstractModule {
private final String runTimeOption;
public ServiceModule(String runTimeOption) {
this.runTimeOption = runTimeOption;
}
#Override protected void configure() {
Class<? extends Service> serviceType = option.equals("aServiceType") ?
AServiceImplementation.class : AnotherServiceImplementation.class;
bind(Service.class).to(serviceType);
}
}
public static void main(String[] args) {
String option = args[0];
Injector injector = Guice.createInjector(new MyRuntimeServiceModule(option));
}