Dependency Injection in Apache Storm topology - dependency-injection

Little background: I am working on a topology using Apache Storm, I thought why not use dependency injection in it, but I was not sure how it will behave on cluster environment when topology deployed to cluster. I started looking for answers on if DI is good option to use in Storm topologies, I came across some threads about Apache Spark where it was mentioned serialization is going to be problem and saw some responses for apache storm along the same lines. So finally I decided to write a sample topology with google guice to see what happens.
I wrote a sample topology with two bolts, and used google guice to injects dependencies. First bolt emits a tick tuple, then first bolt creates message, bolt prints the message on log and call some classes which does the same. Then this message is emitted to second bolt and same printing logic there as well.
First Bolt
public class FirstBolt extends BaseRichBolt {
private OutputCollector collector;
private static int count = 0;
private FirstInjectClass firstInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
firstInjectClass = injector.getInstance(FirstInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
count++;
String message = "Message count "+count;
firstInjectClass.printMessage(message);
log.error(message);
collector.emit("TO_SECOND_BOLT", new Values(message));
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
outputFieldsDeclarer.declareStream("TO_SECOND_BOLT", new Fields("MESSAGE"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
Config conf = new Config();
conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 10);
return conf;
}
}
Second Bolt
public class SecondBolt extends BaseRichBolt {
private OutputCollector collector;
private SecondInjectClass secondInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
secondInjectClass = injector.getInstance(SecondInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
String message = (String) tuple.getValue(0);
secondInjectClass.printMessage(message);
log.error("SecondBolt {}",message);
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
}
}
Class in which dependencies are injected
public class FirstInjectClass {
FirstInterface firstInterface;
private final String prepend = "FirstInjectClass";
#Inject
public FirstInjectClass(FirstInterface firstInterface) {
this.firstInterface = firstInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
firstInterface.printMethod(message);
}
}
Interface used for binding
public interface FirstInterface {
void printMethod(String message);
}
Implementation of interface
public class FirstInterfaceImpl implements FirstInterface{
private final String prepend = "FirstInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Same way another class that receives dependency via DI
public class SecondInjectClass {
SecondInterface secondInterface;
private final String prepend = "SecondInjectClass";
#Inject
public SecondInjectClass(SecondInterface secondInterface) {
this.secondInterface = secondInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
secondInterface.printMethod(message);
}
}
another interface for binding
public interface SecondInterface {
void printMethod(String message);
}
implementation of second interface
public class SecondInterfaceImpl implements SecondInterface{
private final String prepend = "SecondInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Module Class
public class Module extends AbstractModule {
#Override
protected void configure() {
bind(FirstInterface.class).to(FirstInterfaceImpl.class);
bind(SecondInterface.class).to(SecondInterfaceImpl.class);
}
}
Nothing fancy here, just two bolts and couple of classes for DI. I deployed it on server and it works just fine. The catch/problem though is that I have to initialize Injector in each bolt which makes me question what is side effect of it going to be?
This implementation is simple, just 2 bolts.. what if I have more bolts? what impact it would create on topology if I have to initialize Injector in all bolts?
If I try to initialize Injector outside prepare method I get error for serialization.

Related

How to configure Micronaut and Micrometer to write ILP directly to InfluxDB?

I have a Micronaut application that uses Micrometer to report metrics to InfluxDB with the micronaut-micrometer project. Currently it is using the Statsd Registry provided via the io.micronaut.configuration:micronaut-micrometer-registry-statsd dependency.
I would like to instead output metrics in Influx Line Protocol (ILP), but the micronaut-micrometer project does not offer an Influx Registry currently. I tried to work around this by importing the io.micrometer:micrometer-registry-influx dependency and configuring an InfluxMeterRegistry manually like this:
#Factory
public class MyMetricRegistryConfigurer implements MeterRegistryConfigurer {
#Bean
#Primary
#Singleton
public MeterRegistry getMeterRegistry() {
InfluxConfig config = new InfluxConfig() {
#Override
public Duration step() {
return Duration.ofSeconds(10);
}
#Override
public String db() {
return "metrics";
}
#Override
public String get(String k) {
return null; // accept the rest of the defaults
}
};
return new InfluxMeterRegistry(config, Clock.SYSTEM);
}
#Override
public boolean supports(MeterRegistry meterRegistry) {
return meterRegistry instanceof InfluxMeterRegistry;
}
}
When the application runs, the metrics are exposed on my /metrics endpoint as I would expect, but nothing gets written to InfluxDB. I confirmed that my local InfluxDB accepts metrics at the expected localhost:8086/write?db=metrics endpoint using curl. Can anyone give me some pointers to get this working? I'm wondering if I need to manually define a reporter somewhere...
After playing around for a bit, I got this working with the following code:
#Factory
public class InfluxMeterRegistryFactory {
#Bean
#Singleton
#Requires(property = MeterRegistryFactory.MICRONAUT_METRICS_ENABLED, value =
StringUtils.TRUE, defaultValue = StringUtils.TRUE)
#Requires(beans = CompositeMeterRegistry.class)
public InfluxMeterRegistry getMeterRegistry() {
InfluxConfig config = new InfluxConfig() {
#Override
public Duration step() {
return Duration.ofSeconds(10);
}
#Override
public String db() {
return "metrics";
}
#Override
public String get(String k) {
return null; // accept the rest of the defaults
}
};
return new InfluxMeterRegistry(config, Clock.SYSTEM);
}
}
I also noticed that an InfluxMeterRegistry will be available out of the box in the future for micronaut-micrometer as of v1.2.0.

Queues not recreated after broker failure

I'm using Spring-AMQP-rabbit in one of applications which acts as a message-consumer. The queues are created and subscribed to the exchange at startup.
My problem:
When the RabbitMq server is restarted or removed and added completely, the Queue's are not recreated. The connection to the RabbitMq server is re-stored, but not the Queues.
I've tried to do the queue admin within a ConnectionListener but that hangs on startup. I guess the admin is connection aware and should do queue management upon connection restore isn't?
My Queues are created by a service:
#Lazy
#Service
public class AMQPEventSubscriber implements EventSubscriber {
private final ConnectionFactory mConnectionFactory;
private final AmqpAdmin mAmqpAdmin;
#Autowired
public AMQPEventSubscriber(final AmqpAdmin amqpAdmin,
final ConnectionFactory connectionFactory,
final ObjectMapper objectMapper) {
mConnectionFactory = connectionFactory;
mAmqpAdmin = amqpAdmin;
mObjectMapper = objectMapper;
}
#Override
public <T extends DomainEvent<?>> void subscribe(final Class<T> topic, final EventHandler<T> handler) {
final EventName topicName = topic.getAnnotation(EventName.class);
if (topicName != null) {
final MessageListenerAdapter adapter = new MessageListenerAdapter(handler, "handleEvent");
final Jackson2JsonMessageConverter converter = new Jackson2JsonMessageConverter();
converter.setJsonObjectMapper(mObjectMapper);
adapter.setMessageConverter(converter);
final Queue queue = new Queue(handler.getId(), true, false, false, QUEUE_ARGS);
mAmqpAdmin.declareQueue(queue);
final Binding binding = BindingBuilder.bind(queue).to(Constants.DOMAIN_EVENT_TOPIC).with(topicName.value());
mAmqpAdmin.declareBinding(binding);
final SimpleMessageListenerContainer listener = new SimpleMessageListenerContainer(mConnectionFactory);
listener.setQueues(queue);
listener.setMessageListener(adapter);
listener.start();
} else {
throw new IllegalArgumentException("subscribed Event type has no exchange key!");
}
}
}
Part of my handler app:
#Component
public class FooEventHandler implements EventHandler<FooEvent> {
private final UserCallbackMessenger mUserCallbackMessenger;
private final HorseTeamPager mHorseTeamPager;
#Autowired
public FooEventHandler(final EventSubscriber subscriber) {
subscriber.subscribe(FooEvent.class, this);
}
#Override
public void handleEvent(final FooEvent event) {
// do stuff
}
}
I wonder why out-of-the-box feature with the RabbitAdmin and beans for Broker entities doesn't fit your requirements:
A further benefit of doing the auto declarations in a listener is that if the connection is dropped for any reason (e.g. broker death, network glitch, etc.) they will be applied again the next time they are needed.
See more info in the Reference Manual.

Dependency Injection of Primitive Types (Decided at Runtime) With HK2

So basically, I have a situation where I want to inject primitive types into a class (i.e. a String and an Integer). You can think of a URL and port number for an application as example inputs. I have three components:
Now say I have a class, which does take in these params:
public class PrimitiveParamsDIExample {
private String a;
private Integer b;
public PrimitiveParamsDIExample(String a, Integer b) {
this.a = a;
this.b = b;
}
}
So my question here is simple. How do I inject a and b into class PrimitiveParamsDIExample?
In general, this is also asking how to inject parameters that are decided on runtime as well. If I have a and b above, read from STDIN or from an input file, they're obviously going to be different from run to run.
All the more, how do I do the above within the HK2 framework?
EDIT[02/23/15]: #jwells131313, I tried your idea, but I'm getting the following error (this one for the String param; similar one for int):
org.glassfish.hk2.api.UnsatisfiedDependencyException: There was no object available for injection at Injectee(requiredType=String,parent=PrimitiveParamsDIExample,qualifiers
I set up classes exactly as you did in your answer. I also overrode the toString() method to print both variables a and b in PrimitiveParamsDIExample. Then, I added the following in my Hk2Module class:
public class Hk2Module extends AbstractBinder {
private Properties properties;
public Hk2Module(Properties properties){
this.properties = properties;
}
#Override
protected void configure() {
bindFactory(StringAFactory.class).to(String.class).in(RequestScoped.class);
bindFactory(IntegerBFactory.class).to(Integer.class).in(RequestScoped.class);
bind(PrimitiveParamsDIExample.class).to(PrimitiveParamsDIExample.class).in(Singleton.class);
}
}
So now, I created a test class as follows:
#RunWith(JUnit4.class)
public class TestPrimitiveParamsDIExample extends Hk2Setup {
private PrimitiveParamsDIExample example;
#Before
public void setup() throws IOException {
super.setupHk2();
//example = new PrimitiveParamsDIExample();
example = serviceLocator.getService(PrimitiveParamsDIExample.class);
}
#Test
public void testPrimitiveParamsDI() {
System.out.println(example.toString());
}
}
where, Hk2Setup is as follows:
public class Hk2Setup extends TestCase{
// the name of the resource containing the default configuration properties
private static final String DEFAULT_PROPERTIES = "defaults.properties";
protected Properties config = null;
protected ServiceLocator serviceLocator;
public void setupHk2() throws IOException{
config = new Properties();
Reader defaults = Resources.asCharSource(Resources.getResource(DEFAULT_PROPERTIES), Charsets.UTF_8).openBufferedStream();
load(config, defaults);
ApplicationHandler handler = new ApplicationHandler(new MyMainApplication(config));
final ServiceLocator locator = handler.getServiceLocator();
serviceLocator = locator;
}
private static void load(Properties p, Reader r) throws IOException {
try {
p.load(r);
} finally {
Closeables.close(r, false);
}
}
}
So somewhere, the wiring is messed up for me to get an UnsatisfiedDependencyException. What have I not correctly wired up?
Thanks!
There are two ways to do this, but one isn't documented yet (though it is available... I guess I need to work on documentation again...)
I'll go through the first way here.
Basically, you can use the HK2 Factory.
Generally when you start producing Strings and ints and long and scalars like this you qualify them, so lets start with two qualifiers:
#Retention(RUNTIME)
#Target( { TYPE, METHOD, FIELD, PARAMETER })
#javax.inject.Qualifier
public #interface A {}
and
#Retention(RUNTIME)
#Target( { TYPE, METHOD, FIELD, PARAMETER })
#javax.inject.Qualifier
public #interface B {}
then write your factories:
#Singleton // or whatever scope you want
public class StringAFactory implements Factory<String> {
#PerLookup // or whatever scope, maybe this checks the timestamp?
#A // Your qualifier
public String provide() {
// Write your code to get your value...
return whatever;
}
public void dispose(String instance) {
// Probably do nothing...
}
}
and for the Integer:
#Singleton // or whatever scope you want
public class IntegerBFactory implements Factory<Integer> {
#PerLookup // or whatever scope, maybe this checks the timestamp?
#B // Your qualifier
public Integer provide() {
// Write your code to get your value...
return whatever;
}
public void dispose(String instance) {
// Probably do nothing...
}
}
Now lets re-do your original class to accept these values:
public class PrimitiveParamsDIExample {
private String a;
private int b;
#Inject
public PrimitiveParamsDIExample(#A String a, #B int b) {
this.a = a;
this.b = b;
}
}
Note I changed Integer to int, well... just because I can. You can also just use field injection or method injection in the same way. Here is field injection, method injection is an exercise for the reader:
public class PrimitiveParamsDIExample {
#Inject #A
private String a;
#Inject #B
private int b;
public PrimitiveParamsDIExample() {
}
}
There are several ways to bind factories.
In a binder: bindFactory
Using automatic class analysis: addClasses
An EDSL outside a binder: buildFactory

Mbean registered but not found in mbean Server

I have a problem about the mbeans. I have created a simple mbean and I have registered it on the default mBeanServer that is run (Via eclipse or java -jar mbean.jar) and in the same process if I try to fouund the mbean registered with a simple query:
for (ObjectInstance instance : mbs.queryMBeans(ObjectNameMbean, null)) {
System.out.println(instance.toString());
}
the query retuerns my mbean, but if I start another process and try to search this mbean registered the mbeas is not found! why?
The approch is : (Process that is running)
public static void main(String[] args) throws Exception
{
MBeanServer mbeanServer =ManagementFactory.getPlatformMBeanServer();
ObjectName objectName = new ObjectName(ObjectNameMbean);
Simple simple = new Simple (1, 0);
mbeanServer.registerMBean(simple, objectName);
while (true)
{
wait (Is this necessary?)
}
}
So this is the first process that is running (that has the only pourpose to registry the mbean, because there is another process that want to read these informations.
So I start another process to search this mbean but nothing.
I 'm not using jboss but the local Java virtual Machine but my scope is to deploy this simple application in one ejb (autostart) and another ejb will read all informations.
All suggestions are really apprecciated.
This example should be more useful :
Object Hello:
public class Hello implements HelloMBean {
public void sayHello() {
System.out.println("hello, world");
}
public int add(int x, int y) {
return x + y;
}
public String getName() {
return this.name;
}
public int getCacheSize() {
return this.cacheSize;
}
public synchronized void setCacheSize(int size) {
this.cacheSize = size;
System.out.println("Cache size now " + this.cacheSize);
}
private final String name = "Reginald";
private int cacheSize = DEFAULT_CACHE_SIZE;
private static final int DEFAULT_CACHE_SIZE = 200;
}
Interface HelloBean (implemented by Hello)
public interface HelloMBean {
public void sayHello();
public int add(int x, int y);
public String getName();
public int getCacheSize();
public void setCacheSize(int size);
}
Simple Main
import java.lang.management.ManagementFactory;
import java.util.logging.Logger;
import javax.management.MBeanServer;
import javax.management.ObjectName;
public class Main {
static Logger aLog = Logger.getLogger("MBeanTest");
public static void main(String[] args) {
try{
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = new ObjectName("ApplicationDomain:type=Hello");
Hello mbean = new Hello();
mbs.registerMBean(mbean, name);
// System.out.println(mbs.getAttribute(name, "Name"));
aLog.info("Waiting forever...");
Thread.sleep(Long.MAX_VALUE);
}
catch(Exception x){
x.printStackTrace();
aLog.info("exception");
}
}
}
So now I have exported this project as jar file and run it as "java -jar helloBean.jar" and by eclipse I have modified the main class to read informations of this read (Example "Name" attribute) by using the same objectname used to registry it .
Main to read :
public static void main(String[] args) {
try{
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = new ObjectName("ApplicationDomain:type=Hello");
System.out.println(mbs.getAttribute(name, "Name"));
}
catch(Exception x){
x.printStackTrace();
aLog.info("exception");
}
}
But nothing, the bean is not found.
Project link : here!
Any idea?
I suspect the issue here is that you have multiple MBeanServer instances. You did not mention how you acquired the MBeanServer in each case, but in your second code sample, you are creating a new MBeanServer instance which may not be the same instance that other threads are reading from. (I assume this is all in one JVM...)
If you are using the platform agent, I recommend you acquire the MBeanServer using the ManagementFactory as follows:
MBeanServer mbs = java.lang.management.ManagementFactory.getPlatformMBeanServer() ;
That way, you will always get the same MBeanServer instance.

More than one singleton instance in Guice injector

We got a Jetty/Jersey application. We are converting it to use Guice for DI. The problem: We need more than one instance of a Singleton classes. The catch: The number of instances is determined dynamically from a configuration file. Therefore we cant use annotations for different instances.
final InjectedClass instance = injector.getInstance(InjectedClass.class);
This is the standard syntax of the injector. I need something like
final String key = getKey();
final InjectedClass instance = injector.getInstance(InjectedClass.class, key);
There is a way to get an instance from a Guice Key.class
final InjectedClass instance = injector.getInstance(Key.get(InjectedClass.class, <Annotation>);
but the problem is that I need some dynamic annotation, not predefined one.
You could try to use Provider, or #Provides method that would have map of all instances already created. When the number of instances is reached number defained in config file, you wont create any new instances, instead you return old instance from map.
For example something like this could help you.
public class MyObjectProvider implements Provider<MyObject> {
private final Injector inj;
private int counter;
private final int maxNum = 5;
private List<MyObject> myObjPool = new ArrayList<MyObject>();
#Inject
public MyObjectProvider(Injector inj) {
this.connection = connection;
}
public MyObject get() {
counter = counter+1%maxNum;
if(myObjPool.size()=<maxNum) {
MyObject myobj = inj.getInstance(MyObject.class);
myObjPool.add(myobj);
return myobj;
} else {
return myObjPool.get(counter);
}
}
}
P.S.
I wrote this from my head so maybe it does not compile, this is just an idea.
You can solve this by creating a factory. In my example I have used the guice extension called multibindings
interface InjectedClassFactory {
public InjectedClass get(String key);
}
class InjectedClass {}
class InjectedClassFactoryImpl implements InjectedClassFactory{
private final Map<String, InjectedClass> instances;
#Inject
InjectedClassFactoryImpl(Map<String, InjectedClass> instances) {
this.instances = instances;
}
#Override
public InjectedClass get(String key) {
return instances.get(key);
}
}
class MyModule extends AbstractModule {
#Override
protected void configure() {
MapBinder<String, InjectedClass> mapBinder =
MapBinder.newMapBinder(binder(), String.class, InjectedClass.class);
//read you config file and retrieve the keys
mapBinder.addBinding("key1").to(InjectedClass.class).in(Singleton.class);
mapBinder.addBinding("key2").to(InjectedClass.class).in(Singleton.class);
}
}

Resources