I'm working on crawler4j using groovy and grails.
I have a BasicCrawler.groovy class in src/groovy and the domain class Crawler.groovy and a controller called CrawlerController.groovy.
I have few properties in BasicCrawler.groovy class like url, parentUrl, domain etc.
I want to persist these values to the database by passing these values to the domain class while crawling is happening.
I tried doing this in my BasicCrawler class under src/groovy
class BasicCrawler extends WebCrawler {
Crawler obj = new Crawler()
//crawling code
#Override
void visit(Page page) {
//crawling code
obj.url = page.getWebURL().getURL()
obj.parentUrl = page.getWebURL().getParentUrl()
}
#Override
protected void handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) {
//crawling code
obj.httpstatus = "not found"
}
}
And my domain class is as follows:
class Crawler extends BasicCrawler {
String url
String parentUrl
String httpstatus
static constraints = {}
}
But I got the following error:
ERROR crawler.WebCrawler - Exception while running the visit method. Message: 'No such property: url for class: mypackage.BasicCrawler
Possible solutions: obj' at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:50)
After this I tried another approach. In my src/groovy/BasicCrawler.groovy class, I declared the url and parentUrl properties on the top and then used databinding (I might be wrong since I am just a beginner):
class BasicCrawler extends WebCrawler {
String url
String parentUrl
#Override
boolean shouldVisit(WebURL url) { //code
}
#Override
void visit(Page page) { //code
}
#Override
protected void handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) {
//code}
}
def bindingMap = [url: url , parentUrl: parentUrl]
def Crawler = new Crawler(bindingMap)
}
And my Crawler.groovy domain class is as follows:
class Crawler {
String url
String parentUrl
static constraints = {}
}
Now, it doesn't show any error but the values are not being persisted in the database. I am using mongodb for the backend.
I think this example is a bit contrived but here is a way you might solve this problem in current situation:
class BasicCrawler extends WebCrawler {
#Override
void visit(Page page) {
Crawler obj = new Crawler()
obj.url = page.getWebURL().getURL()
obj.parentUrl = page.getWebURL().getParentUrl()
obj.save()
}
#Override
protected void handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) {
Crawler obj = Crawler.findByUrl(webUrl)
obj.httpstatus = "not found"
obj.save()
}
}
Key here is not using a member instance variable and using the URL to 'refetch' and update original site 'visited' since I'm assuming that will be a unique constraint on each row.
Related
Little background: I am working on a topology using Apache Storm, I thought why not use dependency injection in it, but I was not sure how it will behave on cluster environment when topology deployed to cluster. I started looking for answers on if DI is good option to use in Storm topologies, I came across some threads about Apache Spark where it was mentioned serialization is going to be problem and saw some responses for apache storm along the same lines. So finally I decided to write a sample topology with google guice to see what happens.
I wrote a sample topology with two bolts, and used google guice to injects dependencies. First bolt emits a tick tuple, then first bolt creates message, bolt prints the message on log and call some classes which does the same. Then this message is emitted to second bolt and same printing logic there as well.
First Bolt
public class FirstBolt extends BaseRichBolt {
private OutputCollector collector;
private static int count = 0;
private FirstInjectClass firstInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
firstInjectClass = injector.getInstance(FirstInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
count++;
String message = "Message count "+count;
firstInjectClass.printMessage(message);
log.error(message);
collector.emit("TO_SECOND_BOLT", new Values(message));
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
outputFieldsDeclarer.declareStream("TO_SECOND_BOLT", new Fields("MESSAGE"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
Config conf = new Config();
conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 10);
return conf;
}
}
Second Bolt
public class SecondBolt extends BaseRichBolt {
private OutputCollector collector;
private SecondInjectClass secondInjectClass;
#Override
public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
collector = outputCollector;
Injector injector = Guice.createInjector(new Module());
secondInjectClass = injector.getInstance(SecondInjectClass.class);
}
#Override
public void execute(Tuple tuple) {
String message = (String) tuple.getValue(0);
secondInjectClass.printMessage(message);
log.error("SecondBolt {}",message);
collector.ack(tuple);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
}
}
Class in which dependencies are injected
public class FirstInjectClass {
FirstInterface firstInterface;
private final String prepend = "FirstInjectClass";
#Inject
public FirstInjectClass(FirstInterface firstInterface) {
this.firstInterface = firstInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
firstInterface.printMethod(message);
}
}
Interface used for binding
public interface FirstInterface {
void printMethod(String message);
}
Implementation of interface
public class FirstInterfaceImpl implements FirstInterface{
private final String prepend = "FirstInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Same way another class that receives dependency via DI
public class SecondInjectClass {
SecondInterface secondInterface;
private final String prepend = "SecondInjectClass";
#Inject
public SecondInjectClass(SecondInterface secondInterface) {
this.secondInterface = secondInterface;
}
public void printMessage(String message){
log.error("{} {}", prepend, message);
secondInterface.printMethod(message);
}
}
another interface for binding
public interface SecondInterface {
void printMethod(String message);
}
implementation of second interface
public class SecondInterfaceImpl implements SecondInterface{
private final String prepend = "SecondInterfaceImpl";
public void printMethod(String message){
log.error("{} {}", prepend, message);
}
}
Module Class
public class Module extends AbstractModule {
#Override
protected void configure() {
bind(FirstInterface.class).to(FirstInterfaceImpl.class);
bind(SecondInterface.class).to(SecondInterfaceImpl.class);
}
}
Nothing fancy here, just two bolts and couple of classes for DI. I deployed it on server and it works just fine. The catch/problem though is that I have to initialize Injector in each bolt which makes me question what is side effect of it going to be?
This implementation is simple, just 2 bolts.. what if I have more bolts? what impact it would create on topology if I have to initialize Injector in all bolts?
If I try to initialize Injector outside prepare method I get error for serialization.
I have a combo box over my GUI in JavaFX.
This Combo Box is composed of a complex type elements :
public class DureeChoiceBoxElement extends ObservableValueBase<DureeChoiceBoxElement> {
private IntegerProperty duree;
#Override
public String toString() {
return duree.get() + " an";
}
}
I want to map (or bind) the selected complex element with my model which contains the simple type :
public class Pel {
private IntegerProperty duree = new SimpleIntegerProperty(1);
public Property<Number> dureeProperty() {
return duree;
}
public void setDuree(Integer duree) {
this.duree.setValue(duree);
}
public Integer getDuree() {
return duree.getValue();
}
}
How to do it ?
I tried in the controller with :
public class PelController {
#FXML
private ChoiceBox<DureeChoiceBoxElement> duree;
//etc..
pel.dureeProperty().bind(createElapsedBindingByBindingsAPI2(duree.getValue()));
/*
* #return an ObjectBinding of immutable TimeElapsed objects for the player
*/
private ObjectBinding<Property<Number>> createElapsedBindingByBindingsAPI2(
final DureeChoiceBoxElement dureeChoiceBoxElement) {
return Bindings.createObjectBinding(new Callable<Property<Number>>() {
#Override
public IntegerProperty call() throws Exception {
return dureeChoiceBoxElement.dureeProperty();
}
}, dureeChoiceBoxElement.dureeProperty());
}
}
But it doesn't work (even not compile). I want to say that "Bind this simple property to this complex Object calling the method I give you through the method named "createElapsedBindingByBindingsAPI2(..)".
It is logical read but I didn't managed to make it works anyway.
That's poor ....
Any help please :).
Example that (obviously) works with legacy code style (Swing coding) :
duree.getSelectionModel().selectedItemProperty().addListener(new ChangeListener<DureeChoiceBoxElement>() {
#Override
public void changed(ObservableValue<? extends DureeChoiceBoxElement> observable,
DureeChoiceBoxElement oldValue, DureeChoiceBoxElement newValue) {
// changement durée
log.debug("Durée sélectionnée : {}", duree.getSelectionModel().getSelectedItem().getDuree());
log.debug("Durée bindée ? : {}", pel.getDuree());
pel.setDuree(duree.getSelectionModel().getSelectedItem().getDuree());
}
});
Like this my model is set to selected item. But it implies some boilerplate code. Any better idea based on high level bindings of JavaFX ?
So basically, I have a situation where I want to inject primitive types into a class (i.e. a String and an Integer). You can think of a URL and port number for an application as example inputs. I have three components:
Now say I have a class, which does take in these params:
public class PrimitiveParamsDIExample {
private String a;
private Integer b;
public PrimitiveParamsDIExample(String a, Integer b) {
this.a = a;
this.b = b;
}
}
So my question here is simple. How do I inject a and b into class PrimitiveParamsDIExample?
In general, this is also asking how to inject parameters that are decided on runtime as well. If I have a and b above, read from STDIN or from an input file, they're obviously going to be different from run to run.
All the more, how do I do the above within the HK2 framework?
EDIT[02/23/15]: #jwells131313, I tried your idea, but I'm getting the following error (this one for the String param; similar one for int):
org.glassfish.hk2.api.UnsatisfiedDependencyException: There was no object available for injection at Injectee(requiredType=String,parent=PrimitiveParamsDIExample,qualifiers
I set up classes exactly as you did in your answer. I also overrode the toString() method to print both variables a and b in PrimitiveParamsDIExample. Then, I added the following in my Hk2Module class:
public class Hk2Module extends AbstractBinder {
private Properties properties;
public Hk2Module(Properties properties){
this.properties = properties;
}
#Override
protected void configure() {
bindFactory(StringAFactory.class).to(String.class).in(RequestScoped.class);
bindFactory(IntegerBFactory.class).to(Integer.class).in(RequestScoped.class);
bind(PrimitiveParamsDIExample.class).to(PrimitiveParamsDIExample.class).in(Singleton.class);
}
}
So now, I created a test class as follows:
#RunWith(JUnit4.class)
public class TestPrimitiveParamsDIExample extends Hk2Setup {
private PrimitiveParamsDIExample example;
#Before
public void setup() throws IOException {
super.setupHk2();
//example = new PrimitiveParamsDIExample();
example = serviceLocator.getService(PrimitiveParamsDIExample.class);
}
#Test
public void testPrimitiveParamsDI() {
System.out.println(example.toString());
}
}
where, Hk2Setup is as follows:
public class Hk2Setup extends TestCase{
// the name of the resource containing the default configuration properties
private static final String DEFAULT_PROPERTIES = "defaults.properties";
protected Properties config = null;
protected ServiceLocator serviceLocator;
public void setupHk2() throws IOException{
config = new Properties();
Reader defaults = Resources.asCharSource(Resources.getResource(DEFAULT_PROPERTIES), Charsets.UTF_8).openBufferedStream();
load(config, defaults);
ApplicationHandler handler = new ApplicationHandler(new MyMainApplication(config));
final ServiceLocator locator = handler.getServiceLocator();
serviceLocator = locator;
}
private static void load(Properties p, Reader r) throws IOException {
try {
p.load(r);
} finally {
Closeables.close(r, false);
}
}
}
So somewhere, the wiring is messed up for me to get an UnsatisfiedDependencyException. What have I not correctly wired up?
Thanks!
There are two ways to do this, but one isn't documented yet (though it is available... I guess I need to work on documentation again...)
I'll go through the first way here.
Basically, you can use the HK2 Factory.
Generally when you start producing Strings and ints and long and scalars like this you qualify them, so lets start with two qualifiers:
#Retention(RUNTIME)
#Target( { TYPE, METHOD, FIELD, PARAMETER })
#javax.inject.Qualifier
public #interface A {}
and
#Retention(RUNTIME)
#Target( { TYPE, METHOD, FIELD, PARAMETER })
#javax.inject.Qualifier
public #interface B {}
then write your factories:
#Singleton // or whatever scope you want
public class StringAFactory implements Factory<String> {
#PerLookup // or whatever scope, maybe this checks the timestamp?
#A // Your qualifier
public String provide() {
// Write your code to get your value...
return whatever;
}
public void dispose(String instance) {
// Probably do nothing...
}
}
and for the Integer:
#Singleton // or whatever scope you want
public class IntegerBFactory implements Factory<Integer> {
#PerLookup // or whatever scope, maybe this checks the timestamp?
#B // Your qualifier
public Integer provide() {
// Write your code to get your value...
return whatever;
}
public void dispose(String instance) {
// Probably do nothing...
}
}
Now lets re-do your original class to accept these values:
public class PrimitiveParamsDIExample {
private String a;
private int b;
#Inject
public PrimitiveParamsDIExample(#A String a, #B int b) {
this.a = a;
this.b = b;
}
}
Note I changed Integer to int, well... just because I can. You can also just use field injection or method injection in the same way. Here is field injection, method injection is an exercise for the reader:
public class PrimitiveParamsDIExample {
#Inject #A
private String a;
#Inject #B
private int b;
public PrimitiveParamsDIExample() {
}
}
There are several ways to bind factories.
In a binder: bindFactory
Using automatic class analysis: addClasses
An EDSL outside a binder: buildFactory
I am studying orchard architecture.i have faced with a strange concept in display management section.
in Partial view page there is a 'function call like' syntax like so Display(Model.Head). that is not a function thought, it is a dynamic object defined in orchard WebViewPage.
I am wondering how to define a dynamic object so that you can call it (and pass it an argument as well) like a function as i mentioned.
thanks in advance.
A lighter weight way to do it without clay would be to subclass the built-in DynamicObject class.
public static dynamic Display;
void Main()
{
Display = new MyCallableObject();
//this is what i was after
Console.Write(Display("bla bla bla"));
}
public class MyCallableObject:DynamicObject
{
public override bool TryInvoke(InvokeBinder binder, object[] args, out Object result)
{
result = string.Format("This is response for {0}",args.FirstOrDefault());
return true;
}
}
I finally found it my self!
all the operations have done with Clay Library behind the scene.i have wrote a sample console app for demonstrating the process.
class Program
{
static void Main(string[] args)
{
Display = ClayActivator.CreateInstance<MyResponser>(new List<IClayBehavior> {new MyFunctionCallBehavior()});
//this is what i was after
Console.Write(Display("bla bla bla"));
}
public static dynamic Display;
}
public class MyFunctionCallBehavior : IClayBehavior
{
public object InvokeMember(Func<object> proceed, object self, string name, INamedEnumerable<object> args)
{
return ((MyResponser)self).ResponseForRequest(args.FirstOrDefault().ToString());
}
}
public class MyResponser
{
public string ResponseForRequest(string param)
{
return string.Format("This is response for {0}",param);
}
}
We got a Jetty/Jersey application. We are converting it to use Guice for DI. The problem: We need more than one instance of a Singleton classes. The catch: The number of instances is determined dynamically from a configuration file. Therefore we cant use annotations for different instances.
final InjectedClass instance = injector.getInstance(InjectedClass.class);
This is the standard syntax of the injector. I need something like
final String key = getKey();
final InjectedClass instance = injector.getInstance(InjectedClass.class, key);
There is a way to get an instance from a Guice Key.class
final InjectedClass instance = injector.getInstance(Key.get(InjectedClass.class, <Annotation>);
but the problem is that I need some dynamic annotation, not predefined one.
You could try to use Provider, or #Provides method that would have map of all instances already created. When the number of instances is reached number defained in config file, you wont create any new instances, instead you return old instance from map.
For example something like this could help you.
public class MyObjectProvider implements Provider<MyObject> {
private final Injector inj;
private int counter;
private final int maxNum = 5;
private List<MyObject> myObjPool = new ArrayList<MyObject>();
#Inject
public MyObjectProvider(Injector inj) {
this.connection = connection;
}
public MyObject get() {
counter = counter+1%maxNum;
if(myObjPool.size()=<maxNum) {
MyObject myobj = inj.getInstance(MyObject.class);
myObjPool.add(myobj);
return myobj;
} else {
return myObjPool.get(counter);
}
}
}
P.S.
I wrote this from my head so maybe it does not compile, this is just an idea.
You can solve this by creating a factory. In my example I have used the guice extension called multibindings
interface InjectedClassFactory {
public InjectedClass get(String key);
}
class InjectedClass {}
class InjectedClassFactoryImpl implements InjectedClassFactory{
private final Map<String, InjectedClass> instances;
#Inject
InjectedClassFactoryImpl(Map<String, InjectedClass> instances) {
this.instances = instances;
}
#Override
public InjectedClass get(String key) {
return instances.get(key);
}
}
class MyModule extends AbstractModule {
#Override
protected void configure() {
MapBinder<String, InjectedClass> mapBinder =
MapBinder.newMapBinder(binder(), String.class, InjectedClass.class);
//read you config file and retrieve the keys
mapBinder.addBinding("key1").to(InjectedClass.class).in(Singleton.class);
mapBinder.addBinding("key2").to(InjectedClass.class).in(Singleton.class);
}
}