Apache Storm, Twitter - twitter

Question

I am processing twitter tweets by using twiiter4j.properties through storm-bolts. My topology looks like:
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("KafkaSpout", new KafkaSpout(kafkaConfig), 2).setNumTasks(4);
builder.setBolt("Preprocesing", new preprocessBolt2(), 2)
.setNumTasks(4).shuffleGrouping("KafkaSpout");
builder.setBolt("AvgScoreAnalysis",
new AvgScoringBolt(), 4).setNumTasks(8)
.fieldsGrouping("Preprocesing",new Fields("tweetId"));
builder.setBolt("PrinterBolt", new LocalFile(), 6).setNumTasks(4)
.shuffleGrouping("AvgScoreAnalysis");
Where I am taking tweets from KafkaSpout and sending it to bolt for pre-processing, My problem is in the avgScoring where I am calling S3 in that I am having csv for each user and calculating the scoring for each user for each single tweet. I am having 100 users means my avg scoring has to calculate avg score for each tweet for all the number of users in the s3. It is pretty slow how can I increase the performance in this bolt and there are so many duplicates in the file how can I remove duplicates?

Answer 1

Calling S3 from the AvgScoringBolt is not a great idea if you want high performance: unless you're filtering the tweets with some criteria, there's no way to make a connection to S3 for every tweet and still parse thousands tweet per second.
Since there are only 100 users, maybe you could download the CVS of the users at the start of application, make the computation inside the bolt without connecting to S3 (using the downloaded CSV), and periodically upload the updated CSV to S3 to have a loosely synchronization with S3. I don't know if this scenario fits your requirements.