Twitter data streaming - dynamic keyword value change - twitter

Question

I have currently doing Sentiment Analysis on Twitter data in Hadoop.
I have configured FLUME to bring data based upon certain keywords, which I have maintained in "FLUME.conf" file, as shown below
TwitterAgent.sources.Twitter.keywords = Kitkat, Nescafe, Carnation, Milo, Cerelac
With mentioning this, I would like to know whether it is possible to dynamically change the keywords, based upon a java program that will pop-up asking for keywords from user.
Also, apart from this method, please provide any other methods you folks would suggest to hide this complexity of updating the Flume.conf file.
Best Regards,
Ram

Answer 1

Flume provides an Application class to run its configuration file using a Java program.
public class Flume{
public static void main(String[] args) throws Exception
{
new Conf().setConfiguration();
String[] args1 = new String[] { "agent","-nTwitterAgent",
"-fflume.conf" };
BasicConfigurator.configure();
Application.main(args1);
System.setProperty("hadoop.home.dir", "/");
}
}
Here you will get a change to edit your conf file using some file utils like this
class Conf{
int ch;
String keyword ="";
Scanner sc= new Scanner(System.in);
public void setConfiguration(){
System.out.println("Enter the keyword");
keyword=sc.nextLine();
byte[] key= keyword.getBytes();
FileOutputStream fp=null;
FileInputStream src=null;
try{
fp= new FileOutputStream("flumee.conf");
src= new FileInputStream("flume.conf");
while((ch=src.read())!=-1){
fp.write((char)ch);
}
fp.write(key);
}catch(Exception e){
System.out.println("file Exception:"+ e);
}finally{
try{
if(fp!=null){
fp.close();
}
if(src!=null){
src.close();
}
}catch(Exception e){
System.out.println("file closing Exception:"+ e);
}
}
}
}
Now you need to keep the TwitterAgent.sources.Twitter.keywords= line at the end of your flume configuration file so that it will be easier for you to add the keywords into the file. My flume.conf file looks like this
TwitterAgent.sources= Twitter
TwitterAgent.channels= MemChannel
TwitterAgent.sinks=HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels=MemChannel
TwitterAgent.sources.Twitter.consumerKey=xxx
TwitterAgent.sources.Twitter.consumerSecret=xxx
TwitterAgent.sources.Twitter.accessToken=xxx
TwitterAgent.sources.Twitter.accessTokenSecret=
TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:9000/user/flume/direct
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=1000
TwitterAgent.sources.Twitter.keywords=
When i run the program it will ask for the keywords first and then it will start collecting the tweets about that keyword.