My goal is to collect all tweets containing the words "France" and "Germany" and to also collect associated metadata (e.g., the geo coordinates attached to the tweet). I know that this metadata is available, but I can't figure out how to access it with the Java library I'm using : "twitter4j".
Ok, so what I have so far is taken from code samples on the twitter4j site. It prints out all tweets containing my chosen keywords, as they are provided in real-time by Twitter's Streaming API. I call the filter method on my TwitterStream object, and this provides the stream. But I need more control. Namely, I would like to be able to:
1) write the tweets to a file;
2) only print out the first 1000 tweets;
3) access other metadata attached to the tweet (the filter method just prints out the username and the tweet itself).
Here is the code I have so far:
import twitter4j.FilterQuery;
import twitter4j.Status;
import twitter4j.StatusDeletionNotice;
import twitter4j.StatusListener;
import twitter4j.TwitterException;
import twitter4j.TwitterStream;
import twitter4j.TwitterStreamFactory;
import twitter4j.conf.ConfigurationBuilder;
public class Stream {
public static void main(String[] args) throws TwitterException {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true);
cb.setOAuthConsumerKey("bbb");
cb.setOAuthConsumerSecret("bbb");
cb.setOAuthAccessToken("bbb");
cb.setOAuthAccessTokenSecret("bbb");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
public void onStatus(Status status) {
System.out.println("#" + status.getUser().getScreenName() + " - " + status.getText());
}
public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
}
public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
}
public void onScrubGeo(long userId, long upToStatusId) {
System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
}
public void onException(Exception ex) {
ex.printStackTrace();
}
};
FilterQuery fq = new FilterQuery();
String keywords[] = {"France", "Germany"};
fq.track(keywords);
twitterStream.addListener(listener);
twitterStream.filter(fq);
}
}
After looking at this with fresh eyes I realised the solution (which was pretty obvious). Editing the following part of the code:
public void onStatus(Status status) {
System.out.println("#" + status.getUser().getScreenName() + " - " + status.getText());
}
allows me to access other metadata. For example, if I want to access the tweet's date, I simply need to add the following:
System.out.println(status.getCreatedAt());
The Error 401 comes when the API is trying to access some information which is unable to fetch at present. So you need to check the permission which are allowed on twitter. Change it to READ, WRITE and ... for full API access. Or there might be problem as you might be using the proxy server. Hence mention the proxy details using the following commands.
System.getProperties().put("http.proxyHost", "10.3.100.211");
System.getProperties().put("http.proxyPort", "8080");
To write tweets on file:
FileWriter file = new FileWriter(....);
public void onStatus(Status status) {
System.out.println("#" + status.getUser().getScreenName() + " - " + status.getText() + " -> "+ status.getCreatedAt());
try {
file.write(status.getUser().getScreenName() + " - " + status.getText() + " -> "+ status.getCreatedAt() +"\n");
file.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Related
I am using reactor in a project, and one of the features calls a blocking service, which connects to a device and gets an infinite stream of events.
I am trying to do a load test to see how many calls can I make to the blocking service.
I am generating around 1000 requests to the blocking service
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250)
The problem is that reactor is only processing the first 256 requests, after that it isn't making any more requests.
When I added the .log("preConnect") I can see that it is logging only one request(256) from the downstream subscriber.
I don't understand what I am doing wrong.
I am attaching simplified example which can reproduce the issue.
package test.reactor;
import org.junit.jupiter.api.Test;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Supplier;
import java.util.stream.Stream;
public class ReactorTest {
#Test
void testLoad() throws InterruptedException {
AtomicInteger id = new AtomicInteger(0);
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250) // will create a total of 1004 messages
.map(str -> str + " id: " + id.incrementAndGet())
.log("preConnect")
.flatMap(this::blocking)
.log()
.subscribeOn(Schedulers.parallel())
.subscribe();
new CountDownLatch(1).await();
}
private Flux<String> blocking(String ip) {
Mono<String> connectMono = Mono.fromCallable(this::connect)
.subscribeOn(Schedulers.boundedElastic())
.map(msg -> "Connected: "+ip + msg);
Flux<String> streamFlux = Mono.fromCallable(this::infiniteNetworkStream)
.subscribeOn(Schedulers.boundedElastic())
.flatMapMany(Flux::fromStream)
.map(msg -> ip + msg);
return connectMono.concatWith(streamFlux);
}
private Stream<String> infiniteNetworkStream() {
return Stream.generate(new Supplier<String>() {
#Override
public String get() {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return "Hello";
}
});
}
private String connect() throws Exception{
Thread.sleep(100);
return "success";
}
}
Figured out the issue, it has to do with flatmap, the default concurrency for flatmap is 256. It will not request more items from the upstream publisher until the current subscriptions go below 256.
In my case since my flux is infinite, it wasn't processing any after 256.
The solution I found was to increase the concurrency
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250) // will create a total of 1004 messages
.map(str -> str + " id: " + id.incrementAndGet())
.log("preConnect")
.flatMap(this::blocking, 1000) // added 1000 here to increase concurrency
.log()
.subscribeOn(Schedulers.parallel())
.subscribe();
I'm trying to use the twitter4j API to get the stream of tweets on a specific topic.
This is my code:
TwitterStream twitterStream = inizialize();
StatusListener listener = new StatusListener(){
public void onStatus(Status status) {
System.out.println(status.getUser().getName() + " ====> " + status.getText());
}
public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {}
public void onTrackLimitationNotice(int numberOfLimitedStatuses) {}
public void onException(Exception ex) {
ex.printStackTrace();
}
#Override
public void onScrubGeo(long userId, long upToStatusId) {
// TODO Auto-generated method stub
}
#Override
public void onStallWarning(StallWarning warning) {
// TODO Auto-generated method stub
}
};
FilterQuery filterQuery = new FilterQuery("GAME");
twitterStream.addListener(listener);
twitterStream.filter(filterQuery);
twitterStream.sample(); // sample() method internally creates a thread which manipulates TwitterStream and calls these adequate listener methods continuously.
}
The stream of tweets works well, but I cannot set any query. So, I wonder, is it possible to do what I'm trying to?
Of course, the inizialize() returns a twitterStream configured with a valid oauth token.
You'll want to manually filter statuses coming from the stream. For example, if you want to show tweets that contains 'vanilla' only then you could approach like this in your onStatus:
public void onStatus(Status status) {
String statusText = status.getText();
if (statusText.toLowerCase().contains("vanilla")) {
System.out.println(status.getUser().getName() + " ====> " + statusText);
}
}
// Expecting a String[] of topics to track:
filterQuery.track(keywords);
// Within a bounding box of geo-coordinates:
filterQuery.locations(new double[][] {{lng1, lat1}, {lng2, lat2}});
// Specifies a language to track:
filterQuery.language("en");
// Number of previous statuses to stream before transitioning to the live stream:
filterQuery.count(10);
Good day
me and my team are new to coding languages, we are trying through multiple methods to make an arduino based indicator that shows which keyword out of two is more mentioned during the last 5 minutes on twitter
we tried using adafruit + IFTTT and we managed to have a stream of real time tweets of two hashtags but we are trying to find a way to collect that info and make a code that compare the total number of both hashtags and send command to arduino to spine the servo motor based on the result.
and then we tried to do it through processing language and we found this code that makes displays related hashtag tweets on screen but we couldn't make it search for two words and compare the numbers and then send signal to arduino :
//http://codasign.com/tutorials/processing-and-twitter
import twitter4j.conf.*;
import twitter4j.*;
import twitter4j.auth.*;
import twitter4j.api.*;
import java.util.*;
Twitter twitter;
String searchString = "#poznan";
List<Status> tweets;
int currentTweet;
void setup()
{
size(800, 600);
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("");
cb.setOAuthConsumerSecret("");
cb.setOAuthAccessToken("");
cb.setOAuthAccessTokenSecret("");
TwitterFactory tf = new TwitterFactory(cb.build());
twitter = tf.getInstance();
getNewTweets();
currentTweet = 1;
thread("refreshTweets");
}
void draw()
{
fill(0, 40);
rect(0, 0, width, height);
currentTweet = currentTweet + 1;
if (currentTweet >= tweets.size())
{
currentTweet = 0;
}
Status status = tweets.get(currentTweet);
fill(200);
text(status.getText(), random(width), random(height), 300, 200);
delay(250);
}
void getNewTweets()
{
try
{
Query query = new Query(searchString);
//query.setSince("2016-03-17");
//query.setCount(100);
query.setResultType(Query.RECENT);
QueryResult result = twitter.search(query);
tweets = result.getTweets();
println(tweets.size());
}
catch (TwitterException te)
{
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
void refreshTweets()
{
while (true)
{
getNewTweets();
println("Updated Tweets");
delay(60000);
}
}
We are looking for alternative codes and methods to make our concept work
we are open to suggestions, don't hesitate to write to us.
You should probably split this problem in smaller ones.
A. How do I connect to Twitter and "listen" to two hashtags, counting them and comparing them.
B. How do I use arduino to move a servo according to some arbitary number in a specific range.
C. How do I communicate a number between Arduino and processing, via desired channel (wi-fi? bluetooth? usb?)
Of course this can be break even further, and perhaps it should.
Doing like this is going to be much easier to develop and debug your code. Once you got all that figured out, start to combine them.
For B and C I can help very little, it's been sometime since I last touched my Arduino. But those are not really hard to be done. In the Processing forum you can find a lot of answers about serial communication with Arduino. The search of the forum is not so good. But you can always use Google. Something like: serial arduino site:processing.org will do the search in all forums (this is like the third version) and give you easier to navigate results.
For A, I'd suggest you try a "stream" from Twitter's streamingAPI. Once you start getting results, just add them to different Lists, like
hashtag1 and hashtag2.
The size of each list is what you are looking for (if I get this right)
hashtag1.size() - hashtag2.size() will give you the "balance" between them.
edit: If, you are not going to need those status you can just add to two ints... (h1++, h2++), and forget the lists.
Here some stream sample code to get you started:
import twitter4j.util.*;
import twitter4j.*;
import twitter4j.management.*;
import twitter4j.api.*;
import twitter4j.conf.*;
import twitter4j.json.*;
import twitter4j.auth.*;
TwitterStream twitterStream;
// if you enter keywords here it will filter, otherwise it will sample
String keywords[] = {
//all you need is...
"love"
};
void setup() {
size(100, 100);
background(0);
openTwitterStream();
}
void draw() {
background(0);
}
// Stream it
void openTwitterStream() {
ConfigurationBuilder cb = new ConfigurationBuilder();
//fill oAuth data below
cb.setOAuthConsumerKey("");
cb.setOAuthConsumerSecret("");
cb.setOAuthAccessToken("");
cb.setOAuthAccessTokenSecret("");
cb.setDebugEnabled(true);
cb.setJSONStoreEnabled(true);
twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
FilterQuery filtered = new FilterQuery();
filtered.track(keywords);
twitterStream.addListener(listener);
if (keywords.length==0) {
// sample() method internally creates a thread which manipulates TwitterStream
// and calls these adequate listener methods continuously. With a sample of the fireRose
twitterStream.sample();
} else {
twitterStream.filter(filtered);
}
println("connecting...");
}
// Implementing StatusListener interface
StatusListener listener = new StatusListener() {
//#Override
public void onStatus(Status status) {
System.out.println("#" + status.getUser().getScreenName() + " - " + status.getText());
}
//#Override
public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
}
//#Override
public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
}
//#Override
public void onScrubGeo(long userId, long upToStatusId) {
System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
}
//#Override
public void onStallWarning(StallWarning warning) {
System.out.println("Got stall warning:" + warning);
}
//#Override
public void onException(Exception ex) {
ex.printStackTrace();
}
};
I have a specific requirement that i want to collect all the tweets according to the following parameters
1) Im using search API , for example i want to search for "Iphone6"
2) Region wise , ie if i specify the latitude and longitude as per city I get the results, is it possible to fetch all the results country wise( as in the code when i specity the latitude and longitude of india
it doesnt work !)
3) At what intervals should I run my application , so that I get the newly updated tweets , without getting the previously fetched tweets.
This is the code that I have written
public final class twitterdate {
public static void main(String[] args)throws Exception {
double res;
double lat,lon;
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true)
.setOAuthConsumerKey("MyKEY")
.setOAuthConsumerSecret("MySecret")
.setOAuthAccessToken("MyAccesstoken")
.setOAuthAccessTokenSecret("MyTokenSecret").setHttpConnectionTimeout(100000);
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
lat=18.9750; // THis works , but it doenst work when I specify latitude and longitude of India
lon=72.8258;
res=1;
try {
QueryResult result=twitter.search(new Query("iphone").since("2014-11-19").until("2014-11-22").geoCode(new GeoLocation(lat, lon), res,"1mi"));
// Since and untill doesnt work as expected sometimes it fetches the date tweets specified on "since" method sometimes fetches the tweets specified on the date of until method
// Also since and until doesnt work when i specify a time stamp.
List<Status> qrTweets = result.getTweets();
System.out.println("hi");
for (Status tweet : qrTweets )
{
System.out.println( tweet.getId() + " " + "#" + tweet.getUser().getScreenName() + " : " + tweet.getText() + " :::" + tweet.getCreatedAt() );
}
} catch (TwitterException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I would be greatful if somebody can help me with the requirement that I have as I have googled a lot but couldnt find any proper solution.
Thanks in advance !
Have you tried using FilterQuery? The code snippet below would give you a continuous stream of tweets.
As per my understanding you want to fetch tweets from India with content related to iphone6.
With Search query you may end up getting same set of tweets over and again.
You can try something like this below, I am not sure what co-ordinates you have used to fetch tweets from India !, you have to do some trial and error and fine tune the co-ordinates you want to locate your tweets around.
StatusListener listener = new StatusListener(){
public void onStatus(Status status) {
//if (status.getText().contains)
if(status.getUser().getLang().equalsIgnoreCase("en")
|| status.getUser().getLang().equalsIgnoreCase("en_US")) {
System.out.println(status.getUser().getName() + " :: " + status.getText() + " :: " + status.getGeoLocation());
}
}
public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {}
public void onTrackLimitationNotice(int numberOfLimitedStatuses) {}
public void onException(Exception ex) {
ex.printStackTrace();
}
public void onScrubGeo(long arg0, long arg1) {
}
public void onStallWarning(StallWarning arg0) {
}
};
ConfigurationBuilder config = new ConfigurationBuilder();
config.setOAuthConsumerKey("");
config.setOAuthConsumerSecret("");
config.setOAuthAccessToken("");
config.setOAuthAccessTokenSecret("");
TwitterStream twitterStream = new TwitterStreamFactory(config.build()).getInstance();
twitterStream.addListener(listener);
FilterQuery query = new FilterQuery();
// New Delhi India
double lat = 28.6;
double lon = 77.2;
double lon1 = lon - .5;
double lon2 = lon + .5;
double lat1 = lat - .5;
double lat2 = lat + .5;
double box[][] = {{lon1, lat1}, {lon2, lat2}};
query.locations(box);
String[] trackArray = {"iphone"};
query.track(trackArray);
twitterStream.filter(query);
There is however one caveat with FilterQuery that it uses location OR trackList for fetching data. To counter this may be you can put a content filter logic in onStatus() method.
Hope this helps you.
Ok. Each time we get only 100 tweets. I want to show them on a map based on the Geolocation. Is there any way I can get 100 tweets that all of them have Geolocation. Because now only 3-4 out of 100 has Geolocation.
I read some examples with FilterQuery but Processing throws me an error on it, as unknown.
Here is an working example. This was tested using twitter4j 4.0.2
The interface StatusListener is where you can do whatever you want with your tweets Kind of... is there that they arrive
Using stream API:
import twitter4j.util.*;
import twitter4j.*;
import twitter4j.management.*;
import twitter4j.api.*;
import twitter4j.conf.*;
import twitter4j.json.*;
import twitter4j.auth.*;
TwitterStream twitterStream;
double[][] boundingBox= {
{
-180, -90
}
, {
180, 90
}
}; /// whole world;
void setup() {
size(100, 100);
background(0);
openTwitterStream();
}
void draw() {
background(0);
}
// Stream it
void openTwitterStream() {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey(FILL_IN);
cb.setOAuthConsumerSecret(FILL_IN);
cb.setOAuthAccessToken(FILL_IN);
cb.setOAuthAccessTokenSecret(FILL_IN);
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
twitterStream.addListener(listener);
FilterQuery filter = new FilterQuery();
filter.locations(boundingBox);
twitterStream.filter(filter);
println("connected");
}
// Implementing StatusListener interface
StatusListener listener = new StatusListener() {
//#Override
public void onStatus(Status status) {
GeoLocation loc = status.getGeoLocation();
System.out.println("#" + status.getUser().getScreenName() + " - " + loc);
}
//#Override
public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
}
//#Override
public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
}
//#Override
public void onScrubGeo(long userId, long upToStatusId) {
System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
}
//#Override
public void onStallWarning(StallWarning warning) {
System.out.println("Got stall warning:" + warning);
}
//#Override
public void onException(Exception ex) {
ex.printStackTrace();
}
};
import twitter4j.conf.*;
import twitter4j.*;
import twitter4j.auth.*;
import twitter4j.api.*;
//import twitter4j.FilterQuery;
import java.util.*;
Twitter twitter;
String searchString = "shopping";
Query query;
ArrayList tweets,tweets2;
void setup()
{
frameRate(0.2);
tweets = new ArrayList();
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("*");
cb.setOAuthConsumerSecret("*");
cb.setOAuthAccessToken("*");
cb.setOAuthAccessTokenSecret("*");
twitter = new TwitterFactory(cb.build()).getInstance();
query = new Query(searchString);
getNewTweets();
}
void draw()
{
try{
if (current >= tweets.size()) //if u read all the received tweets make a new query
{
tweets.clear();
tweets2.clear();
getNewTweets();
current = 0;
}
Status currentTweet = (Status) tweets.get(current);
GeoLocation loc = currentTweet.getGeoLocation();
User currentUser=(User) currentTweet.getUser(); //Get User info
double latitude = currentTweet.getGeoLocation().getLatitude();
double longitude = currentTweet.getGeoLocation().getLongitude();
println( "Longtitude: "+longitude);
println( "Latitude: "+latitude);
String user = currentUser.getScreenName();
String msg = currentTweet.getText();
println( "User: "+user);
println("Message: "+ msg);
}
catch (Exception te) {
println(te);
}
}
void getNewTweets()
{
try
{
// FilterQuery filtro = new FilterQuery();
/* filtro.locations(boundingBox);
twitter.addListener(listener);
twitter.filter(filtro);
*/
query.setCount(100); //sets the number of tweets to return per page
QueryResult result = twitter.search(query);
tweets2 = (ArrayList) result.getTweets();
for (int i = 0; i < tweets2.size(); i++) {
Status currentTweet = (Status) tweets2.get(i);
GeoLocation loc = currentTweet.getGeoLocation();
if(loc!=null){ //add to list only tweets with geo location
tweets.add(tweets2.get(i));
}
}
}
catch (TwitterException te)
{
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
Not an answer but a Simple workaround:
I know most of people don't have GPS enabled when they tweet and others would not like to share their location!
But they are still sharing their location!! Guess how? On their profiles! Their hometown, their country is mostly visible, which can give you an approximate location of where the tweet came from! You can query for the user's profile and thus his/her location using the Rest API
twitter.showUser(userScreenName).getLocation();
I think if you setGeocode before u search,then the majority of result will has Geolocation(up to 90%). but i dont know why not whole of result with geolocation. maybe it will a little helpful?
GeoLocation obj = new GeoLocation(35.8007019, -97.3383211);
query.setGeoCode(obj, 2000, Unit.valueOf("km"));
result = twitter.search(query);