Rate limit exceeded - twitter

I ran my code that get some tweets with number = 50000 tweets but after getting some of them i got this error . I reviewed the below links in the error message but couldn't get anything help
[WARNING]
429:Returned in API v1.1 when a request cannot be served due to the application's rate limit having been exhausted for the resource. See Rate Limiting in API v1.1.(https://dev.twitter.com/docs/rate-limiting/1.1)
message - Rate limit exceeded
code - 88
Relevant discussions can be found on the Internet at:
http://www.google.co.jp/search?q=d35baff5 or
http://www.google.co.jp/search?q=12c94134
TwitterException{exceptionCode=[d35baff5-12c94134], statusCode=429,
message=Rate limit exceeded, code=88, retryAfter=-1,
rateLimitStatus=RateLimitStatusJSONImpl{remaining=0, limit=180,
resetTimeInSeconds=1497756414, secondsUntilReset=148}, version=3.0.3}
at
twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:177)
part 2 of error
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-
plugin:1.6.0:java (default-cli) An exception occured while executing the Java class. 429:Returned in
API v1.1 when a request cannot be served due to the application's rate
limit having been exhausted for the resource. See Rate Limiting in API
v1.1.(https://dev.twitter.com/docs/rate-limiting/1.1)
[ERROR] message - Rate limit exceeded
I found some similar posts without solutions except one that i didn't get it well and i'm not able to write a comment due to my reputation !

in twitter4j there is RateLimitStatus object. You can access this object after some api calls. For example:
User user = twitter.showUser(userId);
user.getRateLimitStatus();
//OR
IDs followerIDs = twitter.getFollowersIDs(user.getScreenName(), -1);
followerIDs.getRateLimitStatus();
//OR
QueryResult result = twitter.search(query);
result.getRateLimitStatus();
And maybe you can use a function to handle rate limit like this:
private void handleRateLimit(RateLimitStatus rateLimitStatus) {
//throws NPE here sometimes so I guess it is because rateLimitStatus can be null and add this condition
if (rateLimitStatus != null) {
int remaining = rateLimitStatus.getRemaining();
int resetTime = rateLimitStatus.getSecondsUntilReset();
int sleep = 0;
if (remaining == 0) {
sleep = resetTime + 1; //adding 1 more seconds
} else {
sleep = (resetTime / remaining) + 1; //adding 1 more seconds
}
try {
Thread.sleep(sleep * 1000 > 0 ? sleep * 1000 : 0);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
or maybe this:
private void handleRateLimit(RateLimitStatus rateLimitStatus) {
int remaining = rateLimitStatus.getRemaining();
if (remaining == 0) {
int resetTime = rateLimitStatus.getSecondsUntilReset() + 5;
int sleep = (resetTime * 1000);
try {
Thread.sleep(sleep > 0 ? sleep : 0);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Hope this helps.
Any other/better methods would also be appreciated.

Related

netmq TryReceiveMultipartMessage() works abnormal

I used the netmq (VisualStudio 2022, by Nuget install netmq) as https://www.nuget.org/packages/NetMQ/ described.
One SubscriberSocket one thread to connect and receive message from one publisher. source code like below:
public void ZMQReceiveThread(string serverIP, int port)
{
//Create SubscriberSocket
SubscriberSocket subSocket = new SubscriberSocket();
//Connect to Publisher
subSocket.Connect("tcp://" + serverIP + ":" + port.ToString());
//Subscribe all topics
subSocket.Subscribe("");
//set timeout value
int timeout = 10000 * 300; //300ms
TimeSpan ts = new TimeSpan(timeout);
while (!_isStopEnabled)
{
NetMQMessage recvMessage = null;
bool bSuccess = subSocket.TryReceiveMultipartMessage(ts, ref recvMessage, 1);
if(bSuccess == true) //Recieve data successfully
{
//Handle the recvMessage
}
else //Timeout
{
//output log message
Loger.Error($"TryReceiveMultipartMessage({ts.TotalMilliseconds} ms) timeout...");
continue;
}
}
}
sometimes the subSocket.TryReceiveMultipartMessage() timeout although the publisher sent message continuously (we used another test app written in C language linked libzmq(https://github.com/zeromq/libzmq) which can receive the continuous message).
Any comments about this topic?
thanks a lot in advance.
I looked through the netmq source code(https://github.com/zeromq/netmq) but cannot find any clues about TryReceiveMultipartMessage()

Graph API: What is the correct way to interrupt Pagination

I am using this script to fetch Chats. I need 100 chats maximum but it may happen that a chat do not have 100 messages. How can I handle that case in this script?
I am using Node Package Microsoft Graph Client.
const { Client, PageIterator } = require('#microsoft/microsoft-graph-client');
async getChatList(GroupChatId) {
let messages = [];
let count = 0;
let pauseAfter = 100; // 100 messages limit
let response = await this.graphClient
.api(`/chats/${GroupChatId}/messages`)
.version('beta')
.get();
let callback = (data) => {
messages.push(data);
count++;
return count < pauseAfter;
}
let pageIterator = new PageIterator(this.graphClient, response, callback);
await pageIterator.iterate();
return messages;
}
As I answered on the GitHub issue you opened, the iterator should stop all by itself if it runs out of items to iterate before hitting your "maximum". However, I think you're hitting a bug in the specific API you're using /chats/id/messages.
The problem is that this API is returning a nextLink value in it's response even if there are no next pages. It shouldn't be, and I'm reporting that to the Teams folks. That's causing the pageIterator to try to get the next set of results, which returns 0 items and a nextLink. You're stuck in an infinite loop.
So because of this, using the pageIterator just won't work for this API. You'll need to do the iteration yourself. Here's some TypeScript code to show it:
let keepGoing: Boolean = true;
do
{
// If there are no items in the page, then stop
// iterating.
keepGoing = currentPage.value.length > 0;
// Loop through the current page
currentPage.value.forEach((message) => {
console.log(JSON.stringify(message.id));
});
// If there's a next link follow it
if (keepGoing && !isNullOrUndefined(currentPage["#odata.nextLink"]))
{
currentPage = await client
.api(currentPage["#odata.nextLink"])
.get();
}
} while (keepGoing);
You need to check with a conditional statement if the message has value or not.
The pseudo code is given below:
let callback = (data) => {
if(data != "" || data != null)
{
messages.push(data);
count++;
return count < pauseAfter;
}
else{
return;
}
}

Get Publish Response/PubAck latency with paho org.eclipse.paho.client.mqttv3.MqttClient publish

I am using paho library Classes for Mqtt Connections org.eclipse.paho.client.mqttv3.MqttClient. (not MqttAsyncClient)
In my case when I publish using
mqttClient.publish(uid + "/p", new MqttMessage(payload.toString().getBytes()));
This method does the task for me but doesn't return anything so I can't check the latency between publish and pubAck.
To get the latency I use the following instead of directly calling publish function of mqttClient.
public long publish(JsonObject payload , String uid, int qos) {
try {
MqttTopic topic = mqttClient.getTopic(uid + "/p");
MqttMessage message = new MqttMessage(payload.toString().getBytes());
message.setQos(qos);
message.setRetained(true);
long publishTime = System.currentTimeMillis();
MqttDeliveryToken token = topic.publish(message);
token.waitForCompletion(10000);
long pubCompleted = System.currentTimeMillis();
if (token.getResponse() != null && token.getResponse() instanceof MqttPubAck) {
return pubCompleted-publishTime;
}
return -1;
} catch (Exception e) {
e.printStackTrace();
return -1;
}
}
This gets the work done, but I am not sure whether this is the right approach or not. Please let me know in case there is some other way to to do this.

Twitter4j get followers and following of any user

As a part of my final year project in university I'm analysing Twitter data using graph entropy. To briefly outline the purposes:
I want to collect all tweet from a certain area (London) containing keywords "cold", "flu" etc. This part is done using Streaming API.
Then I want to access each of the user's (who tweeted about being ill, collected in previous section) list of followers and following to be able to build a graph for further analysis. And here I'm stuck.
I assume for the second part I should be using Search API, but I keep getting error 88 even for a single user.
Below is the code I use for the first part:
final TwitterStream twitterStream = new TwitterStreamFactory(cb.build())
.getInstance();
StatusListener listener = new StatusListener() {
public void onStatus(Status status) {
User user = status.getUser();
long userid = user.getId();
String username = status.getUser().getScreenName();
String content = status.getText();
GeoLocation geolocation = status.getGeoLocation();
Date date = status.getCreatedAt();
if (filterText(content)) {
System.out.println(username+"\t"+userid);
System.out.println(content);
System.out.println(geolocation);
System.out.println(date);
try {
getConnections(userid);
} catch (TwitterException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
//OTHER LISTENER METHODS
};
twitterStream.addListener(listener);
// London
double lat3 = 51.23;
double lat4 = 51.72;
double lon3 = -0.56;
double lon4 = 0.25;
double[][] bb = { { lon3, lat3 }, { lon4, lat4 } };
FilterQuery fq = new FilterQuery();
fq.locations(bb);
twitterStream.filter(fq);
private static boolean filterText(String tweet) {
return tweet.contains("flu")
|| tweet.contains("cold")
|| tweet.contains("cough")
|| tweet.contains("virus");
}
And this is what I'm trying to complete the second part with:
private static void getConnections(long id) throws TwitterException {
Twitter twitter = new TwitterFactory().getInstance();
long lCursor = -1;
IDs friendsIDs = twitter.getFriendsIDs(id, lCursor);
System.out.println(twitter.showUser(id).getName());
System.out.println("==========================");
do
{
for (long i : friendsIDs.getIDs())
{
System.out.println("follower ID #" + i);
System.out.println(twitter.showUser(i).getName());
}
}while(friendsIDs.hasNext());
}
Any suggestions?
When you receive error 88, that's Twitter telling you that you're being rate limited:
The request limit for this resource has been reached for the current rate limit window.
The search call is limited to either 180 or 450 calls in a 15 minute period. You can see the rate limits here and this documentation explains the rate limiting in detail.
As for how to get around it, you may have to throttle your search calls to the API. Twitter4J provides ways to inspect current limits/exhaustion which may help - see Twitter#getRateLimitStatus().

Why is this iteration of an Outlook Folder only processing a maximum of half the number of items in the folder?

Outlook rules are putting all Facebook originating mail into a Facebook folder, an external process is running as detailed here to separate the contents of that folder in a way that was not feasible through Outlook rules process, originally I had this process running in VBA in outlook but it was a pig choking outlook resources. So I decided to throw it out externally and as I want to improve my c# skill set, this would be a conversion at the same time. Anyway the mail processing is working as it should items are going to correct sub-folders but for some reason the temporary constraint to exit after i number of iterations is not doing as it should. If there are 800 mails in the Facebook folder ( I am a member of many groups) it only runs through 400 iterations, if there are 30 it only processes 15 etc.
I cant for the life of me see why - can anyone put me right?
Thanks
private void PassFBMail()
{
//do something
// result = MsgBox("Are you sure you wish to run the 'Allocate to FB Recipient' process", vbOKCancel, "Hold up")
//If result = Cancel Then Exit Sub
var result = MessageBox.Show("Are you sure you wish to run the Are you sure you wish to run the 'Allocate to SubFolders' process","Sure?",MessageBoxButtons.OKCancel,MessageBoxIcon.Question,MessageBoxDefaultButton.Button2);
if (result == DialogResult.Cancel)
{
return;
}
try
{
OutLook._Application outlookObj = new OutLook.Application();
OutLook.MAPIFolder inbox = (OutLook.MAPIFolder)
outlookObj.Session.GetDefaultFolder(OutLook.OlDefaultFolders.olFolderInbox);
OutLook.MAPIFolder fdr = inbox.Folders["facebook"];
OutLook.MAPIFolder fdrForDeletion = inbox.Folders["_ForDeletion"];
// foreach (OutLook.MAPIFolder fdr in inbox.Folders)
// {
// if (fdr.Name == "facebook")
// {
// break;
// }
// }
//openFacebookFolder Loop through mail
//LOOPING THROUGH MAIL ITEMS IN THAT FOLDER.
Redemption.SafeMailItem sMailItem = new Redemption.SafeMailItem();
int i = 0;
foreach ( Microsoft.Office.Interop.Outlook._MailItem mailItem in fdr.Items.Restrict("[MessageClass] = 'IPM.Note'"))
{
//temp only process 500 mails
i++;
if (i == 501)
{
break;
}
// eml.Item = em
// If eml.To <> "" And eml.ReceivedByName <> "" Then
// strNewFolder = DeriveMailFolder(eml.To, eml.ReceivedByName)
// End If
sMailItem.Item = mailItem;
string strTgtFdr = null;
if (sMailItem.To != null && sMailItem.ReceivedByName != null)
{
strTgtFdr = GetTargetFolder(sMailItem.To, sMailItem.ReceivedByName );
}
// If fdr.Name <> strNewFolder Then
// If dDebug Then DebugPrint "c", "fdr.Name <> strNewFolder"
// eml.Move myInbox.Folders(strNewFolder)
// If dDebug Then DebugPrint "w", "myInbox.Folders(strNewFolder) = " & myInbox.Folders(strNewFolder)
// Else
// eml.Move myInbox.Folders("_ForDeletion")
// End If
if (fdr.Name != strTgtFdr)
{
OutLook.MAPIFolder destFolder = inbox.Folders[strTgtFdr];
mailItem.Move(destFolder);
}
else
{
mailItem.Move(fdrForDeletion);
}
}
//allocate to subfolders
//Process othersGroups
//Likes Max 3 per day per user, max 20% of group posts
//Comments one per day per user, max 10% of group posts
//Shares one per day per user, max 10% of group posts
}
catch(System.Exception crap)
{
OutCrap(crap);
MessageBox.Show("MailCamp experienced an issues while processing the run request and aborted - please review the error log","Errors during the process",MessageBoxButtons.OK,MessageBoxIcon.Error,MessageBoxDefaultButton.Button1);
}
}
Do not use a foreach loop it you are modifying the number of items in the collection.
Loop from MAPIFolder.Items.Count down to 1.

Resources