I am trying to get the complete list of follower IDs for a twitter user. This user has 170,000+ followers. I am using the code bird library. Here's the code:
\Codebird\Codebird::setConsumerKey(TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET);
$cb = \Codebird\Codebird::getInstance();
$cb->setToken(OAUTH_TOKEN, OAUTH_SECRET);
$cb->setReturnFormat(CODEBIRD_RETURNFORMAT_ARRAY);
$next_cursor = -1;
while ($next_cursor) {
$results = $cb->followers_ids("screen_name=some_tw_name","count=5000","cursor=$next_cursor");
$next_cursor = $results['next_cursor_str'];
}
The first call returns 5000 IDs, as it's supposed to. Subsequent calls (through the while loop) return 5000 IDs, but they are almost all the same as the previous call (each new call I get 1 to about 10 news IDs, with the remainder the same as before).
Anyone else run into this issue? resolution?
I think the problem is that you're not really combining the results from each of the requests. so you could do something like this:
function getFollowers(){
$next_cursor = -1;
$output = array();
while ($next_cursor) {
$results = $cb->followers_ids(array("screen_name"=>"some_tw_name","count"=>5000,"cursor"=>$next_cursor));
if(!isset($results['errors'])){
foreach($results['ids'] as $id){
$output[]=$id;
}
}
$next_cursor = $results['next_cursor_str'];
}
return $output;
}
Note: I also changed the argument notation to an array as opposed to strings.
Related
I am just starting with Spring Reactor and want to implement something that I would call 'standard pagination', don't know if there is technical term for this. Basically no matter what start and end date is passed to method, I want to return same amound of data, evenly distributed.
This will be used for some chart drawing in the future.
I figured out rough copy with algorithm that does exactly that, unfortunatelly before I can filter results I need to either count() or take last index() and block to get this number.
This block is surelly not the reactive way to do this, also it makes flux to call DB twice for data (or am I missing something?)
Is there any operator than can help me and get result from count() somehow down the stream for further usage, it would need to compute anyway before stream can be processed, but to get rid of calling DB two times?
I am using mongoDB reactive driver.
Flux<StandardEntity> results = Flux.from(
mongoCollectionManager.getCollection(channel)
.find( and(gte("lastUpdated", begin), lte("lastUpdated", end))))
.map(d -> new StandardEntity(d.getString("price"), d.getString("lastUpdated")));
Long lastIndex = results
.count()
.block();
final double standardPage = 10.0D;
final double step = lastIndex / standardPage;
final double[] counter = {0.0D};
return
results
.take(1)
.mergeWith(
results
.skip(1)
.filter(e -> {
if (lastIndex > standardPage)
if (counter[0] >= step) {
counter[0] = counter[0] - step + 1;
return true;
} else {
counter[0] = counter[0] + 1;
return false;
}
else
return true;
}));
I am using corona (lua) with parse.com and I have hit a problem constructing an $in query using values from another table / array.
My code is a little like this:
local usersToFetch = {}
table.insert( usersToFetch, "KnVvDiV2Cj")
table.insert( usersToFetch, "Paf6LDmykp")
and the working query I want to perform is the following lua table (which will get encoded before heading to parse). As I said, this works when I am hard coding the values as shown
local queryTable = {
["where"] = {
["objectId"] = { ["$in"] = {"KnVvDiV2Cj","Paf6LDmykp" }}
},
["limit"] = 1000
}
My problem is how do I incorporate my 'usersToFetch' table in the above table so that it will work the same as hard coding the values?
I swore I tried that but clearly I did not.. I think that I placed it inside the braces whereas they are not needed which is where I went wrong.
Thanks, hjpotte92 - what you put worked fine but this is my final solution in a single declaration:
Where I was going wrong before was because I had too many braces ["objectId"] = { ["$in"] = { usersToFetch } }
local queryTable = {
["where"] = {
["objectId"] = { ["$in"] = usersToFetch}
},
["limit"] = 1000
}
I've created a simple script that reads through an xml file and posts the results to an SQL database. This works perfectly.
I've put a little if statement in the script to identify orders that have already been posted to SQL. Basically if the transactionID in the input array is higher than the highest transactionID on the SQL server it adds the row values to the output array.
It seems that I am missing a trick here because I am getting "TypeError: Cannot call method "getAttribute" of undefined. (line 18, file "Code")" when trying to compare the current xml row to the last transaction ID.
I've done some searching and whilst I can see people with similar problems the explanations don't make a whole lot of sense to me.
Anyway, here is the relevant part of the code. Note that this all works perfectly without the if() bit.
function getXML() {
var id = lastTransactionID();
var xmlSite = UrlFetchApp.fetch("https://api.eveonline.com/corp/WalletTransactions.xml.aspx?KeyID=1111&vCode=1111&accountKey=1001").getContentText();
var xmlDoc = XmlService.parse(xmlSite);
var root = xmlDoc.getRootElement();
var row = new Array();
row = root.getChild("result").getChild("rowset").getChildren("row");
var output = new Array();
var i = 0;
for (j=0;i<row.length;j++){
if(row[j].getAttribute("transactionID").getValue()>id){ //Produces: TypeError: Cannot call method "getAttribute" of undefined. (line 18, file "Code")
output[i] = new Array();
output[i][0] = row[j].getAttribute("transactionDateTime").getValue();
output[i][1] = row[j].getAttribute("transactionID").getValue();
output[i][2] = row[j].getAttribute("quantity").getValue();
output[i][3] = row[j].getAttribute("typeName").getValue();
output[i][4] = row[j].getAttribute("typeID").getValue();
output[i][5] = row[j].getAttribute("price").getValue();
output[i][6] = row[j].getAttribute("clientID").getValue();
output[i][7] = row[j].getAttribute("clientName").getValue();
output[i][8] = row[j].getAttribute("stationID").getValue();
output[i][9] = row[j].getAttribute("stationName").getValue();
output[i][10] = row[j].getAttribute("transactionType").getValue();
output[i][11] = row[j].getAttribute("transactionFor").getValue();
output[i][12] = row[j].getAttribute("journalTransactionID").getValue();
output[i][13] = row[j].getAttribute("clientTypeID").getValue();
i++;
}
}
insert(output,output.length);
}
I have seen my mistake and corrected.
Mistake was in the for loop.
for (j=0;i
I want to get all video information posible from Youtube for my proyect. I know that the limit page is 100.
I do the next code:
ArrayList<String> videos = new ArrayList<>();
int i = 1;
String peticion = "http://gdata.youtube.com/feeds/api/videos?category=Comedy&alt=json&max-results=50&page=" + i;
URL oracle = new URL(peticion);
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(
yc.getInputStream()));
String inputLine = in.readLine();
while (in.readLine() != null)
{
inputLine = inputLine + in.readLine();
}
System.out.println(inputLine);
JSONObject jsonObj = new JSONObject(inputLine);
JSONObject jsonFeed = jsonObj.getJSONObject("feed");
JSONArray jsonArr = jsonFeed.getJSONArray("entry");
while(i<=100)
{
for (int j = 0; j < jsonArr.length(); j++) {
videos.add(jsonArr.getJSONObject(j).getJSONObject("id").getString("$t"));
System.out.println("Numero " + videosTotales + jsonArr.getJSONObject(j).getJSONObject("id").getString("$t"));
videosTotales++;
}
i++;
}
When the program finish, I have 5000 videos per category, but I need much more, much much more, but the limit is page = 100.
So, how can I get more than 10 millions of videos?
Thank you!
Are those 5000 also unique id's ?
I see the use of max-results=50, but not a start-index parameter in your url.
There is a limit on the results you can get per request. There is also a limit on the number of requests that you can send within some time interval. By checking the statuscode of the response and any error message you can find these limits, as they may change in time.
Besides the category parameter, use some other parameters too. For instance, you may vary the q parameter (used with some keywords) and/or order parameter to get a different results set.
See the documentation for available parameters.
Note, that you are using api version 2, which is deprecated. There is an api version 3.
I'm pulling a paged dataset on an ASP.NET MVC3 application which uses JQuery to get data for endlesss scroll paging via $ajax call. The backend is a Azure SQL database. Here is the code:
[Authorize]
[OutputCache(Duration=0,NoStore=true)]
public PartialViewResult Search(int page = 1)
{
int batch = 10;
int fromRecord = 1;
int toRecord = batch;
if (page != 1)
{
//note these are correctly passed and set
toRecord = (batch * page);
fromRecord = (toRecord - (batch - 1));
}
IQueryable<TheTable> query;
query = context.TheTable.Where(m => m.Username==HttpContext.User.Identity.Name)
.OrderByDescending(d => d.CreatedOn)
.Skip(fromRecord).Take(toRecord);
//this should always be the batch size (10)
//but seems to concatenate the previous results ???
int count = query.ToList().Count();
//results
//call #1, count = 10
//call #2, count = 20
//call #3, count = 30
//etc...
PartialViewResult newPartialView = PartialView("Dashboard/_DataList", query.ToList());
return newPartialView;
}
The data returned from each call from Jquery $ajax continues to GROW on each subsequent call rather then returning only 10 records per call. So the results return contain all of the earlier calls data as well. Also, I've added 'cache=false' to the $ajax call as well. Any ideas on what is going wrong here?
The values you're passing to Skip and Take are wrong.
The argument to Skip should be the number of records you want to skip, which should be 0 on the first page;
The argument to Take needs to be the number of records you want to return, which will always be equal to batch;
Your code needs to be:
int batch = 10;
int fromRecord = 0;
int toRecord = batch;
if (page != 1)
{
fromRecord = batch * (page - 1);
}