So I'm trying to build a real time monitoring tool for twitter key words using tweet sharp. I'm using the search API to collect queries every 10-15 seconds. When I make the calls, I only want to collect tweets that have appeared since the pervious update.
var twitter = FluentTwitter.CreateRequest().AuthenticateAs("username", "password").Search().Query().Containing("key word").Take(1000);
var response = twitter.Request();
currentResponseDateTime= Convert.ToDateTime(response.ResponseDate);
var messages = from m in response.AsSearchResult().Statuses
where m.CreatedDate > lastUpdateDateTime
select m;
lastUpdateDateTime = currentResponseDateTime;
My issue is that the twitter server time is different from the client times by a few seconds. I looked around and tried to get the datetime I recieved the response from the Response.ResponseDate property, but it looks like that is set based on the local computer time. I.e currentResponseDateTime is a few seconds ahead of the Twitter Server time. So I end up not collecting a few of the tweets.
Does anyone know how I can get the current server time from twitter search or REST API?
Thanks
I'm not sure how you would get the local server time of the twitter service, but one approach you could take is to store the date of the most recent twitter update seen in the "lastUpdateDateTime" field. That way, you're guaranteed to get all the messages since the last one you saw, regardless of the offset of the twitter server.
var twitter = FluentTwitter.CreateRequest().AuthenticateAs("username", "password").Search().Query().Containing("key word").Take(1000);
var response = twitter.Request();
currentResponseDateTime= Convert.ToDateTime(response.ResponseDate);
var messages = from m in response.AsSearchResult().Statuses
where m.CreatedDate > lastUpdateDateTime
select m;
lastUpdateDateTime = messages.Select(m => m.CreatedDate).Max();
Another approach (and one that Twitter recommends) is to pull the Date header from their API server's response, which provides Twitter's notion of time in GMT. This assumes that you can access the server response headers, and that depends on the method you're using to access the API.
For example, hitting https://api.twitter.com/1/help/test.json
$ lynx --dump --head https://api.twitter.com/1/help/test.json
HTTP/1.0 200 OK
Date: Tue, 22 Jan 2013 13:30:36 GMT
...
Reference: how to get the twitter server time (synchronize)? on dev.twitter.com support site.
Quoting Taylor Singletary:
The current time that Twitter "thinks" it is is returned in the "Date" HTTP header of every response to an API call you make. You can also issue a simple HTTP HEAD request to GET help/test to get the header as an initial syncing step for your app.
Related
I am working on a Rails app that has a form that makes 1st party API requests to Odoo v12, where I have all of my data stored. When a user submits 2 forms simultaneously, if there's any delay in the API response (as few as .2 seconds), the server begins work on the other request, which is expected.
However, when the responses come back in the order they were made, rails assign the response to the request with an open connection. (For e.g. response 1 is being assigned to request 2, and response 2 is being assigned to request 1.)
Inspecting the request_ids confirms that this is what is happening.
Example of created record IDs returning to the wrong request
res.partner create
Starting api call for res.partner on request
63f02dd4-8242-4043-9926-1e34133be2e1 at 1576616788523
[{:odoo_po=>"7643153000",
:quickbooks_invoice_url=>"", :deal_name=>"Delete ME PLEASE",
:deal_id=>"427759097", :reorder=>0, :company=>333, :sale_contact=>337,
:submitted_by=>98, :approver=>119, :hubspot_id=>"427759097",
:shipping_deadline=>"2020-01-07", :delivery_deadline=>"2020-01-13",
:knitting_deadline=>"2020-01-03", :internal_notes=>"",
"distributor_po"=>"14215",
"kit_arrays"=>"1,85996,15578;2,15578,85996;"}]
sale_form.sale_form create
Starting api call for sale_form.sale_form on request
673fa655-907a-416f-8ed9-7c97d220fc0b at 1576616788650
18805 _
673fa655-907a-416f-8ed9-7c97d220fc0b
51 ms api call for
sale_form.sale_form on request 673fa655-907a-416f-8ed9-7c97d220fc0b
9067 _ 63f02dd4-8242-4043-9926-1e34133be2e1
680 ms api call for res.partner
on request 63f02dd4-8242-4043-9926-1e34133be2e1
In the code above, the '9067' should be record_id associated with the sale_form.sale_form request(request_id 673fa655-907a-416f-8ed9-7c97d220fc0b), while the '18805' should be the record_id associated with the res.partner request(request_id 63f02dd4-8242-4043-9926-1e34133be2e1)
The code runs fine when forms are submitted more than a few seconds apart so I don't think it is an issue with the code I've written. Is this an issue with Rails/Puma/some other Gem? Or should I shift my focus towards how Odoo is handling the requests/responses?
So this is sort of weird. For every 1 request sent from my website using our YouTube API key, the developer console shows 102 queries actually being made. Here is the query format (using Python) -
search_q = '<query-string-here>'
service = build('youtube', 'v3', developerKey='<api-key>')
results = service.search().list(
part='snippet',
channelId='<specific-channel-id-to-search-through>',
type='video',
q=search_q,
).execute()
My logs show only one request being sent using this but my query count on the quotas page increases by 102.
Is there something I'm doing wrong? Or is this a bug on Google's end?
You can use the Quota Calculator to approximate the quota costs your request is using. Sure enough the search API request quota is on 100 range:
I've been trying to get this to work for probably 6 hours now to no avail, read every stackoverflow question I could find on the topic.
I'm trying to get 100, 200, or maybe 500 photos from a single tag:
func hashtags(hashtag: String, nextMaxTagId: String?) -> RequestParamters {
var params = "/tags/\(hashtag)/media/recent|access_token=\(accessToken)"
var parameters = Dictionary<String, AnyObject>()
parameters["access_token"] = accessToken
let urlString = "https://api.instagram.com/v1/tags/\(hashtag)/media/recent"
if let nextMaxTagId = nextMaxTagId {
params += "|max_tag_id=\(nextMaxTagId)"
parameters["max_tag_id"] = nextMaxTagId
}
let sig = HMAC.signWithKey(C.InstagramClientSecret(), usingData: params)
parameters["sig"] = sig
return (urlString: urlString, parameters: parameters)
}
This is what I use to construct my urls and parameters for my request. My first request does not have a nextMaxTagId, and that request goes through, returns 20 images and a pagination json.
Then, when I extract the next_max_tag_id from the pagination block, and create a request using that parameter, I get another 20 images, but they are the same images as before and now I do not get a pagination block.
I am signing my requests correctly (as all my other API requests throughout the app go through no problem) and I am not in Sandbox mode.
Edit: I've also tried using min_tag_id=\(nextMinTagId), still do not receive pagination in the next request.
Seems like:
1) You are using the Instagram Developer API with what seems like an authorized APIKey, and you mentioned you are NOT in Sandbox, so you're in a the Production environment for that api.
I'm trying to get 100, 200, or maybe 500 photos from a single tag
2) This means, combined with returns 20 images and a pagination json, that for 100, you need to make 5 calls minimum (100/20 == 5), 200 == 10, 500 = 25.
3) According to the developer documentation rate limits, the overall cap on Production is 5000 req/hour, with several APIs restricted to a much smaller limit (some are 30/60 req/hour). I'm not sure I see the exact tag rate limit you are hitting, but since the question mentions:
for probably 6 hours now to no avail
it's also possible you've just been hitting the overall hourly request limit each hour.
I definitely know that this is not an answer that I enjoy giving, because it's essentially saying: you're stuck. I've actually played with the rate limits myself before, and I find them extremely limiting (pun fully intended). The only other option, albeit not as "above board", is to scrape Instagram itself for the information you need. I say it's not as "above board" because if you needed info not found on a web scrape, you could theoretically scrape the mobile API through some minor reverse engineering (ie using an HTTP proxy to spoof mobile traffic systematically).
In the end, the API Instagram publishes is definitely very limited, and will face rate limits for the foreseeable future (unless you can get those somehow lifted in a specific partnership they somehow deem worthy, although I'm not sure how this could be approached).
I am using Zend's gdata library for the Google Apps provisioning API. Since Zend doesn't yet support fetching org users (no retrieve function provided by the library for this feed), I am making a custom gdata query to the url (as suggested in the documentation developers.google.com/google-apps/provisioning/#retrieving_organization_users_experimental):
apps-apis.google.com/a/feeds/orguser/2.0/'.$customerId.'?get=all
This works well for <= 100 users.
Now, I have created a domain with 125 users across 5 OUs. When I fetch the above URI, I get the 1st 100 users (as documented and expected). However, I could not find the pagination link mentioned here: developers.google.com/google-apps/provisioning/reference#Results_Pagination
Here's the start of my orguser feed:
<?xml version='1.0' encoding='UTF-8'?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' xmlns:apps='http://schemas.google.com/apps/2006'><id>https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx</id><updated>2013-01-06T08:17:43.520Z</updated><**link rel='next' type='application/atom+xml' href='https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx?get=all&startKey=RASS03jtnz0s2orxmbn.**'/><link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxx'/>
I tried the https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx?get=all&startKey=RASS03jtnz0s2orxmbn. link but it gives me the exact same 100 users that the https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx?get=all link gives. This is the only occurrence of the word "next" in my feed and so there is not other URI I can try to fetch the next 25 users.
So I have only been able to get 100 users from this API call. How do I go about fetching the next 25 users? Examples/code would be really appreciated. Or what am I doing wrong?
Please help - this is blocking an urgent delivery.
Thanks!,
Vinay.
Your 2nd request should look like:
https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxx?startKey=RASS03jtnz0s2orxmbn&get=all
startKey should be set to the value of the next parameter and get should continue to be all for each page request.
Also, make sure the URL is decoded, if & is encoded as & in the request, then Google's servers will see all of all&startKey=RASS03jtnz0s2orxmbn as the value of get and it won't see a startKey parameter at all.
I have problems with the new Twitter API: v1.0 is working without problems, but if I change the URL to v1.1 I get all the time a error "400 Bad request" (seen with Firebug).
Example:
https://api.twitter.com/1/statuses/user_timeline.json?screen_name=twitterapi
This is working like a charm, everything works as excepted.
Simply changing the URL to .../1.1/... and I get a Bad request error and even to JSON error response or even some content at all.
https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=twitterapi
Note: It couldn't be a rate limitation, because I accessed the URL the first time ever.
https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=twitterapi redirects me to https://api.twitter.com/1/statuses/user_timeline.json?screen_name=twitterapi
Looks like 1.1 is the same thing as 1
UPD: Looks like this is a rate limit (as 1.1 link worked for me 2 hours ago). Even if you hit API page for the first time, some of your apps (descktop or mobile) could use API methods.
UPD2: in 1.1 400 Bad request means you are not autorized (https://dev.twitter.com/docs/error-codes-responses, https://dev.twitter.com/docs/auth/oauth#user-context). So you need to get user context
You need to authenticate and authorize using oauth before using v1.1 apis
Here is something which works with python tweepy - gets statuses from users timeline
def twitter_fetch(screen_name = "BBCNews",maxnumtweets=10):
'Fetch tweets from #BBCNews'
# API described at https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline
consumer_token = '' #substitute values from twitter website
consumer_secret = ''
access_token = ''
access_secret = ''
auth = tweepy.OAuthHandler(consumer_token,consumer_secret)
auth.set_access_token(access_token,access_secret)
api = tweepy.API(auth)
#print api.me().name
#api.update_status('Hello -tweepy + oauth!')
for status in tweepy.Cursor(api.user_timeline,id=screen_name).items(2):
print status.text+'\n'
if __name__ == '__main__':
twitter_fetch('BBCNews',10)
For me the cause was the size of the media that was attached to the tweet. If it was <1.2MB it went through OK, but if it was over, I would get a 400 error every time.
Strange considering Twitter says the tweet limit is 3MB https://twittercommunity.com/t/getting-media-parameter-is-invalid-after-successfully-uploading-media/58354