I'm having trouble getting Twitter's rate_limit_status https://dev.twitter.com/docs/api/1/get/account/rate_limit_status to work correctly. I've used to curl and a web browser https://api.twitter.com/1/account/rate_limit_status.json to test it and it does returns a JSON object, but when I run the code below all I get is a 200 so I know I got the response successfully. It's not rated limited so even if I was over my limit it would still work. I've googled it and searched this forum but I can't seem to find why my specific code isn't working. As you can see I've put an alerts in the .getJSON function but neither goes off. I'm using the other Twitter APIs get Friends, get Followers successfully using basically the same code as below. I'm new to JQuery so I'm wondering if I'm missing something. Thanks in advance for any help!
function getRateLimit (){
var uri = "https://api.twitter.com/1/account/rate_limit_status.json";
$.getJSON(uri, function(data) {
alert("Alert before.");
console.log(data["reset_time"]);
console.log(data["remaining_hits"]);
alert("For testing purposes.");
});
}
The Same origin policy prevents that from working. Even if it would work, it would show the rate limit for each user and not for your application. (the rate limit status for the requester's IP address is returned)
You have to to get this data on server-side, everything else makes no sense.
Related
I'm working on a website to load multiple youtube channels live streams. At first i was trying to figure out a way to do this without utilizing youtube's api but have decided to give in.
To find whether a channel is live streaming and to get the live stream links I've been using:
https://www.googleapis.com/youtube/v3/search?part=snippet&channelId={CHANNEL_ID}&eventType=live&maxResults=10&type=video&key={API_KEY}
However with the minimum quota being 10000 and each search being worth 100, Im only able to do about 100 searches before I exceed my quota limit which doesn't help at all. I ended up exceeding the quota limit in about 10 minutes. :(
Does anyone know of a better way to figure out if a channel is currently live streaming and what the live stream links are, using as minimal quota points as possible?
I want to reload youtube data for each user every 3 minutes, save it into a database, and display the information using my own api to save server resources as well as quota points.
Hopefully someone has a good solution to this problem!
If nothing can be done about links just determining if the user is live without using 100 quota points each time would be a big help.
Since the question only specified that Search API quotas should not be used in finding out if the channel is streaming, I thought I would share a sort of work-around method. It might require a bit more work than a simple API call, but it reduces API quota use to practically nothing:
I used a simple Perl GET request to retrieve a Youtube channel's main page. Several unique elements are found in the HTML of a channel page that is streaming live:
The number of live viewers tag, e.g. <li>753 watching</li>. The LIVE NOW
badge tag: <span class="yt-badge yt-badge-live" >Live now</span>.
To ascertain whether a channel is currently streaming live requires a simple match to see if the unique HTML tag is contained in the GET request results. Something like: if ($get_results =~ /$unique_html/) (Perl). Then, an API call can be made only to a channel ID that is actually streaming, in order to obtain the video ID of the stream.
The advantage of this is that you already know the channel is streaming, instead of using thousands of quota points to find out. My test script successfully identifies whether a channel is streaming, by looking in the HTML code for: <span class="yt-badge yt-badge-live" > (note the weird extra spaces in the code from Youtube).
I don't know what language OP is using, or I would help with a basic GET request in that language. I used Perl, and included browser headers, User Agent and cookies, to look like a normal computer visit.
Youtube's robots.txt doesn't seem to forbid crawling a channel's main page, only the community page of a channel.
Let me know what you think about the pros and cons of this method, and please comment with what might be improved rather than disliking if you find a flaw. Thanks, happy coding!
2020 UPDATE
The yt-badge-live seems to have been deprecated, it no longer reliably shows whether the channel is streaming. Instead, I now check the HTML for this string:
{"text":" watching"}
If I get a match, it means the page is streaming. (Non-streaming channels don't contain this string.) Again, note the weird extra whitespace. I also escape all the quotation marks since I'm using Perl.
Here are my two suggestions:
Check my answer where I explain how you can check how retrieve videos from channels who are livesrteaming.
Another option could be use the following URL and somehow make request(s) each time for check if there's a livestreaming.
https://www.youtube.com/channel/<CHANNEL_ID>/live
Where CHANNEL_ID is the channel id you want check if that channel is livestreaming1.
1 Just notice that maybe the URL wont work in all channels (and that depends of the channel itself).
For example, if you check the channel_id UC7_YxT-KID8kRbqZo7MyscQ - link to this channel livestreaming - https://www.youtube.com/channel/UC4nprx9Vd84-ly7N-1Ce6Og/live, this channel will show if he is livestreaming, but, with his channel id UC4nprx9Vd84-ly7N-1Ce6Og - link to this channel livestreaming -, it will show his main page instead.
Adding to the answer by Bman70, I tried eliminating the need of making a costly search request after knowing that the channel is streaming live. I did this using two indicators in the HTML response from channels page who are streaming live.
function findLiveStreamVideoId(channelId, cb){
$.ajax({
url: 'https://www.youtube.com/channel/'+channelId,
type: "GET",
headers: {
'Access-Control-Allow-Origin': '*',
'Accept-Language': 'en-US, en;q=0.5'
}}).done(function(resp) {
//one method to find live video
let n = resp.search(/\{"videoId[\sA-Za-z0-9:"\{\}\]\[,\-_]+BADGE_STYLE_TYPE_LIVE_NOW/i);
//If found
if(n>=0){
let videoId = resp.slice(n+1, resp.indexOf("}",n)-1).split("\":\"")[1]
return cb(videoId);
}
//If not found, then try another method to find live video
n = resp.search(/https:\/\/i.ytimg.com\/vi\/[A-Za-z0-9\-_]+\/hqdefault_live.jpg/i);
if (n >= 0){
let videoId = resp.slice(n,resp.indexOf(".jpg",n)-1).split("/")[4]
return cb(videoId);
}
//No streams found
return cb(null, "No live streams found");
}).fail(function() {
return cb(null, "CORS Request blocked");
});
}
However, there's a tradeoff. This method confuses a recently ended stream with currently live streams. A workaround for this issue is to get status of the videoId returned from Youtube API (costs a single unit from your quota).
I found youtube API to be very restrictive given the cost of search operation. Apparently the accepted answer did not work for me as I found the string on non live streams as well. Web scraping with aiohttp and beautifulsoup was not an option since the better indicators required javascript support. Hence I turned to selenium. I looked for the css selector
#info-text
and then search for the string Started streaming or with watching now in it.
To reduce load on my tiny server that would have otherwise required lot more resources, I moved this test of functionality to a heroku dyno with a small flask app.
# import flask dependencies
import os
from flask import Flask, request, make_response, jsonify
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
base = "https://www.youtube.com/watch?v={0}"
delay = 3
# initialize the flask app
app = Flask(__name__)
# default route
#app.route("/")
def index():
return "Hello World!"
# create a route for webhook
#app.route("/islive", methods=["GET", "POST"])
def is_live():
chrome_options = Options()
chrome_options.binary_location = os.environ.get('GOOGLE_CHROME_BIN')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--remote-debugging-port=9222')
driver = webdriver.Chrome(executable_path=os.environ.get('CHROMEDRIVER_PATH'), chrome_options=chrome_options)
url = request.args.get("url")
if "youtube.com" in url:
video_id = url.split("?v=")[-1]
else:
video_id = url
url = base.format(url)
print(url)
response = { "url": url, "is_live": False, "ok": False, "video_id": video_id }
driver.get(url)
try:
element = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#info-text")))
result = element.text.lower().find("Started streaming".lower())
if result != -1:
response["is_live"] = True
else:
result = element.text.lower().find("watching now".lower())
if result != -1:
response["is_live"] = True
response["ok"] = True
return jsonify(response)
except Exception as e:
print(e)
return jsonify(response)
finally:
driver.close()
# run the app
if __name__ == "__main__":
app.run()
You'll however need to add the following buildpacks in settings
https://github.com/heroku/heroku-buildpack-google-chrome
https://github.com/heroku/heroku-buildpack-chromedriver
https://github.com/heroku/heroku-buildpack-python
Set the following Config Vars in settings
CHROMEDRIVER_PATH=/app/.chromedriver/bin/chromedriver
GOOGLE_CHROME_BIN=/app/.apt/usr/bin/google-chrome
You can find supported python runtime here but anything below python 3.9 should be good since selenium had problems with improper use of is operator
I hope youtube will provide better alternatives than workarounds.
I know this is a old thread, but i thought i share my way of checking to for example grab the status code to use in an app.
This is for a single Channel, but you could easly do a foreach with it.
<?php
#####
$ytchannelID = "UCd0BTXriKLvOs1ANx3puZ3Q";
#####
$ytliveurl = "https://www.youtube.com/channel/".$ytchannelID."/live";
$ytchannelLIVE = '{"text":" watching now"}';
$contents = file_get_contents($ytliveurl);
if ( strpos($contents, $ytchannelLIVE) !== false ){http_response_code(200);} else {http_response_code(201);}
unset($ytliveurl);
?>
Adding onto the other answers here, I use a GET request to https://www.youtube.com/c/<CHANNEL_NAME>/live and then search for "isLive":true (rather than {"text":" watching"})
i want to show public contents from instagram related to a specific hashtag (everything works fine with that) but i can't to renew the access_token everytime it expires.
("do not assume your access_token is valid forever." -
https://www.instagram.com/developer/authentication/)
To renew it manually is not an option i have to make sure there is a valid access_token at ANY time without re-authenticating.
Any ideas or questions? :)
I have one idea, but without API (and access_token). You can make requests to the web-version of Instagram with ?__a=1 parameter. I do not know how long it will work but now there is workflow:
You want to show public content with hashtag space, for example.
Add it to url and add GET-parameter ?__a=1: https://www.instagram.com/explore/tags/space/?__a=1
Make the GET-request. It returns json with nodes in top_posts (8) and media (18). Each node has owner, caption, number of comments and likes. But the most important part is in thumbnail_src and display_src.
There is page_info in media object which helps to paginate results. You need end_cursor (for example, J0HWE9rjAAAAF0HWE9qvgAAAFiYA)
Add the value from end_cursor to the url: https://www.instagram.com/explore/tags/space/?__a=1&max_id=J0HWE9rjAAAAF0HWE9qvgAAAFiYA
Repeat 3-6 to get newest posts with specific hashtag.
Update to the ?__a=1 url param. This appears to have stopped working with users '/account/?__a=1' endpoints.:( Still works on tags apparently.
Instagram shut down their public API. Here's a quick and dirty workaround in PHP:
<?php
function getPublicInfo($username) {
$url = sprintf("https://www.instagram.com/$username");
$content = file_get_contents($url);
$content = explode("window._sharedData = ", $content)[1];
$content = explode(";</script>", $content)[0];
$data = json_decode($content, true);
return $data['entry_data']['ProfilePage'][0];
}
Not sure for how long it's gonna work. Here's one for Javascript.
I can't seem to get the embedded URL in a status, for example, in id=780804331608109057 -
https://twitter.com/i/web/status/780804331608109057
When I retrieve this via GET /statuses/:id, with include_entities set to true, the response looks like this:
"text":"Here\u2019s WSJ \"An Underwhelming Trump-Clinton Debate\u201d trying to spin this as something other than a Clinton triumph\u2026 https:\/\/t.co\/dpkmphGI8k",
"truncated":true,
"entities":
{"hashtags":[],"symbols":[],"user_mentions":[],"urls":
[{"url":"https:\/\/t.co\/dpkmphGI8k",
"expanded_url":"https:\/\/twitter.com\/i\/web\/status\/780804331608109057",
"display_url":"twitter.com\/i\/web\/status\/7\u2026","indices":[114,137]}]},
"source":"\u003ca href=\"https:\/\/about.twitter.com\/products\/tweetdeck\"rel=\"nofollow\"\u003eTweetDeck\u003c\/a\u003e",....
When viewed on my web client, the status instead displays the link to WSJ (referred through t.co) What I would like is one or both of these URLs to show up in my API response:
https://pbs.twimg.com/media/CtX5Sz8WIAAm4tq.jpg
what would be the short URL that looks like "t.co" followed by "/HJs4kbmTKz" (I have to break this up so SO doesn't complain.)
What am I doing wrong here?
The incredibly fast response from a staffer on the TwitterCommunity website was most gratifying:
You need to use tweet_mode=extended for the new longer Tweet format.
Ref: https://twittercommunity.com/t/missing-media-property-in-entities/70388/4
A search on this new parameter yields the appropriate documentation on dev.twitter.com - more documentation links on this mode probably exist out there. The most significant change appears to be that the key text is no longer used in non-compatibility mode - that's where the status text is, and unless you turn on compatibility mode, you have to now use the key full_text
https://dev.twitter.com/overview/api/upcoming-changes-to-tweets
I am using Zend's gdata library for the Google Apps provisioning API. Since Zend doesn't yet support fetching org users (no retrieve function provided by the library for this feed), I am making a custom gdata query to the url (as suggested in the documentation developers.google.com/google-apps/provisioning/#retrieving_organization_users_experimental):
apps-apis.google.com/a/feeds/orguser/2.0/'.$customerId.'?get=all
This works well for <= 100 users.
Now, I have created a domain with 125 users across 5 OUs. When I fetch the above URI, I get the 1st 100 users (as documented and expected). However, I could not find the pagination link mentioned here: developers.google.com/google-apps/provisioning/reference#Results_Pagination
Here's the start of my orguser feed:
<?xml version='1.0' encoding='UTF-8'?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' xmlns:apps='http://schemas.google.com/apps/2006'><id>https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx</id><updated>2013-01-06T08:17:43.520Z</updated><**link rel='next' type='application/atom+xml' href='https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx?get=all&startKey=RASS03jtnz0s2orxmbn.**'/><link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxx'/>
I tried the https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx?get=all&startKey=RASS03jtnz0s2orxmbn. link but it gives me the exact same 100 users that the https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxxx?get=all link gives. This is the only occurrence of the word "next" in my feed and so there is not other URI I can try to fetch the next 25 users.
So I have only been able to get 100 users from this API call. How do I go about fetching the next 25 users? Examples/code would be really appreciated. Or what am I doing wrong?
Please help - this is blocking an urgent delivery.
Thanks!,
Vinay.
Your 2nd request should look like:
https://apps-apis.google.com/a/feeds/orguser/2.0/C00xxxxxx?startKey=RASS03jtnz0s2orxmbn&get=all
startKey should be set to the value of the next parameter and get should continue to be all for each page request.
Also, make sure the URL is decoded, if & is encoded as & in the request, then Google's servers will see all of all&startKey=RASS03jtnz0s2orxmbn as the value of get and it won't see a startKey parameter at all.
i've got an android app and a really simple web service that make an insert in a DB with 3 values.
the titanium code is most like the example given on the docs
var xhr = Ti.Network.createHTTPClient();
xhr.onload = function(e) {};
xhr.open('POST','http://www.micheleantonaci.altervista.org/test/foobar.php');
xhr.send({
"latitude":00100,
"longitude":10000,
"type":'From Nexus'
});
and the web service is just
<?php
$con=mysql_connect('http://www.micheleantonaci.altervista.org/','***','***');
mysql_select_db('***',$con);
if(isset($_REQUEST['submit']))
{
$latitude=$_POST['latitude'];
$longitude=$_POST['longitude'];
$kind=$_POST['type'];
$sql="insert into foobar (latitude,longitude,type) values ('$latitude','$longitude','$kind')";
$res=mysql_query($sql) or die (mysql_error());
}
?>
now, when i try the webservice giving the values with the browser it works good, but with the app I get no results at all, any suggestions? tha app doesn't crash or log any error
You must use PHP function json_decode to get values.
Try adding the header like this:
xhr.setRequestHeader("Content-Type", "application/json");
You should also do a var_dump($_POST) in your PHP to see what's in it, not sure you'll get your stuff in separated variables...
When I see a problem like this, I setup 'Charles' or a similar proxy and have the device send it's request through the proxy. Then you can see if the device is send what is expected.
You could also try
xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
The code is right, the problem was on the framework itself, at that day the post method wasn't working.
Nowadays it has been fixed.