I'm on migration from the old Google AdWords API to the new Google Ads API, using PHP-SDK by Google.
This is the use case, where I'm stuck:
I feed an amount of keywords (paginating them by keyword plans a 10k) to generateHistoricalMetrics($keywordPlanResource) and collect the results.
To do so I followed instructions at https://developers.google.com/google-ads/api/docs/keyword-planning/generate-historical-metrics and, especially, https://developers.google.com/google-ads/api/docs/keyword-planning/generate-historical-metrics#mapping_to_the_ui, with using of KeywordPlanAdGroupKeywords (with a single ad group) and avoiding to pass a specific date range for now, relying on the default value.
Further I had to apply some filters on my keywords because of KEYWORD_HAS_INVALID_CHARS and KEYWORD_TEXT_TOO_LONG, but all the errors which I'm aware of are gone now.
Now, I found out, that the KeywordPlanHistoricalMetrics object does not contain any keyword id (of the form customers//keywordPlanAdGroupKeywords/) So, I have to rely on the correct ordering. This is ok as it seems, that the original ordering of keywords is preserved within the results, as here https://developers.google.com/protocol-buffers/docs/encoding#optional
But still I have the problem, that
count($keywordPlanServiceClient->generateHistoricalMetrics($keywordPlanResource)->getMetrics()) is lower then count($passedKeywords), where each of $passedKeywords where passed to
new KeywordPlanAdGroupKeyword([
'text' => $passedKeyword,
'match_type' => KeywordMatchType::EXACT
'keyword_plan_ad_group' => $planAdGroupResource
]);
Q: So I have two questions here:
Why getMetrics() does not yield the same amount of results as the amount of passed keywords?
I'm struggling with debugging at this moment: Say, I want to know which keywords are let out. Either for providing more information at this place or just to skip them, and let my customer know, that these particular keywords were not queried. How to do this, when although I have a keyword id for every passed keyword I cannot match the returned metrics to them, because the KeywordPlanHistoricalMetrics object does not contain any keyword id.
Detail: While testing I found out, that the reducing of an amount of queried keywords reduces the amount of lost keyword data:
10k of queried keywords - 4,72% loss,
5k - 2,12%,
2,5k - 0,78%,
1,25k - 0,43%,
625 - 0,3%,
500 - 0,24%,
250 - 0,03%
200 - 0,03% of lost keywords.
But I can't imagine, that keywords should be queried one by one.
Related
I use the JSON data from a Google spreadsheet, for 2 mobile applications (iOS and Android). The same information can be outputted using HTML or XML, in this case I am using HTML so the information shown (from the spreadsheet) can be understood by everyone. The only logical way to do this is without Authentication (O’Auth) is through public URL Injects. Information about what I’m talking can be found here. In order to understand what I’m asking, you have to actually click the links and see for yourself. I do not know what to “call” some of the things I’m asking as Google’s documation is poor, no fault of my own.
In my app I have a search feature that queries the spreadsheet (USING A URL REQUEST) along the lines of this,
https://docs.google.com/spreadsheets/d/1yyHaR2wihF8gLf40k1jrPfzTZ9uKWJKRmFSB519X8Bc/gviz/tq?tqx=out:html&tq=select+A,B,C,D,E+where+(B+contains"Cat")&gid=0
I select my columns (A, B, C, D, and, E) and ask (Google) that only the rows where column B contains the word cat be return. Again I’m stressing the point that this is done via a URL address (inject being the proper term). I CANNOT use almost any function/formulas that would normally work within a spreadsheet like, ArrayFormula or ImportRange. In fact I only access to 10 language clauses (Read link from before). I have a rather well knowledge of spreadsheets and databases, and as the URL method of getting information from them is similar they are in NO way the same thing.
Now, I would like to point out this part within the URL
tq?tqx=out:html&tq=select+A,B,C,D,E+where+(B+contains"Cat")&gid=0
Type of output, HTML in this case
tqx=out:html
The start of query
&tq=
Select columns A-E
select+A,B,C,D,E
For returning specific information about Cat
where+(B+contains"Cat")
This is probably the most important part of my question. This is used for specifying what table (Tab) is being queried.
&gid=0
If the gid is changed from gid=0 to gid=181437435 the data returned is from the spreadsheets second table. Instead of having to make 2 requests to search both tables is there a way to do both in one request? (like combining the 2) <— THIS IS WHAT I’M ASKING.
There is a AND clause that I have tried all over the url
select+A,B,C,D,E+where+(B+contains%20"Cat")&gid=181437435+AND+select+A,B,C,D,E+where+(B+contains%20"Cat")&gid=0
I have even flipped the gid around and put in other places but it seems to only go by the last one (gid) in the url, and no matter what is done only 1 table is returned. Grouping is allowed by the way. If that doesn’t clear my question up then let me know where you’re lost. Also I would have posted more URLs for easy access but I am kind of on this 2 URL maximum program.
If I understand your requirement, indeed it is, with syntax like this for example:
=ArrayFormula(QUERY({Sheet1!A1:C4;Sheet2!B1:D4},"select * order by Col1 desc"))
The ; stacks one array above the other (, for side by side).
My confusions is with "URL Query Language" as what here is called Google Query Language (there is even the tag though IMO almost all those Qs belong on Web Applications - including this one, by my understanding!) is not confined to use with URLs.
In the example above the sheet references might be replaced with data import functions.
I am making a bulk call with 30 posts and daily data of all. Is there any limits to the number of rows that will be returned by the API?
I am having problem getting the results.
Can anyone please help.
YouTube doesn't return any rows ... it's not relational data. That may sound like a pedantic thing to point out, but it's crucial for this next point; the API will return 50 videos at a time, along with tokens to get more results based on the same query, up to a total of 500 ... because the data isn't relational, you can't just "select all rows" that match a certain criteria. Rather, it is probabilistically determining relevance to your search parameters, and after about 500 results the algorithms don't have enough certainty to make additional results relevant.
So in your case, where you can change the date as needed (to allow the algorithms to be more specific), you'll want to do a series of calls; perhaps one at a time (since you have to paginate anyway to get more than 50 results, it's probably not that much more expensive in terms of network bandwidth).
I am currently trying to pull data about videos from a YouTube user upload feed. This feed contains all of the videos uploaded by a certain user, and is accessed from the API by a request to:
http://gdata.youtube.com/feeds/api/users/USERNAME/uploads
Where USERNAME is the name of the YouTube user who owns the feed.
However, I have encountered problems when trying to access feeds which are longer than 1000 videos. Since each request to the API can return 50 items, I am iterating through the feed using max_length and start_index as follows:
http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?start-index=1&max-results=50&orderby=published
http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?start-index=51&max-results=50&orderby=published
And so on, incrementing start_index by 50 on each call. This works perfectly up until:
http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?start-index=1001&max-results=50&orderby=published
At which point I receive a 400 error informing me that 'You cannot request beyond item 1000.' This confused me as I assumed that the query would have only returned 50 videos: 1001-1051 in the order of most recently published. Having looked through the documentation, I discovered this:
Limits on result counts and accessible results
...
For any given query, you will not be able to retrieve more than 1,000
results even if there are more than that. The API will return an error
if you try to retrieve greater than 1,000 results. Thus, the API will
return an error if you set the start-index query parameter to a value
of 1001 or greater. It will also return an error if the sum of the
start-index and max-results parameters is greater than 1,001.
For example, if you set the start-index parameter value to 1000, then
you must set the max-results parameter value to 1, and if you set the
start-index parameter value to 980, then you must set the max-results
parameter value to 21 or less.
I am at a loss about how to access a generic user's 1001st last uploaded video and beyond in a consistent fashion, since they cannot be indexed using only max-results and start-index. Does anyone have any useful suggestions for how to avoid this problem? I hope that I've outlined the difficulty clearly!
Getting all the videos for a given account is supported, but you need to make sure that your request for the uploads feed is going against the backend database and not the search index. Because you're including orderby=published in your request URL, you're going against the search index. Search index feeds are limited to 1000 entries.
Get rid of the orderby=published and you'll get the data you're looking for. The default ordering of the uploads feed is reverse-chronological anyway.
This is a particularly easy mistake to make, and we have a blog post up explaining it in more detail:
http://apiblog.youtube.com/2012/03/keeping-things-fresh.html
The nice thing is that this is something that will no longer be a problem in version 3 of the API.
I'd like to retrieve the tweets for given a hashtag and sort them from the most retweeted to the less retweeted.
The closest thing I've found is using the search call and use the type tag:
E.g.: http://search.twitter.com/search.json?q=TheHashTagHere&result_type=popular
However, I'm not sure on how "popular" option works.
For instance, if it finds 100 tweets with that hashtag I believe it should show the X most retweeted tweets, and if none of those tweets have been retweeted then it should show X of them randomly (or sorted in some other way like the most recent).
Unfortunately, if follows some kind of unknown rule to identify what's popular and what not and even hashtags with thousands of tweets might return only one or two results.
I hope I made myself clear. Thanks in advance :)
PS: I'll use PHP but I think that shouldn't affect the question?
Results will sometimes contain a
result_type field into the metadata
with a value of either "recent" or
"popular". Popular results are derived
by an algorithm that Twitter computes,
and up to 3 will appear in the default
mixed mode that the Search API
operates under. Popular results
include another node in the metadata
called recent_retweets. This field
indicates how many retweets the Tweet
has had.
Source (Emphasis are mine)
Just call with result_type=popular and check the recent_retweets node to see how popular it is. result_type=popular will become the default in an upcome release so beware if you omit this parameter.
Results with popular tweets aren't ordered chronologically. *
If you would like to always have results to show, use result_type=mixed: they will have the result_type in the "metadata" section with a value of "recent", and popular results will have "popular". A small reference about result_types:
mixed: Include both popular and real time results in the response.
recent: return only the most recent results in the response
popular: return only the most popular results in the response.
If a search query has any popular results, those will be returned at the top, even if they are older than the other results. *
*[Twitter API Announcements]
This isn't a programmatic method but rather works in the browser with a chrome extension (HackyBird) :
Install the extension
Search for a phrase e.g. #Social (twitter.com/search?q=%23Social)
Click the extension to sort it (you can adjust the ratio of retweets/likes used for sorting in extension options).
P.S. It'll also sort your or any other user's timeline.
I would like to get the adjusted price (adjusting for splits and dividends) for a group of stock symbols using Yahoo! Finance. It looks like the historical prices call is limited to one symbol at a time. Could please let me know if there is a way to get multiple symbols in one call?
I would like to get this data so I can do some back testing on that data. Since I may require quite a few symbols (say 500-1000), it will be easier if I can make just a few batch calls to Yahoo!'s servers instead of making one call per symbol everyday.
Another way of getting the adjusted price is to use their daily stock price api and adjust it manually using dividend and splits information (they allow multiple symbols for their daily stock quotes). Unfortunately I cannot find any way to get splits information from the http call (guessing based on 50% or 200% is one option but if you deal with penny stocks, this can be dangerous and cannot figure out uneven splits). Also, the dividend information returned by it is not easy to decode. They seem to be returning the total over 4 quarters and the dividend date doesn't really correspond with the actual dividend date based on the historical price. The various options for the call can be found here: http://www.gummy-stuff.org/Yahoo-data.htm
Any suggestions on getting adjusted price for multiple symbols? Or Am I unnecessarily worrying about making 100s of calls to Yahoo! everyday? Ideally I would like to download all the required data within a couple of hours each day - that would be 10-20 calls per minute. Is that too much? I couldn't find any documentation on the permissible number of requests per second.
I am open to other places where I can get similar data. However, since I am just trying to learn the basics of quant trading and not trade, I would prefer free downloads.
Thanks
-e
This is an old question, but I did find a source where split data is available. Not sure how comprehensive these announcements are though:
http://biz.yahoo.com/c/09/s1.html
In the url, the "09" part is the year (2009), and the "s1" part is the month (s1 = Jan, s2 = Feb., s3 = Mar., etc.)
It isn't a nice clean CSV, but the format of the page is consistent and should be parseable. Just make a query each day for the current month, parse the page, and process any splits that you didn't see the day before.
ETA: And another source (probably less reliable than Yahoo, but can be queried by ticker):
http://getsplithistory.com/
I am not sure which language you are using but I have a sample in C#. I think it will give you the idea at least or may be help some one else
private string BASE_URL = "http://query.yahooapis.com/v1/public/yql?q=" + "select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20({0})" + "&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";
Collection<Quote> quotes;
string symbolList = String.Join("%2C", quotes.Select(w => "%22" + w.Symbol + "%22").ToArray());
string url = string.Format(BASE_URL,symbolList);
XDocument doc = XDocument.Load(url);
Parse(quotes,doc);
What we are doing here is appending "," to each array item then passing that symbol list to yahoo. I have successfully fetched prices for 700 symbols in each call. Hitting yahoo servers for each ticker is a pain. I fetch stock prices for all of 6500+ tickers everyday. Earlier it use to take 3 hours now it is less than 2 mins.....sweet
Source link for that code is here - http://www.jarloo.com/get-yahoo-finance-api-data-via-yql/
P.S. Please get a api key to work smoothly. The above url is a public link where tables are timed out most of the time. Once you get an api key then your url will be (minus "public")
http://query.yahooapis.com/v1/yql