I have been trying to use YQL to access the fundamentals of listed companies. But what is showing up in the Yahoo finance page is not being returned from YQL queries. Specifically I need to retrieve data from balance sheet and income statement, a sample query for Apple in YQL is like:
SELECT * FROM yahoo.finance.balancesheet WHERE symbol='AAPL'
This, however, only returns a time frame (quarterly), and nothing else.
Link to YQL console of my sample query is here
Is the data inaccessible to YQL or is there anything wrong with the way I am running the query? How can I get a complete list of data as in http://finance.yahoo.com/q/bs?s=AAPL through YQL?
Your query used to work fine. However, a month or two ago, a number of yahoo.finance YQL "tables" stopped working.
IOW, you are doing it right, but YQL is broken.
If you mouse over the yahoo.finance.balancesheet entry in the left column of the YQL console, buttons labeled desc and src appear. If you click src, it fetches the scraping code for you: http://www.datatables.org/yahoo/finance/yahoo.finance.balancesheet.xml. To make the E4X JavaScript legible, right-click and select View Source or use wget or curl from the command line.
Notice that the code fetches http://finance.yahoo.com/q/bs?s=AAPL&quarterly and then uses an XPath query to find the data:
var query = y.xpath(rawresult, "//table[#class='yfnc_tabledata1']/tr/td/table/tr");
If you fetch the page into your browser and inspect the HTML, you find that there is indeed a table with class yfnc_tabledata1. However, it has no tr direct child. Apparently, Yahoo must have decided to add a tbody element. That probably explains why the query no longer scrapes any data.
The code page lists Ryan Hoium as the author. A little googling leads to the github repository where the code lives, alongside the code for the other Yahoo Finance tables.
Sadly, only the yahoo.finance.sectors table has received recent attention. The change was to add double slashes to its XPath expression. Double-slashes relax the "direct child" requirement, allowing, for example, tr to still be found even if there is an intervening tbody. However, it appears the new version has not been pushed out to the public site.
Related
Im trying to scrape this website (https://kamadan.gwtoolbox.com/) with google sheets for material costs for a game that I play. There are two tables; "Common Materials" and "Rare Materials" in a drop down in the top right corner. I am trying to pull the values for both as the prices update. I copied the full Xpath and used the function below in an empty cell on a sheet.
=importxml("https://kamadan.gwtoolbox.com/","/html/body/div[2]/div[1]/div/div[2]/table/tbody")
This returns a #N/A error saying it is returning an empty value.
I also tried it with the regular xpath...
=importxml("https://kamadan.gwtoolbox.com/","//*[#id='trader-overlay-items']")
Which just returns a blank cell. I have also tried both methods using the inspect function through chrome on the ancestors and children they return either of the two errors above.
Sorry if this is a really easy one. I am not familiar at all with Xpaths or html. I mostly dabble in VBA in excel.
Answer:
IMPORTXML can not retrieve data which is populated by a script, and so using this formula to retrieve data from this table is not possible to do.
More Information:
As you've already mentioned, you can attempt to get the data directly from the table using:
=IMPORTXML("https://kamadan.gwtoolbox.com/","//table[#id='trader-overlay-items']")
Which just gets a blank cell.
I went a step further and tried to reverse-engineer this by calling IMPORTXML on the HTML elements on the page in steps:
=IMPORTXML("https://kamadan.gwtoolbox.com/","html")
=IMPORTXML("https://kamadan.gwtoolbox.com/","html/body")
=IMPORTXML("https://kamadan.gwtoolbox.com/","html/body/div[1]")
=IMPORTXML("https://kamadan.gwtoolbox.com/","html/body/div[1]/div[0]")
...
html/body/div[1]/div[0] is the first path which gives no imported content, and we can see from importing html/body that the full body does not contain the imformation and only a template of it - in cell B1 we have references to 'Common materials' and 'Rare materials':
And in D1 we start to see JavaScript and JSON objects which are not called by IMPORTXML and so the results of which can not be retrieved:
As you can see if you disable JavaScript on the site, almost nothing is actually rendered and so can't be obtained using IMPORTXML:
References:
IMPORTXML - Docs Editors Help
I use the JSON data from a Google spreadsheet, for 2 mobile applications (iOS and Android). The same information can be outputted using HTML or XML, in this case I am using HTML so the information shown (from the spreadsheet) can be understood by everyone. The only logical way to do this is without Authentication (O’Auth) is through public URL Injects. Information about what I’m talking can be found here. In order to understand what I’m asking, you have to actually click the links and see for yourself. I do not know what to “call” some of the things I’m asking as Google’s documation is poor, no fault of my own.
In my app I have a search feature that queries the spreadsheet (USING A URL REQUEST) along the lines of this,
https://docs.google.com/spreadsheets/d/1yyHaR2wihF8gLf40k1jrPfzTZ9uKWJKRmFSB519X8Bc/gviz/tq?tqx=out:html&tq=select+A,B,C,D,E+where+(B+contains"Cat")&gid=0
I select my columns (A, B, C, D, and, E) and ask (Google) that only the rows where column B contains the word cat be return. Again I’m stressing the point that this is done via a URL address (inject being the proper term). I CANNOT use almost any function/formulas that would normally work within a spreadsheet like, ArrayFormula or ImportRange. In fact I only access to 10 language clauses (Read link from before). I have a rather well knowledge of spreadsheets and databases, and as the URL method of getting information from them is similar they are in NO way the same thing.
Now, I would like to point out this part within the URL
tq?tqx=out:html&tq=select+A,B,C,D,E+where+(B+contains"Cat")&gid=0
Type of output, HTML in this case
tqx=out:html
The start of query
&tq=
Select columns A-E
select+A,B,C,D,E
For returning specific information about Cat
where+(B+contains"Cat")
This is probably the most important part of my question. This is used for specifying what table (Tab) is being queried.
&gid=0
If the gid is changed from gid=0 to gid=181437435 the data returned is from the spreadsheets second table. Instead of having to make 2 requests to search both tables is there a way to do both in one request? (like combining the 2) <— THIS IS WHAT I’M ASKING.
There is a AND clause that I have tried all over the url
select+A,B,C,D,E+where+(B+contains%20"Cat")&gid=181437435+AND+select+A,B,C,D,E+where+(B+contains%20"Cat")&gid=0
I have even flipped the gid around and put in other places but it seems to only go by the last one (gid) in the url, and no matter what is done only 1 table is returned. Grouping is allowed by the way. If that doesn’t clear my question up then let me know where you’re lost. Also I would have posted more URLs for easy access but I am kind of on this 2 URL maximum program.
If I understand your requirement, indeed it is, with syntax like this for example:
=ArrayFormula(QUERY({Sheet1!A1:C4;Sheet2!B1:D4},"select * order by Col1 desc"))
The ; stacks one array above the other (, for side by side).
My confusions is with "URL Query Language" as what here is called Google Query Language (there is even the tag though IMO almost all those Qs belong on Web Applications - including this one, by my understanding!) is not confined to use with URLs.
In the example above the sheet references might be replaced with data import functions.
In Google sheets there is a function called IMPORTHTML which allows you to return tables and lists from HTML pages. However I have no idea how to find out the indexes of the tables or lists I am looking at. Is there a way to figure out ALL of the tables and lists quickly and there respective indexes? I have tried looking at the source on some of the web pages in Chrome and it is unreadable due to how they load the page but somehow the google function knows how to get the 17th table for example. This is a generic question and doesn't apply to any one website but rather ANY website I might want to extract some unknown table number. Right now I am just brute forcing through a bunch of indices which doesn't seem right.
This won't find the exact table you want, but it will give you a list of the tables.
From google developers console enter below to find table numbers
var i = 1; [].forEach.call(document.getElementsByTagName("table"), function(x) { console.log(i++, x); });
I have created and API on deevloper.yahoo.com and in yql console https://developer.yahoo.com/yql/console.
Tick show community table and search for finance.
I got result of stock data.Here I got exact data for historical data in josn and xml format.
Link for historical data yahoo.finance.historicaldata.
like I got all data
http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices and rest query for this stock is
yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22YHOO%22%20and%20startDate%20%3D%20%222009-09-11%22%20and%20endDate%20%3D%20%222010-03-10%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=
when I try to other stocks like yahoo.finance.balancesheet and other stock
I see the parameter I send in url is wrong specially where condition in other stock option
Can any one tell what will be exact url for other stock option.
What you appear to want is:
https://query.yahooapis.com/v1/public/yql?q=select * from yahoo.finance.historicaldata where symbol = "YHOO" and startDate = "2009-09-11" and endDate = "2010-03-10"&format=json&diagnostics=true&env=store://datatables.org/alltableswithkeys&callback=
You can feed that URL to, for instance, wget, and it will return a blob of JSON.
Where did I get that URL? I started the YQL Console with Show Community Tables checked. Next, I entered your query (the q parameter in the above) and received back valid looking JSON. Finally, I went down to the bottom of the page, where it says THE REST QUERY in green, and copied the URL out of the box.
I've posted a demo Access db at http://www.derekbeck.com/Database0.accdb . I'm using Access 2007.
I am importing an excel spreadsheet, which my organization gets weekly, importing it into Access. It gets imported the table [imported Task list]. From there, an append query reformats it and appends it to my [Master Task List] table.
Previously, we have had a form, where we would manually go through the newest imports, and manually select whether our department was the primary POC for a tasking. I want to automate this.
What syntax do I require, such that the append query will parse the text from [imported Task list].[Department], searching for those divisions listed on [OurDepartments] table (those parts of our company for which we are tracking these tasks), and then select the appropriate Lookup field (connected to [OurDepartments] table) in our [Master Task List] table?
I know that's a mouth full... Put another way, I want the append query update the [Master Task List].[OurDepartments], which is a lookup, based on parsing the text of [imported Task list].[Department].
Note the tricky element: we have to parse the text for "BA" as well as "BAD", "BAC", etc. The shorter "BA" might be an interesting issue for this query.
Hoping for a Non-VBA solution.
Thanks for taking a look!
Derek
PS: Would be very helpful if anyone might be able to respond within the work week. Thx!
The answer is here: http://www.utteraccess.com/forum/Append-Query-Selects-L-t1984607.html