Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I need a way from within client-side Javascript to find a full company name given a ticker symbol. I am aware of Yahoo Finance's interface at:
http://finance.yahoo.com/d/quotes.csv?s=TKR&f=n
and am able to access that via YQL (since this is cross-domain). However, that doesn't return the full company name, yet Yahoo Finance has such because it appears in their charts for the company and on their pages about the company.
I don't need for the solution to be via Yahoo Finance... just mention it here as I already know about it (and am accessing it for other data).
One of the community-provided YQL tables looks like it will work for you: yahoo.finance.stocks.
Example YQL query:
select CompanyName from yahoo.finance.stocks where symbol="TKR"
Update 2012-02-10: As firebush points out in the comments, this YQL community table (yahoo.finance.stocks) doesn't seem to be working correctly any more, probably because the HTML page structures on finance.yahoo.com have changed. This is a good example of the downside of any YQL tables that rely on HTML scraping rather than a true API. (Which for Yahoo Finance doesn't exist, unfortunately.)
It looks like the community table for Google Finance is still working, so this may be an alternative to try: select * from google.igoogle.stock where stock='TRK';
I have screen scrapped this information in the past either using Yahoo Finance or MSN Money. For instance you can get this information for ExxonMobil by going to (link). As far as an API you might need to build one yourself. For an API checkout Xignite.
You can use the "Company Search" operation in the Company Fundamentals API here: http://www.mergent.com/servius/
You can use Yahoo's look up service using Jonathan Christian's .NET api that is available on NuGet under "Yahoo Stock Quotes".
https://github.com/jchristian/yahoo_stock_quotes
//Create the quote service
var quote_service = new QuoteService();
//Get a quote
var quotes = quote_service.Quote("MSFT", "GOOG").Return(QuoteReturnParameter.Symbol,
QuoteReturnParameter.Name,
QuoteReturnParameter.LatestTradePrice,
QuoteReturnParameter.LatestTradeTime);
//Get info from the quotes
foreach (var quote in quotes)
{
Console.WriteLine("{0} - {1} - {2} - {3}", quote.Symbol, quote.Name, quote.LatestTradePrice, quote.LatestTradeTime);
}
EDIT: After posting this I tried this exact code and it was not working for me so instead I used the Yahoo Finance Managed Api however it's not available via NuGet. A good example of use here
QuotesDownload dl = new QuotesDownload();
DownloadClient<QuotesResult> baseDl = dl;
QuotesDownloadSettings settings = dl.Settings;
settings.IDs = new string[] { "MSFT", "GOOG", "YHOO" };
settings.Properties = new QuoteProperty[] { QuoteProperty.Symbol,
QuoteProperty.Name,
QuoteProperty.LastTradePriceOnly
};
SettingsBase baseSettings = baseDl.Settings;
Response<QuotesResult> resp = baseDl.Download();
Also if you just want to download the stuff stocktwits api has a download link for the symbology and industries under "Resources" http://stocktwits.com/developers/docs
It also possible to use Quandl.com resources. Their WIKI database contains 3339 major stocks and can be fetched via secwiki_tickers.csv file. For a plain file portfolio.lst storing the list of your tickers (stocks in US markets), e.g.:
AAPL
IBM
JNJ
MSFT
TXN
you can scan the .csv file for the name, e.g:
import pandas as pd
df = pd.read_csv('secwiki_tickers.csv')
dp = pd.read_csv('portfolio.lst',names=['pTicker'])
pTickers = dp.pTicker.values # converts into a list
tmpTickers = []
for i in range(len(pTickers)):
test = df[df.Ticker==pTickers[i]]
if not (test.empty):
print("%-10s%s" % (pTickers[i], list(test.Name.values)[0]))
what returns:
AAPL Apple Inc.
IBM International Business Machines Corporation
JNJ Johnson & Johnson
MSFT Microsoft Corporation
TXN Texas Instruments Inc.
It is possible to combine more stocks from other Quandl's resources. See the documentation online.
Related
I need to fetch the closing price of given stock at specified dates from Yahoo Finance API. However, there does not seem to be any good documentation on how to achieve this.
Install the following packages:
pandas_datareader
fix_yahoo_finance
Example script:
import pandas_datareader.data as web
import fix_yahoo_finance as yf
yf.pdr_override()
print(web.get_data_yahoo('AAPL', start="2018-01-01", end="2018-01-07"))
This is somewhat of a duplicate question of Does YouTube API forbid to download video captions if you are not it's owner?, Get YouTube captions and Does YouTube API forbid to download video captions if you are not it's owner?, which all basically say it's not possible unless to download captions via the YouTube API unless you are the owner or third-party contributions are not enabled; however, my question is how to sites like http://downsub.com/ or http://www.lilsubs.com/ have access to all captions?
In other words, when I access the YouTube API myself (even with youtubepartner and youtube.force-ssl scopes), I can only download the captions of some videos, but when I try the same videos that failed for me with 403: The permissions associated with the request are not sufficient to download the caption track. The request might not be properly authorized, or the video order might not have enabled third-party contributions for this caption. on these other sites, it works fine. I'm assuming they are using the YouTube API to access the captions, but what special sauce are they using? Some special partner key? An different API version? Are they just scraping from the videos themselves or something?
Send a GET request on:
http://video.google.com/timedtext?lang={LANG}&v={VIDEOID}
Example for your video in comment: http://video.google.com/timedtext?lang=ko&v=0db1_qWZjRA
Let's look at another example of yours, i.e. https://www.youtube.com/watch?v=7068mw-6lmI (and I agree about differentiation part in your comment).
There are multiple subtitles available for the video
English
Korean
Spanish
Korean (auto-generated) also called asr (automatic speech recognition)
These stand for the subtitle name parameter (i.e., name=English).
lang stands for the country code.
In your example: https://www.youtube.com/api/timedtext?lang=es-MX&v=7068mw-6lmI&name=Spanish
If subtitle track is available, it is possible to do translation form it, namely using tlang parameter.
https://www.youtube.com/api/timedtext?lang=en&v=7068mw-6lmI&name=English&tlang=lv
https://www.youtube.com/api/timedtext?lang=ko&v=7068mw-6lmI&name=Korean&tlang=lv
This would be my bid for what these sites are using, i.e. translation of the available subtitle track (confirm by trying to use a video without subtitle track as input for one of their sites).
As for asr signature seems to always be needed, but as long as one of the subtitle tracks are available, you could use that for translation. E.g. in your OP comment example:
https://www.youtube.com/api/timedtext?lang=en&v=vx6NCUyg1NE&tlang=lv
Looks like the last example is special with both of subtitle tracks being asr (checked with Chrome -> Inspect -> Network) therefore you need to omit the subtitle name parameter part. This difference unfortunately is not visible in YouTube video's settings wheel.
A 2022 answer:
Option 1: Send a curl request to the webpage: curl -L "https://youtu.be/YbJOTdZBX1g", search for timedtext in the result, and you would get a URL. replace \u0026 with & and you get the link for the subtitle.
Option 2: Use the yt-dlp package:
# For installing see: https://github.com/yt-dlp/yt-dlp#with-pip
from yt_dlp import YoutubeDL
ydl_opts = {
"skip_download": True,
"writesubtitles": True,
"subtitleslangs": ["all", "-live_chat"],
# Looks like formats available are vtt, ttml, srv3, srv2, srv1, json3
"subtitlesformat": "json3",
# You can skip the following option
"sleep_interval_subtitles": 1,
}
with YoutubeDL(ydl_opts) as ydl:
ydl.download(["YbJOTdZBX1g"])
There is this unofficial API used by Youtube :
https://www.youtube.com/api/timedtext?lang={LANG}&v={VIDEO_ID}
LANG here is ISO 639-1 2 letter country code. For your example it would be :
https://www.youtube.com/api/timedtext?lang=ko&v=0db1_qWZjRA
You can check it in network tab while toggling the closed caption button :
I have used youtube-transcript-api successfully to retrieve transcripts. The below is a demo to dump the transcript into HTML with links back to the timestamps in the video:
import sys
from youtube_transcript_api import YouTubeTranscriptApi
video_id = sys.argv[1]
# Retrieve the available transcripts
transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)
# Just use the first transcript, let it raise an exception if none exist.
transcript = next(iter(transcript_list))
print("<html><body>")
for line_map in transcript.fetch():
st_sec = int(line_map['start'] / 60)
st_msec = int(line_map['start'] - st_sec * 60)
tstmp = f"{st_sec}:{st_msec}"
link_to_tstmp = f"https://youtu.be/{video_id}?t={st_sec*60}"
tstmp_str = ("%2d:%-2d" % (st_sec, st_msec)).replace(" ", " ")
#print(f"{st_sec}:{st_msec} {line_map['text']}")
print("""%s %s<br/>""" % (link_to_tstmp, tstmp_str, line_map['text']))
print("</html></body>")
If there are multiple transcripts, the library provides API to search by language etc.
You can further tweak the logic to merge text so you only get one link every so many minutes. I got good results for a lecture by linking at every 1 min and format the lines into a HTML table.
I've been puzzling on this on and off for months and can't find a solution.
The MTA claims to provide historical data in form of daily dumps in GTFS format here:
[http://web.mta.info/developers/MTA-Subway-Time-historical-data.html][1]
See for yourself by downloading the example they provide, in this case Sep, 17th , 2014:
[https://datamine-history.s3.amazonaws.com/gtfs-2014-09-17-09-31][1]
My problem? The file is gobbledygook. It does not follow GTFS specifications, has no extension, and when I open it using a text editor it looks like 7800 lines of this:
n
^C1.0^X �枪�^Eʞ>`
^C1.0^R^K
^A1^R^F^P����^E^R^K
^A2^R^F^P����^E^R^K
^A3^R^F^P����^E^R^K
^A4^R^F^P����^E^R^K
^A5^R^F^P����^E^R^K
^A6^R^F^P����^E^R^K
^AS^R^F^P����^E^R[
^F000001^ZQ
6
^N050400_1..S02R^Z^H20140917*^A1�>^V
^P01 0824 242/SFY^P^A^X^C^R^W^R^F^Pɚ��^E"^D140Sʚ>^F
^AA^R^AA^RR
^F000002"H
6
Per the MTA site (appears untrue)
All data is formatted in GTFS-realtime
Any idea on the steps necessary to transform this mystery file into usable GTFS data? Is there some encoding I am missing? I have looked for 10+ and been unable to come up with a solution.
Also, not to be a stickler but I am NOT referring to the MTA's realtime data feed, which is correctly formatted and usable. I am specifically referring to the historical data dumps I reference above (have received many "solutions" referring only to realtime data feed)
The file you link to is in GTFS-realtime format, not GTFS, and the page you linked to does a very bad job of explaining which format their data is actually in (though it is mentioned in your quote).
GTFS is used to store schedule data, like routes and scheduled arrival times.
GTFS-realtime is generally used to transfer actual transit performance data in real-time, like vehicle locations and expected or actual arrival times. It is a protobuf, a specification for compiled binary data publicized by Google, which means you can't usefully read it in a text editor, but you instead have to load it programmatically using the Google protobuf tools. It can be used as a historical data format in the way MTA is here, by making daily dumps of the GTFS-rt feed publicly available. It's called GTFS-realtime because various data fields in the realtime like route_id, trip_id, and stop_id are designed to link to the published GTFS schedules.
I confirmed the validity of the data you linked to by decompiling it using the gtfs-realtime.proto specification and the Google protobuf tools for Python. It begins:
header {
gtfs_realtime_version: "1.0"
timestamp: 1410960621
}
entity {
id: "000001"
trip_update {
trip {
trip_id: "050400_1..S02R"
start_date: "20140917"
route_id: "1"
}
stop_time_update {
arrival {
time: 1410960713
}
stop_id: "140S"
}
}
}
...
and continues in that vein for a total of 55833 lines (in the default string output format).
EDIT: the Python script used to convert the protobuf into string representation is very simple:
import gtfs_realtime_pb2 as gtfs_rt
f = open('gtfs-rt.pb', 'rb')
raw_str = f.read()
msg = gtfs_rt.FeedMessage()
msg.ParseFromString(raw_str)
print msg
This requires gtfs-realtime.proto to have been compiled into gtfs_realtime_pb2.py using protoc (following the instructions in the Python protobuf documentation under "Compiling Your Protocol Buffers") and placed in the same directory as the Python script. Furthermore, the binary protobuf downloaded from the MTA needs to be named gtfs-rt.pb and located in the same directory as the Python script.
I am using python and Beautiful soup to read html pages. Unfortunately some sites redirect to my Geo region (AU) so I can't retrieve the target countries version i.e. (UK, US, FR, NZ...)
I have tried using a VPN service but this requires me to manually change the region so I can't automate the process. I have tried using the python quartz.Coregraphics library to click the options on screen but this is temperamental.
Is there a way I can achieve this programmatically?
I have manage to nut this one out myself. Best answered by example for reading a uk based site.
import urllib2
url = 'Some-uk-url'
req = urllib2.Request(url)
req.add_header('Accept-Language', 'en-gb')
req.add_header('X-Forwarded-For', [a uk proxy ipaddress here])
htmltext = urllib2.urlopen(req).read()
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
There are a million different XML parsers for the iPhone. I have a medium sized XML file that contains alot of duplicate tags (at different points in the hierarchy). I was thinking about TBXML but I was concerned about its lack of XPath support. For instance lets say my XML file looked like this.
<blog>
<author> foo1 </author>
<comments>
<comment>
<text>
<![CDATA[ HTML code that must be extracted ]]>
</text>
<author>foo2</author>
</comment>
<comment>
<text>
<![CDATA[ Here is another post ]]>
</text>
<author>foo1</author>
</comment>
</comments>
</blog>
Basically my requirements are that I need to be able to extract that cdata. And know whether it is the blog author a comment author.
Best comparison I've seen on the subject:
http://www.raywenderlich.com/553/how-to-chose-the-best-xml-parser-for-your-iphone-project
The difference between the slowest and fastest parsers in his case was a factor of 2. Depending on your feature requirements, there are definitely parsers that can cut your parsing time in half.
Check out RaptureXML. It's not official XPath behavior, but it's pretty close. Your query would be:
[rxml iterate:#"comments.comment" with: ^(RXMLElement *e) {
if ([[e child:#"author"].text isEqualToString:#"foo1"]) {
NSLog(#"Blog Text is %#.", e.text);
} else {
NSLog(#"Comment Text is %#.", e.text);
}
}];
UPDATE: RaptureXML now supports XPath!
You should look at these libraries, in order :)
Ono, powered by libxml2
XMLDictionary
RaptureXML
Ono has the the cool AFOnoResponseSerializer which can be integrated with AFNetworking
I made this new simple and lightweight XML parser for iOS in Swift - AEXML
You can use it to read XML data like this:
let someValue = xmlDocument["element"]["child"]["anotherChild"].value
or you can use it to build XML string like this:
let document = AEXMLDocument()
let element = document.addChild("element")
element.addChild("child")
document.xmlString // returns XML string of the entire document structure
I hope it will be useful for someone.
As you have requirement for medium sized XML
TinyXML could be an good choice for medium sized documents if you already have experience with the API and are comfortable with C as it ports quite easily over to the iPhone.
For More information you can go through this link