I'm trying to write a script that would download currency rates from yahoo finance. The problem is ... i can't find any information on the limitations of this service. Especially i'm interested in how often i can query yahoo finance to access the quotes.csv file. Would yahoo kill my script if i executed it periodically every minute or so? Does anyone know where i could find some official yahoo information of things like that? I've been searching for hours, but it's either well hidden or it's just hiding in plain sight and i don't see it...
Usually its against the TOS of the website. However if you want to collect data that way on a small scale it is fairly trivial. I have mined yahoo finance in the past and have never been turned off. Don't hammer the site...space out your requests. If you want to be even more clever about it script a web browser to do it for you. The page requests will then look identical.
Related
How can I get finance gata from yahoo.finance in realtime? I tried select * from yahoo.finance.historicaldata but there only data by day not by hours.
You're querying a table called historicaldata, it is probably only updated once a day. Another table, quotes, may have data updated on a more frequent timer (hourly, as you've mentioned).
Unfortunately, I think you've run into a limitation of Yahoo's free financial data feeds. They have to purchase market feeds from the source (NASDAQ, NYSE, etc.) (or close to it) and those feeds aren't cheap. If you want truly real-time data, that's something you'll probably have to cough good money out for.
There are other sources besides Yahoo though that might get you down to 15 minute updates. I will leave that research as an exercise for you since that's out of scope for StackOverflow. If you do find a different source and are having trouble reading from it, do please feel free to ask another question here! The more closely related to actual programming the better response you'll get from the community.
Can anyone provide how to get historical options data with strikes by Google Finance API? Mbe Yahoo API can do it?
Thx.
AFAIK, there's no free API that lets you query for historical option prices easily.
Your best shot may be to collect it daily for the stocks you're interested in:
http://www.google.com/finance/option_chain?q=AAPL&output=json
I've never seen free historical option data prices. Neither Yahoo or Google provide them. I downloaded 2012 end of day data from optiondata.net. It includes daily prices, greeks and IV. As far as I can tell it seems accurate. There are several other paid sources out there but I have not tested their data.
I have an iOS app which can be used offline. I need to do anonymous page view tracking, so our customers can tell which pages people are most interested in (to drive future investments). So when the user is offline, we save a timestamped page view list, and if the user happens to be online when they use the app, we send these historic records up, and also do real-time tracking.
I'm keeping some summary statistics in my GAE app, so I can report the page views with historic accuracy. However, I'm also feeding these views into google analytics, using some python code I ported from google's server-side samples.
That all works great (except for language tracking, which I may have solved thanks to a separate question here on SO). However, I'd love for google analytics to be able to understand the historical hits in context. Right now, if I connect up after looking at several pages offline, GA thinks I just popped through a bunch of pages over the course of a couple seconds.
There is no documented utm variable for timestamping. The google analytics SDK for iOS (which I'm not using) has this ominous note:
Known Issues
Possible inaccurate timestamps: timestamps are recorded at the time the application dispatches to Google Analytics, so if a user experiences long periods of offline use, the timestamps may not be 100% accurate.
That seems like a bit of an understatement. Wouldn't offline timestamps be 100% inaccurate?
Anyway, the fact that the SDK doesn't handle this right makes me think I'm not going to be able to solve this. But I figured some SO wizard might have an idea...
In fact, timestamp is a "relative" (client side) information used by Analytics to compute things like "time on page".
When the page is view in "absolute" (date and time) is always the time you send the request.
I'd like to get a big list (say 1,000 or more) of word phrases that people search for on the internet recently (anything from the most recent month or week or day is ok). Results from Google or any of the bigger search sites would be okay. And is there a way to do this programmatically? Python would be first choice, shell scripts works too. Thanks!
Bonus points for historical results too.
http://www.google.com/trends
google is pretty data friendly
they even provide rss feeds
http://www.google.com/trends/hottrends/atom/hourly
Yes, It's python friendly with API and easy_install to boot!
http://pypi.python.org/pypi/pyGTrends/0.81
Along with what TelsaBoil post, Google Insights looks to give historical results too
http://www.google.com/insights/search/
I think you should check this ones:
http://www.google.com/trends/hottrends
http://www.google.com/trends/hottrends/atom/hourly [RSS Hourly Feed]
http://pypi.python.org/pypi/pyGTrends/0.81 [Python Google Trends Information Retrieval]
I plan to write a thesis about using sentiment information to enhance the predictivity of some financial trading model for currency.
The sentiment data should be twitter threads including some keyword, like "EUR.USD". And I will filter out some sentiment words to identify the sentiment. Simple idea. Then we try to see whether here is any relation between the degree of sentiment and the movement of EUR.USD.
My big concern is on twitter data. As we all know that the twitter set up the limit to see the history data. You could only browser back for like 5 days. It is not enough since our strategy based on daily sentiment.
I noticed that google have some fantastic thing like timeline about the twitter updates: http://www.readwriteweb.com/archives/googles_twitter_timeline_lets_you_explore_the_past.php
But first of all, I am in Switzerland and seems I have no such function on my google which is too smart to identify my location and may block some US google version function like this. Secondly, even I could see some fancy interactive google timeline control on my firefox, How could I dig out data from my query and save them? Does google supply such api?
The Google service you mentioned has shut down recently so you won't be able to use it. (http://www.searchenginejournal.com/google-realtime-shuts-down-as-twitter-deal-expires/31007/)
If you need a longer timespan of data to analyze I see the following options:
pay for historical data :) (https://dev.twitter.com/docs/twitter-data-providers)
if you don't want to pay, you need to fetch tweets containing EUR/USD whatever else (you could use the streaming API for this) and store them somehow. Run this service for a while (if possible) and you'll have more than just 5 days of data.