I'd like to get a big list (say 1,000 or more) of word phrases that people search for on the internet recently (anything from the most recent month or week or day is ok). Results from Google or any of the bigger search sites would be okay. And is there a way to do this programmatically? Python would be first choice, shell scripts works too. Thanks!
Bonus points for historical results too.
http://www.google.com/trends
google is pretty data friendly
they even provide rss feeds
http://www.google.com/trends/hottrends/atom/hourly
Yes, It's python friendly with API and easy_install to boot!
http://pypi.python.org/pypi/pyGTrends/0.81
Along with what TelsaBoil post, Google Insights looks to give historical results too
http://www.google.com/insights/search/
I think you should check this ones:
http://www.google.com/trends/hottrends
http://www.google.com/trends/hottrends/atom/hourly [RSS Hourly Feed]
http://pypi.python.org/pypi/pyGTrends/0.81 [Python Google Trends Information Retrieval]
Related
Can anyone provide how to get historical options data with strikes by Google Finance API? Mbe Yahoo API can do it?
Thx.
AFAIK, there's no free API that lets you query for historical option prices easily.
Your best shot may be to collect it daily for the stocks you're interested in:
http://www.google.com/finance/option_chain?q=AAPL&output=json
I've never seen free historical option data prices. Neither Yahoo or Google provide them. I downloaded 2012 end of day data from optiondata.net. It includes daily prices, greeks and IV. As far as I can tell it seems accurate. There are several other paid sources out there but I have not tested their data.
In it's current version, is it possible to use Bing's "Narrow By Date" feature when accessing it's API?
I cannot find any information about how to narrow the results such that it only shows results from the "past 24 hours" or "past week" (and so on).
The website/documentation isn't exactly clear on what I can and cannot do, and how. Do any of you know whether it's possible or not?
I can see on their Advanced Search Keywords page that you can use other narrowing features (region, language, hasfeed, etc.) here: http://onlinehelp.microsoft.com/en-ca/bing/ff808421.aspx
If you need any more context or information please ask. Thank you for your patience and help.
The Bing search API is pretty vague in terms of limiting results and what you can and cannot use. After testing various words and placement to try and get date to work, I'm fairly certain that there is not a way to use a date to narrow results.
Basically I'm looking for a search engine that searches through a given database. The content will be text being searched.
You will probably want to use a service such as Solr. The easiest way to get started using it is to find a 'cloud' based version, such as Websolr. However, the solution will depend on what language you wish to use when programming your site.
Solutions depend somewhat on language:
1. For java/c#, you have lucene/solr
2. for python you have haystack
You could do text search in the DB directly via LIKE/ILIKE, but the performance depends on DB.
Iconfinder was coded specifically for icon search and at the time (launched in 2007) there were no scripts that worked well for this.
Building a search engine like Iconfinder is not rocket science. I think the hardest part is getting the SQL tuned and figure out how to rank the content. At the moment I collect data about impressions and downloads and calculate a value from that. The icons' rank is based on this value (download/impression) and how well keywords match the tags for the given icon.
I was browsing through fflick, nicely made app on top of twitter. How do they
collect millions of tweets?
accurately (mostly) categorize tweets into postive or negative sentiments?
The collect millions of tweets probably by crawler twitter with their API. Probably searching with Streaming API for keywords related to films, or just searching their own timeline looking for what their followers have to say about films.
Don't know. Probably using some natural language processing techniques from good old AI textbooks. :-)
2) look for smileys - ;), :), :D, :(
A few places provide the latter vas a service now. Check out ViralHeat and Evri:
http://www.viralheat.com/home/features
http://www.readwriteweb.com/archives/sentiment_analysis_is_ramping_up_in_2009.php
I'm trying to write a script that would download currency rates from yahoo finance. The problem is ... i can't find any information on the limitations of this service. Especially i'm interested in how often i can query yahoo finance to access the quotes.csv file. Would yahoo kill my script if i executed it periodically every minute or so? Does anyone know where i could find some official yahoo information of things like that? I've been searching for hours, but it's either well hidden or it's just hiding in plain sight and i don't see it...
Usually its against the TOS of the website. However if you want to collect data that way on a small scale it is fairly trivial. I have mined yahoo finance in the past and have never been turned off. Don't hammer the site...space out your requests. If you want to be even more clever about it script a web browser to do it for you. The page requests will then look identical.