I am working on a data mining system and one of the requirements is it being able to perform the analysis without the use of API. Is there a way to download the Twitter database (or a big part of it, at least) and work with it locally?
There is a paper about creating corpora from twitter. It is called “TWORPUS – An Easy-to-Use Tool for the Creation of Tailored Twitter Corpora”. I recommend to read it because it also covers licensing issues etc. They also provide there code on Github.
In fact, you cannot download the twitter data dumps directly. I can download single tweets and stored them in a corpus. But, it is also not allowed to share that data. Therefore, the authors built the Tworpus client to create private twitter corpora.
APIs are the official way of getting Twitter data and they work really well so it is not comprehensible why you do not want to use APIs. The web scraping is a work around but not recommended, in addition you would like to get a big part of it, so I do not think you will be satisfied with it. You can also buy the data from Gnip.
Related
i implemented a chat app in swift using firebase real time db, there user can send images, emojis and Text.Now i have a requirement of export chat or get the conversation's backup with media and text as per whatsApp.help me to solving it out.
While Firebase offers a backup for the Realtime Database, this doesn't fit your needs here, since you'll want a per-user export of the data.
Since this is specific to your application, you will have to code it yourself, just like the good folks at WhatsApp have probably done. It should be a matter of iterating over all data sources for the user, getting the data through the relevant API (that you're already using to display that data), and then writing to a local file). You can do this either client-side in your Swift code, or server-side on a server you already may have, or using Cloud Functions.
If you're looking for some inspiration for the latter, there is a sample repository that shows how to clean up a user's data, based on a set of wipe-out rules. You'll need to significantly modify this example though, so I'm not convinced this will be less work than rolling your own from scratch.
After making a bunch of CRUD-like apps, I've decided to venture out a bit and play with APIS. The Riot game API seems pretty simple to implement, as it is just a link follow by my API key.
ex: https://na.api.pvp.net/api/lol/na/v1.4/summoner/by-name/RiotSchmick?api_key=<key>
after googling for quite a while yesterday, I am still at lost how I can integrate this information.
Where should I put the above link, how do I store the information from the JSON, and how exactly does the MVC structure fit into this?
If I'm not storing any data (no users, just instant display of info based on API) Do I even need a database?
My project is reliant on several API's, like Twitter and Youtube for example. Recently, Youtube deprecated their old API, and it caused issues with my team's iPad app.
We could have stayed ahead of the change if we were paying attention to Youtube's announcements of the upcoming deprecation. But alas, we were not and the idea of staying up to date with all of our dependencies manually(browsing the web) seems exhausting and inefficient.
I have found the following tool to help notify when changes occur with external library dependencies, https://libraries.io. However, this does not help with API dependencies.
Besides checking the API source webpages every so often, I was wondering if anyone had suggestions on how to stay notified and up-to-date with news regarding updates to a specified list of external API's?
After some time looking at different options, I have found a solution that is not perfect, but seems to work best at fitting this need.
Solution Description
This solution uses a combination of Twitter, Google Scripts, and website blogtrottr.com. I am creating a twitter list of reliable dev handles that often post updates on new API. For example I made a list that contained #twitterapi and #YouTubeDev. Used Google Scripts to create an online feed out of the twitter list. Then used blogtrottr to email me every time that feed gets a new posting.
Steps to Implement
Create a twitter list of reliable handles that often post about updates to their API
Create an RSS Feed from that Twitter list. The details for how to do this can be found here.
Plug that url that you get from Google Script into blogtrotter.
I did find some other ways to do this, but so far this is the only solution that was 100% free!
I'm developing some website using Rails. I want to add "our users' tweets" part to the main page. I need an advice how I can do it better. I hoped to get standard way to do it, may be some Twitter widget or something else. I used Google, but I've found nothing. Please, point me to the right path. Sorry if my questions is very simple, but I don't really know how to do it. I hope that I needn't parse JSON and add styles independently; I need simple design from Twitter :)
To answer your [ambiguous] question, there are a number of things to consider:
How will you retrieve the tweets?
How will you store / access them?
How will the data be displayed on front-end?
The two methods you have are either to use the Twitter gem, or the TwitterFetcher JS plugin:
Gem
The Twitter gem uses the Twitter API to pull data from the official Twitter API. This means you've got the throttling & authentication to build into your app
The benefits of using this gem is it gives you a HUGE amount of flexibility with the data. You can pull as much data as you need / want, in whatever format you want - all formatted in JSON & can be displayed on your site
This gem is best suited to storing your tweets, either in a DB or in Redis etc, otherwise you'll have massive synchronous dependency on Twitter's API - which is never good for performance
JS
The TwitterFetcher JS plugin is epic - basically takes a Twitter widget & strips out the HTML, allowing you to style it how you like
This is the most effective way to retrieve Twitter data on-the-fly, as it's asynchronous, relies on Twitter's widget system (far more robust than API), and stores no data locally
I would like to build a web application that tracks some user defined search terms in real-time and provides a real-time visualization. http://www.monitter.com/ is an app I've found that is similar in its requirements. What is the appropriate API to use for it? Initially I thought the streaming API was the obvious choice, but the limitation of one concurrent connection means that I can only track one search term at a time(with one user account). I could get around this by making multiple user accounts, but that seems like the wrong approach.
I looked at user streams but the language for that API seems to be more geared towards desktop applications.
So, what is the most best API for my use case? Thanks.
Actually you can track up to 400 keywords/terms via one streaming API connection.
https://dev.twitter.com/docs/streaming-api/methods#track
Depending on language you are using there are multiple interfaces you can use.
If you are using PHP, then I can suggest Phirehose as it works quite well and has multiple examples for different usages scenarios included.
http://code.google.com/p/phirehose/wiki/Introduction
Whats not there - when processing received tweets you will need to figure out how to match which tweet corresponds to which keyword/term because twitter streaming API gives all matching tweets in one stream.
Investigating further using Firebug, I found that monitter.com simply polls the REST search api every second or so on the client side. This is what I ended up doing as well.