I want to parse this RSS (from a bit.ly user stream) to grab all bit.ly links and related data, and store all new items into a database. What are examples of easy ways to parse RSS (e.g. simple helpers with small footprint)? I'm a Code Igniter rookie, so please be gentle :)
I ended up by simply using json_decode(file_get_contents('http://bitly.com/u/joaoramos.json')); and then dug into the object to get what I needed.
Check out SimplePie. Integrating this in CI , shouldn't be hard .
Or If you want one with less foot print, check this Thread on CI forums , This is very light and does the job.
Related
I've been searching high and low on how to make use of Webhook in API.AI. I'm new to API.AI in general.
My objective is to integrate API.AI into my Swift Application which i have already done. I want to get a JSON Object from a webpage and use those data to manually add them as Entities/Intents/Response/etc by sending back a request to API.AIand then create them.
Tried posting on DialogFlow forums but i guess it's pretty inactive.
The whole workflow is..
Integrating API.AI into my Swift Application for users to use.
I have the base of the AI now.. Intents lead to Entities in a sense whereby users ask for a type of study, and an Entity [Studies] is triggered, so meaning from the Intent 'asking for a type of study' the response would be like Science, or something. This reply i want to get it from the JSON that i mentioned above, from a webpage. Is there any way to actually set these responses? Create entities etc, with my Swift Codes.
I hope it's clear enough.. And hope that anyone can point me in a direction as i can't seem to find any articles or somewhere to follow on it?
May be its late for answering this question, but few days back, I came across a similar situation. I countered it and wrote a tutorial series for the same. Here's the link for tutorial. It might be useful for those who are looking to use API.AI, or want to integrate it in iOS project.
Building a Chat Bot having AI is easy
I'm trying to find a good alternative at Parse.
FireBase seems to be good, but I can't find how to upload image/file from the dashboard like in Parse ?
Can we upload image/file from the firebase dashboard like in Parse ?
No You Can Not because firebase doesn't provide such type of feature.
Edit:
Ah! My old accepted answer got unaccepted because I gave straight answer :) Well OP didn't ask about programming way of solution so I assumed not to provide one.
There are couple of approaches to handle images with firebase, I usually upload images to s3 bucket and store urls in firebase. Again there is choice to implement above approach i.e. do all on server-side, do all on client-side or do with mix of client-side and server-side. But It is better to do signaturing on server to keep thing secure.
Node-Cheat Available:
For complete code, get working node-cheat at github node-cheat aws, choose any folder and run node file_name.
While the above is an answer, it's does't provide a solution, so here's a bit more data.
Uploading/storing data in firebase is super simple and requires a minimal amount of code.
For images, encode the image as a string and store the string in firebase. Then to retrieve the image, retrieve the string and decode it.
You should be able to do both in about 12 lines of code (depending on platform).
There are several examples already published here, I like this one with Swift.
Swift2 retrieving images from Firebase
Hope that helps!
No, you cannot upload directly, but you could use some opensource project out there that will make a simple dashboard for you to upload it easily like this tutorial
How could I access a website and turn components of the website into strings. For example taking information from Facebook posts. I have done a little searching but can't find any good tutorials or anything useful.
Try looking at this tutorial. It should get you more familiar on the subject and start you off on the right track.
As it states at the beginning of the tutorial...
How to Parse HTML on iOS
Let’s say you want to find some information inside a web page and
display it in a custom way in your app. This technique is called
“scraping.” Let’s also assume you’ve thought through alternatives to
scraping web pages from inside your app, and are pretty sure that’s
what you want to do. Well then you get to the question – how can you
programmatically dig through the HTML and find the part you’re looking
for, in the most robust way possible? Believe it or not, regular
expressions won’t cut it! Well, in this tutorial you’ll find out how!
You’ll get hands-on experience with parsing HTML into an Objective-C
data model that your apps can use.
http://www.raywenderlich.com/14172/how-to-parse-html-on-ios
I am working on a web application and want to use SimplePie for parsing a lot of RSS feeds.
One problem is that I want to know if SimplePie automatically removes all the dead links while parsing RSS feeds. So that the final output of the RSS feed doesn't have any invalid links.
No, it does not. In order to do so, you'd have to send a HEAD request for every link in the feed. This is too expensive for SimplePie to do, so you'll need to find code to do that on your own. Try this example on SO.
I'm using RSS library so i can parse Atom and RSS in Ruby and Rails and store it in a model.
I've looked at the standard RSS library, but is there one library that will auto-detect that there is a new rss feed so i can update my database ?
what are the best practice to trigger an instruction in order to store the new rss feed ?
should i use threads to handle that problem ?is it going to be slow?
thank you for your help
OK heres the deal.
If you want a real fast feed parser go for Feedzirra. Does not work on windows. http://github.com/pauldix/feedzirra
Autodiscovery?
-Theres truffle-hog if you don't want to do GET redirects. http://github.com/pauldix/truffle-hog
-Theres feedbag if you want to do GET redirects to find feeds from given urls. This is slower though. http://github.com/damog/feedbag
Feedzirra is the best bet if you want to poll for new entries for your feed. But if you want a more non-polling solution to your problem then i would suggest going through the pubsubhubbub spec. Make sure while parsing your feeds they are pubsubhubbub enabled. Check for the link tag. If it points to pubsubhubbub.appspot.com or any other pubsub enabled hub then just subscribe to the feed by sending a subscription request to the hub. You can then define a endpoint in your app which will in turn receive updated entry pings for your feed subscription from the hub. Just read the raw POST data and store it in your database. Stats are that 95% of the blogger blogs are pubsub enabled. That is a lot of data in your hands already. :)
If you are polling for changes then you should check the last-modified or etag from the header rather than parse the entire feed again. Saves you from wasting resources. Feedzirra takes care of this for you.
I am not sure what you mean by "auto-detect" a new feed?
Are you looking for code that can discover when someone creates a new feed on a site? Or, do you mean discover when an existing feed has a new article?
The first is tough because your code needs to know what site to look at so it needs some sort of auto-discovery of sites with new feeds. Searching the google for "new rss feeds" doesn't return anything that looks useful, at least not on the first page. If you, or your users, know of a new site then you can have an interface to add new sites to search. Then you grab the page at that URL, look for the RSS/Atom auto-discovery links, and go from there. Auto-discovery links can open a can of worms because of duplicate content being served using different protocols (RDF, RSS and Atom), so you have to determine which to use, or multiple feeds with alternate content listed.
If you mean you want to discover when an existing feed has new articles, then you have to keep track of the last time your code looked at the feed, and the last article that was seen, then retrieve the feed and see if any articles were not in your list of previously seen articles. Your code needs to be sensitive to the time-to-live information in a lot of feeds too. Hitting the feed every fifteen minutes when they update once a week is bad form. Most aggregation code can do those things already but you might need to configure a database and tell the code how to find it.
Generally, for this sort of task I set up a crontab entry on a production Linux or Unix system and fire off the job periodically, looking in the database for feeds whose last-run-time plus the stored time-to-live value is in the past.
Does that help any?
Very easy solution is to use Dynamic attribute-based finders
When you are filling your model with RSS feed data, instead of Model.create(...) use Model.find_or_create_by_column(value, :other_column => other_value).
You can specify a date as unique value or RSS message title ... (whatever you want)
I think this is pretty easy. You can make some cron task to fill your model once per hour for example. Only new feeds will be added.
There is no chance to get some "event" when RSS is updated without downloading whole RSS feed again.