I'm looking to add feeds via the front-end to be fetched by Feedjira, I've followed the basic Railscast tutorial and got 1 feed set up and working. What I can't wrap my head around is allowing me to add a feed to feedjira via the front end (e.g. text box) and then letting the cron job pick it up and parse it.
I have a user model and posts model, with a default Feedjira set-up. I'm guessing I need a 'feeds' table in the user model which stores the URL's of the users desired RSS feeds and then passes them into Feedjira to be parsed?
Pretty new at rails/ruby and would love some help/guidance on this matter.
Thanks
Once you have URL store in a table, or in a field of your User model, you can parse it in your rake task.
In your task, itinerates users, or feeds, and do that you did with one feed inside your loop.
So finally, let your cron job do the job for your at interval time.
Related
I am new to Ruby and have a question about how to do a specific task on Rails.
I have a list of inventory and each item has a specific Stock ID that is emailed to my personal Gmail account. I want my web application to listen for emails from a specific email account. When my gmail receives an email from that specific account I want my application to parse it for a couple of fields and insert the stock ID into my database.
For example:
Let's say my database has an item with style code: A5U31 and size:10.
The email will say something like item with style code: A5U31 and size:10 has Stock ID:329193020.
I want my Rails application to search the database for an entry with that specific style code and size, and when it finds an entry to simply insert the stock ID into the row.
I am trying to using the Google-API-Client gem to this, but I am struggling, because I am still a beginner. So far I have done this quick-start guide to authenticate my gmail account with my rails app.
https://developers.google.com/gmail/api/quickstart/ruby?authuser=2
If someone could help me figure out how to write this sort of code and where to put it in my app(models, controllers, views) that would be very appreciated. Thanks!
I know it's been several months since you posted this, so you probably already have it worked out, but in case you still need a solution (or if someone else wants to do something similar), I'll share my thoughts:
At a high level, it sounds like your plan is
Identify when a new email has come in (either by polling or by using a push notification).
Grab the new email's content.
Parse the email's content in order to extract relevant data.
Use the data to query and update a database.
Based on the documentation for the Gmail API, it does look like you should be able to set up push notifications, so you won't have to poll the endpoint to get the information you need.
However, my big takeaway from this list is that none of the items on it really require Rails, since you're not exposing an external web API for requests. I suppose that you could leverage ActiveRecord to create an item model and use that to manage the database; however, since it seems like you'd only need to make some basic SQL queries (and the same ones each time), I'm not sure that bringing in ActiveRecord adds much value.
If I were trying to solve this problem myself, I would probably create a simple Ruby program that (a) uses the gem you mentioned to handle push notifications from the Gmail API, and (b) uses another gem to connect to whatever kind of database you're using (e.g. pg for Postgres) and make the necessary queries.
(All of this assumes, of course, that you aren't specifically using Rails for some other reason, e.g. adding this feature to an existing Rails application).
I can successfully pull different feeds using the Feedzirra gem and get feed updates. However, each feed that I'd like to pull has different content (ie: Github Public Feed, last.fm recently played, etc.).
What is the best way to go about combining all of these feeds into one? Right now I have different models for different types of feeds and some feeds use different timestamps than the others.
m,
You could add multiple extra fields to hold each of the unique attributes in an uber-feed object, only filling in the ones that come from each particular feed at time of processing. (It's kind of like the NoSQL model in that way, though not quite, since you have to define the fields ahead of time, but you can add any arbitrary field as a data-holder.)
This is how you add a new field to all instances of a feed...
Feedzirra::Feed.add_common_feed_entry(:my_custom_field)
You'll find a little more dialog about this here...
https://groups.google.com/forum/?fromgroups#!msg/feedzirra/_h4y8_vwDGc/N8sjym6NouEJ
You are creating an activity feed -- here are several gems that you can research on how to create activity feeds: https://www.ruby-toolbox.com/categories/Rails_Activity_Feeds
I'm working on a life-streaming type app in Rails and am constantly parsing several different RSS feeds using Feedzirra. To support scaling of the app, I have to implement some kind of caching while also allowing to paginate the cached results in the view. The cache can expire as little as once a day.
Being a novice to caching in Rails, what types of caching would be recommended for this? Also, I'm doing most of the feed parsing operations in modules in my lib/ directory, not sure if this would have affect / not be ideal for caching. Ideally I'd like to cache the array of results the RSS feed returns and do some post-processing to them before I send it to the view.
Any help would be much appreciated!
I suggest you to use a gem for run a schedule task in the cron, collect all desired results from all your rss feeds and save it to an xml or even in a table.
For the next time, load the results from this xml or table and create an static cached pages (html files).
And everytime you run your schedule task, erase your previous saved files, preventing old results tyo be displayed.
I want to be able to store media RSS and iTunes podcast RSS feeds into the database. The requirement here is that I don't want to miss out on ANY element or its attributes in the feed. It would make sense to find all most common elements in the feed and have them stored in database as separate columns. The catch here is that there can be feed specific elements that may not be standard. I want to capture them too. Since I don't know what they can be, I won't have a dedicated column for them.
Currently I have 2 tables called feeds and feed_entries. For RSS 2.0 tags like enclosures, categories, I have separate tables that have associations with feeds/feed_entries. I am using feedzirra for parsing the feeds. Feedzirra requires us to know the elements in the feed we want to parse and hence we would not know if feed contains elements beyond what feedzirra can understand.
What would be the best way to go about storing these feeds in the database and not miss single bit of information? (Dumping of the whole feed into the database as is won't work as we want to query most of the attributes). What parser would be the best fit? Feedzirra was chosen for performance, however, getting all data in the feed into the database is a priority.
Update
I'm using MySQL as the database.
I modeled my database on feeds and entries also, and cross-mapped the fields for RSS, RDF and Atom, so I could capture the required data fields as a starting point. Then I added a few others for tagging and my own internal-summarizations of the feed, plus some housekeeping and maintenance fields.
If you move from Feedzirra I'd recommend temporarily storing the actual feed XML in a staging table so you can post-process it using Nokogiri at your leisure. That way your HTTP process isn't bogged down processing the text, it's just retrieving content and filing it away, and updating the records for the processing time so you know when to check again. The post process can extract the feed information you want from the stored XML to store in the database, then delete the record. That means there's one process pulling in feeds periodically as quickly as it can, and another that basically runs in the background chugging away.
Also, both Typhoeus/Hydra and HTTPClient can handle multiple HTTP requests nicely and are easy to set up.
Store the XML as a CLOB, most databases have XML processing extensions that allow you to include XPath type queries as part of a SELECT statement.
Otherwise if your DBMS does not support XML querying, use your languages XPath implementation to query the CLOB. You will probably need to extract certain elements into table columns for speedy querying.
I am a newbie to rails and I have been watching Rails Casts videos.
I am interested to know a little bit more on FeedZirra (Rails casts episode 168) and especially feed parsing.
For example, I need to Parse feeds from Telegraph and Guardian
I want to put all the sports news from both the newspapers in one table, just football news in another table, cricket news in another table etc
How can I achieve that using feed-zirra?
How do I display only football news in one view and only cricket news in another view?
Also, I want the user to know which website he is gonna visit before he actually clicks the link and finds out.
Something like this
Ryder Cup 2010: Graeme McDowell the perfect hero for Europe
5 min ago | Telegraph.co.uk
How do I display Telegraph.co.uk
Looking forward for your help and support
Thanks
There are many questions there, but I'll take this one:
I just know how to put all feeds in
table. I dont know how to keep feeds
in different tables
Create different models to suit your data model, based on what information you need to show rather than what is provided in the feed. (Different tables for each models if required or Single Table Inheritance if possible)
Write a wrapper class that will use FeedZirra (or any other parser for that matter) to read the parsed feeds and process them. These are generally kept in the lib folder.
Create a rake task which can be called to run this script OR if you are familiar with delayed_job, then create a job.
Schedule your rake task through cron or your job through delayed_job, so that you can periodically update your data.