I need to read the content of a website, and save HTML of a part of this page.
For example, let's say I want to get only the description of an athlete on this page : https://www.olympic.org/usain-bolt : the section.text-content element.
How can I do in Rails to store that HTML in my database, to be able to provide it later via an API?
Anyone have a clue about this?
You can get the description easily openning the url, parsing the html and accessing the element you pointed, like:
require 'nokogiri'
require 'open-uri'
url = 'https://www.olympic.org/usain-bolt'
doc = Nokogiri.HTML(open(url))
puts doc.css('section.text-content').text
As you already have the data then you need a model where to store, you can create a new one, just as example called Athlete, using the rails generate command and migrate, like
$ rails g model Athlete description:text
$ rails db:migrate
The description is a text data type attribute, which allow you to store large texts, as the description.
Then you need to insert it, or update it. You can create a new record, and then update it. In the rails console, just:
Athlete.create
This will create a new athlete without description, but necessary to get it by its id. After that you can create a task, in the lib/tasks folder, you can create a file with .rake extension and add your code, using the way to create a task, like:
require 'nokogiri'
require 'open-uri'
namespace :feed do
desc 'Gets the athlete description and insert it in database.'
task athlete_description: :environment do
url = 'https://www.olympic.org/usain-bolt'
doc = Nokogiri.HTML(open(url))
description = doc.css('section.text-content').text
Athlete.find(1).update description: description
end
end
You have the libraries, get the data, and update to the record using ActiveRecord, you can easily run:
rails feed:athlete_description
# or
rake feed:athlete_description
Nokogiri might be able to do what you need by way of CSS selectors.
If not, you can use Net:HTTP to get the page contents into a local variable, then you can use string manipulation to find the piece you want and store it. Unfortunately, I don't think there's any straightforward way to select that element specifically with this method.
Related
Ruby on Rails 3
I have a table that is not shown on my application. I want to export the table to excel from the console.
Is there way to export a database table to an excel file from the console?
Thank you
Here is what I have so far:
require 'csv'
file = "#{Rails.root}/public/survey.csv"
userrefs = Survey.all
CSV.open( file, 'w' ) do |writer|
userrefs.each do |ur|
writer << ur.attributes.values_at(*column_names)
end
end
When I enter require 'csv' it returns false. How do you make it true?
Also, the *column_names is undefined.
As mentioned in the comments an easy approach is using format.xls function to render an excel file from the console. Ryan Bates video covers Excel outputs extensively.
You can connect to the database table using the existing driver or, if you prefer a more high-level API, you can create an ActiveRecord model or use the Sequel gem.
Once connected, simply use the Ruby CSV library (it's in the standard library, so no additional library is required) to dump the content into a CSV file.
CSV files can be easily read from Excel.
PS. Just to use the appropriate words, that is not a Rails table. It's a database table.
Another interesting approach could be using the activeadmin gem, it's easy to install and allow you to export your tables to csv.
http://www.activeadmin.info/
I have a table that I'm going to store units of measurements in and the corresponding category. For example, "cups, volume" and "ounce, mass". I only have about 12 measurements I care about, and I want them to be in the database without manually entering them because anytime I want to deploy it or move the code somewhere, I want this to recreate.
So, I figure the best way to do this is to create a csv file with them, and then use rake db:seed. But I've never done this before so can someone guide me as to how to create this csv, and how to write the seeds.rb file to generate that? Is this the right way to do it?
SOLUTION:
#db/seeds.rb
require 'open-uri'
#Supply all the units of measurement to be used
Unit.delete_all
open("db/seeds/measurements.csv") do |measurements|
measurements.read.each_line do |measurement|
unit, uof = measurement.chomp.split(",")
Unit.create!(:name => unit, :unit_of => uof)
end
end
Just write the code you'd use to generate them manually and put it in the seeds.rb. You can use normal ruby in that file; so either just write plain create's or load a csv you store in, e.g. db/seeds/measurements.csv. To get the path to that file, use Rails.root.join("db/seeds/measurements.csv").
Example:
File.open(Rails.root.join("db/seeds/measurements.csv")) do |f|
# use f here
end
There's a file called seeds.rb in db directory.
You can add your default data in it and when you execute rake db:seed, that data will be populated in your database.
Here's an example of how you can add seeds to it.
product_types = ProductType.create([
{:name => 'clock',:organisation_id => 1}
])
I coded a little Ruby script that would parse a remote XML file and extract some data from it using Nokogiri. Now I'm trying to code a more advanced version as a Rails application.
I have my code inside of a controller. It's similar to the code that I used in my Ruby script, however it's not working. I believe the error is because it's trying to load the XML locally rather then externally.
Here is the error that Rails is giving me:
No such file or directory - http://mal-api.com/anime/10?format=xml
Here is a sample of the code in my controller: (I can provide the whole thing if needed, but it's mainly just the default Rails scaffold code.)
def create
require 'nokogiri'
#anime = Anime.new(params[:anime])
doc = Nokogiri::XML(open("http://mal-api.com/anime/#{#anime.mal_id}?format=xml"))
#Title
title = doc.css("anime english_title").inner_html
#Snipped rails scaffold code
end
mal_id is passed in through a form. Nokogiri is added in my Gemfile.
Is there something I'm missing or that I've done wrong?
Any help is appreciated.
By default the open method in ruby is used to open files. If you want to directly open an URL you need to require 'open-uri'. More information can be found in the docs: http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open-uri/rdoc/OpenURI.html
I have a bit of code that I need to run when a user loads a page. What it does specifically is update my database based on an xml file using a gem. So far I've found answers that tell me I should put it in everything from a rake task to the lib folder itself to the model. It's all a little confusing.
Here's the code in question:
require 'rubygems'
require 'eaal'
EAAL.cache = EAAL::Cache::FileCache.new
api = EAAL::API.new("id", "vcode", "char")
result = api.MarketOrders("characterID" => "id")
result.orders.each do |order|
#found = MarketItem.find_by_typeid(order.typeID.to_i)
MarketItem.update(#found.id, :remaining => order.volRemaining.to_i)
end
I'm sorry if this is an obvious question and I'm sure my code is horrible. I'm really new to rails and the only way I seem to be able to learn new languages is the bull-in-a-china-shop method.
There is a route to that page that hits a function first
for example. show_user_path will hit the function show. You can put your code at the beginning of the function that renders the page.
I have created a ruby on rails application and would like to automate the creation of new model instances. I am currently using python scripts to log information on many machines and would like to find a way to send these log results to the rails server. I need to populate the forms with the contents of my logs in order to generate the associated database entries.
Is there an easy way to send HTTP Post requests that contain the log attributes I want in order to create these new model instances and database entries?
require "net/http"
require "uri"
uri = URI.parse("http://example.com/search")
# Shortcut
response = Net::HTTP.post_form(uri, {"q" => "My query", "per_page" => "50"})
http://www.rubyinside.com/nethttp-cheat-sheet-2940.html