I created a web scraper.
I'm struggling to save the results into the model's columns.
How do I push the results of the scrapped results into the columns?
Do you map it? Would like to understand it so I can eventually post to index perhaps ... or show previous saved results, etc ...
Schema
ActiveRecord::Schema.define(version: 20170308223314) do
# These are extensions that must be enabled in order to support this database
enable_extension "plpgsql"
create_table "links", force: :cascade do |t|
t.string "link_info"
t.string "date"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
end
end
The two columns that I'm trying to save to are the ones you see in the schema above: link_info and date ...
Current Controller
class LinksController < ApplicationController
def craigslist_scrape
require 'open-uri'
url = "https://losangeles.craigslist.org/search/web"
page = Nokogiri::HTML(open(url))
#craigslist_info = page.css("ul.rows")
#link_info = page.css("li.result-row p.result-info a.result-title.hdrlnk")
#date = page.css("li.result-row p.result-info time.result-date")
end
end
craiglist_scrape.html.erb
<% #link_info.each_with_index do |link, index| %>
<h2><%= "Title of the job: #{link.text}" %></h2>
<p><%= "Date: #{#date[index].text}" %></p>
<% end %>
routes
Rails.application.routes.draw do
root 'links#craigslist_scrape'
end
model
class Link < ApplicationRecord
end
In the controller you can add:
def craigslist_scrape
require 'open-uri'
url = "https://losangeles.craigslist.org/search/web"
page = Nokogiri::HTML(open(url))
#craigslist_info = page.css("ul.rows")
#link_info = page.css("li.result-row p.result-info a.result-title.hdrlnk")
#date = page.css("li.result-row p.result-info time.result-date")
#link_info.each_with_index do |link, index|
Link.new(:link => link.text, :date => #date[index].text).save
end
end
I'm rather new to Ruby and don't know much about Rails but what I always do when I'm trying to save something into a "database" is I create a hash. For example:
database = {}
puts "What is your name?"
input = gets.chomp
puts "What's your age?"
input_2 = Integer(gets.chomp)
database[input.to_sym] = input_2
In the example above I create a hash called "database" and I ask the user for their name and age. Storing the user's name as the string value for database and their age as an integer value. I don't know if this helps at all, like I said I'm fairly new to Ruby.
Related
I am trying to check if a record is stale and update if it is. Here's my code:
#listing = Listing.where(listing_id: listing['listing_id'])
if #listing.exists?
if #listing.created_at < 7.days.ago # line where error shows
#listing.update(state: listing['state'])
end
end
And I'm getting the following error:
undefined method `updated_at' for #<Listing::ActiveRecord_Relation:0x00007fb3ebabbac0> Did you mean? update_all
Here's my DB record:
<Listing id: 5, listing_id: 188996174, state: "active", created_at: "2018-03-13 20:43:35", updated_at: "2018-03-13 20:46:48">
And my migration:
class CreateListings < ActiveRecord::Migration[5.1]
def change
create_table :listings do |t|
t.integer :listing_id
t.string :state
t.timestamps
end
end
end
Any ideas?
Thanks in advance!
using where, you get an ActiveRecord_Relation :
#listing = Listing.where(listing_id: listing['listing_id'])
You want an single object here.
I am unsure about exactly how you've set things, but assuming the listing_id column is unique, the following should do the job :
#listing = Listing.find_by(listing_id: listing['listing_id'])
the the rest of your code is okay. As it would only return the first object matching the listing-id. So then you could try to call updated_at on this object
If they're not unique, you might want to do the following :
#listings = Listing.where(listing_id: params['listing_id'])
#listings.each do |listing|
listing.update(state: params['state']) if listing.created_at < 7.days.ago
end
Or shorter :
Listing
.where(
'listing_id = ? AND updated_at > ?',
params['listing_id'],
Time.zone.now - 7.days
)
.update_all(state: params['state'])
My problem is that i have to export an excel sheet save some rows to the database without duplication or redundancy
so i started it with importing CSV instead of XLS then when i finish i might be able to parse the xls
this is my model code:
require 'csv'
class Machine < ActiveRecord::Base
def self.assign_row(row)
a, b, c, d = row
#c = c.slice(1,4)
Machine.create(name: c, mid: #c)
end
def self.import(file)
CSV.foreach(file.path) do |row|
machine = Machine.assign_row(row)
end
end
end
Import method in machines_controller
def import
count = Machine.import params[:file]
redirect_to machines_path, notice: "file imported successfully!"
end
Migration code
def change
create_table :machines do |t|
t.string :name
t.string :mid
t.timestamps null: false
end
add_index :machines, :name, :unique => true
end
and the view code
<%= form_tag import_machines_path, multipart: true do %>
<%= file_field_tag :file %>
<%= submit_tag "upload" %>
<% end %>
routes
Rails.application.routes.draw do
resources :errors
resources :machines do
collection do
post :import
end
end
root "machines#index
end
any thoughts on how to skip duplicated records from saving into database would be appreciated
thanks
Unique Identifier:
To avoid duplicate records saving to database you should maintain a unique identifier other than primary key. This helps you to identify if the record already available in DB, if it is available you can skip that record from saving again.
I guess you can use name in this case, which should be unique for each record in database. write a uniqueness validation in model to implement this.
After changes:
validates_uniqueness_of :name
def self.assign_row(row)
a, b, c, d = row
#c = c.slice(1,4)
machine = Machine.find_by(name: c)
Machine.create(name: c, mid: #c) if machine.blank?
end
Hope it helps!!
Thank you.
I am trying to make an app in Rails 4. I use simple form for forms and country_select gem for country lists.
I have an address model, which includes this method:
def country_name
self.country = ISO3166::Country[country]
country.translations[I18n.locale.to_s] || country.name
end
When I try to save a new address, I get this error:
undefined method `translations' for "Australia":String
Can anyone see what's wrong with this method definition?
If I change my view to:
<% if #profile.addresses.any? %>
<%= #profile.addresses.first.country.titlecase %>
<% else %>
<span class="profileeditlink">
<%= link_to "Add your location", new_address_path %>
</span>
<% end %>
Then the record displays - but as AU instead of Australia (which is what the method in the address model provides).
Address table:
create_table "addresses", force: :cascade do |t|
t.string "unit"
t.string "building"
t.string "street_number"
t.string "street"
t.string "city"
t.string "region"
t.string "zip"
t.string "country"
t.boolean "main_address"
t.boolean "project_offsite"
t.string "time_zone"
t.float "latitude"
t.float "longitude"
t.integer "addressable_id"
t.integer "addressable_type"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
end
add_index "addresses", ["addressable_type", "addressable_id"], name: "index_addresses_on_addressable_type_and_addressable_id", unique: true, using: :btree
TAKING JAEHYEN'S SUGGESTION,
I changed my country name method in the address model to:
def country_name
# self.country = ISO3166::Country(country)
# country.translations[I18n.locale.to_s] || country.name
iso_country = ISO3166::Country.find_by_name[country] # `country` should be name like 'Australia'
iso_country.translations[I18n.locale.to_s] || iso_country.name
end
I get this error:
undefined method `translations' for nil:NilClass
ANOTHER ATTEMPT:
I found this resource: http://www.scriptscoop.net/t/4ee6d5ef4577/displaying-countries-using-country-select-gem-in-rails-4.html
I tried changing my form input to:
<%= f.country_select :country, priority: [ "Australia", "New Zealand", "United Kingdom" ] %>
It still just displays the country code instead of the country name. I'm stuck.
ANOTHER ATTEMPT
I found this post:
Rails Simple_Form: How to show long name of the country
The answer in this post suggests defining country_name as:
def country_name
country = ISO3166::Country[country_code]
country.translations[I18n.locale.to_s] || country.name
end
This is slightly different to my previous attempts, however, when I try this, I get this error:
undefined local variable or method `country_code' for #<Address:0x007fbae5bfb290>
I tried changing the method to:
def country_name
self.country = ISO3166::Country[country_code]
country.translations[I18n.locale.to_s] || country.name
end
This gives the same error as the formulation that does not use 'self'. I think these attempts don't work because the attribute in my address table is called 'country'.
When i change the method to:
def country_name
self.country = ISO3166::Country[country]
country.translations[I18n.locale.to_s] || country.name
end
I get the error with the word 'translations'. When I delete '.translations' from the method, I get an error with the word 'name'.
I'm losing my marbles trying to figure this out.
ANOTHER ATTEMPT
I tried adding the countries gem to my gem file (above country_select).
Nothing changes when I bundle this gem. Same problem.
ANOTHER ATTEMPT
Trying again (as I originally had the method defined), but with countries gem installed (above country_select gem):
def country_name
self.country = ISO3166::Country[country]
country.translations[I18n.locale.to_s] || country.name
end
I get this error: undefined method `translations' for "Cayman Islands":String
This is the same problem that I originally started with, so I don't think that adding the countries gem has helped advance toward a solution.
This code work well for me. I had country column in my User table.
On my model :-
def country_name
country = self.country
ISO3166::Country[country]
end
in the form :-
<%= form_for #user do |f| %>
<%= country_select("user", "country") %>
<% end %>
You Question is not enough, there are some point that should make error.
First,
def country_name
self.country = ISO3166::Country[country]
country.translations[I18n.locale.to_s] || country.name
end
In this methods, you should make clear country / self.country variable or instance. I can not imagine country is ISO3166::Country instance or String
Second,
Make clear it use ISO code or country name as hash key
Update:
you have country column that is string.
calling self.country = ISO3166::Country[country] means assigning ISO3166::Country instance in country(string) variable. so it makes error.
I do not know what you expect.
But you should use key of ISO03166::Country as ISO code. like ISO3166::Country['AU']. And you can not assign this in self.country.
def country_name
iso_county = ISO3166::Country[country] # `country` should be iso_code
iso_country.translations[I18n.locale.to_s] || iso_country.name
end
If you have country name, not ISO code in country column. use ISO3166::Country.find_by_name
def country_name
iso_county = ISO3166::Country.find_by_name(country) # `country` should be name like 'Australia'
iso_country.translations[I18n.locale.to_s] || iso_country.name
end
It looks like the answer will be a certain combination of your previous attempts.
Country_select uses the country codes instead of full country names:
country_select("user", "country", priority_countries: ["GB", "FR", "DE"])
This is shown on the github page for that gem:
https://github.com/stefanpenner/country_select
Since the select returns a country code then we can assume that the country attribute will be a country code and not a full country name.
With this in mind we'll modify a previous answer:
def country_name
iso_country = ISO3166::Country[country] # `country` should be code like 'AU'
iso_country.translations[I18n.locale.to_s] || iso_country.name
end
It's best practice to avoid creating a variable with the same name as an attribute in that model, so stick with iso_country instead of country in your testing.
The issue that you've been encountering is that when you assign the ISO3166::Country object to an object's attribute (self.country = ISO3166::Country[country]) it's not assigning the ISO3166::Country object itself to that attribute, but instead is assigning the ISO3166::Country object's name to it.
I tested this in Rails console:
user = User.new
#<User:0x007ff7de29d1b8
id: nil,
...
user.country = ISO3166::Country['AU']
#<ISO3166::Country:0x007ff7de3c6a08
#data=
{"continent"=>"Australia",
...
user.country
"Australia"
As you can see, it assigned the country name (not the country object) to the attribute. Therefore when you try to access country.translations or country.name it will throw an error because it is accessing the address attribute and not the country object. So definitely keep the variable name different from the attribute name.
One last thing, if you do have the country name then use the method find_country_by_name instead of find_by_name since find_by_name will return an array, while find_country_by_name will return a Country object. This should not be necessary in your current scenario, but keep it in mind if you need to use it later.
Getting the following error message in the log
ActionController::ParameterMissing - param is missing or the value is empty: user_evaluation_result:
Within the Drills controller under certain conditions I am trying to insert a row in the user_evaluation_results. The lines of code from the Drills Controller are below. I have the feeling I am missing something obvious
....
user_evaluation_result_req = Rails.cache.read("user_evaluation_result_req")
if user_evaluation_result_req
id = Rails.cache.read("evaluation_assumption_id")
#user_evaluation_results = UserEvaluationResult.get_user_evaluation_results(id)
#result_list.each do |result|
if #user_evaluation_results.present?
puts "user evaluation result present "
else
#user_evaluation_result = UserEvaluationResult.new
#user_evaluation_result.evaluation_assumption_id = id
#user_evaluation_result.company_listing_id = 124
#user_evaluation_result.target_share_price_dollars = 7333
#user_evaluation_result = userEvaluationResult.new(user_evaluation_result_params)
end
end
....
# further down in Drills Controller
def user_evaluation_result_params
params.require(:user_evaluation_result).
permit(:evaluation_assumption_id,
:company_listing_id,
:target_share_price_dollars )
end
The value for user_evaluation_results currently hard coded but that changes when I get it working.
The model for user_evaluation_result
class UserEvaluationResult < ActiveRecord::Base
include ActiveModel::ForbiddenAttributesProtection
before_validation :clean_data
belongs_to :company_listing
belongs_to :evaluation_assumption
validates :evaluation_assumption_id, :company_listing_id,
:target_share_price_dollars, presence: true
validates :target_share_price_dollars, :numericality => { :only_integer => true,
:greater_than_or_equal_to => 0}
validates :company_listing_id, uniqueness: {scope: [:evaluation_assumption_id]}
def self.get_user_evaluation_results(id)
where("evaluation_assumption_id = ?", id)
end
def target_share_price_dollars
target_share_price.to_d/1000 if target_share_price
end
def target_share_price_dollars=(dollars)
self.target_share_price = dollars.to_d*1000 if dollars.present?
end
private
def clean_data
# trim whitespace from beginning and end of string attributes
attribute_names.each do |name|
if send(name).respond_to?(:strip)
send("#{name}=", send(name).strip)
end
end
end
end
table for user_evaluation_result
create_table "user_evaluation_results", force: true do |t|
t.integer "company_listing_id"
t.integer "target_share_price"
t.datetime "created_at"
t.datetime "updated_at"
t.integer "evaluation_assumption_id"
end
any help welcome and of course can post further details
thanks Pierre
Your UserEvaluationResult called here: #user_evaluation_result = userEvaluationResult.new(user_evaluation_result_params) is lowercase and must be capital so Rails knows you are referring to the User Evaluation Result class. You are getting that error because the params you defined require a UserEvaluationResult to be acting on yet you are not providing one.
I need to upload a text file without saving it in the database. My goal is to upload this file and automatically have to take your content and save it in my database.
my file: data.txt
name age occupation
julio 19 student
ramon 20 student
my database:
class CreateStudents < ActiveRecord::Migration
def change
create_table: students do |t|
t.string "name"
t.integer "age"
t.string "occupation"
t.timestamps
end
end
end
Does anyone have any idea how this can be done? I searched on the internet, but found no solution to my case. I need help.
= form_tag url, {multipart: true} do
= file_field_tag :file
....
in controller
if params[:file]
lines = params[:file].tempfile.readlines.map(&:chomp) #readlines from file & removes newline symbol
lines.shift #remove first line
lines.each do |l|
m = l.match(/(\S+)\s(\d+)\s(\S+)/) #parse line
Student.create {name: m[1],age: m[2], occupation: m[3]}
end
end