So I have the controller who scrapes the entire html of a page and stores it into mysql database. Before I store the data I want to encode it using the htmlentities gem. My issue is that with some websites it works ok e.g https://www.lookagain.co.uk/ but with others I get invalid byte sequence in UTF-8 such as https://www.google.co.uk/ and I do not know why. At first I though it might be something wrong with the database so I have changed all the fields to LONGTEXT but the problem still persists
Controller:
class PageScraperController < ApplicationController
require 'nokogiri'
require 'open-uri'
require 'diffy'
require 'htmlentities'
def scrape
#url = watched_link_params[:url].to_s
puts "LOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOG#{#url}"
#page = Nokogiri::HTML(open(#url))
coder = HTMLEntities.new
#encodedHTML = coder.encode(#page)
create
end
def index
#savedHTML = ScrapedPage.all
end
def show
#savedHTML = ScrapedPage.find(id)
end
def new
#savedHTML = ScrapedPage.new
end
def create
#savedHTML = ScrapedPage.create(domain: #url, html: #encodedHTML, css: '', javascript: '')
if #savedHTML.save
puts "ADDED TO THE DATABASE"
redirect_to(root_path)
else
puts "FAILED TO ADD TO THE DATABASE"
end
end
def edit
end
def upadate
end
def delete
#watched_links = ScrapedPage.find(params[:id])
end
def destroy
#watched_links = ScrapedPage.find(params[:id])
#watched_links.destroy
redirect_to(root_path)
end
def watched_link_params
params.require(:default).permit(:url)
end
end
Related
I'm making an export to csv file functionality in a Ruby on Rails repo and I'm almost done. However, when I press the "Export all" button, I get the undefined method `export' for nil:NilClass error. The log shows that format.csv { send_data #foos.export, filename: "foos-#{Date.today}.csv" } went wrong. What am I missing please?
This is model
class Foo < ApplicationRecord
has_many :bars
def export
[id, name, foos.map(&:name).join(' ')]
end
end
This is part of controller
def index
#foos = Foo.all
end
def export
all = Foo.all
attributes = %w{name}
CSV.generate(headers: true) do |csv|
csv << attributes
all.each do |foo|
csv << attributes.map{ |attr| foo.send(attr) }
end
respond_to do |format|
format.csv { send_data #foos.export, filename: "foos-#{Date.today}.csv" }
end
end
end
def name
"#{foo_id} #{name}"
end
This is View
<button class="btn btn-success">export all</button>
This is Routes
Rails.application.routes.draw do
resources :foos
get :export, controller: :foos
root "foos#index"
end
This is Rake (lib/tasks/export.rb)
namespace :export do
task foo: :environment do
file_name = 'exported_foo.csv'
csv_data = Foo.to_csv
File.write(file_name, csv_data)
end
end
Start by creating a service object that takes a collection of records and returns CSV so that you can test the CSV generation in isolation:
# app/services/foo_export_service.rb
# Just a Plain Old Ruby Object that converts a collection of foos into CSV
class FooExportService
# The initializer gives us a good place to setup our service
# #param [Enumerable] foo - an array or collection of records
def initialize(foos)
#headers = %w{name} # the attributes you want to use
#foos = foos
end
# performs the actual work
# #return [String]
def perform
CSV.generate do |csv|
#foos.each do |foo|
csv << foo.serializable_hash.slice(#headers).values
end
end
end
# A convenient factory method which makes stubbing the
# service easier
# #param [Enumerable] foos - an array or collection of records
# #return [String]
def self.perform(foos)
new(foos).perform
end
end
# example usage
FooExportService.perform(Foo.all)
Not everything in a Rails application needs to be jammed into a model, view or controller. They already have enough responsiblities. This also lets you resuse the code for example in your rake task if you actually need it.
This simply iterates over the collection and uses Rails built in serialization features to turn the model instances into hashes that can be serialized as CSV. It also uses the fact that Hash#slice also reorders the hash keys.
In your controller you then just use the service object:
class FoosController
def export
#foos = Foo.all
respond_to do |format|
format.csv do
send_data FooExportService.perform(#foos),
filename: "foos-#{Date.today}.csv"
end
end
end
end
You don't even really need a separate export action in the first place. Just use MimeResponds to add CSV as an availble response format to the index:
class FoosController
def index
# GET /foos
# GET /foos.csv
#foos = Foo.all
respond_to do |format|
format.html
format.csv do
send_data FooExportService.perform(#foos),
filename: "foos-#{Date.today}.csv"
end
end
end
end
<%= link_to("Export as CSV", foos_path(format: :csv)) %>
using ruby 2.6.5, Rails 6.0.3.7
There is before_action filter which are working fine when running the project in the development server. But while running the integration tests of the rails application.
The call back do not execute and the request goes directly to the called function rather than going to the before action first.
Here attaching my controller and integration test case and error output.
Controller
class TvSeriesController < ApplicationController
before_action :check_file, only: [:create, :tv_seriel_comments]
require 'roo'
def index
#tv_series = TvSeriel.includes(:comments).all
end
def show
#series = TvSeriel.includes({ comments: [:user] }).find(params[:id])
end
def create
spreadsheet = Roo::Spreadsheet.open(params[:file])
header = spreadsheet.row(1)
(2..spreadsheet.last_row).each do |line|
row = HashWithIndifferentAccess[[header, spreadsheet.row(line)].transpose]
tv_serial = TvSeriel.new
tv_serial.name = row['TV Series']
tv_serial.genre = row["Genre"]
tv_serial.seasons = row["No of seasons"]
tv_serial.release_date = row["Date of First Release"]
tv_serial.director = row["Director"]
tv_serial.actor = row["Actor"]
tv_serial.shoot_location = row["Shoot location"]
tv_serial.country = row["Country"]
tv_serial.save
end
flash[:notice] = 'TV series uploaded!'
redirect_back(fallback_location: tv_series_index_path)
end
def tv_seriel_comments
spreadsheet = Roo::Spreadsheet.open(params[:file])
header = spreadsheet.row(1)
(2..spreadsheet.last_row).each do |line|
row = HashWithIndifferentAccess[[header, spreadsheet.row(line)].transpose]
comment = Comment.new
user = User.find_by_name(row["User"])
comment.user = user
tv_serial = TvSeriel.find_by_name(row['TV Series'])
comment.tv_seriel = tv_serial
comment.stars = row['Stars']
comment.review = row['Review']
comment.save
end
flash[:notice] = 'TV series comment uploaded!'
redirect_back(fallback_location: tv_series_index_path)
end
def destroy
TvSeriel.find(params[:id]).destroy
flash[:notice] = 'TV series removed success!'
redirect_back(fallback_location: tv_series_index_path)
end
def search_actor
#tv_series = TvSeriel.includes(:comments).where("actor like ?", "%#{params[:actor]}%")
render 'tv_series/index'
end
## check if uploaded file is a cvs
def check_file
unless params[:file].blank?
if params[:file].content_type.eql?("text/csv")
else
flash[:alert] = "file format Invalid, expected: text/csv Got #{params[:file].content_type}"
redirect_back(fallback_location: :tv_series_index_path)
end
else
flash[:alert] = 'Please provide a file to be uploaded !'
end
end
end
Integration test:
require 'test_helper'
class UserFlowsTest < ActionDispatch::IntegrationTest
fixtures :all
include Devise::Test::IntegrationHelpers
include Warden::Test::Helpers
test 'Get list of all users' do
#user = users(:vidur)
sign_in(#user)
get "/users"
assert_response :success
assert_select "h2", "User List"
end
test 'Update Series' do
#user = users(:vidur)
sign_in(#user)
post '/tv_series'
assert_response :success
end
end
Error:
# Running:
E
Error:
UserFlowsTest#test_Update_Series:
TypeError: no implicit conversion of nil into String
app/controllers/tv_series_controller.rb:15:in `create'
test/integration/user_flows_test.rb:19:in `block in <class:UserFlowsTest>'
Here the before_action filter do not execute and the request goes directly to the defined action can any body give the reason why only running test, and how to correct it, Thanks in advance.
Here is the git repo for complete code: https://github.com/vidurpunj/sopra_test.
It's because you don't have a file parameter.
So your check_file is using this part:
flash[:alert] = 'Please provide a file to be uploaded !'
And since you don't render or redirect, the controller action is called normally. See the Rails documentation on filters for more details:
If a "before" filter renders or redirects, the action will not run
I recently started getting complaints that locked users cannot reset their accounts from the email I am sending.
I'm getting
NameError (uninitialized constant Unlock):
config/initializers/quiet_assets.rb:6:in `call_with_quiet_assets'
Nothing has changed on my routesr and nothing has changed in my links... what could be causing this error message and how can I fix it?
quiet_assets.rb:
Rails.application.assets.logger = Logger.new('log/logger.txt')
Rails::Rack::Logger.class_eval do
def call_with_quiet_assets(env)
previous_level = Rails.logger.level
Rails.logger.level = Logger::ERROR if env['PATH_INFO'].index("/assets/") == 0
call_without_quiet_assets(env).tap do
Rails.logger.level = previous_level
end
end
alias_method_chain :call, :quiet_assets
end
Stacktrace:
INFO Started GET "/users/unlock?unlock_token=CzpxHwV5kL7EyDZb32Ex" for 127.0.0.1 at 2018-11-22 21:08:56 +0200
INFO Processing by Devise::UnlocksController#show as HTML
INFO Parameters: {"unlock_token"=>"CzpxHwV5kL7EyDZb32Ex"}
INFO -- store_location: /users/unlock?unlock_token=CzpxHwV5kL7EyDZb32Ex
INFO Completed 500 Internal Server Error in 8.1ms
FATAL NameError (uninitialized constant Unlock):
config/initializers/quiet_assets.rb:6:in `call_with_quiet_assets'
INFO Rendered /Users/david/.rvm/gems/ruby-1.9.3-p551/gems/actionpack-3.2.17/lib/action_dispatch/middleware/templates/rescues/_trace.erb (1.7ms)
INFO Rendered /Users/david/.rvm/gems/ruby-1.9.3-p551/gems/actionpack-3.2.17/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (0.8ms)
Controller
require 'white_label_utils'
include ERB::Util
#Need to include these helpers or image_path won't work
include Sprockets::Helpers::RailsHelper
include Sprockets::Helpers::IsolatedHelper
include ActionView::Helpers::AssetTagHelper
class CustomMailer < Devise::Mailer
include Devise::Mailers::Helpers
include Devise::Controllers::UrlHelpers # Optional. eg. `confirmation_url`
helper :application # gives access to all helpers defined within `application_helper`.
default from: 'no-reply#ourcompany.com'
# Sends a simple email (without attachments and call to action button)
# to - array of recipients
# subject - subject title of email
# message - body of email
# section - section used for white labeling purposes
# file_attachments - array of files attachments to be included in email. eg. [{name: 'myFileName.csv', content: fileData}, {name: 'anotherFileName.zip', content: anotherFileData}] Empty by default
def simple_email(to, subject, message, section = nil, file_attachments = [])
get_whitelabel_details(section)
set_email_global_params(to, subject, message)
# if there are files, attach them
file_attachments = [] if file_attachments.nil?
if file_attachments.length > 0
file_attachments.each do |file|
attachments["#{file[:name]}"] = file[:content]
end
end
mail(to: to, subject: #subject, from: #whitelabel.email_from).delivery_method.settings.merge!(Dynamic::Application.config.send_grid_smtp_settings)
end
# Sends a devise invitation_instructions email located under app/views/user/mailer/invitation_instructions.html
def invitation_instructions(record, opts = {})
to = record.email
begin
section = record.publisher.sections.first
rescue => e
Rails.logger.warn "Could not find any sections associated with user #{record}"
section = nil
end
initialize_from_record(record)
get_whitelabel_details(section)
set_email_global_params(to, invitation_email_subject, I18n.t("devise.mailer.invitation_instructions.message", invitee_name: record.first_name.capitalize, inviter_name: #resource.invited_by.is_dy_admin ? #resource.invited_by.last_name.capitalize : #resource.invited_by.full_name, product_name: #whitelabel.product_name))
uri = URI.parse(edit_invitations_url({invitation_token: #resource.invitation_token}))
#cta_text = I18n.t("devise.mailer.invitation_instructions.call_to_action_text")
#cta_link = whitelabel_links(uri)
super record, opts
end
# Sends a devise unlock_instructions email located under app/views/user/mailer/unlock_instructions.html
def unlock_instructions(record, opts = {})
to = record.email
begin
section = record.publisher.sections.first
rescue => e
Rails.logger.warn "Could not find any sections associated with user #{record}"
section = nil
end
initialize_from_record(record)
get_whitelabel_details(section)
set_email_global_params(to, I18n.t("devise.mailer.unlock_instructions.subject"), I18n.t("devise.mailer.unlock_instructions.message"))
uri = URI.parse(unlock_url(#resource, :unlock_token => #resource.unlock_token))
#cta_text = I18n.t("devise.mailer.unlock_instructions.call_to_action_text")
#cta_link = whitelabel_links(uri)
super record, opts
end
# Sends a devise reset_password_instruction email located under app/views/user/mailer/reset_password_instructions.html
def reset_password_instructions(record, token = nil, opts = {})
to = record.email
begin
section = record.publisher.sections.first
rescue => e
Rails.logger.warn "Could not find any sections associated with user #{record}"
section = nil
end
initialize_from_record(record)
get_whitelabel_details(section)
set_email_global_params(to, I18n.t("devise.mailer.reset_password_instructions.subject"), I18n.t("devise.mailer.reset_password_instructions.message"))
uri = URI.parse(edit_password_url(#resource, {reset_password_token: #resource.reset_password_token}))
#cta_text = I18n.t("devise.mailer.reset_password_instructions.call_to_action_text")
#cta_link = whitelabel_links(uri)
super record, opts
end
private
def devise_mail(record, action, opts={})
initialize_from_record(record)
(mail headers_for(action, opts)).delivery_method.settings.merge!(Dynamic::Application.config.send_grid_smtp_settings)
end
# Customize the subject and sender display name by the white-label profile
def headers_for(action, opts)
headers = {:from => #email_from,
:reply_to => #email_from}
super.merge!(headers)
end
# Overrides the default subject line for devise emails (reset_password_instructions, invitation_instructions, etc)
def subject_for(key)
return super unless key.to_s == 'invitation_instructions'
invitation_email_subject
end
# Gets the whitelabel details associated with the section
def get_whitelabel_details(section)
#section = section.blank? ? err_message('section') : section
begin
#whitelabel = WhiteLabelUtils::get_profile(nil, section.site.publisher)
rescue => e
Rails.logger.warn "Could not determine WhiteLabel profile when sending email"
#whitelabel = WhiteLabelUtils::get_profile(nil, nil)
end
end
# Validates the existence of parameters and assigns them to global vars that will be used in the email template itself
def set_email_global_params(to, subject, message)
#errors = nil
#to = to.blank? ? err_message('to') : to
#subject = subject.blank? ? err_message('subject') : subject.slice(0, 1).capitalize + subject.slice(1..-1).chomp('.') #remove trailing period, we add this in the template so this avoids duplicates
#message = message.blank? ? err_message('message') : message.slice(0, 1).capitalize + message.slice(1..-1).chomp('.') #remove trailing period, we add this in the template so this avoids duplicates
#email_from = #whitelabel.email_from
#reply_to = #whitelabel.reply_to
#introduction = create_introduction(to)
unless #errors.blank?
raise #errors
end
end
def invitation_email_subject
I18n.t("devise.mailer.invitation_instructions.subject", product_name: #whitelabel.product_name)
end
# Receives a generic url and replaces it with whitelabelled domains
def whitelabel_links uri
"#{#whitelabel.root_url_info[:protocol]}://#{#whitelabel.root_url_info[:host]}#{uri.path}?#{uri.query}"
# rendered as https://companydomain.com/users/unlock?unlock_token=TOKENVALUE
end
# Searches the system for a user with 'email_address' and returns
# a personalized introduction with the user's first name otherwise
# returns a generic introduction with the wording 'Dear User'
def create_introduction email_address
user = User.where(email: email_address).first
"#{I18n.t("dy.common.general.hi")}#{user.nil? ? '' : " #{user.first_name.capitalize}"},"
end
def err_message(val)
#errors = #errors.blank? ? '<' + val + '> field can not be empty' : #errors + '\n<' + val + '> field can not be empty'
end
end
The route being called is https://example.com/users/unlock?unlock_token=USERS_TOKEN
and the controller is from devise unlockscontroller
class Devise::UnlocksController < DeviseController
prepend_before_filter :require_no_authentication
# GET /resource/unlock/new
def new
build_resource({})
end
# POST /resource/unlock
def create
self.resource = resource_class.send_unlock_instructions(resource_params)
if successfully_sent?(resource)
respond_with({}, :location => after_sending_unlock_instructions_path_for(resource))
else
respond_with(resource)
end
end
# GET /resource/unlock?unlock_token=abcdef
def show
self.resource = resource_class.unlock_access_by_token(params[:unlock_token])
if resource.errors.empty?
set_flash_message :notice, :unlocked if is_navigational_format?
respond_with_navigational(resource){ redirect_to after_unlock_path_for(resource) }
else
respond_with_navigational(resource.errors, :status => :unprocessable_entity){ render :new }
end
end
protected
# The path used after sending unlock password instructions
def after_sending_unlock_instructions_path_for(resource)
new_session_path(resource) if is_navigational_format?
end
# The path used after unlocking the resource
def after_unlock_path_for(resource)
new_session_path(resource) if is_navigational_format?
end
end
This was never properly resolved. I ended overriding the built-in devise callback functions (which is a horrible practice)
Old versions of Rails3 and Ruby 1.9.3 are most probably the culprits - but I don't have the privilege of upgrading - too much legacy code working in production already.
Story of my life :(
I am attempting to download a page from Wikipedia. For such a task, I am using gems. When using net/http, all I get is an empty string. So I tried with open-uri and it works fine.
Nevertheless, I prefer the first option because it gives me a much more explicit control; but why is it returning an empty string?
class Downloader
attr_accessor :entry, :url, :page
def initialize
# require 'net/http'
require 'open-uri'
end
def getEntry
print "Article name? "
#entry = gets.chomp
end
def getURL(entry)
if entry.include?(" ")
#url = "http://en.wikipedia.org/wiki/" + entry.gsub!(/\s/, "_")
else
#url = "http://en.wikipedia.org/wiki/" + entry
end
#url.downcase!
end
def getPage(url)
=begin THIS FAULTY SOLUTION RETURNS AN EMPTY STRING ???
connector = URI.parse(url)
connection = Net::HTTP.start(connector.host, connector.port) do |http|
http.get(connector.path)
end
puts "Body:"
#page = connection.body
=end
#page = open(url).read
end
end
test = Downloader.new
test.getEntry
test.getURL(test.entry)
test.getPage(test.url)
puts test.page
P.S.: I am an autodidact programmer so the code might not fit good practices. My apologies.
Because your request return 301 Redirect (check connection.code value), you should follow redirect manually if you are using net/http. Here is more details.
This code is working correctly. The problem is everything is getting printed to the console. I want to show it on the browser. How to do that? Do I need a template called create.html.erb. How to access the variables and basically the whole controller code in the view? Please help!!
require File.join(Rails.root, 'config/myconfig')
puts Rails.root
class UsersController < ApplicationController
layout 'admin'
#require File.expand_path('././myconfig') #=> C:/ruby/require/expand_path/ok.rb loaded
def list
#users = User.all
end
def new
#user = User.new
end
def create
if EntityList::ENTITIES.include?(params[:user][:entity_name])
puts " Entered entity is: "
#entity = params[:user][:entity_name]
#var = EntityList::RELATION_SHIPS[#entity.to_sym]
puts "Entered entity is related to"
if(#var.nil?)
#do nothing
else
puts #var
checking(#var)
end
##var.split(" ")
##len = #var.length
#puts #var[0]
#puts #var[1]
#puts #var[1]
##var.each {|#var| puts #var}
#for index in 0 ... #var.size
# puts EntityList::RELATION_SHIPS[#var[index].to_sym]
# end
# #var.each_with_index {|val, index| puts "#{val} => #{index}" }
##var2= EntityList::RELATION_SHIPS[#var.to_sym]
#puts "Entity2 is related to"
#puts #var2
flash[:notice] = "The entity you entered is valid!!"
puts "Before Redirection"
redirect_to(:action => "helloworld")
puts "After redirection"
puts "done"
else
redirect_to(:action => "SorryPage")
end
end
def checking(array)
array.split(" ")
for index2 in 0 ... array.size
if EntityList::RELATION_SHIPS[array[index2].to_sym].nil?
# do nothing
else
puts EntityList::RELATION_SHIPS[array[index2].to_sym]
some = EntityList::RELATION_SHIPS[array[index2].to_sym]
checking(some)
end
end
end
end
Yes, you would create an create.html.erb and can access all Controller Instance variables like #entity or #var from there.
I'd recommend looking at the output from a generate scaffold call to get an example how this works, e.g. by calling it in a new rails app:
rails new tryout
cd tryout
rails generate scaffold User name:string email:string
and then look at the generated controller and view templates.