I have the following script, which I want to automatically update my database.
require 'feedzirra'
require 'sanitize'
namespace :db do
task :update_pitzer => :environment do
feed = Feedzirra::Feed.fetch_and_parse('http://www.cafebonappetit.com/rss/menu/219')
feed.entries.each do |entry|
date = Sanitize::clean(entry.title)
summary = Sanitize::clean(entry.summary)
summary.gsub!(/\s{2}/, " ").to_s
summary.gsub!(/\s{2}/, "").to_s
summary.gsub!("&", "&")
puts "#{date}"
theDay = Day.create(date: date)
#theDay.collins_meals.create()
array = summary.split(/\[/)
array.collect! {|x| "[" + x}
array.reject! {|x| x == "[ "}
array.to_s
puts array
end
end
end
When I run rake db:update_pitzer at the command line, I get:
(in /Users/Alex/rails_projects/dining_hall_api)
rake aborted!
Don't know how to build task 'db:update_pitzer'
Thoughts?
Code seems proper. Make sure you have put 'update_pitzer.rb' file under 'lib/tasks' folder in your rails application
Related
I am trying to import data from CSV into the database using Classes so that I can easily write Test Case for the csv import rake task I created
However, my solution does not work.
And I also feel:
It doesn't make sense
Aside feeling its not a good solution that connotes Ruby mastery, it doesn't work.
Here is what I came up with in my engines/csv_importer/lib/tasks/csv_import.rake
require 'open-uri'
require 'csv'
namespace :csv_import do
desc 'Import users from csv'
task users: :environment do
WebImport.new(url: 'http://blablabla.com/details/people.csv').call.answers
end
end
class WebImport
def initialize(url)
#csv_string = url
end
def call
CSV.parse(#csv_string, headers: true, header_converters: :symbol) do |row|
next unless row[:name].present? && row[:email_address].present?
end
CsvImporter::User.create row.to_h
end
def self.answers
user = []
counter = 0
duplicate_counter = 0
user.persisted? ? counter + 1 : duplicate_counter + 1
p "Email duplicate record: #{user.email_address} - #{user.errors.full_messages.join(',')}" if user.errors.any?
p "Imported #{counter} users, #{duplicate_counter} duplicate rows ain't added in total"
end
end
Error when I run rake csv_import:users
$ rake csv_import:users
rake aborted!
NoMethodError: private method `gets' called for {:url=>"http://blablabla.com/details/people.csv"}:Hash
How do I make this work and write TEST for this at the long run?
You are getting this error because you are passing a hash to CSV.parse while that method accepts a string.
To fix that you need to change argument from a hash to a string: WebImport.new('http://blablabla.com/details/people.csv') and read a remote CSV file before passing it to CSV.parse, for example: CSV.parse(open(url)).
You can try to use
rake db:seed
to import the data to your database using seed file as
require 'csv'
puts "Importing data..."
CSV.foreach(Rails.root.join("file_name.csv"), headers: true) do |row|
Model_name.create! do |model_name|
model_name.name = row[0]
model_name.email_address = row[1]
end
end
csv file should be in your project root folder
I have a large database with 4+ million addresses/records.
The rake command (below) worked fine when the database was a small test set, but now with the large database it just simply stalls.
rake geocode:all CLASS=YourModel
2 questions:
1. Is there any simple method to have geocoder code a null/nil lat and long when the records are called (on the fly). I have a feeling that this would be hard.
2. Anyone else have problems with geocode-ing a large dataset and using the rake command?
Thanks!
Update:
I create pull request based on this answer and now you can use batch in geocoder:
rake geocode:all CLASS=YourModel SLEEP=0.25 BATCH=100
I would use this solution for large database, i take it from geocoder gem rake task:
You can refine this for your needs.
Some example create rake task:
namespace :geocode_my_data do
desc "Geocode all objects in my databse."
task all: :environment do
klass = User
klass.where(geocoded: false).find_each(limit: 100) do |obj|
obj.geocode; obj.save
end
end
end
$> rake geocode_my_data:all
Used below code and put it into my lib/tasks folder as geocode_my_data.rake
To run:
rake geocode:all CLASS=YourModel
Works great!
namespace :geocode_my_data do
desc "Geocode all objects without coordinates."
task :all => :environment do
class_name = ENV['CLASS'] || ENV['class']
sleep_timer = ENV['SLEEP'] || ENV['sleep']
raise "Please specify a CLASS (model)" unless class_name
klass = class_from_string(class_name)
klass.not_geocoded.find_each(batch_size: 100) do |obj|
obj.geocode; obj.save
sleep(sleep_timer.to_f) unless sleep_timer.nil?
end
end
end
##
# Get a class object from the string given in the shell environment.
# Similar to ActiveSupport's +constantize+ method.
#
def class_from_string(class_name)
parts = class_name.split("::")
constant = Object
parts.each do |part|
constant = constant.const_get(part)
end
constant
end
I am writing a Ruby on Rails application which has a Rake task that can parse a CSV file.
Here is the code:
desc "Import Channels into DB\n Usage: rake channel_import"
task :import_channels, :path_to_channel_list do |t, args|
require "#{Rails.root}/app/models/channel"
require "csv"
filePath = args.path_to_channel_list
puts "Filepath received = #{filePath}"
csv = CSV.read("#{filePath}", :encoding => 'windows-1251:utf-8')
csv.each_with_index do |row, i|
if [0].include?(i)
puts "Skipping over row #{i}"
next
end
if(row.nil?)
puts "row[#{i}] was nil"
else
channelName = nil
classif = nil
owner = nil
channelName = row[0].force_encoding('UTF-8')
classif = row[1].force_encoding('UTF-8')
owner = row[2].force_encoding('UTF-8')
if (channelName.nil?)
puts "Channel name for row #{i} was nil"
#add entry to Log file or errors database
next #skip this row of the csv file and go to next row
else
channel_hash = Hash.new("name" =>"#{channelName}", "classification" => "#{classif}", "owner" => "#{owner}" )
end
puts "\nChannel Name = #{channelName}\nClassification = #{classif}\n Ownership = #{owner}"
#If channel name exists in the Database, update it
xisting_channel = nil
xisting_channel = Channel.find_by channel_name: '#{channelName}'
if(xisting_channel.nil?)
#create a new channel
#new_channel = Channel.create(channel_hash)
puts "Inserted....#{#new_channel.inspect}"
else
#update existing channel
Channel.update(xisting_channel.id, :classification => "#{classif}", :ownership => "#{owner}" )
puts "Updated...."
puts "channel_hash = #{channel_hash.inspect} "
end#end if/else
end#end if/else
end #end CSV.each
end
When I run this code I get the following error message:
MRMIOMP0903:am AM$ rake import_channels[/XXXXXXXX/Channellist.csv]
Filepath received = /XXXXXXX/Channellist.csv
Skipping over row 0
Channel Name = APTN HD+
Classification = Specialty
Ownership = Aboriginal Peoples Television Network
rake aborted!
ActiveRecord::ConnectionNotEstablished
I tried to create a Channel object using IRB and it worked just fine. The DB is created and I'm seeing it using MySQL WorkBench. All tables are there, however, I am unable to create the Channel object from the Rake task.
My hypothesis is, perhaps outside of a certain folder/hierarchy the app cannot access the ActiveRecord::Base class or something like this.
Any suggestions on how to make this work?
UPDATE:
BAsed on the answer by Phillip Hallstrom
I changed the top line to load the environment
task :import_channels => :environment do|t, args|
Then tried rake import_channels and got this error:
rake aborted!
undefined method `safe_constantize' for #<Hash:0x007fbeacd89920>
You need to load the environment prior to running your Rake task. I'm not familiar with your multiple task name options there, but for a simple example, you'd want this:
task : import_channels => :environment do
I've got a rake task that changes data on the homepage every few hours. I've tested it out and it works fine in development. But it doesn't work in production. What do I have to do to get the changes I want to see? Should I add a command that restarts the server? Would that make the server acknowledge the change? Is there a smarter way to do this?
The rake task is below. It'll be run by heroku's scheduler add on, so it's currently in the lib/tasks/scheduler.rake file.
desc 'changes the meta tags'
task :mixup_meta_tags => :environment do
regex = /#meta_tag/
file = File.open('app/controllers/site_controller.rb', 'r')
lines = []
file.each_line do |line|
(line =~ regex) ? (lines << replace_line(line)) : (lines << line)
end
file.close
file = File.open('app/controllers/site_controller.rb', 'w')
lines.each{|line| file.write line}
file.close
end
def replace_line(line)
meta_tags = MetaTag.all.map { |tag| tag["tag"] }
new_tag = meta_tags.sample(1)[0]
line = " #meta_tag = \"#{new_tag}\" \n" # added the newline
end
Yes, changes to your Rails application in Production require a restart for them to get picked up by the server. To get this to work on the fly you might want to try the solution mentioned in this post why-does-code-need-to-be-reloaded-in-rails-3
I am trying to run a ruby script that parses a text file with album name, artist name, year on each line. It should then save a new record into a Rails model called Album.
(buildDB.rb)
fh = File.open('albums.txt')
while line = fh.gets
if ( line =~ /^(.+)\~\s(.+) \(\'(\d\d)\)/ )
a = Album.new
a.name = $1
a.artist = $2
a.year = $3
a.save
end
end
I am running ruby buildDB.rb in terminal which produces the message
buildDb.rb:12:in '<main>': uninitialized constant Album (NameError)
This made me think that the script could not find the model. So I tried loading the rails environment by using
require "C:/ruby192/www/Project02/config/environment.rb"
at the top of the ruby script. The script will run without errors but nothing is committed to the sqlite database. I can also run find on the already existing Albums, it just seems I can't create new ones.
I am a rails noob so, there probably is a better way to do this (seeds.rb or a rake task maybe). Any help or a direction to look into would be greatly appreciated.
I would use a rake task:
task :create_albums => :environment do
fh = File.open('albums.txt')
while line = fh.gets
if ( line =~ /^(.+)\~\s(.+) \(\'(\d\d)\)/ )
a = Album.new
a.name = $1
a.artist = $2
a.year = $3
a.save!
end
end
fh.close
end
I added a ! to the save method so that any errors will throw an exeception. Also, make sure you close your files.
Rake task is the way ahead. I would use FasterCSV as it will handle some of the data import for you.
namespace :import do
desc "Import from a csv file"
task :album_csv, [:filename] => :environment do |task, args|
lines = FasterCSV.read(args[:filename]) rescue nil
if lines
lines.slice!(0) # remove the CSV header if there is one
puts "# Processing #{lines.count} records"
lines.each do |line|
a = Album.new
a.name = line[0]
a.artist = line[1]
a.year = line[2]
a.save!
end
end
end
You can then call your rake task as:
rake import:album_csv[filename]