I'm working on a project that is migrating data from a customers old_busted DB into rails objects to be worked on later. Similarly, I need to convert these objects into a CSV and upload it to a neutral FTP (this is to allow a coworker to build the example pages through Sugar CRM). I've created rake files to do all of this, and it was successful. Now, I'm going to continue this process for each object that I create in rails (relative to the previous DB) and, best case, wanted these generated when I run rake generate scaffold <object>.
Here is my import rake:
desc "Import Clients from db"
task :get_busted_clients => [:environment] do
#old_clients = Busted::Client.all
#old_clients.each do |row|
#client = Client.new();
#client.client_id = row.NUMBER
#client.save
end
end
Here is my CSV convert/FTP upload rake:
desc "Exports db's to local CSV and uploads them to FTP"
task :export_clients_CSV => [:environment] do
# Required libraries for CSV read/write and NET/FTP IO #
require 'csv'
require 'net/ftp'
# Pull all Editor objects into clients for reading #
clients = Client.all
puts "Creating CSV file for <Clients> and updating column names..."
# Open a new CSV file that uses the column headers from Client #
CSV.open("clients.csv", "wb",
:write_headers => true, :headers => Client.column_names) do |csv|
puts "--Loading each entry..."
# Load all entries from Client into the CSV file row by row #
clients.each do |client|
# This line specifically puts the attributes in the rows WITH RESPECT TO#
# THE COLUMNS
csv << client.attributes.values_at(*Client.column_names)
end
puts "--Done loading each entry..."
end
puts "...Data populated. Finished bulding CSV. Closing File."
puts "------------------------"
# Upload CSV File to FTP server by requesting new FTP connection, assigning credentials
# and informing the client what file to look for and what to name it
puts "Uploading <Clients>..."
ftp = Net::FTP.new('192.168.xxx.xxx')
ftp.login(user = "user", passwd = "passwd")
ftp.puttextfile("clients.csv", "clients.csv")
ftp.quit()
puts "...Finished."
end
I ran rake generate g get_busted and put this in my get_busted_generator.rb:
class GetBustedGenerator < Rails::Generators::NamedBase
source_root File.expand_path('../templates', __FILE__)
def generate_get_busted
copy_file "getbusted.rake", "lib/tasks/#{file_name}.rake"
end
end
After that, I got lost. I can't find anything on templating a rake file or the syntax included to do so.
Rails has been a recent endeavor and I may be overlooking something in terms of design of the solution to my problem.
TL;DR: Is templating a rake file a bad thing? Solution alternatives? If not, whats the syntax for generating either script custom to the object (or point me in the direction, please).
Related
I have a simple rails application where I import data from csv into my rails app which is functioning properly, but I have no idea where to start with testing this rake task, as well as where in a modular rails app. Any help would be appreciated. Thanks!
Hint
My Rails structure is a little different from traditional rails structures, as I have written a Modular Rails App. My structure is in the picture below:
engines/csv_importer/lib/tasks/web_import.rake
The rake task that imports from csv..
require 'open-uri'
require 'csv'
namespace :web_import do
desc 'Import users from csv'
task users: :environment do
url = 'http://blablabla.com/content/people.csv'
# I forced encoding so avoid UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8
csv_string = open(url).read.force_encoding('UTF-8')
counter = 0
duplicate_counter = 0
user = []
CSV.parse(csv_string, headers: true, header_converters: :symbol) do |row|
next unless row[:name].present? && row[:email_address].present?
user = CsvImporter::User.create row.to_h
if user.persisted?
counter += 1
else
duplicate_counter += 1
end
end
p "Email duplicate record: #{user.email_address} - #{user.errors.full_messages.join(',')}" if user.errors.any?
p "Imported #{counter} users, #{duplicate_counter} duplicate rows ain't added in total"
end
end
Mounted csv_importer in my parent structure
This makes the csv_importer engine available in the root of the application.
Rails.application.routes.draw do
mount CsvImporter::Engine => '/', as: 'csv_importer'
end
To correctly migrate in the root of the application, I added initializer
/engines/csv_importer/lib/csv_importer/engine.rb
module CsvImporter
class Engine < ::Rails::Engine
isolate_namespace CsvImporter
# This enables me to be able to correctly migrate the database from the parent application.
initializer :append_migrations do |app|
unless app.root.to_s.match(root.to_s)
config.paths['db/migrate'].expanded.each do |p|
app.config.paths['db/migrate'] << p
end
end
end
end
end
So with this explanation am able to run rails app like every other rails application. I explained this so anyone who will help will understand what to help me with as regards writing test for the rake task inside the engine.
What I have done as regards writing TEST
task import: [:environment] do
desc 'Import CSV file'
task test: :environment do
# CSV.import 'people.csv'
Rake::Task['app:test:db'].invoke
end
end
How do someone write test for a rake task in a modular app? Thanks!
I haven't worked with engines, but is there a way to just put the CSV importing logic into it's own class?
namespace :web_import do
desc 'Import users from csv'
task users: :environment do
WebImport.new(url: 'http://blablabla.com/content/people.csv').call
end
end
class WebImport # (or whatever name you want)
def initialize(url) ... end
def call
counter, CSV parse, etc...
end
end
That way you can bump into the Rails console to do the WebImport and you can also do a test isolating WebImport. When you do Rake tasks and Jobs (Sidekiq etc), you want to make the Rake task act as as thin a wrapper as possible around the actual meat of the code (which is in this case CSV parsing). Separate the "trigger the csv parse" code from the "actually parse the csv" code into their own classes or files.
I would like to update a cities table every week to reflect changes in cities across the world. I am creating a Rake task for the purpose. If possible, I would like to do this without adding another gem dependency.
The zipped file is a publicly available zipped file at geonames.org/15000cities.zip.
My attempt:
require 'net/http'
require 'zip'
namespace :geocities do
desc "Rake task to fetch Geocities city list every 3 days"
task :fetch do
uri = URI('http://download.geonames.org/export/dump/cities15000.zip')
zipped_folder = Net::HTTP.get(uri)
Zip::File.open(zipped_folder) do |unzipped_folder| #erroring here
unzipped_folder.each do |file|
Rails.root.join("", "list_of_cities.txt").write(file)
end
end
end
end
The return from rake geocities:fetch
rake aborted!
ArgumentError: string contains null byte
As detailed, I'm trying to unzip the file and save it to a list_of_cities.txt file. Once I the methodology down for accomplishing this, I believe I can figure out how to update my db, based on the file. (But if you have opinions on how best to handle the actual db update, other than my planned way, I'd love to hear them. But that seems like a different post entirely.)
This will save zipped_folder to disk, then unzip it and save its contents:
require 'net/http'
require 'zip'
namespace :geocities do
desc "Rake task to fetch Geocities city list every 3 days"
task :fetch do
uri = URI('http://download.geonames.org/export/dump/cities15000.zip')
zipped_folder = Net::HTTP.get(uri)
File.open('cities.zip', 'wb') do |file|
file.write(zipped_folder)
end
zip_file = Zip::File.open('cities.zip')
zip_file.each do |file|
file.extract
end
end
end
This will extract all files inside the zip file, in this case cities15000.txt.
You can then read the contents of cities15000.txt and update your database.
If you want to extract to a different file name, you can pass it to file.extract like this:
zip_file.each do |file|
file.extract('list_of_cities.txt')
end
I think it can be done more easily without ruby, just using wget and unzip:
namespace :geocities do
desc "Rake task to fetch Geocities city list every 3 days"
task :fetch do
`wget -c --tries=10 http://download.geonames.org/export/dump/cities15000.zip | unzip`
end
end
I have just created a Rails app with a model app/models/post.rb and have written a scraper scrapers/base_scraper.rb (class BaseScraper) that collect data from the target site to the hash variable data. Now I want to insert values of data into the Post model. How to do it properly in Rails? I have heard smth about Rake but have no idea how to utilize it properly. Help me please!
Assuming that data stores just one post and that each of the key stored in the datahash are valid Post fields (column_name), you can do simply this:
Post.create(data)
If you want to launch the whole process from console, you can create a rake task under lib/tasks directory of your process with the following:
# scraper.rake
namespace :scraper do
desc "Run scraper"
task :run => :environment do
data = BaseScraper.your_collect_data_class_method
Post.create(data) if data
end
end
task :default => 'scraper:run'
And then run it from console as a rake task with rake scraper
Of course I also assume that scrapers dir is in your Rails load path.
If not, add it to your application.rbfile.
# application.rb
...
module YourApp
class Application < Rails::Application
...
config.autoload_paths += Dir["#{config.root}/scrapers/"]
...
end
end
I've been knocking my head around with Heroku, while trying to download a zip file with all my receipt files data.
The files are stored on amazon s3 and it all works fine on my development machine..
I thought it had to do with Tempfile, and abandoned that previous solution, since heroku has some strict policies with their filesystem, so i used the tmp folder, but the problem doesn't seem to be there. I already tried to load directly from s3 (using openUri) to the zip file, but it doesn't seem to work either on Heroku.
What might be wrong with my code for Heroku not loading the files to the zip?
Here is my model method :
def zip_receipts(search_hash=nil)
require 'zip/zip'
require 'zip/zipfilesystem'
t=File.open("#{Rails.root}/tmp/#{Digest::MD5.hexdigest(rand(12).to_s)}_#{Process.pid}",'w')
# t = Tempfile.new(Digest::MD5.hexdigest(rand(12).to_s))
# Give the path of the temp file to the zip outputstream, it won't try to open it as an archive.
Zip::ZipOutputStream.open(t.path) do |zos|
logger.debug("search hash Zip: #{search_hash.inspect}")
self.feed(search_hash).each do |receipt|
begin
require 'open-uri'
require 'tempfile'
#configures filename
filen = File.basename(receipt.receipt_file_file_name)
ext= File.extname(filen)
filen_noext = File.basename(receipt.receipt_file_file_name, '.*')
filen=filen_noext+SecureRandom.hex(10)+ext
logger.info("Info Zip - Filename: #{filen}")
# Create a new entry on the zip file
zos.put_next_entry(filen)
# logger.info("Info Zip - Added entry: #{zos.inspect}")
# Add the contents of the file, reading directly from amazon
tfilepath= "#{Rails.root}/tmp/#{File.basename(filen,ext)}_#{Process.pid}"
open(tfilepath,"wb") do |file|
file << open(receipt.authenticated_url(:original),:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE).read
end
zos.print IO.binread tfilepath
# logger.info("Info Zip - Extracted from amazon: #{zos.inspect}")
rescue Exception => e
logger.info("exception #{e}")
end # closes the exception begin
end #closes receipts cycle
end #closes zip file stream cycle
# The temp file will be deleted some time...
t.close
#returns the path for send file controller to act
t.path
end
My controller:
def download_all
#user = User.find_by_id(params[:user_id])
filepath = #user.zip_receipts
# Send it using the right mime type, with a download window and some nice file name.
send_file(filepath,type: 'application/zip', disposition: 'attachment',filename:"MyReceipts.zip")
end
And I write also my view and routes, so that it might serve anyone else trying to implement a download all feature
routes.rb
resources :users do
post 'download_all'
end
my view
<%= link_to "Download receipts", user_download_all_path(user_id:user.id), method: :post %>
The problem seemed to be with the search hash, and the sql query, and not the code itself. For some reason, the receipts get listed, but aren't downloaded. So it is an all different issue
In the end i have this code for the model
def zip_receipts(search_hash=nil)
require 'zip/zip'
require 'zip/zipfilesystem'
t=File.open("#{Rails.root}/tmp/MyReceipts.zip_#{Process.pid}","w")
# t = Tempfile.new(Digest::MD5.hexdigest(rand(12).to_s))
#"#{Rails.root}/tmp/RecibosOnline#{SecureRandom.hex(10)}.zip"
puts "Zip- Receipts About to enter"
# Give the path of the temp file to the zip outputstream, it won't try to open it as an archive.
Zip::ZipOutputStream.open(t.path) do |zos|
self.feed(search_hash).each do |receipt|
begin
require 'open-uri'
require 'tempfile'
filen = File.basename(receipt.receipt_file_file_name)
ext= File.extname(filen)
filen_noext = File.basename(receipt.receipt_file_file_name, '.*')
filen=filen_noext+SecureRandom.hex(10)+ext
# puts "Info Zip - Filename: #{filen}"
# Create a new entry on the zip file
zos.put_next_entry(filen)
zos.print open(receipt.authenticated_url(:original),:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE).read
rescue Exception => e
puts "exception #{e}"
end # closes the exception begin
end #closes receipts cycle
end #closes zip file stream cycle
# The temp file will be deleted some time...
t.close
#returns the path for send file controller to act
t.path
end
I need to create a recurring task that creates (or edits) product records from an external text file. From inside irb:
>> f = File.open(<filename>) # file in same directory path
No issues.
But when pasted into a Rake task file, the script always bombs "File not found". (Rails 3.1, Ubuntu.)
namespace :sap do
desc "uploads data from raw SAP file"
task :upload => :environment do
f = File.open("sap_pnlist_20111010a.csv")
records = f.readlines
records.each {|row|
... etc etc ...
}
end
end
Suggestions?
If the file is somewhere inside your Rails root, use
Rails.root.join('grandparent_dir', 'parent_dir', 'file.txt')
If the file is not in your Rails root, you must give it the full path.