I've written my first rake task in rails, I've set it up with the Heroku scheduler and it is getting run, however the mail isn't getting sent. My mail settings are fine as I'm using it for various other things, I would imagine it's a problem with my code in the rake task. Any help would be much appreciated.
lib/tasks/uncomplete_form.rake
desc "Remind users if they haven't completed quote form"
task uncomplete_form: :environment do
puts 'Reminding users of uncomplete quote form'
date = Date.parse('december 18 2016')
quickcontacts = Quickcontact.where(created_at: date.midnight..Time.now)
quickcontacts.each do |quickcontact|
next unless quickcontact.created_at > 1.hour.ago
if quickcontact.p_p = nil
QuickcontactMailer.uncomplete_form(#quickcontact).deliver
end
end
puts 'done.'
end
By running rake uncomplete_form I get
Reminding users of uncomplete quote form
done.
And running
heroku run rake uncomplete_form
I get
Reminding users of uncomplete quote form
Quickcontact Load (1.5ms) SELECT "quickcontacts".* FROM "quickcontacts" WHERE ("quickcontacts"."created_at" BETWEEN '2016-12-18 00:00:00.000000' AND '2016-12-20 12:09:23.683977')
done.
It doesn't seem to be picking up any quickcontacts - however if in the console I run:
date = Date.parse('december 18 2016')
followed by
quickcontacts = Quickcontact.where(created_at: date.midnight..Time.now)
It does find the expected contacts
Have you tried
QuickcontactMailer.uncomplete_form(#quickcontact).deliver_now
?
Edit: What's .p_p supposed to be? You're doing assignment in that if clause instead of what I presume should have been comparison (?)
if quickcontact.p_p = nil
Solved: Two bad errors on my part. Thanks for the help with the first #matija. The problem was with my rake task code. The first error was where I had 'if quickcontact.p_p = nil' needed to be changed to 'if quickcontact.p_p.nil?' - I was accidentally assigning nil value, rather than checking it.
The second error was that in the next line down, quickcontact should not have been an instance variable. This is the updated, functioning code:
desc "Remind users if they haven't completed quote form"
task uncomplete_form: :environment do
puts 'Reminding users of uncomplete quote form'
date = Date.parse('december 18 2016')
quickcontacts = Quickcontact.where(created_at: date.midnight..Time.now)
quickcontacts.each do |quickcontact|
next unless quickcontact.created_at > 1.hour.ago
if quickcontact.p_p.nil?
QuickcontactMailer.uncomplete_form(quickcontact).deliver
end
end
puts 'done.'
end
Related
I have an issue with importing a lot of records from a user provided excel file into a database. The logic for this is working fine, and I’m using ActiveRecord-import to cut down on the number of database calls. However, when a file is too large, the processing can take too long and Heroku will return a timeout. Solution: Resque and moving the processing to a background job.
So far, so good. I’ve needed to add CarrierWave to upload the files to S3 because I can’t just hold the file in memory for the background job. The upload portion is also working fine, I created a model for them and am passing the IDs through to the queued job to retrieve the file later as I understand I can’t pass a whole ActiveRecord object through to the job.
I’ve installed Resque and Redis locally, and everything seems to be setup correctly in that regard. I can see the jobs I’m creating being queued and then run without failing. The job seems to run fine, but no records are added to the database. If I run the code from my job line by line in the console, the records are added to the database as I would expect. But when the queued jobs I’m creating run, nothing happens.
I can’t quite work out where the problem might be.
Here’s my upload controller’s create action:
def create
#upload = Upload.new(upload_params)
if #upload.save
Resque.enqueue(ExcelImportJob, #upload.id)
flash[:info] = 'File uploaded.
Data will be processed and added to the database.'
redirect_to root_path
else
flash[:warning] = 'Upload failed. Please try again.'
render :new
end
end
This is a simplified version of the job with fewer sheet columns for clarity:
class ExcelImportJob < ApplicationJob
#queue = :default
def perform(upload_id)
file = Upload.find(upload_id).file.file.file
data = parse_excel(file)
if header_matches? data
# Create a database entry for each row, ignoring the first header row
# using activerecord-import
sales = []
data.drop(1).each_with_index do |row, index|
sales << Sale.new(row)
if index % 2500 == 0
Sale.import sales
sales = []
end
end
Sale.import sales
end
def parse_excel(upload)
# Open the uploaded excel document
doc = Creek::Book.new upload
# Map rows to the hash keys from the database
doc.sheets.first.rows.map do |row|
{ date: row.values[0],
title: row.values[1],
author: row.values[2],
isbn: row.values[3],
release_date: row.values[5],
units_sold: row.values[6],
units_refunded: row.values[7],
net_units_sold: row.values[8],
payment_amount: row.values[9],
payment_amount_currency: row.values[10] }
end
end
# Returns true if header matches the expected format
def header_matches?(data)
data.first == {:date => 'Date',
:title => 'Title',
:author => 'Author',
:isbn => 'ISBN',
:release_date => 'Release Date',
:units_sold => 'Units Sold',
:units_refunded => 'Units Refunded',
:net_units_sold => 'Net Units Sold',
:payment_amount => 'Payment Amount',
:payment_amount_currency => 'Payment Amount Currency'}
end
end
end
I can probably have some improved logic anyway as right now I’m holding the whole file in memory, but that isn’t the issue I’m having – even with a small file that has only 500 or so rows, the job doesn’t add anything to the database.
Like I said my code worked fine when I wasn’t using a background job, and still works if I run it in the console. But for some reason the job is doing nothing.
This is my first time using Resque so I don’t know if I’m missing something obvious? I did create a worker and as I said it does seem to run the job. Here’s the output from Resque’s verbose formatter:
*** resque-1.27.4: Waiting for default
*** Checking default
*** Found job on default
*** resque-1.27.4: Processing default since 1508342426 [ExcelImportJob]
*** got: (Job{default} | ExcelImportJob | [15])
*** Running before_fork hooks with [(Job{default} | ExcelImportJob | [15])]
*** resque-1.27.4: Forked 63706 at 1508342426
*** Running after_fork hooks with [(Job{default} | ExcelImportJob | [15])]
*** done: (Job{default} | ExcelImportJob | [15])
In the Resque dashboard the jobs aren’t logged as failed. They get executed and I can see an increment in the ‘processed’ jobs on the stats page. But as I say the DB remains untouched. What’s going on? How can I debug the job more clearly? Is there a way to get into it with Pry?
It looks like my problem was with Resque.enqueue(ExcelImportJob, #upload.id).
I changed my code to ExcelImportJob.perform_later(#upload.id) and now my code actually runs!
I also added a resque.rake task to lib/tasks as described here: http://bica.co/2015/01/20/active-job-resque/.
That link also notes how to use rails runner to call the job without running the full Rails server and triggering the job, which is useful for debugging.
Strangely, I didn't quite manage to get the job to print anything to STDOUT as suggested by #hoffm but at least it led me down a good avenue of inquiry.
I still don't fully understand the difference between why calling Resqueue.enqueue still added my jobs to the queue and indeed seemed to run them, but the code wasn't executed, so if someone has a better grasp and an explanation, that would be much appreciated.
TL;DR: calling perform_later rather than Resque.enqueue fixed the problem but I don't know why.
The code below, stored at config/initializers/console.rb works only at first time I exec rails console CLI. When exit and enter again, no selection message is displayed, but the preview tenant selected is loaded.
if defined?(Rails::Console) || $PROGRAM_NAME.include?('spring')
tenants = Apartment.tenant_names.sort
default = tenants.first
puts "Available tenants: #{tenants.join(', ')}"
print "Select tenant (#{default}): "
tenant = gets.strip
Apartment::Tenant.switch! tenants.include?(tenant) ? tenant : default
end
I wish every time when enter at rails console ask for what tenant will be loaded.
Thanks!
The only way I could get Apartment::Tenant.switch! to work in the Rails console was by creating the following .irbrc file in the project's root directory:
IRB.conf[:IRB_RC] = Proc.new do
tenants = Apartment.tenant_names.sort
puts "Available tenants: #{tenants.join(', ')}"
print "Select tenant: "
tenant = gets.strip
unless tenant.empty?
if tenants.include?(tenant)
Apartment::Tenant.switch!(tenant)
else
puts "Tenant not found in list '#{tenant}'"
end
end
puts "Tenant set to '#{Apartment::Tenant.current}'"
end
I faced similar issue. If you are using the Apartment Gem. In your rails console you can switch between tenants by first connecting to DB and then using schema_search_path
e.g.
c = Company.connection
c.schema_search_path = "tenant1"
To check tenant has been switched use ActiveRecord::Base.connection.schema_search_path
=> "\"tenant1\""
Company is just a table in my DB.
Here is a simple code (pry version) usable at launch or while at console
`Apartment::Tenant.switch!` during `bin/rails console` using `pry`
This happens because of Spring, by default it's configured only for the development environment. Just remove it from your Gemfile and it should work as you expected.
Our test suite has a problem.
Whenever the whole suite starts up, a stray record is created.
Whenever I run a single file or context of specs, this stray record does not get created.
I wrote config.before(:each) { puts "COUNT = #{Model.count}\n" } in spec_helper.rb and no matter which order the tests ran, this record persists somehow. Seemingly before the examples even started.
We seed our database, so I tried a clean setup.
RAILS_ENV=test rake db:setup
RAILS_ENV=test rake db:seed
echo "Model.count" | rails c test
=> 0
Then whenever I run all tests (checked the order and checked for before(:all)).
rspec
COUNT = 1
.
COUNT = 1
.
COUNT = 1
.
etc
I've meticulously examined spec_helper.rb (RSpec2, pre rails_helper.rb) and commented out every support file with no luck.
I'm suspecting the code base at this point, maybe some weird call to something where someone left behind a find_or_create, a weird callback or something, I have no idea.
What I would like to know is:
How do I tackle breaking down the suites startup?
Can I get a backtrace of my test suite just starting up?
Anyone successfully chased down this kind of persistent record in RSpec?
EDIT
I added config.before(:suite) { debugger } and the record is still created before this!
How can I break down this code even further?
[23, 32] in /Users/yourname/.rvm/gems/ruby-1.9.3-p327#projectname/gems/rspec-core-2.14.6/lib/rspec/core/command_line.rb
23 #world.announce_filters
24
25 #configuration.reporter.report(#world.example_count, #configuration.randomize? ? #configuration.seed : nil) do |reporter|
26 begin
27 #configuration.run_hook(:before, :suite)
=> 28 #world.example_groups.ordered.map {|g| g.run(reporter)}.all? ? 0 : #configuration.failure_exit_code
29 ensure
30 #configuration.run_hook(:after, :suite)
31 end
32 end
(rdb:1) ModelName.count
1
For the record of how I found the persistent record:
By using before(:suite) { debugger }, I removed the record and ran the entire test suite and notice a test was failing.
Within this test, was something similar to the following:
context "blah" do
model = FactoryGirl.create(:model)
end
I simply put the code within a before(:each) block, and the test was passing and now so were mine.
TIL: Whenever RSpec loads the suite, it evaluates all the code that is not within a transaction. Which is why I could not try to pin point a problematic file by looking at when the record was being created.
I am trying to run a rake task to get all the data with a specific tag from Instagram, and then input some of the data into my server.
The task runs just fine, except sometimes I'll get an error response. It's sort of random, so I think it just happens sometimes, and since it's a fairly long running task, it'll happen eventually.
This is the error on my console:
Instagram::BadGateway: GET https://api.instagram.com/v1/tags/xxx/media/recent.json?access_token=xxxxx&max_id=996890856542960826: 502: The server returned an invalid or incomplete response.
When this happens, I don't know what else to do except run the task again starting from that max_id. However, it would be nice if I could get the whole thing to automate itself, and retry itself from that point when it gets that error.
My task looks something like this:
task :download => :environment do
igs = Instagram.tag_recent_media("xxx")
begin
sleep 0.2
igs.each do |ig|
dl = Instadownload.new
dl.instagram_url = ig.link
dl.image_url = ig.images.standard_resolution.url
dl.caption = ig.caption.text if ig.caption
dl.taken_at = Time.at(ig.created_time.to_i)
dl.save!
end
if igs.pagination.next_max_id?
igs = Instagram.tag_recent_media("xxx", max_id: igs.pagination.next_max_id)
moreigs = true
else
moreigs = false
end
end while moreigs
end
Chad Pytel and Tammer Saleh call this "Fire and forget" antipattern in their Rails Antipatterns book:
Assuming that the request always succeeds or simply not caring if it
fails may be valid in rare circumstances, but in most cases it's
unsufficient. On the other hand, rescuing all the exceptions would be
a bad practice aswell. The proper solution would be to understand the
actual exceptions that will be raised by external service and rescue
those only.
So, what you should do is to wrap your code block into begin/rescue block with the appropriate set of errors raised by Instagram (list of those errors can be found here). I'm not sure which particular line of your code snippet ends with 502 code, so just to give you and idea of what it could look like:
begin
dl = Instadownload.new
dl.instagram_url = ig.link
dl.image_url = ig.images.standard_resolution.url
dl.caption = ig.caption.text if ig.caption
dl.taken_at = Time.at(ig.created_time.to_i)
dl.save!
rescue Instagram::BadGateway => e # list of acceptable errors can be expanded
retry # restart from beginning
end
I build a website-crawler that (later on) uses these links to read out information.
The current rake-task goes through all the possible pages one by one and checks if the requests goes trough (valid response) or returns a 404/503 error (invalid page). If it's valid the pages url gets saved into my database.
Now as you can see the task requests 50,000 pages in total thus requires some time.
I have read about Sidekiq and how it can perform these tasks asynchronously thus making this a lot faster.
My question: As you can see my task builds the counter after each loop. This will not work with Sidekiq I guess as it will only perform this script independent of itself various times, am I right?
How would I go around the problem of each instance needing its own counter then?
Hopefully my question makes sense - Thank you very much!
desc "Validate Pages"
task validate_url: :environment do
require 'rubygems'
require 'open-uri'
require 'nokogiri'
counter = 1
base_url = "http://example.net/file"
until counter > 50000 do
begin
url = "#{base_url}_#{counter}/"
open(url)
page = Page.new
page.url = url
page.save!
puts "Saved #{url} !"
counter += 1
rescue OpenURI::HTTPError => ex
logger ||= Logger.new("validations.log")
if ex.io.status[0] == "503"
logger.info "#{ex} # #{counter}"
end
puts "#{ex} # #{counter}"
counter += 1
rescue SocketError => ex
logger ||= Logger.new("validations.log")
logger.info "#{ex} # #{counter}"
puts "#{ex} # #{counter}"
counter += 1
end
end
end
A simple Redis INCR operation will create and/or increment a global counter for your jobs to use. You can use Sidekiq's redis connection to implement a counter trivially:
Sidekiq.redis do |conn|
conn.incr("my-counter")
end
If you want to use it async - that means you will have many instances of same job. The fastest approach - to use something like redis. This will give you simple and fast way to check\update counter for your needs. But also make sure you took care about counter: If one of your jobs using it, lock it for other jobs, so there wont be wrong results, etc