We have a Rails application that runs several jobs in the background using Sidekiq. Those jobs will create a processing log and attempt to attach it to a database object using ActiveStorage. The following code illustrates the issue:
class Import < ApplicationRecord
...
has_many_attached :log_files
...
end
class ImportJob
include Sidekiq::Job
queue_as :default
def perform(id)
import = Import.find(id)
log_file_name = "test_#{DateTime.now.iso8601}.log"
log_file = Tempfile.new(log_file_name)
log_file.write 'Testing...'
import.log_files.attach(io: log_file, filename: log_file_name, content_type: 'text/plain')
end
end
If this job is run synchronously, the log file is successfully attached to the job's record.
ImportJob.new.perform(123)
If however, the job is run asynchronously, the log file attach fails:
ImportJob.perform_async(123)
The error is ActiveRecord::RecordNotSaved: Failed to replace log_files_attachments because one or more of the new records could not be saved.
This code seems to run fine in development (WLS) and on an AWS EC2 server. But cannot get past this error when running on a Heroku server.
Has anyone run into this before and found a solution?
Related
I'm trying to send email campaigns in a rails app with the Heya gem and GoodJob. The example in the Heya readme as well as the Heya example app uses Sidekiq as the Active Job backend.
I'm confused about how to actually send the Heya campaigns with GoodJob.
The docs for Heya show this example of starting Sidekick: bundle exec sidekiq -q default -q heya
I assume that there is a Job queue somewhere in the gem called "Heya", but I can't find this in the source code. Do I need to create one?
Do I need to create a job that runs the Heya scheduler? While the example app uses Sidekiq, I also don't see any custom jobs in that app.
I have the following setup for GoodJob and it appears to be running fine with good_job start which should run all of the jobs and queues, but I've also tried good_job start --queues=heya,default.
Here is the relevant code:
Profile.dev
web: bin/rails server -p 3000
css: bin/rails tailwindcss:watch
worker: bundle exec good_job start
config/initializers/heya.rb
Heya.configure do |config|
config.user_type = "User"
config.campaigns.priority = [
"WelcomeCampaign",
]
end
app/jobs/application_job.rb
class ApplicationJob < ActiveJob::Base
# Automatically retry jobs that encountered a deadlock
# retry_on ActiveRecord::Deadlocked
# Most jobs are safe to ignore if the underlying records are no longer available
# discard_on ActiveJob::DeserializationError
end
app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
segment :email_subscriber?
default from: "#{I18n.t('settings.site_name')} <#{I18n.t('settings.newsletter_email')}>"
end
app/campaigns/welcome_campaign.rb
class WelcomeCampaign < ApplicationCampaign
default wait: 5.minutes,
layout: "newsletter"
step :intro, wait: 0.minutes,
subject: "Welcome to #{I18n.t('settings.site_name')}"
end
I also have a layout and views for the campaign similar to the Heya example app, and I'm using Mailcatcher to see if any email is being sent.
What am I missing to send these emails with Heya and GoodJob?
Note that I'm subscribing the users on signups like this:
class User < ApplicationRecord
after_create_commit :add_user_to_newsletters
private
def add_user_to_newsletters
WelcomeCampaign.add(self)
EvergreenCampaign.add(self)
self.update(email_subscriber: true)
end
end
And the default segment in campaigns/application_campaign.rb is segment :email_subscriber?
If I run User.last.email_subscriber? in the console to check this it returns true.
I feel like I'm missing something about how Heya connects to Active Job that is not obvious in the Heya docs.
Also, not sure if this is related, but I added this to config/puma.rb
# https://github.com/bensheldon/good_job#execute-jobs-async--in-process
before_fork do
GoodJob.shutdown
end
on_worker_boot do
GoodJob.restart
end
on_worker_shutdown do
GoodJob.shutdown
end
MAIN_PID = Process.pid
at_exit do
GoodJob.shutdown if Process.pid == MAIN_PID
end
preload_app!
Are you running the heya scheduler periodically? $ rails heya:scheduler
It looks like you could create your own background job to be run using GoodJob Cron, by executing Heya::Campaigns::Scheduler.new.run to run the scheduler and enqueue the emails.
Reading the "Running the Scheduler" part of the README explains what's happening:
To start queuing emails, run the scheduler task periodically:
rails heya:scheduler
Heya uses ActiveJob to send emails in the background. Make sure your
ActiveJob backend is configured to process the heya queue. By default, GoodJob runs from all queues "*".
You can change Heya's default queue using the queue option:
# app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
default queue: "custom"
end
I built a simple test job for Sidekiq and added it to my schedule.yml file for Sidekiq Cron.
Here's my test job:
module Slack
class TestJob < ApplicationJob
queue_as :default
def perform(*args)
begin
SLACK_NOTIFIER.post(attachments: {"pretext": "test", "text": "hello"})
rescue Exception => error
puts error
end
end
end
end
The SLACK_NOTIFIER here is a simple API client for Slack that I initialize on startup.
And in my schedule.yml:
test_job:
cron: "* * * * *"
class: "Slack::TestJob"
queue: default
description: "Test"
So I wanted to have it run every minute, and it worked exactly as I expected.
However, I've now deleted the job file and removed the job from schedule.yml, and it still tries to run the job every minute. I've gone into my sidekiq dashboard, and I see a bunch of retries for that job. No matter how many times I kill them all, they just keep coming.
I've tried shutting down both the redis server and sidekiq several times. I've tried turning off my computer (after killing the servers, of course). It still keeps scheduling these jobs and it's interrupting my other jobs because it raises the following exception:
NameError: uninitialized constant Slack::TestJob
I've done a project-wide search for "TestJob", but get no results.
I only had the redis server open with this job for roughly 10 minutes...
Is there maybe something lingering in the redis database? I've looked into the redis-cli documentation, but I don't think any of it helps me.
Try calling $ FLUSHALL in the redis cli...
Other than that...
The sidekiq-cron documentation seems to expect that you check for the existence of schedule.yml explicitly...
#initializers/sidekiq.rb
schedule_file = "config/schedule.yml"
if File.exist?(schedule_file) && Sidekiq.server?
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end
The default for Sidekiq is to retry a failed job 25 times. Because the job doesn't exist, it fails... every time; Sidekiq only knows that the job fails (because attempting to perform a job that doesn't exist would raise an exception), so it flags it for retry... 25 times. Because you were scheduling that job every minute, you probably had a TON of these failed jobs queued up for retry. You can:
Wait ~3 weeks to hit the maximum number of retries
If you have a UI page set up for Sidekiq, you can see and clear these jobs in the Retries tab
Dig through the Redis CLI documentation for a way to identify these specific jobs and delete them (or flush the whole thing if you're not worried about losing other jobs)
I have a model called Task.
rails runner -e development "Task.myClassMethod"
Results in this:
Please specify a valid ruby command or the path of a script to run.
Run 'bin/rails runner -h' for help.
undefined method `myClassMethod' for Thor::Command:Class
This works in any of my other models. I tried clearing out my Task model:
class Task < ApplicationRecord
def self.myClassMethod
pp 'Hello'
end
end
I've also tried running rails app:update:bin per this thread: https://github.com/javan/whenever/issues/663
But still getting the same errors. Is this a naming conflict?
While using ActiveJob with sidekiq,
# application.rb
config.active_job.queue_adapter = :sidekiq
I made a job using redis.
class SubmissionJob < ActiveJob::Base
queue_as :default
def perform(*args)
While checking through the log, after calling job as
SubmissionJob_perform_later args
But this doesn't work after giving message when doing rails s
Enqueued SubmissionJob (Job ID: 63320b9d-de2a-4791-99f6-e58f2adf73d7) to Sidekiq(development_default) with arguments: ["12345", "12346"]
when at last, I render the result calling with job_id from redis, the result supposed to be return the arguments, but it only returns empty array, and it seems SubmissionJob never runned.
However, when I call with perform_now , SubmissionJob.perform_now args the function works properly so it seems the method SubmissionJob itself is not the problem.
Where should I start to find problem and what is the possible solution to solve this.
Solved problem.
This was due to Work did not successfully dequeued job from suggested queue. When running the sidekiq, I gave -q option.
For example bundle exec sidekiq -e develop -q develop_submit_job_queue
I am developing a mailing web application in Ruby on Rails and I am in front of an issue with delayed_job gem:
In the settings of my application, I give the ability for the customer to update the email address from where mailing are sent. But I discovered there was something like a cache from delayed_job which doesn’t use the update email address for the « from » header.
When I use the delayed_job task by Capistrano manually it works so I tried to add a callback after_update in my model to handle the restart of delayed_job but without any success.
Capistrano command:
cap <my_env> delayed_job:restart # this works but it’s a manual command so useless in my case
What I tried is to dynamically restart delayed_job from the model:
class Setting < ActiveRecord::Base
after_save :restart_delayed_job
def restart_delayed_job
if email_changed?
system "RAILS_ENV=#{Rails.env} do bundle exec bin/delayed_job -n 1 restart"
end
end
end
My Mailing class:
class MyMailer < ApplicationMailer
default from: Setting.first.email # After updating the email value in setting, it still the old one used.
# more code skipped
end
Does anyone knows how can I restart delayed_job from Rails ?
Is there a way to do it exclusively in Ruby without writing shell script ?
In order to help me to understand better, is there several instance of delayed_job (one by website in the server) or one for all website ?
Thanks for your help !
My project:
- Rails 4.2.5
- Ruby 2.2.2
- ActiveAdmin 1.0.0 pre2
- Delayed job 4.1.1
- Capistrano 3.4.0
The solution was to move the from header into the mail function.
The reason is the default from is set only once when the application start and cannot be changed this way.
mail(from: email, subject: subject, ...) do
# skipped code
end
Here is the answer who helped me to understand that: https://github.com/collectiveidea/delayed_job/issues/882