How to fix Delayed Job deserialization errors after upgrading from Rails 4.2 to 5.1+? - ruby-on-rails

Question

I am working on upgrading an app from Rails 4.2 to 5.2. I have am running into an issue were jobs that were created in 4.2 are raising errors when they are invoked under Rails 5.2.
Delayed::DeserializationError (Job failed to load: not delegated...
I have narrowed it down to a problem after moving from 5.0 to 5.1. In 5.0.7 there is no problem but there is in 5.1.0. I can reproduce on a simple test case (taken from job.handler) by doing YAML.load(yml) where yml:
object: !ruby/object:Account
raw_attributes:
id: '8469'
attributes: !ruby/object:ActiveRecord::AttributeSet
attributes: !ruby/object:ActiveRecord::LazyAttributeHash
types:
id: &4 !ruby/object:ActiveRecord::Type::Integer
precision:
scale:
limit: 8
range: !ruby/range
begin: -9223372036854775808
end: 9223372036854775808
excl: true
values:
id: '8469'
created_at: '2019-11-15 21:16:15.257401'
additional_types: {}
materialized: true
delegate_hash:
id: !ruby/object:ActiveRecord::Attribute::FromDatabase
name: id
value_before_type_cast: '8469'
type: *4
value: 8469
created_at: !ruby/object:ActiveRecord::Attribute::FromDatabase
name: created_at
value_before_type_cast: '2019-11-15 21:16:15.257401'
type: !ruby/object:ActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter
subtype: !ruby/object:ActiveRecord::ConnectionAdapters::PostgreSQL::OID::DateTime
precision:
scale:
limit:
value: 2019-11-15 21:16:15.257401000 Z
new_record: false
active_record_yaml_version: 0
That gives the error ArgumentError (not delegated). I have found that removing the subtype under created_at makes the problem go away but not idea why. I have tried changing the subtype to something simple like an integer and get the same problem.
Anyone have thoughts on how to approach this? I am really surprised that I have not found any info on others running into the same problem.

Answer 1

Turned out it was bigger than just 5.0 being able to serialize but 5.1 couldn't. There are a bunch of methods and classes that have been moved or removed in Rails 5 so it is just difficult to deserialize. With some help by posting an issue in the delayed job issues (https://github.com/collectiveidea/delayed_job/issues/1111) I was able to come up with a solution. I ended up writing the following migration that deserializes and then serializes all delayed_jobs.
migrate_dj_to_rails5.rb
require 'ruby-progressbar'
class MigrateDjToRails5 < ActiveRecord::Migration[5.2]
include MigrationHelper
def up
execute create_backup('delayed_jobs')
migrator = DelayedJobMigrator.new(Delayed::Job.all)
migrator.migrate
end
def down
execute copy_field_from_backup('delayed_jobs', 'handler')
execute load_dropped_records_from_backup('delayed_jobs')
end
class DelayedJobMigrator
def initialize(jobs = Delayed::Job.all)
#exceptions = []
#corrected_job_ids = []
#deleted_job_ids = []
#jobs = jobs
#progress_bar = ProgressBar.create(total: jobs.count, format: "%t: |%w| %e")
end
def migrate_job(job)
begin
data = YAML.load_dj(job.handler)
rescue
begin
job.payload_object = TempToRuby.create.accept(Psych.parse(job.handler))
rescue => exception
if exception.message =~ /Couldn't find (.+) with 'id'=/
# job is no longer valid
#deleted_job_ids << job.id
job.delete
end
#exceptions << exception
return
end
job.save
#corrected_job_ids << job.id
end
end
def migrate
#jobs.find_each do |job|
migrate_job(job)
#progress_bar.increment
end
#corrected_job_ids.each do |id|
puts "Corrected job_id: #{id}"
end
#exceptions.each do |exception|
puts "Exceptions:: #{exception}"
end
puts "#{#corrected_job_ids.count} jobs corrected"
puts "#{#exceptions.count} exceptions encountered (should be same as jobs deleted)"
puts "#{#deleted_job_ids.count} jobs deleted"
end
class TempToRuby < Delayed::PsychExt::ToRuby
def visit_Psych_Nodes_Mapping(object)
if %r{^!ruby/object:ActiveRecord::AttributeSet}.match(object.tag.to_s)
{}
elsif %r{^!ruby/object:(.+)$}.match(object.tag.to_s)
klass = resolve_class(Regexp.last_match[1])
if klass < ActiveRecord::Base
payload = Hash[*object.children.map { |c| accept c }]
return super unless payload['raw_attributes']
id = payload['raw_attributes'][klass.primary_key]
klass.unscoped.find(id)
else
super
end
else
super
end
end
end
end
end
migration_helper.rb
# Helper module for commonly used migration related methods
module MigrationHelper
def create_backup(table_name, suffix='_copy')
table_name_copy = table_name + suffix
<<-SQL
DROP TABLE IF EXISTS #{table_name_copy};
CREATE TABLE #{table_name_copy} AS TABLE #{table_name};
SQL
end
def copy_field_from_backup(table_name, field, suffix='_copy')
table_name_copy = table_name + suffix
records_to_update = <<-SQL
UPDATE #{table_name}
SET #{field} = #{table_name_copy}.#{field}
FROM #{table_name_copy}
WHERE #{table_name_copy}.id = #{table_name}.id
SQL
end
def load_dropped_records_from_backup(table_name, suffix='_copy')
table_name_copy = table_name + suffix
<<-SQL
INSERT INTO #{table_name}
SELECT #{table_name_copy}.* FROM #{table_name_copy}
LEFT JOIN #{table_name} on #{table_name}.id = #{table_name_copy}.id
where #{table_name}.id is null
SQL
end
end