How would I silence/ignore these Delayed Job query logs from log/development.log?
Delayed::Backend::ActiveRecord::Job Load (1.0ms) UPDATE "delayed_jobs" SET locked_at = '2013-11-19 19:55:45.053991', locked_by = 'host:desktop-virtual pid:22277' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2013-11-19 19:55:45.053435' AND (locked_at IS NULL OR locked_at < '2013-11-19 15:55:45.053519') OR locked_by = 'host:desktop-virtual pid:22277') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Delayed::Backend::ActiveRecord::Job Load (1.4ms) UPDATE "delayed_jobs" SET locked_at = '2013-11-19 19:55:50.056977', locked_by = 'host:desktop-virtual pid:22277' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2013-11-19 19:55:50.056484' AND (locked_at IS NULL OR locked_at < '2013-11-19 15:55:50.056530') OR locked_by = 'host:desktop-virtual pid:22277') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I've tried adding this to config/initializers/delayed_job.rb and it routes everything but the interval query logs which still get put in log/development.log.
if Rails.env == "development"
Delayed::Worker.logger = Logger.new(File.join(Rails.root, "log", "delayed_job.log"))
end
Thank you.
The log line still shows up because that line is logged by ActiveRecord, not Delayed Job. See the github bug report for more info on that. Here's a workaround:
in config/initializers/delayed_job_silencer.rb:
if Rails.env.development?
module Delayed
module Backend
module ActiveRecord
class Job
class << self
alias_method :reserve_original, :reserve
def reserve(worker, max_run_time = Worker.max_run_time)
previous_level = ::ActiveRecord::Base.logger.level
::ActiveRecord::Base.logger.level = Logger::WARN if previous_level < Logger::WARN
value = reserve_original(worker, max_run_time)
::ActiveRecord::Base.logger.level = previous_level
value
end
end
end
end
end
end
end
Change ActiveRecord log level only for Delayed job.
This will allow you to see other ActiveRecord logs.
Use:
Delayed::Backend::ActiveRecord::Job.logger.level = 1
I think you can set log level to one as follows in the initializers. It helps me to ignoring delayed job query info from the production as well as development log too.
ActiveRecord::Base.logger.level = 1
Related
This Rails code is supposed to prevent duplicate records from being recorded by the server within 20 seconds:
#transit = Transit.new(tag: params[:tag])
if Transit.where(tag: #transit.tag).where("created_at > ?", 20.seconds.ago).first
logger.warn "Duplicate tag"
else
#transit.save!
end
However, this is not working. I can see in my production database (hosted on Heroku) two different records getting created with the same tag 10 seconds apart.
Logs show the correct query is executed on the second request, but it returns no results and saves a new record anyway.
Why does this happen? I thought Postgres' default isolation level of read_committed would prevent this from happening. The query that returns no records should miss Rails' SQL cache. Logs show both requests were handled by the same WEB.1 Dyno on Heroku, and my Puma.rb is set up for 4 workers and 5 threads.
What am I missing?
Here are the two records in the db:
=> #<Transit id: 1080116, tag: 33504,
created_at: "2019-01-30 12:36:11",
updated_at: "2019-01-30 12:41:23">
=> #<Transit id: 1080115, tag: 33504,
created_at: "2019-01-30 12:35:56",
updated_at: "2019-01-30 12:35:56">
Log of the first insert:
30 Jan 2019 07:35:56.203132 <190>1 2019-01-30T12:35:56.050681+00:00 app web.1 - - [1m [36m (0.8ms) [0m [1mBEGIN [0m
30 Jan 2019 07:35:56.203396 <190>1 2019-01-30T12:35:56.055097+00:00 app web.1 - - [1m [35mSQL (1.0ms) [0m INSERT INTO "transits" ("tag", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"
30 Jan 2019 07:35:56.269133 <190>1 2019-01-30T12:35:56.114572+00:00 app web.1 - - [1m [36m (2.0ms) [0m [1mCOMMIT [0m
Log of the query from right before the duplicate is inserted:
30 Jan 2019 07:36:12.160359 <190>1 2019-01-30T12:36:11.863973+00:00 app web.1 - - [1m [35mTransit Load (5.1ms) [0m SELECT "transits".* FROM "transits" WHERE "transits"."tag" = 33504 AND created_at > '2019-01-30 12:35:51.846431' ORDER BY "transits"."id" ASC LIMIT 1
And here is the postgres transaction isolation level, which to be clear is for a different connection opened after this issue came up:
SHOW default_transaction_isolation;
default_transaction_isolation
-------------------------------
read committed
(1 row)
One way to prevent duplicates in Rails is with validations:
Correct way of prevent duplicate records in Rails
However your criteria is more complex as it deals with spanning more than one row.
I believe your criteria is, don't allow entry of a transit record if the most recent transit record was created less than 20 seconds ago. Is that right?
Trying to enforce a constraint that involves looking at data from many rows is mentioned as undesirable here:
SQL Sub queries in check constraint
A trigger could be used to enforce your constraint at the database level.
One could catch the trigger in an exception.
There's a gem named HairTrigger that might be useful, not sure.
Taking ideas from here:
https://karolgalanciak.com/blog/2016/05/06/when-validation-is-not-enough-postgresql-triggers-for-data-integrity/
Example with Postgresql trigger:
bin/rails generate model transit tag:text
rails generate migration add_validation_trigger_for_transit_creation
class AddValidationTriggerForTransitCreation < ActiveRecord::Migration[5.2]
def up
execute <<-CODE
CREATE FUNCTION validate_transit_create_time() returns trigger as $$
DECLARE
age int;
BEGIN
age := (select extract(epoch from current_timestamp - t.created_at)
from transits t
where t.tag = NEW.tag
and t.id in (select id from transits u
where u.id = t.id
and u.tag = t.tag
and u.created_at = (select max(v.created_at) from transits v where v.tag = u.tag)
));
IF (age < 20) THEN
RAISE EXCEPTION 'created_at too early: %', NEW.created_at;
END IF;
RETURN NEW;
END;
$$ language plpgsql;
CREATE TRIGGER validate_transit_create_trigger BEFORE INSERT OR UPDATE ON transits
FOR EACH ROW EXECUTE PROCEDURE validate_transit_create_time();
CODE
end
def down
execute <<-CODE
drop function validate_transit_create_time() cascade;
CODE
end
end
user1#debian8 /home/user1/rails/dup_test > ../transit_test.rb ; sleep 20; ../transit_test.rb
dup_test_development=> select * from transits;
id | tag | created_at | updated_at
-----+----------+----------------------------+----------------------------
158 | test_tag | 2019-01-31 18:38:10.115891 | 2019-01-31 18:38:10.115891
159 | test_tag | 2019-01-31 18:38:30.609125 | 2019-01-31 18:38:30.609125
(2 rows)
Here is the portion of our query that gives the latest transit entry with our tag
dup_test_development=> select * from transits t
where t.tag = 'test_tag' and t.id in
(select id from transits u where u.id = t.id and u.tag = t.tag and u.created_at =
(select max(v.created_at) from transits v where v.tag = u.tag));
id | tag | created_at | updated_at
-----+----------+----------------------------+----------------------------
159 | test_tag | 2019-01-31 18:38:30.609125 | 2019-01-31 18:38:30.609125
(1 row)
Modifying to give the difference between the current_timestamp (now) and the latest transit entry with our tag. This difference is an interval in postgresql. Using UTC to match Rails:
dup_test_development=> select current_timestamp at time zone 'utc' - created_at
from transits t where t.tag = 'test_tag' and t.id in
(select id from transits u where u.id = t.id and u.tag = t.tag and u.created_at =
(select max(v.created_at) from transits v where v.tag = u.tag));
?column?
-----------------
00:12:34.146536
(1 row)
Adding Extract(epoch) to convert this to seconds:
dup_test_development=> select extract(epoch from current_timestamp at time zone 'utc' - created_at)
from transits t where t.tag = 'test_tag' and t.id in
(select id from transits u where u.id = t.id and u.tag = t.tag and u.created_at =
(select max(v.created_at) from transits v where v.tag = u.tag));
date_part
------------
868.783503
(1 row)
We store the seconds as age, and if the age is < 20, we raise a database exception
Running 2 inserts with a second delay less than 20:
user1#debian8 /home/user1/rails/dup_test > ../transit_test.rb ; sleep 5; ../transit_test.rb
#<ActiveRecord::StatementInvalid: PG::RaiseException: ERROR: created_at too early: 2019-01-31 18:54:48.95695
: INSERT INTO "transits" ("tag", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id">
"ERROR: created_at too early: 2019-01-31 18:54:48.95695\n"
Short test outside of rails:
#!/usr/bin/env ruby
require 'active_record'
require 'action_view'
path = "/home/user1/rails/dup_test/app/models"
require "#{path}/application_record.rb"
Dir.glob(path + "/*.rb").sort.each do | file |
require file
end
ActiveRecord::Base.establish_connection(
:adapter => "postgresql",
:database => 'dup_test_development',
encoding: "unicode",
username: "user1",
password: nil
)
class Test
def initialize()
end
def go()
begin
t = Transit.new(tag: 'test_tag')
t.save
rescue ActiveRecord::StatementInvalid => e
p e
p e.cause.message
end
end
end
def main
begin
t = Test.new()
t.go()
rescue Exception => e
puts e.message
end
end
main
Using someting like Redis has been mentioned - may be better for peformance
I believe this was a concurrency issue.
Rails transactions continue asynchronously after ActiveRecord returns. Anytime the commit takes 15 seconds to apply it will cause this problem. This is long and unlikely, but possible.
I cannot prove this is what happened, but it appears to be the only explanation. Preventing it would require a dB stored procedure or like #PhilipWright suggested or a distributed lock like you and #kwerle suggested.
This is what testing is for.
class Transit < ActiveRecord::Base
def new_transit(tag: tag)
<your code>
end
end
You test code:
test 'it saves once' do
<save it once. check the count, etc>
end
test 'it does not save within 10 seconds' do
<save it once. Set the created at to 10 seconds ago. try to save again. check the count, etc>
end
etc
p.s. Consider using redis or something like that. Otherwise you're wanting to do something like table locks to make sure you're not stepping on yourself. And you probably don't want to do table locks.
I am trying to use a very basic delayed job on rails:
class ImportJob < Struct.new(:job_params)
def perform
#blank perform
end
end
And When I queue that job all I see in rake:jobs is that the job is RUNNING, but it never finishes:
Delayed::Job.enqueue ImportJob.new(job_params)
Furthermore if I run the job worker from the rails console I see the job get 'locked' and then immediately deleted:
2.1.2 :001 > work = Delayed::Worker.new
=> #<Delayed::Worker:0x007f8a8c4d2ee8 #quiet=true, #failed_reserve_count=0>
2.1.2 :002 > work.start
Delayed::Backend::ActiveRecord::Job Load (1.8ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2014-07-16 19:10:54.367527' AND (locked_at IS NULL OR locked_at < '2014-07-16 15:10:54.367862') OR locked_by = 'host:jerrods-mbp.raleigh.ibm.com pid:36564') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
Delayed::Backend::ActiveRecord::Job Load (3.5ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2014-07-16 19:10:59.386290' AND (locked_at IS NULL OR locked_at < '2014-07-16 15:10:59.386563') OR locked_by = 'host:jerrods-mbp.raleigh.ibm.com pid:36564') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
SQL (5.9ms) UPDATE "delayed_jobs" SET "locked_at" = '2014-07-16 19:10:59.387743', "locked_by" = 'host:jerrods-mbp.raleigh.ibm.com pid:36564' WHERE "delayed_jobs"."id" IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE "delayed_jobs"."id" = 6 AND ((run_at <= '2014-07-16 19:10:59.386290' AND (locked_at IS NULL OR locked_at < '2014-07-16 15:10:59.386563') OR locked_by = 'host:jerrods-mbp.raleigh.ibm.com pid:36564') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC)
Delayed::Backend::ActiveRecord::Job Load (0.7ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE "delayed_jobs"."id" = ? LIMIT 1 [["id", 6]]
(0.1ms) begin transaction
SQL (0.5ms) DELETE FROM "delayed_jobs" WHERE "delayed_jobs"."id" = ? [["id", 6]]
(2.6ms) commit transaction
Has anyone ever seen this behavior before?
Alright so after reading various other threads on here and other places I came by the following solution:
I realized it was failing silently for whatever reason so I added Delayed::Worker.destroy_failed_jobs = false config/application.rb
With that line added I was able to go in the console and get Delayed::Job.last.last_error and it told me that my job file wasn't being loaded in rake for whatever reason
To solve this I created a custom.rb in initlializers and added the following line:
require 'my_class'
Hopefully this will help someone else
I don't see anything out of the ordinary here. When a job completes successfully it is deleted from the delayed_jobs queue / database table. You could have your job raise an error and see what happens when one doesn't complete successfully:
def perform
raise "BOOM, goes the dynamite!"
end
It should be outputting a count of jobs completed successfully in any given "run period", however. See the source here for that.
I am using delayed jobs like Something.delay.some_function()
I am running [Foreman][1] Procfile to start both rails server and a worker
web: bundle exec rails s
worker: bundle exec rake jobs:work
And all the time I see this in my terminal, though I always run rake jobs:clear if something bad happened during the job. And now I have no delayd jobs
irb(main):010:0> Delayed::Job.all
Delayed::Backend::ActiveRecord::Job Load (0.7ms) SELECT "delayed_jobs".* FROM "delayed_jobs"
=> []
but I still see those strange things in my terminal. What is this and how can I get rid of it?
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.1ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:52.491892' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:52.492124') OR locked_by = 'host:kik-VirtualBox pid:5010') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.5ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:52.706307' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:52.706499') OR locked_by = 'host:kik-VirtualBox pid:4654') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.4ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:52.712781' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:52.713010') OR locked_by = 'host:kik-VirtualBox pid:4313') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (4.8ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:52.710021' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:52.710261') OR locked_by = 'host:kik-VirtualBox pid:4622') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.5ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:54.530083' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:54.530318') OR locked_by = 'host:kik-VirtualBox pid:5307') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.6ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:57.497068' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:57.497275') OR locked_by = 'host:kik-VirtualBox pid:5010') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.5ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:57.723823' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:57.724071') OR locked_by = 'host:kik-VirtualBox pid:4313') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.4ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:57.722732' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:57.722934') OR locked_by = 'host:kik-VirtualBox pid:4654') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (10.0ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:57.718190' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:57.718443') OR locked_by = 'host:kik-VirtualBox pid:4622') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
11:47:01 rails_s.1 | Delayed::Backend::ActiveRecord::Job Load (0.5ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-06-14 09:46:59.534770' AND (locked_at IS NULL OR locked_at < '2013-06-14 05:46:59.535003') OR locked_by = 'host:kik-VirtualBox pid:5307') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
Thanks in advance
Delayed Job checks if there are new tasks for it every once in a while. As long as rake jobs:work is running you'll be seeing those logs. In short - everything is ok!
Here is the actual error from logfile:
2013-02-20T20:56:03+00:00 app[worker.1]: SQL (11.1ms) UPDATE "delayed_jobs" SET locked_by = null, locked_at = null WHERE (locked_by = 'host:05c659a5-86fd-46dd-b139-263e49a96171 pid:2')
2013-02-20T20:56:03+00:00 app[worker.1]: rake aborted!
2013-02-20T20:56:03+00:00 app[worker.1]: PG::Error: ERROR: cannot execute UPDATE in a read-only transaction
2013-02-20T20:56:03+00:00 app[worker.1]: : UPDATE "delayed_jobs" SET locked_at = '2013-02-20 20:56:02.982379', locked_by = 'host:05c659a5-86fd-46dd-b139-263e49a96171 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2013-02-20 20:56:02.882553' AND (locked_at IS NULL OR locked_at < '2013-02-20 16:56:02.882574') OR locked_by = 'host:05c659a5-86fd-46dd-b139-263e49a96171 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
This error also appears in the followers log. That's why assumed there is a problem with that. Delayed Job version: 3.0.5
It seems that issue came with recent update. See this Github issue for details:
https://github.com/collectiveidea/delayed_job_active_record/issues/34
For me downgrading to 0.3.3 worked.
I use the following code to execute a method with delayed job:
class MyClass
def send_a_text
t = TwilioCommunicator.new
t.send_sms("+15177416150","sent at #{Time.now}")
end
handle_asynchronously :send_a_text
end
t = MyClass.new
t.send_a_text
Without delayed job, this sends me a text message 100% of the time. With delayed job, I can see that this job is queued in the database and then dequeued but the code does not appear to run. I am testing locally and here is a snippet from my development.log:
AREL (0.5ms) INSERT INTO "delayed_jobs" ("priority", "attempts", "handler", "last_error", "run_at", "locked_at", "failed_at", "locked_by", "created_at", "updated_at") VALUES (0, 0, '--- !ruby/struct:Delayed::PerformableMethod
object: !ruby/object:MyClass {}
method_name: :send_a_text_without_delay
args: []
', NULL, '2011-07-28 05:24:47.700969', NULL, NULL, NULL, '2011-07-28 05:24:47.701070', '2011-07-28 05:24:47.701070')
AREL (2.9ms) UPDATE "delayed_jobs" SET locked_at = '2011-07-28 05:24:51.191404', locked_by = 'delayed_job host:Stephen-B-Silverbergs-MacBook.local pid:3714' WHERE (id = 13 and (locked_at is null or locked_at < '2011-07-28 01:24:51.191404') and (run_at <= '2011-07-28 05:24:51.191404'))
AREL (0.3ms) DELETE FROM "delayed_jobs" WHERE ("delayed_jobs"."id" = 13)
One thing I notice is the method_name is send_a_text_without_delay instead of send_a_text but this is the only strange thing I see. Any ideas?
Where is the MyClass object in your Rails project? If it is in not in app/model, then you need to reference the class in an initializer for Delayed_Job to work. This issue fails silently in delayed_job.
See: Rails Delayed Job & Library Class