Delayed jobs service is stopping after less than 1 hour of its starting time with the following log:
I, [2018-02-26T06:00:26.580458 #11439] INFO -- : 2018-02-26T06:00:26+0400: [Worker(delayed_job host:myhost pid:11439)] Starting job worker
I, [2018-02-26T06:00:26.664929 #11439] INFO -- : 2018-02-26T06:00:26+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41019) RUNNING
I, [2018-02-26T06:00:27.342994 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41019) COMPLETED after 0.6779
I, [2018-02-26T06:00:27.346526 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41020) RUNNING
I, [2018-02-26T06:00:27.470858 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41020) COMPLETED after 0.1242
I, [2018-02-26T06:00:27.474937 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41024) RUNNING
I, [2018-02-26T06:00:27.603043 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41024) COMPLETED after 0.1280
I, [2018-02-26T06:00:27.606702 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41025) RUNNING
I, [2018-02-26T06:00:27.725715 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41025) COMPLETED after 0.1189
I, [2018-02-26T06:00:27.728021 #11439] INFO -- : 2018-02-26T06:00:27+0400: [Worker(delayed_job host:myhost pid:11439)] 4 jobs processed at 3.4871 j/s, 0 failed
I, [2018-02-26T06:14:48.287220 #11439] INFO -- : 2018-02-26T06:14:48+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41027) RUNNING
I, [2018-02-26T06:14:48.414079 #11439] INFO -- : 2018-02-26T06:14:48+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41027) COMPLETED after 0.1267
I, [2018-02-26T06:14:48.416335 #11439] INFO -- : 2018-02-26T06:14:48+0400: [Worker(delayed_job host:myhost pid:11439)] 1 jobs processed at 7.3771 j/s, 0 failed
I, [2018-02-26T06:16:33.492435 #11439] INFO -- : 2018-02-26T06:16:33+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41028) RUNNING
I, [2018-02-26T06:16:33.613684 #11439] INFO -- : 2018-02-26T06:16:33+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41028) COMPLETED after 0.1211
I, [2018-02-26T06:16:33.615953 #11439] INFO -- : 2018-02-26T06:16:33+0400: [Worker(delayed_job host:myhost pid:11439)] 1 jobs processed at 7.8121 j/s, 0 failed
I, [2018-02-26T06:22:33.853678 #11439] INFO -- : 2018-02-26T06:22:33+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41030) RUNNING
I, [2018-02-26T06:22:33.967338 #11439] INFO -- : 2018-02-26T06:22:33+0400: [Worker(delayed_job host:myhost pid:11439)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=41030) COMPLETED after 0.1136
I, [2018-02-26T06:22:33.970307 #11439] INFO -- : 2018-02-26T06:22:33+0400: [Worker(delayed_job host:myhost pid:11439)] 1 jobs processed at 8.2735 j/s, 0 failed
I, [2018-02-26T06:38:24.595215 #11439] INFO -- : 2018-02-26T06:38:24+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:24.593926', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:24.593351' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:24.593398') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:29.597026 #11439] INFO -- : 2018-02-26T06:38:29+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:29.596061', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:29.595477' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:29.595524') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:34.598775 #11439] INFO -- : 2018-02-26T06:38:34+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:34.597856', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:34.597278' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:34.597325') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:39.600772 #11439] INFO -- : 2018-02-26T06:38:39+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:39.599713', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:39.599063' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:39.599110') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:44.602546 #11439] INFO -- : 2018-02-26T06:38:44+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:44.601568', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:44.601024' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:44.601072') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:49.604286 #11439] INFO -- : 2018-02-26T06:38:49+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:49.603369', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:49.602808' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:49.602863') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:54.606189 #11439] INFO -- : 2018-02-26T06:38:54+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:54.605111', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:54.604563' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:54.604613') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:38:59.608610 #11439] INFO -- : 2018-02-26T06:38:59+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:38:59.607243', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:38:59.606483' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:38:59.606539') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:39:04.610465 #11439] INFO -- : 2018-02-26T06:39:04+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:39:04.609457', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:39:04.608876' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:39:04.608926') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I, [2018-02-26T06:39:09.612201 #11439] INFO -- : 2018-02-26T06:39:09+0400: [Worker(delayed_job host:myhost pid:11439)] Error while reserving job: PG::ConnectionBad: PQsocket() can't get socket descriptor: UPDATE "delayed_jobs" SET locked_at = '2018-02-26 02:39:09.611263', locked_by = 'delayed_job host:myhost pid:11439' WHERE id IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE ((run_at <= '2018-02-26 02:39:09.610721' AND (locked_at IS NULL OR locked_at < '2018-02-25 22:39:09.610770') OR locked_by = 'delayed_job host:myhost pid:11439') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
database.yml
production:
adapter: postgresql
encoding: unicode
database: myapp
port: 5432
pool: 5
username: username
password: password
reconnect: true
Please can anyone just explain the reason of this error and how to avoid it:
Error while reserving job: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Update:
I believe this issue is not related to delayed jobs, since I'm getting same error while I'm doing some normal DB queries. So DB is restarting for some reason and this why delayed job service is stopping.
As per commented by #LaurenzAlbe , below are some issues found in /var/log/postgresql/postgresql-9.3-main.log:
LOG: connection received: host=10.10.10.15 port=57322
LOG: replication connection authorized: user=MyDBUser
FATAL: must be superuser or replication role to start walsender
LOG: could not receive data from client: Connection reset by peer
LOG: disconnection: session time: 0:06:18.911 user=MyDBUser database=MyDB host=127.0.0.1 port=34040
./systemd: 36: kill: Operation not permitted
WARNING: skipping "delayed_jobs" --- only table or database owner can analyze it
After some research, I believe delayed jobs have a memory leak issue, which is solved by using config.cache_classes = true because delayed jobs keeps reloading classes every now and then.
I had the same issue where delayed job process used more than 90%+ of memory and it crashed with the same error, even though I had cached classes, but I was running delayed jobs without RAILS_ENV, which caused them to load in development and ignore the other environment settings.
This problem can also be caused by:
Rails's database idle_timeout setting, which is 300 seconds by default. This means that if a connection is idle for more than 300 seconds, the server will close it.
PgBouncer client_idle_timeout setting. This setting controls how long a connection can be idle before PgBouncer closes it. If this setting is set too low, it can cause the same errors as Rails's idle_timeout.
PostgreSQL idle_session_timeout settings (from PostgreSQL 14) and idle_in_transaction_session_timeout(for sessions that were idle in transaction).
TCP or other network or firewall timeouts(e.g. there is a case of the timeout in Azure Cloud)
And to solve this problem, there are a few possible solutions:
Speed up slow processing
Increase the idle_timeout(or another guilty timeout) setting to a higher value to prevent a server from closing idle connections too soon.
Add a database setting called reaping_frequency: 10, which will clear and restore connections more frequently to prevent them from becoming idle.
Add manual reconnection, like the code snippet below, which releases the connection and re-establishes it before saving a record to the database.
ActiveRecord::Base.connection_pool.release_connection
ActiveRecord::Base.connection_pool.with_connection do
# save record in database
end
Links:
(1)(2)(3)(4) Info about reaping_frequency
(5)(6) Info about Rails's idle_timeout
(7)(8)(9) Solutions with manual reconnection
The beginning of my ActiveAdmin rails log for a query looks like this:
Started GET "/things" for 127.0.0.1 at 2015-12-17 15:43:30 -0500
Processing by ThingsController#index as HTML
AdminUser Load (57.1ms) SELECT `admin_users`.* FROM `admin_users` WHERE `admin_users`.`id` = 1 ORDER BY `admin_users`.`id` ASC LIMIT 1
(58.1ms) SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM `things` WHERE `things`.`retired_at` IS NULL AND `things`.`errored_at` IS NULL LIMIT 30 OFFSET 0) subquery_for_count
CACHE (0.0ms) SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM `things` WHERE `things`.`retired_at` IS NULL AND `things`.`errored_at` IS NULL LIMIT 30 OFFSET 0) subquery_for_count
CACHE (0.0ms) SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM `things` WHERE `things`.`retired_at` IS NULL AND `things`.`errored_at` IS NULL LIMIT 30 OFFSET 0) subquery_for_count
CACHE (0.0ms) SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM `things` WHERE `things`.`retired_at` IS NULL AND `things`.`errored_at` IS NULL LIMIT 30 OFFSET 0) subquery_for_count
What are the mechanisms and files in ActiveAdmin or its dependencies that are generating the 58.1ms SELECT COUNT ... subquery_for_count?
You can refer the following URL: documentation
That count query can be reduced with following:
index pagination_total: false do
# ...
end
Hope this helps you.
I am trying to use a very basic delayed job on rails:
class ImportJob < Struct.new(:job_params)
def perform
#blank perform
end
end
And When I queue that job all I see in rake:jobs is that the job is RUNNING, but it never finishes:
Delayed::Job.enqueue ImportJob.new(job_params)
Furthermore if I run the job worker from the rails console I see the job get 'locked' and then immediately deleted:
2.1.2 :001 > work = Delayed::Worker.new
=> #<Delayed::Worker:0x007f8a8c4d2ee8 #quiet=true, #failed_reserve_count=0>
2.1.2 :002 > work.start
Delayed::Backend::ActiveRecord::Job Load (1.8ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2014-07-16 19:10:54.367527' AND (locked_at IS NULL OR locked_at < '2014-07-16 15:10:54.367862') OR locked_by = 'host:jerrods-mbp.raleigh.ibm.com pid:36564') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
Delayed::Backend::ActiveRecord::Job Load (3.5ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2014-07-16 19:10:59.386290' AND (locked_at IS NULL OR locked_at < '2014-07-16 15:10:59.386563') OR locked_by = 'host:jerrods-mbp.raleigh.ibm.com pid:36564') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
SQL (5.9ms) UPDATE "delayed_jobs" SET "locked_at" = '2014-07-16 19:10:59.387743', "locked_by" = 'host:jerrods-mbp.raleigh.ibm.com pid:36564' WHERE "delayed_jobs"."id" IN (SELECT "delayed_jobs"."id" FROM "delayed_jobs" WHERE "delayed_jobs"."id" = 6 AND ((run_at <= '2014-07-16 19:10:59.386290' AND (locked_at IS NULL OR locked_at < '2014-07-16 15:10:59.386563') OR locked_by = 'host:jerrods-mbp.raleigh.ibm.com pid:36564') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC)
Delayed::Backend::ActiveRecord::Job Load (0.7ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE "delayed_jobs"."id" = ? LIMIT 1 [["id", 6]]
(0.1ms) begin transaction
SQL (0.5ms) DELETE FROM "delayed_jobs" WHERE "delayed_jobs"."id" = ? [["id", 6]]
(2.6ms) commit transaction
Has anyone ever seen this behavior before?
Alright so after reading various other threads on here and other places I came by the following solution:
I realized it was failing silently for whatever reason so I added Delayed::Worker.destroy_failed_jobs = false config/application.rb
With that line added I was able to go in the console and get Delayed::Job.last.last_error and it told me that my job file wasn't being loaded in rake for whatever reason
To solve this I created a custom.rb in initlializers and added the following line:
require 'my_class'
Hopefully this will help someone else
I don't see anything out of the ordinary here. When a job completes successfully it is deleted from the delayed_jobs queue / database table. You could have your job raise an error and see what happens when one doesn't complete successfully:
def perform
raise "BOOM, goes the dynamite!"
end
It should be outputting a count of jobs completed successfully in any given "run period", however. See the source here for that.
How would I silence/ignore these Delayed Job query logs from log/development.log?
Delayed::Backend::ActiveRecord::Job Load (1.0ms) UPDATE "delayed_jobs" SET locked_at = '2013-11-19 19:55:45.053991', locked_by = 'host:desktop-virtual pid:22277' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2013-11-19 19:55:45.053435' AND (locked_at IS NULL OR locked_at < '2013-11-19 15:55:45.053519') OR locked_by = 'host:desktop-virtual pid:22277') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Delayed::Backend::ActiveRecord::Job Load (1.4ms) UPDATE "delayed_jobs" SET locked_at = '2013-11-19 19:55:50.056977', locked_by = 'host:desktop-virtual pid:22277' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2013-11-19 19:55:50.056484' AND (locked_at IS NULL OR locked_at < '2013-11-19 15:55:50.056530') OR locked_by = 'host:desktop-virtual pid:22277') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
I've tried adding this to config/initializers/delayed_job.rb and it routes everything but the interval query logs which still get put in log/development.log.
if Rails.env == "development"
Delayed::Worker.logger = Logger.new(File.join(Rails.root, "log", "delayed_job.log"))
end
Thank you.
The log line still shows up because that line is logged by ActiveRecord, not Delayed Job. See the github bug report for more info on that. Here's a workaround:
in config/initializers/delayed_job_silencer.rb:
if Rails.env.development?
module Delayed
module Backend
module ActiveRecord
class Job
class << self
alias_method :reserve_original, :reserve
def reserve(worker, max_run_time = Worker.max_run_time)
previous_level = ::ActiveRecord::Base.logger.level
::ActiveRecord::Base.logger.level = Logger::WARN if previous_level < Logger::WARN
value = reserve_original(worker, max_run_time)
::ActiveRecord::Base.logger.level = previous_level
value
end
end
end
end
end
end
end
Change ActiveRecord log level only for Delayed job.
This will allow you to see other ActiveRecord logs.
Use:
Delayed::Backend::ActiveRecord::Job.logger.level = 1
I think you can set log level to one as follows in the initializers. It helps me to ignoring delayed job query info from the production as well as development log too.
ActiveRecord::Base.logger.level = 1
Here is the actual error from logfile:
2013-02-20T20:56:03+00:00 app[worker.1]: SQL (11.1ms) UPDATE "delayed_jobs" SET locked_by = null, locked_at = null WHERE (locked_by = 'host:05c659a5-86fd-46dd-b139-263e49a96171 pid:2')
2013-02-20T20:56:03+00:00 app[worker.1]: rake aborted!
2013-02-20T20:56:03+00:00 app[worker.1]: PG::Error: ERROR: cannot execute UPDATE in a read-only transaction
2013-02-20T20:56:03+00:00 app[worker.1]: : UPDATE "delayed_jobs" SET locked_at = '2013-02-20 20:56:02.982379', locked_by = 'host:05c659a5-86fd-46dd-b139-263e49a96171 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2013-02-20 20:56:02.882553' AND (locked_at IS NULL OR locked_at < '2013-02-20 16:56:02.882574') OR locked_by = 'host:05c659a5-86fd-46dd-b139-263e49a96171 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
This error also appears in the followers log. That's why assumed there is a problem with that. Delayed Job version: 3.0.5
It seems that issue came with recent update. See this Github issue for details:
https://github.com/collectiveidea/delayed_job_active_record/issues/34
For me downgrading to 0.3.3 worked.