I am using rails "Shoryuken" gem but I am getting errors for validation on queues on my development environment when I started rails server below is the error:-
gems/shoryuken-2.0.11/lib/shoryuken/environment_loader.rb:172:in `validate_queues': The specified queue(s) ["development_worker"] do not exist (ArgumentError)
I have used below settings:-
config/shoryuken.yml
aws:
access_key_id: <%= ENV["SQS_IAM"] %>
secret_access_key: <%= ENV["SQS_IAM_SECRET"] %>
region: <%= ENV["SQS_IAM_REGION"] %>
concurrency: 25 # The number of allocated threads to process messages. Default 25
delay: 0 # The delay in seconds to pause a queue when it's empty. Default 0
queues:
- ["<%= Rails.env %>_worker", 1]
initializers/shoryuken.rb
def parse_config(config_file)
if File.exist?(config_file)
YAML.load(ERB.new(IO.read(config_file)).result)
else
raise ArgumentError, "Config file #{config_file} does not exist"
end
end
config = parse_config([Dir.pwd, 'config/shoryuken.yml'].join('/')).deep_symbolize_keys
Shoryuken::EnvironmentLoader.load(config)
I want the queue should be environment specific.
Ravindra, looking at the code of shoryuken, you are getting the error because you do not have created an SQS queue named development_worker, is that rigth?
You will need to create one queue for each developer, because shoryuken will be running in the computers of each developer. If you don't do that, all shoryuken processes of each computer will be polling messages of the same queue.
Imagine that there are two shoryuken processes, sh1 and sh2, that correspond to the dev1 and dev2 machine respectively.
If the dev1 sends a SQS msg to the dev-queue, the sh2 proccess can be able to poll the message that the dev1 sent.
If you wish to avoid to create the queues in AWS, you can take a look at this.
Related
We have a shoryuken worker process setup in rails application (ruby 2.6.6, rails > 5).
this is the shoryuken.yml.erb file
:aws:
:access_key_id: ?
:secret_access_key: ?
:region: <%= Settings.aws.region %>
:concurrency: 10
:daemon: false
:pidfile: tmp/shoryuken.pid
:logfile: log/shoryuken.log.<%= Time.now.strftime('%Y%m%d') %>
:queues:
- [<%= Settings.aws.sqs.use_case_1_queue_name %>, 2]
- [<%= Settings.aws.sqs.use_case_2_queue_name %>, 1]
development:
:verbose: true
And we start the worker using shoryuken -R -C ./shoryuken.yml Please note the -R option since we would want it to run rails initializers in order to access Settings constant in the above shoryuken.yml.erb
It all looked good but then we figured out that the AWS credentials are not being stored in application setting , but in IAM role. So how will I pass the aws-sdk credential (stored using IAM) to shoryuken process.
If there are no solution, then I will need to finally store aws credentials in application setting which might be a security issue according to infra team.
Thank you for this wonderful library . Awaiting a response on this issue
I have an umbrella project compose by:
gateway - Phoenix application
core - business model layer notifications a dedicated app for delivering sms, email, etc… users
user management, role system, and authentication.
The three components are connected via AMQP, so they can send and receive messages between them.
We are using Docker, and drone.io hosted on google cloud’s kubernetes engine. So, what is happening is that randomly the application raises the following exception while running my tests:
{\"status\":\"error\",\"message\":\"%DBConnection.OwnershipError{message: \\\"cannot find ownership process for #PID<0.716.0>.\\\\n\\\\nWhen using ownership, you must manage connections in one\\\\nof the four ways:\\\\n\\\\n* By explicitly checking out a connection\\\\n* By explicitly allowing a spawned process\\\\n* By running the pool in shared mode\\\\n* By using :caller option with allowed process\\\\n\\\\nThe first two options require every new process to explicitly\\\\ncheck a connection out or be allowed by calling checkout or\\\\nallow respectively.\\\\n\\\\nThe third option requires a {:shared, pid} mode to be set.\\\\nIf using shared mode in tests, make sure your tests are not\\\\nasync.\\\\n\\\\nThe fourth option requires [caller: pid] to be used when\\\\nchecking out a connection from the pool. The caller process\\\\nshould already be allowed on a connection.\\\\n\\\\nIf you are reading this error, it means you have not done one\\\\nof the steps above or that the owner process has crashed.\\\\n\\\\nSee Ecto.Adapters.SQL.Sandbox docs for more information.\\\"}\",\"code\":0}"
It’s a Json since we exchange amqp messages in our tests, for instance:
# Given the module FindUserByEmail
def Users.Services.FindUserByEmail do
use GenAMQP.Server, event: "find_user_by_email", conn_name:
Application.get_env(:gen_amqp, :conn_name)
alias User.Repo
alias User.Models.User, as: UserModel
def execute(payload) do
%{ "email" => email } = Poison.decode!(payload)
resp =
case Repo.get_by(UserModel, email: email) do
%UserModel{} = user -> %{status: :ok, response: user}
nil -> ErrorHelper.err(2011)
true -> ErrorHelper.err(2012)
{:error, %Ecto.Changeset{} = changeset} -> ViewHelper.translate_errors(changeset)
end
{:reply, Poison.encode!(resp)}
end
end
# and the test
defmodule Users.Services.FindUserByEmailTest do
use User.ModelCase
alias GenAMQP.Client
def execute(payload) do
resp = Client.call_with_conn(#conn_name, "find_user_by_email", payload)
data = Poison.decode!(resp, keys: :atoms)
assert data.status == "ok"
end
end
Following details my .drone.yaml file:
pipeline:
unit-tests:
image: bitwalker/alpine-elixir-phoenix:1.6.1
environment:
RABBITCONN: amqp://user:pass#localhost:0000/unit_testing
DATABASE_URL: ecto://username:password#postgres/testing
commands:
- mix local.hex --force && mix local.rebar --force
- mix deps.get
- mix compile --force
- mix test
mix.exs file in every app contains the following aliases
defp aliases do
[
"test": ["ecto.create", "ecto.migrate", "test"]
]
end
All ours model_case files contain this configuration:
setup tags do
:ok = Sandbox.checkout(User.Repo)
unless tags[:async] do
Sandbox.mode(User.Repo, {:shared, self()})
end
:ok
end
How can I debug this? it only occurs while testing code inside the container. Would the issue be related to container’s resources?
Any tip or hint would be appreciated.
Try setting onwership_timeout and timeout to a large numbers in your config/tests.exs
config :app_name, User.Repo,
adapter: Ecto.Adapters.Postgres,
username: ...,
password: ...,
database: ...,
hostname: ...,
pool: Ecto.Adapters.SQL.Sandbox,
timeout: 120_000, # i think the default is 5000
pool_timeout: 120_000,
ownership_timeout: 120_000 #i think the default is 5000
We have a Rails app on Heroku with Sidekiq and are running out of database connections.
ActiveRecord::ConnectionTimeoutError: could not obtain a database
connection within 5.000 seconds (waited 5.000 seconds)
Heroku stuff:
Database plan: Standard0 (120 connections)
Web dynos: 2 Standard-2X
Worker dynos: 1 Standard-2X
heroku config:
MAX_THREADS: 5
(DB_POOL not set)
(WEB_CONCURRENCY not set)
Procfile:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq
database.yml:
...
production:
url: <%= ENV["DATABASE_URL"] %>
pool: <%= ENV["DB_POOL"] || ENV['MAX_THREADS'] || 5 %>
puma.rb:
# https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#adding-puma-to-your-application
workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 2)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
sidekiq.yml:
---
:concurrency: 25
:queues:
- [default]
We also have a couple of rake tasks that fire every 10 minutes, and they finish within a second or two.
The problem seems to happen when we do a lot of message processing in sidekiq. We do something like:
get article headlines from a 3rd party web service
insert each headline into the db inside a single transaction
create a message in sidekiq for each headline (worker.perform_async)
each message is processed, hits an endpoint to get the body and updates the body (can take .5 - 3 seconds)
While number 4 is happening we see the connection issue.
My understanding is we are way, way, way below the connection limit with our configuration above, but did we do something incorrectly? Is something just consuming the pool? Any help would be great, thanks.
Sources:
https://devcenter.heroku.com/articles/concurrency-and-database-connections
https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server
https://github.com/mperham/sidekiq/wiki/Advanced-Options
You are sharing 5 DB connections among 25 Sidekiq threads. Set DB_POOL to 25 or Sidekiq's concurrency to 5.
So I'm trying to process a CSV file via Sidekiq background job processing on a Heroku Worker instance. Whilst I can complete the process, I feel it could certainly be done quicker/more efficiently than I'm currently doing it. This question contains two parts - firstly are the database pools setup correctly and secondly how can I optimise the process.
Application environment:
Rails 4 application
Unicorn
Sidekiq
Redis-to-go (Mini plan, 50 connections max)
CarrierWave S3 implementation
Heroku Postgres (Standard Yanari, 60 connections max)
1 Heroku Web dyno
1 Heroku Worker dyno
NewRelic monitoring
config/unicorn.rb
worker_processes 3
timeout 15
preload_app true
before_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
end
end
after_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
end
if defined?(ActiveRecord::Base)
config = ActiveRecord::Base.configurations[Rails.env] ||
Rails.application.config.database_configuration[Rails.env]
config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
config['pool'] = ENV['DB_POOL'] || 2
ActiveRecord::Base.establish_connection(config)
end
end
config/sidekiq.yml
---
:concurrency: 5
staging:
:concurrency: 5
production:
:concurrency: 35
:queues:
- [default, 1]
- [imports, 10]
- [validators, 10]
- [send, 5]
- [clean_up_tasks, 30]
- [contact_generator, 20]
config/initializers/sidekiq.rb
ENV["REDISTOGO_URL"] ||= "redis://localhost:6379"
Sidekiq.configure_server do |config|
config.redis = { url: ENV["REDISTOGO_URL"] }
database_url = ENV['DATABASE_URL']
if database_url
ENV['DATABASE_URL'] = "#{database_url}?pool=50"
ActiveRecord::Base.establish_connection
end
end
Sidekiq.configure_client do |config|
config.redis = { url: ENV["REDISTOGO_URL"] }
end
The database connection pools are worked out as such:
I have 3 Web processes (unicorn worker_processes), to each of these I am allocating 2 ActiveRecord connections via the after_fork hook (config/unicorn.rb) for 6 total (maximum) of my 60 available Postgres connections assigned to the Web dyno. In the Sidekiq initialiser, I'm allocating 50 Postgres connections via the ?pool=50 param appended to ENV['DATABASE_URL'] as described (somewhere) in the docs. I'm keeping my Sidekiq concurrency value at 35 (sidekiq.yml) to ensure I stay under both the 50 Redis connections and 60 Postgres connection limits. This still needs more finely grained tuning, but I'd rather get the data processing itself sorted before going any further with this.
Now, assuming the above is correct (and it wouldn't surprise me at all if it weren't) I'm handling the following scenario:
A user uploads a CSV file to be processed via their browser. This file can be anywhere between 50 rows and 10 million rows. The file is uploaded to S3 via the CarrierWave gem.
The user then configures a couple of settings for the import via the UI,the culmination of which adds an FileImporter job to the Sidekiq queue to start creating various models based on the rows.
The Import worker looks something like:
class FileImporter
include Sidekiq::Worker
sidekiq_options :queue => :imports
def perform(import_id)
import = Import.find_by_id import_id
CSV.foreach(open(import.csv_data), headers: true) do |row|
# import.csv_data is the S3 URL of the file
# here I do some validation against a prebuilt redis table
# to validate the row without making any activerecord calls
# (business logic validations rather than straight DB ones anyway)
unless invalid_record # invalid_record being the product of the previous validations
# queue another job to actually create the AR models for this row
ImportValidator.perform_async(import_id, row)
# increment some redis counters
end
end
end
This is slow - I've tried to limit the calls to ActiveRecord in the FileImporter worker so I'm assuming it's because I'm streaming the file from S3. It's not processing rows fast enough to build up a queue so I'm never utilising all of my worker threads (usually somewhere between 15-20 of the 35 available threads are active. I've tried splitting this job up and feeding rows a 100 at a time into an intermediary worker which then creates the ImportValidator jobs in a more parallel fashion but that didn't fare much better.
So my question is, what's the best/most efficient method to accomplish a task like this?
It's possible you are at 100% CPU with 20 threads. You need another dyno.
I'm used to using delayed_jobs method of going into the console to see whats in the queue, and the ease of clearing the queue when needed. Are there similar commands in Sidekiq for this? Thanks!
There is an ergonomic API for viewing and managing queues.
It is not required by default.
require 'sidekiq/api'
Here's the excerpt:
# get a handle to the default queue
default_queue = Sidekiq::Queue.new
# get a handle to the mailer queue
mailer_queue = Sidekiq::Queue.new("mailer")
# How many jobs are in the default queue?
default_queue.size # => 1001
# How many jobs are in the mailer queue?
mailer_queue.size # => 50
#Deletes all Jobs in a Queue, by removing the queue.
default_queue.clear
You can also get some summary statistics.
stats = Sidekiq::Stats.new
# Get the number of jobs that have been processed.
stats.processed # => 100
# Get the number of jobs that have failed.
stats.failed # => 3
# Get the queues with name and number enqueued.
stats.queues # => { "default" => 1001, "email" => 50 }
#Gets the number of jobs enqueued in all queues (does NOT include retries and scheduled jobs).
stats.enqueued # => 1051
I haven't ever used Sidekiq, so it's possible that there are methods just for viewing the queued jobs, but they would really just be wrappers around Redis commands, since that's basically all Sidekiq (and Resque) is:
# See workers
Sidekiq::Client.registered_workers
# See queues
Sidekiq::Client.registered_queues
# See all jobs for one queue
Sidekiq.redis { |r| r.lrange "queue:app_queue", 0, -1 }
# See all jobs in all queues
Sidekiq::Client.registered_queues.each do |q|
Sidekiq.redis { |r| r.lrange "queue:#{q}", 0, -1 }
end
# Remove a queue and all of its jobs
Sidekiq.redis do |r|
r.srem "queues", "app_queue"
r.del "queue:app_queue"
end
Unfortunately, removing a specific job is a little more difficult as you'd have to copy its exact value:
# Remove a specific job from a queue
Sidekiq.redis { |r| r.lrem "queue:app_queue", -1, "the payload string stored in Redis" }
You could do all of this even more easily via redis-cli :
$ redis-cli
> select 0 # (or whichever namespace Sidekiq is using)
> keys * # (just to get an idea of what you're working with)
> smembers queues
> lrange queues:app_queue 0 -1
> lrem queues:app_queue -1 "payload"
if there is any scheduled job. You may delete all the jobs using the following command:
Sidekiq::ScheduledSet.new.clear
if there any queues you wanted to delete all jobs you may use the following command:
Sidekiq::Queue.new.clear
Retries Jobs can be removed by the following command also:
Sidekiq::RetrySet.new.clear
There are more information here at the following link, you may checkout:
https://github.com/mperham/sidekiq/wiki/API
There is a API for accessing real-time information about workers, queues and jobs.
Visit here https://github.com/mperham/sidekiq/wiki/API
A workaround is to use the testing module (require 'sidekiq/testing') and to drain the worker (MyWorker.drain).
There were hanged 'workers' in default queue and I was able to see them though web interface. But they weren't available from console if I used Sidekiq::Queue.new.size
irb(main):002:0> Sidekiq::Queue.new.size
2014-03-04T14:37:43Z 17256 TID-oujb9c974 INFO: Sidekiq client with redis options {:namespace=>"sidekiq_staging"}
=> 0
Using redis-cli I was able to find them
redis 127.0.0.1:6379> keys *
1) "sidekiq_staging:worker:ip-xxx-xxx-xxx-xxx:7635c39a29d7b255b564970bea51c026-69853672483440:default"
2) "sidekiq_staging:worker:ip-xxx-xxx-xxx-xxx:0cf585f5e93e1850eee1ae4613a08e45-70328697677500:default:started"
3) "sidekiq_staging:worker:ip-xxx-xxx-xxx-xxx:7635c39a29d7b255b564970bea51c026-69853672320140:default:started"
...
The solution was:
irb(main):003:0> Sidekiq.redis { |r| r.del "workers", 0, -1 }
=> 1
Also in the Sidekiq v3 there is a command
Sidekiq::Workers.new.prune
But for some reason it didn't work for me that day
And if you want to clear the sidekiq retry queue, it's this: Sidekiq::RetrySet.new.clear
$ redis-cli
> select 0 # (or whichever namespace Sidekiq is using)
> keys * # (just to get an idea of what you're working with)
> smembers queues
> lrange queue:queue_name 0 -1 # (queue_name must be your relevant queue)
> lrem queue:queue_name -1 "payload"
Rake task for clear all sidekiq queues:
namespace :sidekiq do
desc 'Clear sidekiq queue'
task clear: :environment do
require 'sidekiq/api'
Sidekiq::Queue.all.each(&:clear)
end
end
Usage:
rake sidekiq:clear
This is not a direct solution for the Rails console, but for a quick monitoring of the Sidekiq task count and queue size, you can use sidekiqmon binary that ships with Sidekiq 6+:
$ sidekiqmon
Sidekiq 6.4.2
2022-07-25 11:05:56 UTC
---- Overview ----
Processed: 20,313,347
Failed: 57,120
Busy: 9
Enqueued: 17
Retries: 0
Scheduled: 37
Dead: 2,382
---- Processes (1) ----
36f993209f93:15:a498f85c6a12 [server]
Started: 2022-07-25 10:49:43 +0000 (16 minutes ago)
Threads: 10 (9 busy)
Queues: default, elasticsearch, statistics
---- Queues (3) ----
NAME SIZE LATENCY
default 0 0.00
elasticsearch 17 0.74
statistics 0 0.00