How does Redis work with Rails and Sidekiq - ruby-on-rails

Problem: need to send e-mails from Rails asynchronously.
Environment: Windows 7, Ruby 2.0, Rails 4.1, Sidekiq, Redis
After setting everything up, starting Sidekiq and starting Redis, I can see the mail request queued to Redis through the monitor:
1414256204.699674 "exec"
1414256204.710675 "multi"
1414256204.710675 "sadd" "queues" "default"
1414256204.710675 "lpush" "queue:default" "{\"retry\":true,\"queue\":\"default\",\"class\":\"Sidekiq::Extensions::DelayedMailer\",\"args\":[\"---\\n- !ruby/class 'UserMailer'\\n- :async_reminder\\n- - 673\\n\"],\"jid\":\"d4024c0c219201e5d1649c54\",\"enqueued_at\":1414256204.709674}"
But the mailer method never seems to get executed. The mail doesn't get sent and none of the log messages show up.
How does Redis know to execute the job on the queue and does something else need to be setup in the environment for it to know where the application resides?
Is delayed_job a better solution?
I started redis in one window, bundle exec sidekiq in another window, and rails server in a third window.
How does an item on the redis queue get picked up and processed? Is sidekiq both putting things on the redis queue and checking to see if something was added that needs to be processed?

Redis is used just for storage. It stores jobs to be done. It does not execute anything. DelayedJob uses your database for job storage instead of Redis.
Rails process pushes new jobs to Redis.
Sidekiq process pops jobs from Redis and executes them.
In your MONITOR output, you should see LPUSH commands when Rails sends mail. You should also see BRPOP commands from Sidekiq.
You need to make sure that both Rails and Sidekiq processes use the same Redis server, database number, and namespace (if any). It's a frequent problem that they don't.

Related

How to handle redis pub/sub in rails

I have a rails app which talks to a socket.io app via Redis.
I can create a message on the rails app and socket.io broadcasts it, but I have no idea how to handle the incoming messages (i.e. I want a service to always be listening to Redis and process the incoming messages).
Can you please tell me how I can achieve this?
you can use sidekiq gem here which is by default using redis so whenever you enqueue a job to redis via perform_later method provided by sidekiq. Sidekiq worker will run it whenever a worker is free.
For Pub/sub as you mentioned in the header you can go for Action Cable feature provided by Rails.

After a deploy to EC2 sidekiq now reports SocketError: getaddrinfo: Name or service not known

Application is Rails 4.1.4, Ruby 2.1.2.
Using sidekiq 3.2.6, redis 3.1.0, celluloid 0.15.2. The sidekiq implementation is as default as can be, with the exception of connecting to a remote redis queue (elastic cache).
When certain events are processed, we use sidekiq to queue up calls to an external API. The API is reachable through curl from the server our application is hosted on. All other functionality seems to still be performing as expected. This functionality has worked for weeks on the current server implementation/architecture.
After a successful deploy (with Capistrano, through Jenkins) to and EC2 instance, which is behind an elastic load balancer, and an auto-scaling group sidekiq will no longer connect(?) to elasticcache.
SocketError: getaddrinfo: Name or service not known
/gems/redis-3.1.0/lib/redis/connection/ruby.rb:152 in getaddrinfo
/gems/redis-3.1.0/lib/redis/connection/ruby.rb:152 in connect
/gems/redis-3.1.0/lib/redis/connection/ruby.rb:211 in connect
/gems/redis-3.1.0/lib/redis/client.rb:304 in establish_connection
/gems/redis-3.1.0/lib/redis/client.rb:85 in block in connect
/gems/redis-3.1.0/lib/redis/client.rb:266 in with_reconnect
/gems/redis-3.1.0/lib/redis/client.rb:84 in connect
/gems/redis-3.1.0/lib/redis/client.rb:326 in ensure_connected
/gems/redis-3.1.0/lib/redis/client.rb:197 in block in process
/gems/redis-3.1.0/lib/redis/client.rb:279 in logging
/gems/redis-3.1.0/lib/redis/client.rb:196 in process
/gems/redis-3.1.0/lib/redis/client.rb:102 in call
/gems/redis-3.1.0/lib/redis.rb:1315 in block in smembers
/gems/redis-3.1.0/lib/redis.rb:37 in block in synchronize
/usr/local/rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/monitor.rb:211 in mon_synchronize
/gems/redis-3.1.0/lib/redis.rb:37 in synchronize
/gems/redis-3.1.0/lib/redis.rb:1314 in smembers
/gems/sidekiq-3.2.6/lib/sidekiq/api.rb:557 in block in cleanup
/gems/connection_pool-2.0.0/lib/connection_pool.rb:58 in with
/gems/sidekiq-3.2.6/lib/sidekiq.rb:72 in redis
/gems/sidekiq-3.2.6/lib/sidekiq/api.rb:556 in cleanup
/gems/sidekiq-3.2.6/lib/sidekiq/api.rb:549 in initialize
/gems/sidekiq-3.2.6/lib/sidekiq/scheduled.rb:79 in new
/gems/sidekiq-3.2.6/lib/sidekiq/scheduled.rb:79 in poll_interval
/gems/sidekiq-3.2.6/lib/sidekiq/scheduled.rb:58 in block in poll
/gems/sidekiq-3.2.6/lib/sidekiq/util.rb:15 in watchdog
/gems/sidekiq-3.2.6/lib/sidekiq/scheduled.rb:23 in poll
/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25 in public_send
/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25 in dispatch
/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122 in dispatch
/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322 in block in handle_message
/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416 in block in task
/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55 in block in initialize
/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13 in block in create
We have restarted sidekiq, restarted elastic cache, restarted the server, inspected the redis queue with redis-cli and seen nothing noteworthy.
As implied, we can connect to elastic cache using redis-cli, however, using sidekiq/apifrom the console, we get the same SocketError
Any ideas on how to remedy? The application is neigh unusable at this point.
Thanks!
Yay for embarrassing errors! There was a typo in the ENV var url. 10 hours later, between me and the devops, and it was a copy and paste issue.
Thanks

rails: deploy workers for delayed_job

Is there any good practices about setting up a queue to work with delayed_job in rails ?
For more precisions: I intend to ping some web hooks with my rails api. If using delayed_jobs, the PseudoCode could look like
get :ping do
present ping: :pong #grape style
# Bad, synchronous idea:
MyAwesomeTracker.send(event: "ping") # thi will wait for the server answer before it goes on
#Better: put it in a queue using delay_job:
MyAwesomeTracker.delay.send(event: "ping") # this will go to the queue
end
Now wether I use job_delay or resque, I'm able to send events into the queue, which is great.
The actual question: Is there any good practices for deploying workers whenever I deploy my api ?
What about worker failures ? Is there any environnement where a worker can be restarted after a crash/failure ?
I've seen a worker can be launched by running rake some_command, but what I'm wondering is how to set up an environment where a simple cap production deploy would both set up the api application, and some workers that listen to the queue.
Thanks in advance !

Ignoring unregister_worker in resque

My rails app consists of an API server and an Admin server. Both make use of resque jobs.
In my local development environment each of them have a Procfile and are using the same redis server. The problem I have is that I cannot control which server is picking up which job. Each server has distinct queues, but they seem to be picked up in the 'unregister_worker' routine.
/ruby-1.9.3-p194/gems/resque-1.24.1/lib/resque/worker.rb:459:in `unregister_worker'
I see this in my call stack and it leads to a ruby error since it doesn't know the class.
How can I tell resque to ignore an `unregister_worker'? Just to clarify I don't have a worker task associated with the '*' queue.
As a workaround I could run two redis servers locally as I do in my production environment, but I like to avoid that if I can.
Redis Namespaces do exactly that. In 'config/initializers/resque.rb' specify the namespace of the queue:
Resque.redis.namespace = "resque:admin_server"
Found the solution here:
How do you start resque to use Resque.redis.namespace?

Not able to make Resque work

I'm trying to make Resque work with my project, but unfortunately it seems that for some reasons Resque is not able to write on Redis.
Redis seems to be configured correctly, I'm able to connect with redis-cli and issue commands, runs on port 6379 as configured inside my Rails 3.0.5 app.
When I try to Resque enqueue something the job is queued, but it doesn't seem that something actually happens on Redis (0 clients connected inside my Redis logs).
When I restart the console, the queue is empty, with no workers running.
Everything fails silently, I have nothing in my rails logs, nothing on the console, nothing if I start a worker, it just (obviously) doesn't find any job to perform.
https://gist.github.com/867620
Any suggestions on how to fix or debug this ?
The problem was that I was including resque_spec in the bundle.
Obviously, resque_spec was stubbing Resque.enqueue, making my mistake very stupid and very difficult to spot.

Resources