Heroku 20
Ruby: 3.1.0
Rails: 6.1.0
Redis: Screen from gemfile.lock
Redis on heroku: 6.2.3
Most likely after changing the version of redis on heroku: from free to premium, the worker stopped working. (Web is working)
heroku run rails c
redis = Redis.new(url: ENV["REDIS_URL"])
# => #<Redis client v5.0.6 for rediss://ec2-5x.xx.xx.xx.eu-west-1.compute.amazonaws.com:12499/0>
redis.ping
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/ruby_connection.rb:62:in `connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=5x.xx.xx.xx:12499 state=error: certificate verify failed (self signed certificate in certificate chain) (Redis::CannotConnectError)
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/ruby_connection.rb:62:in `connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=5x.xx.xx.xx:12499 state=error: certificate verify failed (self signed certificate in certificate chain) (RedisClient::CannotConnectError)
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/ruby_connection.rb:62:in `connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=5x.xx.xx.xx:12499 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
$redis = Redis.new(url: ENV["REDIS_URL"], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })
$redis.ping
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/connection_mixin.rb:48:in `call_pipelined': WRONGPASS invalid username-password pair or user is disabled. (Redis::CannotConnectError)
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/connection_mixin.rb:48:in `call_pipelined': WRONGPASS invalid username-password pair or user is disabled. (RedisClient::AuthenticationError)
Then I attempted to change rediss:// to redis:// in the URL:
redis = Redis.new(url: url_with_one_s, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })
redis.ping
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/ruby_connection.rb:130:in `rescue in read': Connection reset by peer (Redis::CannotConnectError)
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/ruby_connection.rb:130:in `rescue in read': Connection reset by peer (RedisClient::CannotConnectError)
# /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/ruby_connection.rb:130:in `rescue in read': Connection reset by peer (RedisClient::ConnectionError)
# /app/vendor/ruby-3.1.0/lib/ruby/3.1.0/socket.rb:452:in `__read_nonblock': Connection reset by peer (Errno::ECONNRESET)
Procfile:
release: bundle exec rails db:migrate
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -c 5 -C config/sidekiq.yml
When I run in bash:
bundle exec sidekiq -c 5 -C config/sidekiq.yml
I get:
pid=4 tid=1i50 INFO: Booting Sidekiq 6.5.5 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"rediss://:REDACTED#ec2-5x.xx.xx.xx.eu-west-1.compute.amazonaws.com:12499/1"}
W, [2023-02-14T12:22:11.722698 #4]
pid=4 tid=1i50 INFO: Booted Rails 6.1.7.1 application in staging environment
pid=4 tid=1i50 INFO: Running in ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [x86_64-linux]
pid=4 tid=1i50 INFO: See LICENSE and the LGPL-3.0 for licensing details.
pid=4 tid=1i50 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
pid=4 tid=1i50 WARN: Redis::CannotConnectError: WRONGPASS invalid username-password pair or user is disabled.
pid=4 tid=1i50 WARN: /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.12.1/lib/redis_client/connection_mixin.rb:48:in `call_pipelined'
heroku redis:info
› Warning: heroku update available from 7.60.2 to 7.67.1.
=== redis-xxxxx-xxxxx (REDIS_URL)
Plan: Premium 0
Status: available
Created: 2021-xx-xx xx:xx
Version: 6.2.3
Timeout: 300
Maxmemory: noeviction
Maintenance: scheduled for 2023-xx-xx xx:xx:00 +0000, required by 2023-xx-xx xx:xx:xx +0000
Maintenance window: Wednesdays xx:xx to Thursdays xx:xx UTC
Persistence: AOF
HA Status: Available
Requires TLS: Yes
Keyspace Notifications: Disabled
Related
I want to start Puma (v.5.2.2) using SSL in on Windows on Rails 6.
When I execute:
bundle exec puma
I get the following error:
Puma compiled without SSL support (RuntimeError)
I have openssl installed:
>openssl version
OpenSSL 1.1.1j 16 Feb 2021
In the file puma.rb, I inserted:
ssl_bind '0.0.0.0', 3000, {
key: 'mykey.key',
cert: 'mykey.crt',
verify_mode: 'none'
}
I generated mykey.key and mykey.crt and put them in Rails root folder.
What I've already tried, without success:
I reinstalled Puma after the installation of openssl
I ran the server using this command: rails s -b "ssl://localhost:3000?key=mykey.key&cert=mykey.crt"
I tried Puma version 4.x.x
> ruby -rpuma -e "puts Puma.ssl?"
false
gem install puma
> ruby -rpuma -e "puts Puma.ssl?"
true
I'm trying to run Sidekiq 5.2.2 on Rails 5.2 but I always get a connection lost error. Below are my server logs:
# bundle exec sidekiq -q v2_default,2 -q v2_low -e production
TID-2yed0 INFO: Booting Sidekiq 5.2.2 with redis options {:url=>"redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1", :network_timeout=>7, :id=>"Sidekiq-server-PID-3644"}
TID-2yed0 INFO: Running in ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-linux]
TID-2yed0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
TID-2yed0 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
TID-2yed0 INFO: Starting processing, hit Ctrl-C to stop
TID-19spa0 ERROR: heartbeat: Connection lost (ECONNRESET)
TID-xhh1g ERROR: Error fetching job: READONLY You can't write against a read only slave.
TID-xhh1g WARN: Redis::CommandError: READONLY You can't write against a read only slave.
TID-xhh1g WARN: /usr/local/rvm/gems/ruby-2.3.5/gems/redis-4.0.2/lib/redis/client.rb:119:in `call'
and this is my config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1', network_timeout: 7 }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1', network_timeout: 7 }
end
I verify that I can do ping PONG with the server but I don't know why sidekiq can't connect.
# rails c production
Running via Spring preloader in process 7372
Loading production environment (Rails 5.2.1)
2.3.5 :001 > require "redis"
=> false
2.3.5 :002 > redis = Redis.new(url: 'redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1')
=> #<Redis client v4.0.2 for redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1>
2.3.5 :003 > redis.ping
=> "PONG"
2.3.5 :004 >
Any ideas?
I was just pointing to a wrong redis endpoint in AWS. Thanks to Pavel Mikhailyuk for pointing out.
Gem which i have used in this application
gem 'redis', '~> 3.0'
gem 'sidekiq'
gem 'bunny'
This is the consumer part to get the message from queue as a rake task.
task :do_consumer => :environment do
connection = Bunny.new(ENV['CLOUDAMQP_URL'])
connection.start # start a communication session with the amqp server
channel = connection.channel()
queue = channel.queue('order-queue', durable: true)
puts ' [*] Waiting for messages. To exit press CTRL+C'
queue.subscribe(manual_ack: true, block: true) do |delivery_info,
properties, payload|
puts " [x] Received #{payload}"
puts " [x] Done"
channel.ack(delivery_info.delivery_tag, false)
# Call worker to do task
callSidekiqWorker.perform_async(payload)
end
end
Procfile
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e production
worker: bundle exec rake do_consumer
I have added redistogo addon in heroku and i configured REDIS_PROVIDER as the ENV variable.
here the issue this is working fine in local, but after i push to heroku i am getting this error in logs:
[x] Received {"pos_items_cache_id":816,"process_state_id":320,"location_id":604}
14 Sep 2018 16:57:57.439106 <190>1 2018-09-14T11:27:56.885838+00:00 app worker.1 - - [x] Done
14 Sep 2018 16:57:57.509635 <190>1 2018-09-14T11:27:56.885844+00:00 app worker.1 - - E, [2018-09-14T11:27:56.885527 4] ERROR --
<Bunny::Session:0x29beb80 vmeksylf#chimpanzee.rmq.cloudamqp.com:5672, vhost=vmeksylf, addresses=[chimpanzee.rmq.cloudamqp.com:5672]>: Uncaught exception from consumer
<Bunny::Consumer:21834760 #channel_id=1 #queue=order-queue> #consumer_tag=bunny-1536910387000-84532836151>:
<Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED)> # /app/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis/client.rb:345:in `rescue in establish_connection'
am i doing any wrong configuration in heroku? Also i think this rake task is not listening always.
When I deploy my application for the first time using cap [env] deploy, everything works as expected. The code is deployed and the puma server is started successfully with puma:start.
capistrano output of puma starting
* 2015-01-09 12:19:37 executing `puma:start'
* executing "cd /path/to/app/current && bundle exec puma -q -d -e production -C ./config/puma/production.rb"
servers: ["example.com"]
[example.com] executing command
** [out :: example.com] Puma starting in single mode...
** [out :: example.com] * Version 2.9.2 (ruby 2.1.5-p273), codename: Team High Five
** [out :: example.com] * Min threads: 0, max threads: 16
** [out :: example.com] * Environment: production
** [out :: example.com] * Listening on unix:///path/to/app/shared/sockets/puma.sock
** [out :: example.com] * Daemonizing...
If I make a code change and attempt to re-deploy, instead of calling puma:start, capistrano calls puma:restart, which acts as if successful, but puma wasn't actually restarted.
* 2015-01-09 12:27:56 executing `puma:restart'
* executing "cd /path/to/app/current && bundle exec pumactl -S /path/to/app/shared/sockets/puma.state restart"
servers: ["example.com"]
[example.com] executing command
** [out :: example.com] Command restart sent success
At this point, if I refresh the web page, I get a 504 Gateway Time-out error. It's very similar to this issue..
As the person suggested, if I added workers 1 to my puma configuration file, restart would be working, but start/stop would not.
At my current state (without workers), if I do cap [env] puma:stop, it does not stop puma. It also does NOT delete any of these files:
/path/to/app/shared/pids/puma.pid
/path/to/app/shared/sockets/puma.sock
/path/to/app/shared/sockets/puma.state
Important Note
In my Rails server, I am using ActionController::Live with Rails 4.1. I am connected to an event stream with Javascript / Redis. I noticed this in my nginx error log:
nginx error log
2015/01/09 12:29:32 [error] 8992#0: *371 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: [ip], server: example.com, request: "GET /build_configurations/refresh_workers HTTP/1.1",
upstream: "http://unix:///path/to/app/shared/sockets/puma.sock/build_configurations
/refresh_workers", host: "example.com", referrer: "https://example.com/"
All in all, how can I successfully use cap [env] deploy to deploy updates to my code while having puma start and restart okay?
Update
I found this issue that talks about restarting puma with ActionController::Live, but no solution seems present.
Using the information I found in this other Stack Overflow answer, I was able to implement a heartbeat that fixes this situation it seems. I haven't fully understood all of the mechanics myself, but I started with an initializer to start the heartbeat:
config/initializers/redis.rb
REDIS = Redis.new(url: "redist://localhost:6379")
heartbeat_thread = Thread.new do
while true
REDIS.publish('heartbeat','thump')
sleep 2.seconds
end
end
at_exit do
# not sure this is needed, but just in case
heartbeat_thread.kill
REDIS.quit
end
My controller has:
def build_status_events
response.headers["Content-Type"] = "text/event-stream"
redis = Redis.new(url: "redist://localhost:6379")
# blocks the current thread
redis.psubscribe(['my_event', 'heartbeat']) do |on|
on.pmessage do |pattern, event, data|
response.stream.write("event: #{event}\n")
if event == 'heartbeat'
response.stream.write("data: heartbeat\n\n")
else
response.stream.write("data: #{data.to_json}\n\n")
end
end
end
rescue IOError
logger.info 'Events stream closed'
ensure
logger.info 'Stopping Events streaming thread'
redis.quit
response.stream.close
end
I believe what happens is that the heartbeat gets posted, and if there's an error, the ensure block will be called and the subscription is closed. Then puma reboots as expected. If anyone has a better solution, or more information, please comment or add another answer.
I'm using a before_restart.rb hook in opsworks and I have a problem when it run "rake i18n:js:export". I don't know why is running sidekiq with this rake. it fails only in setup stage of opsworks. When I deploy it the error disappears.
[2015-01-09T18:52:17+00:00] INFO: deploy[/srv/www/XXX] queueing checkdeploy hook /srv/www/XXX/releases/20150109185157/deploy/before_restart.rb
[2015-01-09T18:52:17+00:00] INFO: Processing execute[rake i18n:js:export] action run (/srv/www/XXXX/releases/20150109185157/deploy/before_restart.rb line 3)
Error executing action `run` on resource 'execute[rake i18n:js:export]'
Mixlib::ShellOut::ShellCommandFailed
Expected process to exit with [0], but received '1'
---- Begin output of bundle exec rake i18n:js:export ----
STDOUT: 2015-01-09T18:52:30Z 1808 TID-92c6g INFO: Sidekiq client with redis options {}
STDERR: /home/deploy/.bundler/XXXX/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:309:in `rescue in establish_connection': Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED) (Redis::CannotConnectError)
Sidekiq client (NOT sidekiq server) is running because it is defined in an initializer. When rake runs, it loads the entire rails app environment. So either allow for an environment variable to disable sidekiq client in config/initializers/sidekiq.rb or make sure redis-server is properly configured on the instance you're running this on.
unless ENV['DISABLE_SIDEKIQ']
# Sidekiq.configure...
end
DISABLE_SIDEKIQ=true bundle exec rake do:stuff