I'm using the latest version of Sidekiq (2.15.2). I'm not sure how this happened but for some reason, my Sidekiq dashboard always has 2 busy processes.
When I click on busy, I just get the message, "Internal Server Error."
I tried stopping and restarting Sidekiq, but I still get the 2 busy processes. Does anyone have any suggestions? Thanks!
Here is my log after resetting Sidekiq.
2013-11-13T05:09:02Z 2508 TID-osg9yldog INFO: Received USR1, no longer accepting new work
2013-11-13T05:09:02Z 2508 TID-osgbbuwe4 INFO: Shutting down 25 quiet workers
2013-11-13T05:09:04Z 2593 TID-ow9gk2yd0 INFO: Received USR1, no longer accepting new work
2013-11-13T05:09:04Z 2593 TID-ow9hyfa04 INFO: Shutting down 25 quiet workers
2013-11-13T05:09:32Z 2508 TID-osg9yldog INFO: Shutting down
2013-11-13T05:09:32Z 2508 TID-osgbbuwe4 INFO: Shutting down 0 quiet workers
2013-11-13T05:09:36Z 2593 TID-ow9gk2yd0 INFO: Shutting down
2013-11-13T05:09:36Z 2593 TID-ow9hyfa04 INFO: Shutting down 0 quiet workers
2013-11-13T05:10:33Z 15613 TID-osg19rhj4 INFO: Booting Sidekiq 2.15.2 using redis://localhost:6379/0 with options {}
2013-11-13T05:10:33Z 15613 TID-osg19rhj4 INFO: Running in ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]
2013-11-13T05:10:33Z 15613 TID-osg19rhj4 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2013-11-13T05:10:33Z 15613 TID-osg19rhj4 INFO: Starting processing, hit Ctrl-C to stop
2013-11-13T05:10:33Z 15698 TID-ox8r4hzlg INFO: Booting Sidekiq 2.15.2 using redis://localhost:6379/0 with options {}
2013-11-13T05:10:33Z 15698 TID-ox8r4hzlg INFO: Running in ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]
2013-11-13T05:10:33Z 15698 TID-ox8r4hzlg INFO: See LICENSE and the LGPL-3.0 for licensing details.
2013-11-13T05:10:33Z 15698 TID-ox8r4hzlg INFO: Starting processing, hit Ctrl-C to stop
If you are using a default redis.conf you'll find the following lines:
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb
dir ./
This means Redis will write on those periods a snapshot of the current status.
When you are developing probably you'd like to delete this file or commenting all the save lines as per redis.conf document before restarting the server again, or it will try to pick up any previous activity.
Related
I have been trying to understand an issue I've had when running roribio16/alpine-sqs docker image on one of my machines. Whenever I try to run the image without specifying any other settings, docker run roribio16/alpine-sqs
[xxxx#yyyy ~]$ docker run roribio16/alpine-sqs
2021-05-29 15:48:41,216 INFO Included extra file "/etc/supervisor/conf.d/elasticmq.conf" during parsing
2021-05-29 15:48:41,216 INFO Included extra file "/etc/supervisor/conf.d/insight.conf" during parsing
2021-05-29 15:48:41,216 INFO Included extra file "/etc/supervisor/conf.d/sqs-init.conf" during parsing
2021-05-29 15:48:41,216 INFO Set uid to user 0 succeeded
2021-05-29 15:48:41,222 INFO RPC interface 'supervisor' initialized
2021-05-29 15:48:41,222 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-05-29 15:48:41,222 INFO supervisord started with pid 1
2021-05-29 15:48:42,225 INFO spawned: 'sqs-init' with pid 9
2021-05-29 15:48:42,229 INFO spawned: 'elasticmq' with pid 10
2021-05-29 15:48:42,230 INFO spawned: 'insight' with pid 11
cp: can't stat '/opt/custom/*.conf': No such file or directory
> sqs-insight#0.3.0 start /opt/sqs-insight
> node index.js
15:48:42.605 [main] INFO org.elasticmq.server.Main$ - Starting ElasticMQ server (0.15.0) ...
Loading config file from "/opt/sqs-insight/lib/../config/config_local.json"
15:48:42.929 [elasticmq-akka.actor.default-dispatcher-2] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
Unable to load queues for undefined
Config contains 0 queues.
library initialization failed - unable to allocate file descriptor table - out of memorylistening on port 9325
2021-05-29 15:48:43,233 INFO success: sqs-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-29 15:48:43,233 INFO success: elasticmq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-29 15:48:43,234 INFO success: insight entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-29 15:48:43,234 INFO exited: sqs-init (exit status 0; expected)
2021-05-29 15:48:44,318 INFO exited: elasticmq (terminated by SIGABRT (core dumped); not expected)
2021-05-29 15:48:45,322 INFO spawned: 'elasticmq' with pid 67
15:48:45.743 [main] INFO org.elasticmq.server.Main$ - Starting ElasticMQ server (0.15.0) ...
15:48:46.044 [elasticmq-akka.actor.default-dispatcher-2] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
library initialization failed - unable to allocate file descriptor table - out of memory2021-05-29 15:48:47,223 INFO success: elasticmq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-29 15:48:47,389 INFO exited: elasticmq (terminated by SIGABRT (core dumped); not expected)
2021-05-29 15:48:48,393 INFO spawned: 'elasticmq' with pid 89
15:48:48.766 [main] INFO org.elasticmq.server.Main$ - Starting ElasticMQ server (0.15.0) ...
15:48:49.066 [elasticmq-akka.actor.default-dispatcher-3] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
library initialization failed - unable to allocate file descriptor table - out of memory^C2021-05-29 15:48:49,559 INFO success: elasticmq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-05-29 15:48:49,559 WARN received SIGINT indicating exit request
2021-05-29 15:48:49,559 INFO waiting for insight, elasticmq to die
2021-05-29 15:48:49,566 INFO stopped: insight (terminated by SIGTERM)
2021-05-29 15:48:50,431 INFO stopped: elasticmq (terminated by SIGABRT (core dumped))
With a bit of googling I found this post where somebody had the same issue when running some other random image, and then posted that they managed to get the image running by setting some ulimits when running the image, which also worked for me (docker run --ulimit nofile=122880:122880 roribio16/alpine-sqs).
I checked the ulimits set inside the container when I didn't use this configuration
docker exec -it ca bash
$ ulimit -a
and found that the nofile setting was ridiculously high, which I assume is what is causing the container to run out of memory, if too many files are being opened simultaneously. I don't have a particulary good understanding of how this works though so would appreciate any clarification somebody could shed on that particular topic also.
Anyway the point of that ramble is that I want to try and find where the default docker container ulimits are set as I don't understand why they are so high on the machine I am using. I have another machine that does not have this problem.
I can find lots of ways to change the default limits but there does not seem to be much information about where these limits get set in the first place. I understand according to the docker documentation that if custom values are not set then the ulimits should be inherited from my system but as far as I can tell my system nofile settings are much lower than what I'm seeing in the container.
(Both machines run manjaro linux however the one that doesn't have this issue is XFCE and the one that does is KDE).
As you can see in the stack trace below, Reminders::FindStaleJobsJob is causing a problem because of the uninitialized constant Reminders. What I don't get is that I don't call Reminders::FindStaleJobsJob anywhere; rather, I call Recaps::FindStaleJobsJob.
I have flushed out the Sidekiq queue and still get this error repeatedly.
2018-09-25T17:45:14.539Z 12784 TID-oxxicof3s INFO: Running in ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-darwin17]
2018-09-25T17:45:14.539Z 12784 TID-oxxicof3s INFO: See LICENSE and the LGPL-3.0 for licensing details.
2018-09-25T17:45:14.539Z 12784 TID-oxxicof3s INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2018-09-25T17:45:14.541Z 12784 TID-oxxicof3s INFO: Starting processing, hit Ctrl-C to stop
2018-09-25T18:00:05.107Z 12784 TID-oxxi975os Recaps::FindStaleJobsJob JID-ec113586e3f8fe72eb3ca479 INFO: start
2018-09-25T18:00:05.135Z 12784 TID-oxxim1crg ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-4bc5f87567ca3f019b2015e4 INFO: start
2018-09-25T18:00:05.136Z 12784 TID-oxxi970ss Recaps::FindStaleJobsJob JID-3125783fd5da7604b95bb813 INFO: start
2018-09-25T18:00:05.155Z 12784 TID-oxxim1crg ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-4bc5f87567ca3f019b2015e4 INFO: fail: 0.02 sec
2018-09-25T18:00:05.155Z 12784 TID-oxxim1crg WARN: {"context":"Job raised exception","job":{"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","queue":"default","description":"","args":[{"job_class":"Reminders::FindStaleJobsJob","job_id":"d6161fcf-2abd-4e2b-8946-73668a78282f","queue_name":"default","arguments":[]}],"retry":true,"jid":"4bc5f87567ca3f019b2015e4","created_at":1537898405.1336598,"enqueued_at":1537898405.133705},"jobstr":"{\"class\":\"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper\",\"queue\":\"default\",\"description\":\"\",\"args\":[{\"job_class\":\"Reminders::FindStaleJobsJob\",\"job_id\":\"d6161fcf-2abd-4e2b-8946-73668a78282f\",\"queue_name\":\"default\",\"arguments\":[]}],\"retry\":true,\"jid\":\"4bc5f87567ca3f019b2015e4\",\"created_at\":1537898405.1336598,\"enqueued_at\":1537898405.133705}"}
2018-09-25T18:00:05.155Z 12784 TID-oxxim1crg WARN: NameError: uninitialized constant Reminders
My Sidekiq cron initializer:
#/config/initializers/sidekiq_cron_scheduler.rb
jobs_hash = {
'recap' => {
'class' => 'Recaps::FindStaleJobsJob',
'cron' => '0, 15, 30, 45 * * * *',
'active_job' => true
}
}
Sidekiq::Cron::Job.load_from_hash jobs_hash
Am I doing something silly and obvious?
Something was hung up somewhere. I removed the Recaps module from the sidekiq-cron initializer and let it fail on that. Then I reintroduced the module name and with a few redis-cli flushall commands sprinkled and there, and everything seems to be working fine.
When I look to /sidekiq, I can see in the section called Retries that there's a job that has not been processed - failed.
Queue default
Job Order.order_report_for_manufacturers
Arguments
JID d25956cdd486335ecaf8a186
Created At about 5 hours ago
Enqueued about an hour ago
Retry Count 10
Last Retry about an hour ago
Next Retry about 2 hours from now
It's confusing for me, because when I search in my project the method order_report_for_manufacturers, the editor (TextMate) doesn't find anything. Then I though that maybe this method might be used in a CRON job, so I logged in to my Ubuntu server, run crontab -l but there's nothing (no rake task) that would use a method with such a name.
I tried also restart the server (and Sidekiq), but in a while this job (with this failed method) is there again.
How do I find from where is the method called? (it's not in the Order model).
Thank you.
EDIT:
Sidekiq log:
2016-05-15T14:23:40.910Z 25380 TID-163go0 Sidekiq::Extensions::DelayedClass JID-d25956cdd486335ecaf8a186 INFO: fail: 0.006 sec
2016-05-15T14:23:40.910Z 25380 TID-163go0 WARN: {"class"=>"Sidekiq::Extensions::DelayedClass", "args"=>["---\n- !ruby/class 'Order'\n- :order_report_for_manufacturers\n- []\n"], "retry"=>true, "queue"=>"default", "jid"=>"d25956cdd486335ecaf8a186", "created_at"=>1463316907.739199, "enqueued_at"=>1463322220.9033682, "error_message"=>"undefined method `order_report_for_manufacturers' for #<Class:0x000000036eb740>", "error_class"=>"NoMethodError", "failed_at"=>1463316955.7728505, "retry_count"=>8, "retried_at"=>1463322220.9097943}
2016-05-15T14:23:40.911Z 25380 TID-163go0 WARN: NoMethodError: undefined method `order_report_for_manufacturers' for #<Class:0x000000036eb740>
2016-05-15T14:23:40.911Z 25380 TID-163go0 WARN: /home/deployer/apps/myapp-production/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.6/lib/active_record/dynamic_matchers.rb:26:in `method_missing'
EDIT2:
I am using Sidekiq as an Upstart service (Ubuntu server). I am just running the command ps aux | grep '[s]idekiq', I get this output:
deployer 6647 0.4 15.8 1162920 611080 ? Ssl May03 73:59 sidekiq 4.1.1 myapp-production [0 of 25 busy]
deployer 6659 0.4 18.2 1132336 702620 ? Ssl May03 75:06 sidekiq 4.1.1 myapp-production [0 of 25 busy]
deployer 25380 0.4 8.5 936444 329260 ? Ssl May14 5:14 sidekiq 4.1.1 myapp-production [0 of 25 busy]
I don't know why Sidekiq is there three times; I know that I restarted sidekiq yesterday (sudo stop sidekiq index=0 and sudo start sidekiq index=0) - that's the one on the third line.
What are the first two rows I have no idea. Redis is running on a separated server and communicate with this server (where is the application + Sidekiq).
I need something sending emails on demand.
Each user in my app should be able to set a date and hour when he will receive an email periodically, reminding that she or him has to take medicines.
Is it possible to handle this with a custom daemon? I think so, but its my first time with this stuff and I need your opinion to be sure before starting.
http://railscasts.com/episodes/129-custom-daemon
Thanks a lot, guys.
Edit:
medicine_worker.rb:
class MedicineWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(name, count)
puts 'Doing hard work'
end
end
redis.rb:
# bundle exec sidekiq
$redis = Redis.new(:host => 'localhost', :port => 6379)
bundle exec sidekiq:
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: Running in ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: See LICENSE and the LGPL-3.0 for licensing details.
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: Starting processing, hit Ctrl-C to stop
2014-08-26T14:57:17.374Z 1754 TID-zvhqi8 MedicineWorker JID-d3b450f5fd4b926ea70f1c13 INFO: start
Doing hard work
2014-08-26T14:57:17.374Z 1754 TID-zvgu8k MedicineWorker JID-f4958c2fb72aabd91d53b02b INFO: start
Doing hard work
2014-08-26T14:57:17.375Z 1754 TID-zvgbmk MedicineWorker JID-104ee301c6e5a034f42ed06d INFO: start
Doing hard work2014-08-26T14:57:17.375Z 1754 TID-zvfigk MedicineWorker JID-5698a08a24a3def961908dc1 INFO: start
Doing hard work2014-08-26T14:57:17.376Z 1754 TID-zt4ps4 MedicineWorker JID-36d06312e1fdfe8cc439ddbd INFO: start
Doing hard work
2014-08-26T14:57:17.377Z 1754 TID-zvgu8k MedicineWorker JID-f4958c2fb72aabd91d53b02b INFO: done: 0.003 sec
2014-08-26T14:57:17.378Z 1754 TID-zt3qkc MedicineWorker JID-b25e0e7be05580e358c6744a INFO: start
Doing hard work
2014-08-26T14:57:17.379Z 1754 TID-zvhqi8 MedicineWorker JID-d3b450f5fd4b926ea70f1c13 INFO: done: 0.005 sec
2014-08-26T14:57:17.380Z 1754 TID-zvgbmk MedicineWorker JID-104ee301c6e5a034f42ed06d INFO: done: 0.005 sec
2014-08-26T14:57:17.380Z 1754 TID-zvgqac MedicineWorker JID-38a185bc67c673062c475faf INFO: start
Doing hard work2014-08-26T14:57:17.380Z 1754 TID-zvhbbk MedicineWorker JID-d2058fcebd88210c406b49e4 INFO: start
Doing hard work2014-08-26T14:57:17.383Z 1754 TID-zt4ps4 MedicineWorker JID-36d06312e1fdfe8cc439ddbd INFO: done: 0.007 sec
2014-08-26T14:57:17.384Z 1754 TID-zvfigk MedicineWorker JID-5698a08a24a3def961908dc1 INFO: done: 0.009 sec
2014-08-26T14:57:17.385Z 1754 TID-zvf9k0 MedicineWorker JID-cc5e3ee8ddeead76d0c1fc5b INFO: start
Doing hard work2014-08-26T14:57:17.385Z 1754 TID-zvg6c0 MedicineWorker JID-d83dc9cb10b69c1680305417 INFO: start
Doing hard work
2014-08-26T14:57:17.385Z 1754 TID-zt4h14 MedicineWorker JID-9d13fbed1bb0042eea452ec5 INFO: start
Doing hard work
2014-08-26T14:57:17.388Z 1754 TID-zt5uv0 MedicineWorker JID-877e16ded3e18828e3396719 INFO: start
Doing hard work2014-08-26T14:57:17.389Z 1754 TID-zt3qkc MedicineWorker JID-b25e0e7be05580e358c6744a INFO: done: 0.011 sec
2014-08-26T14:57:17.389Z 1754 TID-10h4kzs MedicineWorker JID-6739ea8a340b44c06da9dabf INFO: start
Doing hard work
2014-08-26T14:57:17.390Z 1754 TID-10h4pi0 MedicineWorker JID-dcb9282efabb6c9948014834 INFO: start
Doing hard work
2014-08-26T14:57:17.391Z 1754 TID-zt1wnk MedicineWorker JID-c6fa12391faff774ffda3f5b INFO: start
Doing hard work
2014-08-26T14:57:17.392Z 1754 TID-10h6pco MedicineWorker JID-44d7747775163b4f8bf40af8 INFO: start
Doing hard work
2014-08-26T14:57:17.393Z 1754 TID-10h6u2o MedicineWorker JID-212baee3c6f3526a3ff24d9c INFO: start
Doing hard work
2014-08-26T14:57:17.394Z 1754 TID-10h5cvo MedicineWorker JID-27f2009864a0e63ab4cab2d3 INFO: start
Doing hard work2014-08-26T14:57:17.446Z 1754 T
.......
.......
.......
You better use a Background Queue for this preferably one with a scheduling option.
Sidekiq is pretty good for what you need.
https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
When I create a new user on GitLab the email is not sent to the new user's email inbox.
I am new to GitLab so don't know in-depth of it. here goes the log of "sidekiq.log"
2013-11-06T04:03:55Z 2747 TID--9ygdxi INFO: Booting Sidekiq 2.14.0 using redis://localhost:6379 with options {:namespace=>"resque:gitlab"}
2013-11-06T04:03:55Z 2747 TID--9ygdxi INFO: Running in ruby 2.0.0p247 (2013-06-27 revision 41674) [i686-linux]
2013-11-06T04:03:55Z 2747 TID--9ygdxi INFO: See LICENSE and the LGPL-3.0 for licensing details.
2013-11-06T04:13:08Z 2747 TID--9rl0h4 Sidekiq::Extensions::DelayedMailer JID-6ba7a5bad5ed737cf49d9ad9 INFO: start
2013-11-06T04:13:08Z 2747 TID--9rl0h4 Sidekiq::Extensions::DelayedMailer JID-6ba7a5bad5ed737cf49d9ad9 INFO: done: 0.339 sec
2013-11-06T05:09:09Z 2747 TID--9rl0h4 Sidekiq::Extensions::DelayedMailer JID-312cc90acce9945981cc9cc0 INFO: start
2013-11-06T05:09:09Z 2747 TID--9rl0h4 Sidekiq::Extensions::DelayedMailer JID-312cc90acce9945981cc9cc0 INFO: done: 0.114 sec
Let me know how to come over this issue? do I need to run any other services?
I did it by myself, I found what was wrong. I reinstalled postfix and selected internet site its working fine now. In previous install I selected some local option.