Is there an easy method to see the elapsed time of a sidekiq job?
The log output?
ScheduledWorker JID-072f6c37f240a92ca3c07914 INFO: start
ScheduledWorker JID-072f6c37f240a92ca3c07914 INFO: done: 0.003 sec
Related
Im running sonar in Jenkins job. The analysis stage end successfully, but after that, the job get stuck, there is nothing in the log, but after a few min I get out off memory error and the job fails.
my sonar property file:
sonar.language=javascript
# sources
sonar.sources=src
sonar.exclusions=**/node_modules/**
# tests
sonar.tests=src
sonar.test.inclusions=**/*.test.js
# tests reports
sonar.testExecutionReportPaths=reports/test-reporter.xml
sonar.javascript.lcov.reportPaths=coverage/lcov.info
sonar.verbose=true
the log:
13:31:10 13:31:10.683 INFO: Analysis report generated in 522ms, dir size=4 MB
13:31:12 13:31:12.797 INFO: Analysis report compressed in 2114ms, zip size=2 MB
13:31:12 13:31:12.797 INFO: Analysis report generated in /my_reports_loc
13:31:12 13:31:12.797 DEBUG: Upload report
13:31:12 13:31:12.955 DEBUG: POST 200 http://my-sonar/api/ce/submit?projectKeymyProjt&projectName=projectNamet | time=157ms
13:31:12 13:31:12.958 INFO: Analysis report uploaded in 161ms
13:31:12 13:31:12.959 DEBUG: Report metadata written to /my_reports_loc
13:31:12 13:31:12.959 INFO: ANALYSIS SUCCESSFUL, you can browse http://my-sonar/dashboard?id=my-project
13:31:12 13:31:12.959 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
13:31:12 13:31:12.959 INFO: More about the report processing at http://my-sonar/api/ce/task?id=my_id
13:31:12 13:31:12.964 DEBUG: eslint-bridge server will shutdown
13:31:13 13:31:13.208 DEBUG: stylelint-bridge server will shutdown
13:31:13 13:31:13.209 INFO: Analysis total time: 42.940 s
13:31:13 13:31:13.230 INFO: ------------------------------------------------------------------------
13:31:13 13:31:13.230 INFO: EXECUTION SUCCESS
13:31:13 13:31:13.230 INFO: ------------------------------------------------------------------------
13:31:13 13:31:13.230 INFO: Total time: 44.469s
13:31:13 13:31:13.369 INFO: Final Memory: 43M/1106M
13:31:13 13:31:13.369 INFO: ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] //
[Pipeline] }
[Pipeline] //
13:33:47 java.lang.OutOfMemoryError: GC overhead limit exceeded
Finished: FAILURE
You need to increase the Heap Size.
The error about GC overhead implies that Jenkins is thrashing in Garbage Collection. This means it's probably spending more time doing Garbage Collection than doing useful work. This situation normally comes about when the heap is too small for the application.
This post will help you in case if you need to know, how to increase help size for jenkins.
I am struggling to get Heroku to use Sidekiq as the job handler in production.
Locally - it appears like Sidekiq is taking the email jobs, as the logs below show from Sidekiq's stdout:
2018-09-15T05:10:05.164Z 5256 TID-ovryxfcwc ActionMailer::DeliveryJob JID-156e566ba5bc0136ef18ee45 INFO: start
2018-09-15T05:10:06.838Z 5256 TID-ovryxfcwc ActionMailer::DeliveryJob JID-156e566ba5bc0136ef18ee45 INFO: done: 1.674 sec
2018-09-15T05:10:06.839Z 5256 TID-ovryxfcwc ActionMailer::DeliveryJob JID-c97af1298abad5b8caebfcd4 INFO: start
2018-09-15T05:10:06.935Z 5256 TID-ovryxfcwc ActionMailer::DeliveryJob JID-c97af1298abad5b8caebfcd4 INFO: done: 0.096 sec
2018-09-15T05:10:06.937Z 5256 TID-ovryxfcwc ActionMailer::DeliveryJob JID-0cca4280f85cd9f41325a2af INFO: start
2018-09-15T05:10:07.035Z 5256 TID-ovryxfcwc ActionMailer::DeliveryJob JID-0cca4280f85cd9f41325a2af INFO: done: 0.098 sec
However, in production in Heroku, the logs show that all mail taks are performed by the Web Dyno. I would assume that the Worker Dyno should be processing the ActionMailer job, but never appears to get to sidekiq.
Under config/initializers/active_job.rb I have:
Rails.application.config.active_job.queue_adapter = :sidekiq
Where am I going wrong getting my application to work in production with sidekiq the way it does in test?
It was me being stupid and out of practice - of course the worker is running sidekiq - and it has to call the web worker to actually do the job.
Confirmed by looking at the stats by enabling the sidekiq route to check the jobs were run.
A bazel binary that I am building completes unsuccessfully during the analysis phase. What flags and tools can I use to debug why it fails during analysis.
Currently, clean builds return the following output
ERROR: build interrupted
INFO: Elapsed time: 57.819 s
FAILED: Build did NOT complete successfully (133 packages loaded)
If I retry building after failed completion, I receive the following output
ERROR: build interrupted
INFO: Elapsed time: 55.514 s
FAILED: Build did NOT complete successfully (68 packages loaded)
What flags can I use to identify
what packages are being loaded
what package the build is being interrupted on
whether the interruption is coming from a timeout or an external process.
Essentially, something similar to --verbose_failures but for the analysis phase rather than the execution phrase.
So far I have ran my build through the build profiler, and have not been able to glean any insight. Here is the output of my build:
WARNING: This information is intended for consumption by Blaze developers only, and may change at any time. Script against it at your own risk
INFO: Loading /<>/result
INFO: bazel profile for <> at Mon Jun 04 00:10:11 GMT 2018, build ID: <>, 49405 record(s)
INFO: Aggregating task statistics
=== PHASE SUMMARY INFORMATION ===
Total launch phase time 9.00 ms 0.02%
Total init phase time 91.0 ms 0.16%
Total loading phase time 1.345 s 2.30%
Total analysis phase time 57.063 s 97.53%
Total run time 58.508 s 100.00%
=== INIT PHASE INFORMATION ===
Total init phase time 91.0 ms
Total time (across all threads) spent on:
Type Total Count Average
=== LOADING PHASE INFORMATION ===
Total loading phase time 1.345 s
Total time (across all threads) spent on:
Type Total Count Average
CREATE_PACKAGE 0.67% 9 3.55 ms
VFS_STAT 0.69% 605 0.05 ms
VFS_DIR 0.96% 255 0.18 ms
VFS_OPEN 2.02% 8 12.1 ms
VFS_READ 0.00% 5 0.01 ms
VFS_GLOB 23.74% 1220 0.93 ms
SKYFRAME_EVAL 24.44% 3 389 ms
SKYFUNCTION 36.95% 8443 0.21 ms
SKYLARK_LEXER 0.19% 31 0.29 ms
SKYLARK_PARSER 0.68% 31 1.04 ms
SKYLARK_USER_FN 0.03% 5 0.27 ms
SKYLARK_BUILTIN_FN 5.91% 349 0.81 ms
=== ANALYSIS PHASE INFORMATION ===
Total analysis phase time 57.063 s
Total time (across all threads) spent on:
Type Total Count Average
CREATE_PACKAGE 0.30% 138 3.96 ms
VFS_STAT 0.05% 2381 0.03 ms
VFS_DIR 0.19% 1020 0.35 ms
VFS_OPEN 0.04% 128 0.61 ms
VFS_READ 0.00% 128 0.01 ms
VFS_GLOB 0.92% 3763 0.45 ms
SKYFRAME_EVAL 31.13% 1 57.037 s
SKYFUNCTION 65.21% 32328 3.70 ms
SKYLARK_LEXER 0.01% 147 0.10 ms
SKYLARK_PARSER 0.03% 147 0.39 ms
SKYLARK_USER_FN 0.20% 343 1.08 ms
As far as my command, I am running
bazel build src:MY_TARGET --embed_label MY_LABEL --stamp --show_loading_progress
Use the --host_jvm_debug startup flag to debug Bazel itself during a build.
From https://bazel.build/contributing.html:
Debugging Bazel
Start creating a debug configuration for both C++ and
Java in your .bazelrc with the following:
build:debug -c dbg
build:debug --javacopt="-g"
build:debug --copt="-g"
build:debug --strip="never"
Then you can rebuild Bazel with bazel build --config debug //src:bazel and use your favorite debugger to start debugging.
For debugging the C++ client you can just run it from gdb or lldb as
you normally would. But if you want to debug the Java code, you must
attach to the server using the following:
Run Bazel with debugging option --host_jvm_debug before the command (e.g., bazel --batch --host_jvm_debug build //src:bazel).
Attach a debugger to the port 5005. With jdb for instance, run jdb -attach localhost:5005. From within Eclipse, use the remote
Java application launch configuration.
Our IntelliJ plugin has built-in debugging support
When I look to /sidekiq, I can see in the section called Retries that there's a job that has not been processed - failed.
Queue default
Job Order.order_report_for_manufacturers
Arguments
JID d25956cdd486335ecaf8a186
Created At about 5 hours ago
Enqueued about an hour ago
Retry Count 10
Last Retry about an hour ago
Next Retry about 2 hours from now
It's confusing for me, because when I search in my project the method order_report_for_manufacturers, the editor (TextMate) doesn't find anything. Then I though that maybe this method might be used in a CRON job, so I logged in to my Ubuntu server, run crontab -l but there's nothing (no rake task) that would use a method with such a name.
I tried also restart the server (and Sidekiq), but in a while this job (with this failed method) is there again.
How do I find from where is the method called? (it's not in the Order model).
Thank you.
EDIT:
Sidekiq log:
2016-05-15T14:23:40.910Z 25380 TID-163go0 Sidekiq::Extensions::DelayedClass JID-d25956cdd486335ecaf8a186 INFO: fail: 0.006 sec
2016-05-15T14:23:40.910Z 25380 TID-163go0 WARN: {"class"=>"Sidekiq::Extensions::DelayedClass", "args"=>["---\n- !ruby/class 'Order'\n- :order_report_for_manufacturers\n- []\n"], "retry"=>true, "queue"=>"default", "jid"=>"d25956cdd486335ecaf8a186", "created_at"=>1463316907.739199, "enqueued_at"=>1463322220.9033682, "error_message"=>"undefined method `order_report_for_manufacturers' for #<Class:0x000000036eb740>", "error_class"=>"NoMethodError", "failed_at"=>1463316955.7728505, "retry_count"=>8, "retried_at"=>1463322220.9097943}
2016-05-15T14:23:40.911Z 25380 TID-163go0 WARN: NoMethodError: undefined method `order_report_for_manufacturers' for #<Class:0x000000036eb740>
2016-05-15T14:23:40.911Z 25380 TID-163go0 WARN: /home/deployer/apps/myapp-production/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.6/lib/active_record/dynamic_matchers.rb:26:in `method_missing'
EDIT2:
I am using Sidekiq as an Upstart service (Ubuntu server). I am just running the command ps aux | grep '[s]idekiq', I get this output:
deployer 6647 0.4 15.8 1162920 611080 ? Ssl May03 73:59 sidekiq 4.1.1 myapp-production [0 of 25 busy]
deployer 6659 0.4 18.2 1132336 702620 ? Ssl May03 75:06 sidekiq 4.1.1 myapp-production [0 of 25 busy]
deployer 25380 0.4 8.5 936444 329260 ? Ssl May14 5:14 sidekiq 4.1.1 myapp-production [0 of 25 busy]
I don't know why Sidekiq is there three times; I know that I restarted sidekiq yesterday (sudo stop sidekiq index=0 and sudo start sidekiq index=0) - that's the one on the third line.
What are the first two rows I have no idea. Redis is running on a separated server and communicate with this server (where is the application + Sidekiq).
I need something sending emails on demand.
Each user in my app should be able to set a date and hour when he will receive an email periodically, reminding that she or him has to take medicines.
Is it possible to handle this with a custom daemon? I think so, but its my first time with this stuff and I need your opinion to be sure before starting.
http://railscasts.com/episodes/129-custom-daemon
Thanks a lot, guys.
Edit:
medicine_worker.rb:
class MedicineWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(name, count)
puts 'Doing hard work'
end
end
redis.rb:
# bundle exec sidekiq
$redis = Redis.new(:host => 'localhost', :port => 6379)
bundle exec sidekiq:
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: Running in ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: See LICENSE and the LGPL-3.0 for licensing details.
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
2014-08-26T14:57:17.321Z 1754 TID-znnu9k INFO: Starting processing, hit Ctrl-C to stop
2014-08-26T14:57:17.374Z 1754 TID-zvhqi8 MedicineWorker JID-d3b450f5fd4b926ea70f1c13 INFO: start
Doing hard work
2014-08-26T14:57:17.374Z 1754 TID-zvgu8k MedicineWorker JID-f4958c2fb72aabd91d53b02b INFO: start
Doing hard work
2014-08-26T14:57:17.375Z 1754 TID-zvgbmk MedicineWorker JID-104ee301c6e5a034f42ed06d INFO: start
Doing hard work2014-08-26T14:57:17.375Z 1754 TID-zvfigk MedicineWorker JID-5698a08a24a3def961908dc1 INFO: start
Doing hard work2014-08-26T14:57:17.376Z 1754 TID-zt4ps4 MedicineWorker JID-36d06312e1fdfe8cc439ddbd INFO: start
Doing hard work
2014-08-26T14:57:17.377Z 1754 TID-zvgu8k MedicineWorker JID-f4958c2fb72aabd91d53b02b INFO: done: 0.003 sec
2014-08-26T14:57:17.378Z 1754 TID-zt3qkc MedicineWorker JID-b25e0e7be05580e358c6744a INFO: start
Doing hard work
2014-08-26T14:57:17.379Z 1754 TID-zvhqi8 MedicineWorker JID-d3b450f5fd4b926ea70f1c13 INFO: done: 0.005 sec
2014-08-26T14:57:17.380Z 1754 TID-zvgbmk MedicineWorker JID-104ee301c6e5a034f42ed06d INFO: done: 0.005 sec
2014-08-26T14:57:17.380Z 1754 TID-zvgqac MedicineWorker JID-38a185bc67c673062c475faf INFO: start
Doing hard work2014-08-26T14:57:17.380Z 1754 TID-zvhbbk MedicineWorker JID-d2058fcebd88210c406b49e4 INFO: start
Doing hard work2014-08-26T14:57:17.383Z 1754 TID-zt4ps4 MedicineWorker JID-36d06312e1fdfe8cc439ddbd INFO: done: 0.007 sec
2014-08-26T14:57:17.384Z 1754 TID-zvfigk MedicineWorker JID-5698a08a24a3def961908dc1 INFO: done: 0.009 sec
2014-08-26T14:57:17.385Z 1754 TID-zvf9k0 MedicineWorker JID-cc5e3ee8ddeead76d0c1fc5b INFO: start
Doing hard work2014-08-26T14:57:17.385Z 1754 TID-zvg6c0 MedicineWorker JID-d83dc9cb10b69c1680305417 INFO: start
Doing hard work
2014-08-26T14:57:17.385Z 1754 TID-zt4h14 MedicineWorker JID-9d13fbed1bb0042eea452ec5 INFO: start
Doing hard work
2014-08-26T14:57:17.388Z 1754 TID-zt5uv0 MedicineWorker JID-877e16ded3e18828e3396719 INFO: start
Doing hard work2014-08-26T14:57:17.389Z 1754 TID-zt3qkc MedicineWorker JID-b25e0e7be05580e358c6744a INFO: done: 0.011 sec
2014-08-26T14:57:17.389Z 1754 TID-10h4kzs MedicineWorker JID-6739ea8a340b44c06da9dabf INFO: start
Doing hard work
2014-08-26T14:57:17.390Z 1754 TID-10h4pi0 MedicineWorker JID-dcb9282efabb6c9948014834 INFO: start
Doing hard work
2014-08-26T14:57:17.391Z 1754 TID-zt1wnk MedicineWorker JID-c6fa12391faff774ffda3f5b INFO: start
Doing hard work
2014-08-26T14:57:17.392Z 1754 TID-10h6pco MedicineWorker JID-44d7747775163b4f8bf40af8 INFO: start
Doing hard work
2014-08-26T14:57:17.393Z 1754 TID-10h6u2o MedicineWorker JID-212baee3c6f3526a3ff24d9c INFO: start
Doing hard work
2014-08-26T14:57:17.394Z 1754 TID-10h5cvo MedicineWorker JID-27f2009864a0e63ab4cab2d3 INFO: start
Doing hard work2014-08-26T14:57:17.446Z 1754 T
.......
.......
.......
You better use a Background Queue for this preferably one with a scheduling option.
Sidekiq is pretty good for what you need.
https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs