We have two servers A and B
When we enqueue Active Jobs, Server A shows that a job has been created a minute ago in the Sidekiq web interface, but Server B shows that the job has been created 3 hours ago.
The same is true for
set(wait: 10.hours).perform_later(object_id)
=> Server A in Sidekiq Web UI: In 10 hours
=> Server B in Sidekiq Web UI: In 7 hours
Server A and B both return the same date and /etc/timezone
$ date
Mo 9. Aug 15:32:14 CEST 2021
$ cat /etc/timezone
Europe/Berlin
We have a test job to reproduce the problem:
4 class TimeTestJob < ApplicationJob
5 queue_as :default
6
7 def perform(time)
8 puts "Time in Sidekiq: #{Time.now.to_s}"
9 puts "DateTime in Sidekiq: #{DateTime.now.to_s}"
10 puts "time args: #{time.to_s}"
11 ...
16 end
17
18 end
The output is identical on both servers
TimeTestJob.perform_now(Time.now.to_s)
Performing TimeTestJob (Job ID: b542c55d-7957-48db-83b8-7d0620accb6f) from Sidekiq(default) enqueued at with arguments: "2021-08-09 15:32:14 +0200"
Time in Sidekiq: 2021-08-09 15:32:14 +0200
DateTime in Sidekiq: 2021-08-09T15:32:14+02:00
time args: 2021-08-09 15:32:14 +0200
Enqueued TimeTestJob (Job ID: d3d9b7e7-2d97-4700-9c93-9e61f9fbeae5) to Sidekiq(default) at 2021-08-09 23:33:14 UTC with arguments: "2021-08-09 15:32:14 +0200"
...
Any ideas what could be the issue for the time offset on Server B?
Related
I cannot figure this out. Hope I can get some help
I have a hobby tier in Heroku running django. To this I would like to attach a Redis service. However, I would like to use the recent rejson (redislabs/rejson:latest) docker image (instead of redistogo or heroku-redis)because it has json support. This works great on my local env. I was able to push the docker image into the container registry and actually start the redis server
2021-07-23T00:14:48.576294+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.576 # Current maximum open files is 10000. maxclients has been reduced to 9968 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
2021-07-23T00:14:48.577054+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Running mode=standalone, port=6379.
2021-07-23T00:14:48.577124+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # Server initialized
2021-07-23T00:14:48.577184+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2021-07-23T00:14:48.577671+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # <ReJSON> JSON data type for Redis v1.0.7 [encver 0]
2021-07-23T00:14:48.577789+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Module 'ReJSON' loaded from /usr/lib/redis/modules/rejson.so
2021-07-23T00:14:48.578232+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.578 * Ready to accept connections
Unfortunately, django is unable to connect to it
ConnectionError: Error -2 connecting to redis://localhost:6379. Name or service not known
There are no ENV variable exposed, so am unable to set any (I mean I can set them, but doubt if they will be relevant)
I did experiment with installing redistogo add on and am able to connect to it (this required setting the connection based on the REDIS_URL env variable that gets exposed when redistogo is added)
At my wits end...any help appreciated...I guess the question really boils do to:
I am running the gitlab with docker, but it always exits after a period of time
==> /var/log/gitlab/redis/current <==
2019-06-21_18:00:08.72435 459:signal-handler (1561140008) Received SIGTERM scheduling shutdown...
2019-06-21_18:00:08.81864 459:M 21 Jun 18:00:08.817 # User requested shutdown...
2019-06-21_18:00:08.81866 459:M 21 Jun 18:00:08.817 * Saving the final RDB snapshot before exiting.
2019-06-21_18:00:08.83736 459:M 21 Jun 18:00:08.837 * DB saved on disk
2019-06-21_18:00:08.83741 459:M 21 Jun 18:00:08.837 * Removing the pid file.
2019-06-21_18:00:08.83817 459:M 21 Jun 18:00:08.838 * Removing the unix socket file.
2019-06-21_18:00:08.83935 459:M 21 Jun 18:00:08.839 # Redis is now ready to exit, bye bye...
ok: down: redis-exporter: 0s, normally up
==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
2019-06-21_18:00:09.57615 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Shutting down
2019-06-21_18:00:09.57625 2019-06-21T18:00:09.576Z 807 TID-ovivo05i7 INFO: Scheduler exiting...
2019-06-21_18:00:09.57655 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Terminating quiet workers
This was reported in gitlab-org/omnibus-gitlab issue 4137: "runsv send SIGTERM to redis in docker version"
runsv sends SIGTERM to redis every 60 secs
gitlab-org/omnibus-gitlab issue 1611 suggests a docker restart first.
But the general issue is not conclusively resolved yet.
I am new to chef and installed push-jobs feature on chef-server to run commands from workstation or Jenkins.
When I run the command from Jenkins knife job start 'chef-client' <node>, it is not displaying complete log like when I run "chef-client" on a node.
How to get the complete log in Jenkins console ?? Please help me with this.
Below is the log which i am getting,
C:\chef\chef-repo>knife job start 'chef-client' hybris-server
Started. Job ID: cdaf88e535736bbf816b1a46dd4c119d
.Running (1/1 in progress) ...
.......................Complete.
command: chef-client
created_at: Mon, 16 Jan 2017 11:48:37 GMT
env:
id: cdaf88e535736bbf816b1a46dd4c119d
nodes:
succeeded: hybris-server
run_timeout: 3600
status: complete
updated_at: Mon, 16 Jan 2017 11:49:25 GMT
Regards,
Umesha
When I look to /sidekiq, I can see in the section called Retries that there's a job that has not been processed - failed.
Queue default
Job Order.order_report_for_manufacturers
Arguments
JID d25956cdd486335ecaf8a186
Created At about 5 hours ago
Enqueued about an hour ago
Retry Count 10
Last Retry about an hour ago
Next Retry about 2 hours from now
It's confusing for me, because when I search in my project the method order_report_for_manufacturers, the editor (TextMate) doesn't find anything. Then I though that maybe this method might be used in a CRON job, so I logged in to my Ubuntu server, run crontab -l but there's nothing (no rake task) that would use a method with such a name.
I tried also restart the server (and Sidekiq), but in a while this job (with this failed method) is there again.
How do I find from where is the method called? (it's not in the Order model).
Thank you.
EDIT:
Sidekiq log:
2016-05-15T14:23:40.910Z 25380 TID-163go0 Sidekiq::Extensions::DelayedClass JID-d25956cdd486335ecaf8a186 INFO: fail: 0.006 sec
2016-05-15T14:23:40.910Z 25380 TID-163go0 WARN: {"class"=>"Sidekiq::Extensions::DelayedClass", "args"=>["---\n- !ruby/class 'Order'\n- :order_report_for_manufacturers\n- []\n"], "retry"=>true, "queue"=>"default", "jid"=>"d25956cdd486335ecaf8a186", "created_at"=>1463316907.739199, "enqueued_at"=>1463322220.9033682, "error_message"=>"undefined method `order_report_for_manufacturers' for #<Class:0x000000036eb740>", "error_class"=>"NoMethodError", "failed_at"=>1463316955.7728505, "retry_count"=>8, "retried_at"=>1463322220.9097943}
2016-05-15T14:23:40.911Z 25380 TID-163go0 WARN: NoMethodError: undefined method `order_report_for_manufacturers' for #<Class:0x000000036eb740>
2016-05-15T14:23:40.911Z 25380 TID-163go0 WARN: /home/deployer/apps/myapp-production/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.6/lib/active_record/dynamic_matchers.rb:26:in `method_missing'
EDIT2:
I am using Sidekiq as an Upstart service (Ubuntu server). I am just running the command ps aux | grep '[s]idekiq', I get this output:
deployer 6647 0.4 15.8 1162920 611080 ? Ssl May03 73:59 sidekiq 4.1.1 myapp-production [0 of 25 busy]
deployer 6659 0.4 18.2 1132336 702620 ? Ssl May03 75:06 sidekiq 4.1.1 myapp-production [0 of 25 busy]
deployer 25380 0.4 8.5 936444 329260 ? Ssl May14 5:14 sidekiq 4.1.1 myapp-production [0 of 25 busy]
I don't know why Sidekiq is there three times; I know that I restarted sidekiq yesterday (sudo stop sidekiq index=0 and sudo start sidekiq index=0) - that's the one on the third line.
What are the first two rows I have no idea. Redis is running on a separated server and communicate with this server (where is the application + Sidekiq).
Dataflow job id: 2016-01-13_16_00_09-15016519893798477319
Pipeline was configured with the following worker/scaling config:
min 2 workers
max 50 workers
However, the job scaled to 55 workers. Why was the max worker value of 50 not honoured?
Jan 14, 2016, 11:00:10 AM
(77f7e53b4884ba02): Autoscaling: Enabled for job 2016-01-13_16_00_09-15016519893798477319 between 1 and 1000000 worker processes.
Jan 14, 2016, 11:00:17 AM
(374d4f69f65e2506): Worker configuration: n1-standard-1 in us-central1-a.
Jan 14, 2016, 11:00:18 AM
(28acda8454e90ad2): Starting 2 workers...
Jan 14, 2016, 11:01:49 AM
(cf611e5d4ce4784d): Autoscaling: Resizing worker pool from 2 to 50.
Jan 14, 2016, 11:06:20 AM
(36c68efd7f1743cf): Autoscaling: Resizing worker pool from 50 to 55.
This turned out to be a bug in our code. We were calling the wrong method. We need to call setMaxNumWorkers, and not setNumWorkers.