Cron rake task not ending after finish work - ruby-on-rails

I have configured rake tasks using whenever gem to run every 5 minute. that rake task take 10 seconds to run in local but in server, It's not ending always it's stay in process list
ps -ef | grep some_rake_task`
shows like following output:
user 22628 22626 0 09:00 ? 00:00:00 /bin/sh -c /bin/bash -l -c 'cd app_direc && rake some_rake_task'
user 22630 22628 0 09:00 ? 00:00:00 /bin/bash -l -c cd app_direc && rake some_rake_task
user 22933 22630 0 09:00 ? 00:00:00 rake some_rake_task
user 22934 22933 0 09:00 ? 00:00:04 ruby /opt/user/app/shared/bundle/ruby/2.1.0/bin/rake some_rake_task
user 25261 25260 0 08:00 ? 00:00:00 /bin/sh -c /bin/bash -l -c 'cd app_direc && rake some_rake_task'
user 25263 25261 0 08:00 ? 00:00:00 /bin/bash -l -c cd app_direc && rake some_rake_task
user 25570 25263 0 08:00 ? 00:00:00 rake some_rake_task
user 25571 25570 0 08:00 ? 00:00:04 ruby /opt/user/app/shared/bundle/ruby/2.1.0/bin/rake some_rake_task
user 26570 26569 0 07:00 ? 00:00:00 /bin/sh -c /bin/bash -l -c 'cd app_direc && rake some_rake_task'
user 26573 26570 0 07:00 ? 00:00:00 /bin/bash -l -c cd app_direc && rake some_rake_task
user 26879 26573 0 07:00 ? 00:00:00 rake some_rake_task
user 26880 26879 0 07:00 ? 00:00:04 ruby /opt/user/app/shared/bundle/ruby/2.1.0/bin/rake some_rake_task
user 30915 30691 0 09:16 pts/2 00:00:00 grep --color=auto some_rake_task
I tried to kill process manually but every task initiated by crontab it's stays alive not ending. Any suggestionsm please?
My rake task code
namespace :some_rake_task do
desc "Send the email with content"
task :send => :environment do
User.find_in_batches do |user|
SendMailer.newsletter_email(user).deliver
end
end
end
But there is only below 1000 users

Crond detaches child processes, so when you kill crond process, spawned tasks will continue their work.
Judging by the output of ps -ef, there is no <defunct> near rake task processes, so i can decide that they are alive and running.
Your rake tasks either enter endless loop or just take much more time than you expect. In case of second it can be caused by slow network connection, for example if you call some external API service, poorly optimized SQL query or algorithm, too large database, something like that.
Advice: Try to fix your rake task or crontab entry, in order to not run rake task if same one is already running.

Related

STDOUT logs not working when using symlink for log file to /proc/1/fd/1 on Kubernetes

I have a cronjob that runs every minute which will redirect output to a log file /var/log/cronjob/cron.log. Since this is running in Kubernetes, I want to redirect the log to STDOUT.
The approach I took was to use a symlink using RUN ln -sf /proc/1/fd/1 /var/log/cronjob/cron.log:
# ls -la /var/log/cronjob/cron.log
lrwxrwxrwx 1 root root 12 Jan 21 19:23 /var/log/cronjob/cron.log -> /proc/1/fd/1
When I run kubectl logs it has no output.
If I (within the container), delete the symlink and create as a normal file, my output as expected appears in the /var/log/cronjob/cron.log file.
# tail -f /var/log/cronjob/cron.log
Running scheduled command: '/usr/bin/php7.3' 'artisan' sync:health_check > '/dev/null' 2>&1
Running scheduled command: ('/usr/bin/php7.3' 'artisan' compute:user_preferences > '/dev/null' 2>&1 ; '/usr/bin/php7.3' 'artisan' schedule:finish "framework/schedule-9019c9dc22ad7439efd038277fe8f370f56958e7") > '/dev/null' 2>&1 &
How can I get the my log via symlink write to STDOUT?
I have tried other things such as:
Use /dev/stdout for the symlink
Tail the /var/log/cronjob/cron.log file within the entrypoint
Edit: More information about files/scripts:
crontab:
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
* * * * * /usr/local/bin/schedule-run.sh
# An empty line is required at the end of this file for a valid cron file
/usr/local/bin/schedule-run.sh:
#!/bin/bash
# Source container environment variables
source /tmp/export
# Run Laravel scheduler
php /var/www/api/artisan schedule:run >> /var/log/cronjob/cron.log 2>&1
Edit #2:
Currently my CMD looks like this which spawns multiple child processes:
CMD export >> /tmp/export && crontab /etc/cron.d/realty-cron && cron && tail -f /var/log/cronjob/cron.log
root#workspace-dev-condos-ca-765dc6686-h8vdl:/var/www/api# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 21:55 ? 00:00:00 /bin/sh -c export >> /tmp/export && crontab /etc/cron.d/realty-cron && cron && tail -f /var/log/cronjob/cron.log
root 8 1 0 21:55 ? 00:00:00 cron
root 9 1 0 21:55 ? 00:00:00 tail -f /var/log/cronjob/cron.log
root 170 1 0 21:59 ? 00:00:00 ssh-agent -s
root 233 0 0 22:00 pts/0 00:00:00 bash
root 249 1 0 22:00 ? 00:00:00 ssh-agent -s
root 1277 233 0 22:26 pts/0 00:00:00 ps -ef
I'm not sure if that is relevant but through trial and error testing, I noticed that sometimes echo "test1" >> /proc/1/fd/1 or echo "test2" >> /proc/1/fd/2 will output to stdout (kubectl logs) but not both at the same time. I feel like the child processes are related but don't know why.

monit: failed to restart

I am using monit for sidekiq
while I am running the monit log file, it is showing the error.
[EDT Jun 18 09:50:11] error : 'sidekiq_site' process is not running
[EDT Jun 18 09:50:11] info : 'sidekiq_site' trying to restart
[EDT Jun 18 09:50:11] info : 'sidekiq_site' start: /bin/bash
[EDT Jun 18 09:51:41] error : 'sidekiq_site' failed to start
/etc/monit/conf.d/sidekiq.conf
check process sidekiq_site
with pidfile /var/www/project/shared/pids/sidekiq.pid
start program = "bash -c 'cd /var/www/project/current ; RAILS_ENV=production bundle exec sidekiq --index 0 --pidfile /var/www/project/shared/pids/sidekiq.pid --environment production --logfile /var/www/project/shared/log/sidekiq.log --daemon'" as uid root and gid root with timeout 90 seconds
stop program = "bash -c 'if [ -d /var/www/project/current ] && [ -f /var/www/project/shared/pids/sidekiq.pid ] && kill -0 `cat /var/www/project/shared/pids/sidekiq.pid`> /dev/null 2>&1; then cd /var/www/project/current && bundle exec sidekiqctl stop /var/www/project/shared/pids/sidekiq.pid 1 ; else echo 'Sidekiq is not running'; fi'" as uid root and gid root
if totalmem is greater than 200 MB for 2 cycles then restart # eating up memory?
group site_sidekiq
/etc/monit/monitrc
set daemon 30
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set eventqueue
basedir /var/lib/monit/events
slots 100
set httpd port 2812
allow admin:""
set httpd port 2812 and
use address xx.xxx.xx.xx
allow xx.xx.xx.xx
check system trrm_server
if loadavg(5min) > 2 for 2 cycles then alert
if memory > 75% for 2 cycles then alert
if cpu(user) > 75% for 2 cycles then alert
include /etc/monit/conf.d/*
When running a start/stop event in monit there is no path variable set, therefore all programs must have absolute paths, even your call to bash.
No environment variables are used by Monit

Duplicate process in cron job using Whenever gem for Rails

Using Rails 3.2.21, whenever gem. This is the list of my crontab:
Begin Whenever generated tasks for: abc
0 * * * * /bin/bash -l -c 'cd /home/deployer/abc/releases/20141201171336 &&
RAILS_ENV=production bundle exec rake backup:perform --silent'
Here's the output when the scheduled job is run:
deployer#localhost:~$ ps aux | grep rake
deployer 25593 0.0 0.0 4448 764 ? Ss 12:00 0:00 /bin/sh -c /bin/bash -l -c
'cd /home/deployer/abc/releases/20141201171336 && RAILS_ENV=production bundle exec rake
backup:perform --silent'
deployer 25594 0.0 0.1 12436 3040 ? S 12:00 0:00 /bin/bash -l -c cd
/home/deployer/abc/releases/20141201171336 && RAILS_ENV=production bundle exec rake
backup:perform --silent
deployer 25631 69.2 4.4 409680 90072 ? Sl 12:00 0:06 ruby /home/deployer/abc/
shared/bundle/ruby/1.9.1/bin/rake backup:perform --silent
deployer 25704 0.0 0.0 11720 2012 pts/0 S+ 12:00 0:00 grep --color=auto rake
Notice the the top 2 processes are actually similar processes. Are they running 2 same jobs concurrently? How do I prevent that?
deployer 25593 0.0 0.0 4448 764 ? Ss 12:00 0:00 /bin/sh -c /bin/bash …
deployer 25594 0.0 0.1 12436 3040 ? S 12:00 0:00 /bin/bash …
Notice the the top 2 processes are actually similar processes. Are they running 2 same jobs concurrently?
No, they aren't. The first is a /bin/sh that started the second, the crontab command /bin/bash …. Most probably /bin/sh is just waiting for termination of /bin/bash and not running again before /bin/bash … has finished execution; you can verify this with e. g. strace -p 25593.
Check your scheduled.rb for a duplicate entry, if you find then remove and deploy.
If there is no duplicate entry in scheduled.rb then you need to remove/comment it from cron tab.
To delete or comment jobs in cron take a look at https://help.1and1.com/hosting-c37630/scripts-and-programming-languages-c85099/cron-jobs-c37727/delete-a-cron-job-a757264.html OR http://www.esrl.noaa.gov/gmd/dv/hats/cats/stations/qnxman/crontab.html

Cron job not running using Capistrano & whenever

I'm stumped on this one. I am using capistrano and whenver gems to manage my builds to prod. Cron is being setup correctly on prod. When I look at 'crontab -e' I see...
# Begin Whenever generated tasks for: nso
10 * * * * /bin/bash -l -c 'cd /home/user/nso/releases/20140130161552 && RAILS_ENV=production bundle exec rake send_reminder_emails --silent >> /var/log/syslog 2>&1'
...this looks correct. In /var/log/syslog I see...
Feb 3 16:10:01 vweb-nso CRON[32186]: (user) CMD (/bin/bash -l -c 'cd /home/user/nso/releases/20140130161552 && RAILS_ENV=production bundle exec rake send_reminder_emails --silent >> /var/log/syslog 2>&1')
Feb 3 16:10:01 vweb-nso postfix/pickup[31636]: 8A67B805C7: uid=1001 from=<user>
Feb 3 16:10:01 vweb-nso postfix/cleanup[32191]: 8A67B805C7: message-id=<20140203211001.8A67B805C7#vweb-nso>
Feb 3 16:10:01 vweb-nso postfix/qmgr[31637]: 8A67B805C7: from=<user#abtech.edu>, size=712, nrcpt=1 (queue active)
Feb 3 16:10:01 vweb-nso postfix/local[32193]: 8A67B805C7: to=<user#abtech.edu>, orig_to=<user>, relay=local, delay=0.03, delays=0.02/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Feb 3 16:10:01 vweb-nso postfix/qmgr[31637]: 8A67B805C7: removed
Everything looks OK there..no?
Addtionally I can manually run the command. At the command prompt if I do.
/bin/bash -l -c 'cd /home/user/nso/releases/20140130161552 && RAILS_ENV=production bundle exec rake send_reminder_emails --silent >> /var/log/syslog 2>&1'
...I get my reminder emails as I should. What am I missing?
So this ended up being my issue...
Rails cron whenever, bundle: command not found
Once I added env :PATH, ENV['PATH'] to the top of config/schedule.rb everything worked as advertised!

Managing Resque workers with Monit on RBenv setup

I'm trying to set up Monit to manage Resque workers, but it fails to start saying /home/deployer/.rbenv/shims/bundle: line 4: exec: rbenv: not found
I've checked that it is running commands as deployer user and if I copy and paste the command directly via SSH everything works fine. Below is my Monit configuration. Thanks!
check process resque_worker_1
with pidfile CURRENT_PATH/tmp/pids/resque_worker_1.pid
start program = "/usr/bin/env HOME=/home/deployer RACK_ENV=production PATH=/home/deployer/.rbenv/shims:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c 'cd CURRENT_PATH; bundle exec rake environment resque:work RAILS_ENV=production QUEUE=high,normal,low VERBOSE=1 PIDFILE=CURRENT_PATH/tmp/pids/resque_worker_1.pid >> CURRENT_PATH/log/resque_worker_.log 2>&1'"
as uid deployer and gid admin
stop program = "/bin/sh -c 'cd CURRENT_PATH && kill -9 $(cat tmp/pids/resque_worker_1.pid) && rm -f tmp/pids/resque_worker_1.pid; exit 0;'"
as uid deployer and gid admin
if totalmem is greater than 300 MB for 10 cycles then restart # eating up memory?
group resque_workers
I'm not sure if this helps, but in my monitrc start line, I have to first su to the user I want to run under. I haven't tried to use the uid and gid flags to know if that works well, so this might be a goose-chase of an answer.
I remember having the same issue as you though... everything worked from the command line, but not when monit would do its thing.
For example, in my monitrc, I am monitoring arsendmail with the following:
# arsendmail_rails3
# daemon that watches and sends mail from the rails app
check process ar_sendmail with pidfile /var/www/rak/log/ar_sendmail.pid
start program "/bin/su - mike && /bin/bash -c 'cd /var/www/rak && ar_sendmail_rails3 -b1000 -d -e production'"
stop program "/bin/ps -ef | /bin/grep ar_sendmail_rails3 | /bin/grep -v grep | /usr/bin/awk '{ /usr/bin/print $2}' | /usr/bin/xargs /bin/kill -9"
I saw that the topic was created in 2012 but I had a similar problem and this thread is top ranked by google.
The problem is that monit launch commands with a restricted env (env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/sh to simulate).
To use monit with rbenv you must specify the correct path before your bundle exec command.
PATH=/home/[USER]/.rbenv/bin:/home/[USER]/.rbenv/shims:$PATH bundle exec ...
Example with unicorn:
check process unicorn_dev with pidfile /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid
group dev
start program = "/bin/bash -c 'cd /home/wizville/app/dev.wizville.fr/current && PATH=/home/wizville/.rbenv/bin:/home/wizville/.rbenv/shims:$PATH bundle exec unicorn -c config/unicorn.rb -D'" as uid "wizville"
stop program = "/bin/bash -c 'kill -s QUIT `cat /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid`'"
depends on mysql
This worked for me.
check process app_resque_worker with pidfile <%= resque_pid%>
start program = "/usr/bin/env HOME=/home/subcaster RACK_ENV=production PATH=/home/subcaster/.rvm/rubies/ruby-2.0.0-p247/bin/ruby:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c \'cd <%= current_path %>; bundle exec rake environment resque:work RAILS_ENV=production BACKGROUND=yes QUEUE=* PIDFILE=<%= resque_pid %>\'"
stop program = "kill -9 cat <%= resque_pid%> && rm -f <%= resque_pid%>"
if totalmem is greater than 2000 MB for 10 cycles then restart

Resources