Struggling with cron jobs. Ubuntu 11.10 on the server.
Until recently had whenever cron jobs running successfully several times a day; then due to another problem I had to remove RVM from the server and go back to ruby 1.9.3 installed without RVM (I'm sure this is something to do with it)
There is no .rvmrc file in my app
Now, the cron jobs are somehow failing as I can see from syslog:
Jun 30 08:03:01 ip-10-251-30-96 CRON[18706]: (ubuntu) CMD (/bin/bash -l -c 'cd /var/www/my_app/app/releases/201300629090954 && script/rails runner -e production '\''User.remind_non_confirmed_users'\''')
Jun 30 08:03:01 ip-10-251-30-96 CRON[18705]: (CRON) error (grandchild #18706 failed with exit status 127)
Jun 30 08:03:01 ip-10-251-30-96 CRON[18705]: (CRON) info (No MTA installed, discarding output)
If I run that command manually (with env - /bin/bash -l -c '...' ) it runs fine..
I'm going to add "set :output, 'tmp/whenever.log'" to whenever to see what is going on, but I suspect it is an issue with the ruby version / path or something.
Any idea how I could diagnose / fix this properly??
this is my cron/whenever job:
3 8 * * * /bin/bash -l -c 'cd /var/www/my_app/app/releases/20130629090954 && script/rails runner -e production '\''User.remind_non_confirmed_users'\'''
many thanks
To help diagnose what's going on, I usually capture the cron output into a separate log file. There's probably an error that's just not being recorded anywhere.
#hourly bash -lc 'cd /path/to/app; RAILS_ENV=production bundle exec rake remind_non_confirmed_users' >> /path/to/app/log/tasks.log
Also, I prefer creating rake tasks for cron jobs as opposed to runners. A little easier to invoke via the command line than runners, for me at least.
I'm still not sure what was going on, running Whatever with 'set :output' should have created log files, but it didn't, yet the jobs are still failing (and write permissions were there for the log files).
I got so fed up I redeveloped the solution without using script/runner, in stead have cron just call a URL that then takes care of matters as a delayed job. For our particular situation this has a number of additional benefits, though I know it is not ideal for many.
thanks for the suggestions
Related
Can I invoke "rake jobs:work" automatically after running "rails s" in console?
Currently, after running rails s in cmd I will also run rake jobs:work in the other console, what i want to happen is After running "rails s" the jobs:work will automatically start.
The right way to go about this would be to use a process manager, like Invoker or Foreman. There is ample documentation on the links, but it boils down to the following steps:
Install the software
Create a configuration file where you declare what processes do you intend to run. Both support Procfile style declaration.
Use the command line client to start the process manager.
Based on my personal experience, I highly recommend Invoker, it goes beyond just a process manager, and packs in a few more handy features, like support for .dev local domain.
One you can do is simply:
rails server & rake jobs:work
It'll run rails server as background job, which you can get back to foreground with fg. It can be annoying that you'll get output from both processes mixed.
I'm not sure what are your needs and what you expect but maybe it would be good for you to use screen (or tmux) to run them in parallel and be able to switch between.
You can do your own .screenrc script which will run the server and any other commands when automatically for you.
There is a little problem that if you run the server from it and you close it (ctrl+c) than you'll loose it's screen window. Fortunately there is a solution for that as well (worked-out on the SO as well - you can read more about it here)
So, I use some helper script for that .run_screen (don't forget to chmod +x it):
#!/bin/bash
/bin/bash -i <<<"$*; exec </dev/tty"
Than I have .screenrc_rails file:
#shell -${SHELL}
caption always "%n(%t) %= %{b}#%H[%l] : %{r}%c:%s"
termcapinfo xterm ti#:te#
termcap xterm 'AF=\E[3%dm:AB=\E[4%dm'
terminfo xterm 'AF=\E[3%p1%dm:AB=\E[4%p1%dm'
startup_message off
screen -t server 2 ${HOME}/.run_screen rails s
screen -t spork 3 ${HOME}/.run_screen bundle exec spork
screen -t dev_log 4 ${HOME}/.run_screen tail -f ./log/development.log
screen -t test_log 5 ${HOME}/.run_screen tail -f ./log/test.log
screen -t bash 0
screen -t bash 1
And an alias ( screenr(ails) ) defined at .bash_profile:
alias screenr='screen -c ~/.screenrc_rails'
If you don't know screen than start from ctrl+a, ". ctrl+a, ? will give you some help.
I hope you'll enjoy it.
I've been searching online for days on how to set up my server to automatically rotate the logs for a Rails app my team recently released. I've gotten myself as far as being able to run sudo logrotate -f /etc/logrotate.conf but of course, who wants to do that all the time?
The contents of the config file for the app's log (I want to add more, but don't see a need to when I can't rotate one file yet):
/path/to/app/production.log {
daily
missingok
rotate 7
compress
delaycompress
notifempty
copytruncate
}
I've verified the /etc/logrotate.conf file contains this line:
include /etc/logrotate.d
But this is the part where I'm not too sure where to go. I've found many different approaches at actually automating the process, but none seem to work. For the record, I've verified the server has the anacron command installed, but I don't know how to configure it for any process of my own. Also, root does not have a crontab on the server yet (we haven't needed it), and I'm unsure if that's better to use than /etc/crontab. In the /etc/crontab file, I've added:
15 0 * * * root cd / && run-parts --report /etc/cron.daily
but I've seen other people use
15 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Is the latter a better option? Why? If so, how do I ensure it works? Again, I don't know how to set up anacron for the task at hand.
Finally, here are the previous contents of the /etc/cron.daily/logrotate file:
/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
and after some research, I replaced that with this (which I understand a bit better):
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf
Can someone explain to me what the first config was doing, and which of these two options is better? I'm unsure why I have to force this process just to get it to run. Maybe /etc/crontab doesn't work like I think it does?
Is the latter a better option? Why?
With your cron command, /etc/cron.daily/* will only ever run if the computer is on at midnight (00:15). If you turn it off at night, as some people do, it would never run.
The work around this, and instead run the command when the computer starts later in the day, one can use anacron. This is obviously less useful for servers than desktops.
Of course, you don't want to use both at once, since that would run the jobs twice a day. Therefore, cron, the most brittle mechanism, will yield to anacron by only running the job if anacron is not installed.
This is what Debian and Ubuntu does by default with their test -x /usr/sbin/anacron || crontab job prefixes.
All server distros will come with logrotate correctly set up, so you shouldn't be modifying the crontab, anacrontab, or /etc/cron.daily/logrotate. The only thing you should do is add a file to /etc/logrotate.d.
Try putting this in /etc/crontab file :
-*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1
It works for me.
I've written a Rake script that should run automatically with Crontab. The script runs fine when typed in the command line but fails to run correctly inside the cron.
The script looks like this :
#!/bin/sh
echo `date`
cd /home/mick/myapp/current
rake RAILS_ENV=production mynamespace:myaction
The crontab setting looks like this :
10 0,6,12,18 * * * /home/mick/work/launch.sh >> /home/mick/work/launch.log
After execution, the log file just contains the date but nothing else, and the error I get in the syslog looks like this :
Mar 18 18:10:01 CRON[21773]: (mick) CMD (/home/mick/work/launch.sh >> /home/mick/work/launch.log)
Mar 18 18:10:01 CRON[21772]: (CRON) error (grandchild #21773 failed with exit status 127)
Mar 18 18:10:01 postfix/sendmail[21776]: fatal: open /etc/postfix/main.cf: No such file or directory
Mar 18 18:10:01 CRON[21772]: (mick) MAIL (mailed 1 byte of output; but got status 0x004b, #012)
EDIT :
Thanks to #Holger Just's comment, i've found this link which helped me got a working script.
Here is the updated version of my script
#!/usr/bin/env bash # UPDATED TO GET ACCESS TO 'source'
export PATH=blabla # NOT SURE THIS HELPED AS IT WAS MY FIRST MODIF AND DIDN'T FIX
source /home/mick/.rvm/environments/default # LOADING RVM TO MAKE THINGS WORK
echo `date`
cd /home/mick/myapp/current
rake RAILS_ENV=production mynamespace:myaction
Cron typically has a very restricted $PATH. So you could either set the $PATH on top of your script or use full paths throughout. You can set the path in your script like
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
Of course you probably need to adapt your $PATH. e.g. you could use that of your root user's login shell. You can get that by running echo $PATH.
Also note that if you use RVM, you need to load it in your cron script explicitly as cron doesn't load rvm (or any other shell initialization scripts) by default.
I'm having a rough time executing script/runner with a cron and RVM. I believe the issues lie with the rvm environment not being set before the runner is executed.
currently im throwing the error
/bin/sh: 1.sql: command not found
which is more than i've gotten earlier, so i guess that's good.
I've read this thread Need to set up rvm environment prior to every cron job but im still not really getting it. Part of the problem i think is the error reporting.
this is my runner thus far.
*/1 * * * * * /bin/bash -l -c 'rvm use 1.8.7-p352#2310; cd development/app/my_app2310 && script/runner -e development "Mailer.find_customer"'
as per the above link, i tried making a rvm_cron_runner.
i created a file and placed this in it:
#!/bin/sh
source "/Users/dude/.rvm/scripts/rvm"
exec $1
then i updated my crontab to this.
*/1 * * * * * /bin/bash -l -c '/Users/dude/development/app/my_app2310/rvm_cron_runner; rvm use 1.8.7-p352#2310; cd development/app/my_app2310 && script/runner -e development "Mailer.find_customer"'
This also has made no difference. i get no error. nothing.
Can anyone see what i'm doing incorrectly?
P.S i hope my code formatting worked.
Could you try to place the code you want to run in a separate script, and then use the rvm_cron_runner ?
So place your actions in a file called /path/cron_job
rvm use 1.8.7-p352#2310
cd development/app/my_app2310 && script/runner -e development "Mailer.find_customer"
and then in your crontab write
1 2 * * * /path/rvm_cron_runner /path/cron_job
The differences:
this does not start a separate shell
use the parameter of the rvm_cron_runner
If you would use an .rvmrc file, you could even drop the rvm use ... line, I think.
You don't need to write a second cron runner (following that logic, you might as well write a third cron runner runner). Please keep things simple. All you need to do is configure your cron job to launch a bash shell, and make that bash shell load your environment.
The shebang line in your script should not refer directly to a ruby executable, but to rvm's ruby:
#!/usr/bin/env ruby
This instructs the script to load the environment and run ruby as we would on the command line with rvm loaded.
On many UNIX derived systems, crontabs can have a configuration section before the actual lines that define the jobs to be run. If this is the case, you would then specify:
SHELL=/path/to/bash
This will ensure that the cron job will be spawned from bash. Still, your environment is missing, so to instruct bash to load your environment, you will want to add to the configuration section the following:
BASH_ENV=/path/to/environment (typically .bash_profile or .bashrc)
HOME is automatically derived from the /etc/passwd line of the crontab owner, but you can override it.
HOME=/path/to/home
After this, a cron job might look like this:
15 14 1 * * $HOME/rvm_script.rb
What if your crontab doesn't support the configuration section. Well, you will have to give all the environment directives in one line, with the job itself. For example,
15 14 1 * * export BASH_ENV=/path/to/environment && /full/path/to/bash -c '/full/path/to/rvm_script.rb'
Full blog post on the subject
You can use rvm wrappers:
/home/deploy/.rvm/wrappers/ruby-2.2.4/ruby
Source: https://rvm.io/deployment/cron#direct
I have a Rails script that I would like to run daily. I know there are many approaches, and that a cron'd script/runner approach is frowned upon by some, but it seems to meet my needs.
However, my script is not getting executed as scheduled.
My application lives at /data/myapp/current, and the script is in script/myscript.rb. I can run it manually without problem as root with:
/data/myapp/current/script/runner -e production /data/myapp/current/script/myscript.rb
When I do that, the special log file (log/myscript.log) gets logged to as expected:
Tue Mar 03 13:16:00 -0500 2009 Starting to execute script...
...
Tue Mar 03 13:19:08 -0500 2009 Finished executing script in 188.075028 seconds
I have it set to run with cron every morning at 4 am. root's crontab:
$ crontab -l
0 4 * * * /data/myapp/current/script/runner -e production /data/myapp/current/script/myscript.rb
In fact, it looks like it's tried to run as recently as this morning!
$ tail -100 /var/log/cron
...
Mar 2 04:00:01 hostname crond[8894]: (root) CMD (/data/myapp/current/script/runner -e production /data/myapp/current/script/myscript.rb)
...
Mar 3 04:00:01 hostname crond[22398]: (root) CMD (/data/myapp/current/script/runner -e production /data/myapp/current/script/myscript.rb)
...
However, there is no entry in my log file, and the data that it should update has not been getting updated. The log file permissions (as a test) were even set to globally writable:
$ ls -lh
total 19M
...
-rw-rw-rw- 1 myuser apps 7.4K Mar 3 13:19 myscript.log
...
I am running on CentOS 5.
So my questions are...
Where else can I look for information to debug this?
Could this be a SELinux issue? Is there a security context that I could set or change that might resolve this error?
Thank you!
Update
Thank you to Paul and Luke both. It did turn out to be an environment issue, and capturing the stderr to a log file enabled me to find the error.
$ cat cron.log
/usr/bin/env: ruby: No such file or directory
$ head /data/myapp/current/script/runner
#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'commands/runner'
Adding the specific Ruby executable to the command did the trick:
$ crontab -l
0 4 * * * /usr/local/bin/ruby /data/myapp/current/script/runner -e production /data/myapp/current/script/myscript.rb >> /data/myapp/current/log/cron.log 2>&1
By default cron mails its output to the user who ran it. You could look there.
It's very useful to redirect the output of scripts run by cron so that you can look at the results in a log file instead of some random user's local mail on the server.
Here's how you would redirect stdout and stderr to a log file:
cd /home/deploy/your_app/current; script/runner -e production ./script/my_cron_job.rb >> /home/deploy/your_app/current/log/my_file.log 2>&1
The >> redirect stdout to a file, and and the 2>&1 redirects stderr to stdout so any error messages will be logged as well.
Having done this, you will be able to examine the error messages to see what's really going on.
The usual problem when somebody discovers their script won't run in a cron job when it will run from the command line is that it relies on some piece of the environment that an interactive session has but cron doesn't get. Some frequent candidates are the "PATH" environment, and possibly "HOME".
On Linux, make sure all the config files (/etc/crontab, /etc/crond.{daily,hourly,etc}/* and /etc/cron.d/*) are only writeable to user root and are not symlinks, otherwise they will not even be considered.
To allow non-root and/or symlinks, specify the -p option to the crond daemon.