For a Rails 3.1 (will be 3.2 very soon), I have exceptionally verbose logs that have a lot of additional worker information spewing forth into them.
I routinely end up with multigigabyte development.log files. I've seen some various chatter around about rotating production logs, however I've not found anything that seems applicable to development.log rotation.
How do you rotate your development.log at every 100.megabytes or so? OR WHAT I WOULD PREFER is to actually truncate the head of the file so that only the most recent items remain in the log, up to 100MB of the most recent entries.
I have played with this a little and am thinking more and more than nothing quite like this exists at present and that perhaps I should implement something that will use the ruby File.truncate somehow, however I'm not sure of the efficacy of this yet on the tail end of the file as of yet.
You can actually tell the Ruby Logger class to rotate the files in the constructor:
http://corelib.rubyonrails.org/classes/Logger.html#M000163
Example:
Logger.new(name, shift_age = 7, shift_size = 1048576)
In one of my enrivornment files I have the following line:
config.logger = Logger.new("#{RAILS_ROOT}/log/#{ENV['RAILS_ENV']}.log", 10, 1048576)
This keeps the last 10 logfiles which are rotated every 1 MB.
On OSX i would use newsyslog
/etc/newsyslog.conf
On a Linux OS: logrotate
logrotate
Related
What is the best way to rotate Ruby on Rails log files daily?
I saw posts in the past. They recommended using logrotate and enable copytruncate option to avoid restarting Rails.
However, logs are lost at small time slice between copying the file and truncating it.
My project requirement does not lose logs, so we chose cronolog with ruby's stdlib Logger.
The code like the following:
config.logger = Logger.new(IO.popen("/usr/sbin/cronolog #{config.paths['log'].first}.%Y%m%d", "w"))
It's mostly works correctly, but logs are mixed when logging data size more than PIPE_BUF that because using PIPE.
So what is recommended way to rotate logs in such situation?
I got my production log file working. But wanted to get some suggestions onto how to work with it. I currently look at it view console and doing either
tail log/*
tail -f log/production.log
to look at it. I would like to either 1. find a better way to be able to look at the logs and perhaps with colors. 2. Can the production log file be outputted in a view where an admin user can view it?
Lastly, is there a gem that's great for emailing me when an error occurred and what the error was / how do you currently keep track of when errors happening in production?
Thanks!
For just having a nicer alternative to tail which shows the colours instead of those ESC[… codes, try less -R production.log. If you want to follow the end of the file like tail -f, use:
less -R +F production.log
There's nothing built-in for getting the log into a view as far as I know, but I suppose you could write a controller which shows (some of) the log in a view. Make sure it's secure with authentication though; you don't want to be exposing your log to the internet! Using an external service should remove this concern and it will have a load of other features. For example, I haven't tried papertrail (https://papertrailapp.com) but it's been on my to-try list for a while. It looks good and they're free for up to 100MB of logs a month. I've used airbrake (https://airbrake.io/) in the past for exception notifications but I don't think they have a free tier - just a free trial.
I'm running a Rails 3.0 application.
My problem is that the production.log is growing very fast.
Because I'm using caching, I get a lot of "Exist fragment?" entries in the log file.
I modified in config/environments/production.rb the following line:
config.log_level = :fatal, but this seems to have no effect.
In production mode, the site runs under Apache (with PhusionPassenger).
The log information seems to be the same as in development mode.
Does anybody has any idea?
You can delete this log file once it is too big. Make a backup if the data is important to you.
As per the below question...
Ruby on rails log file size too large
Alternatively you can rotate the log file using logrotate, see the below answer...
Ruby on Rails production log rotation
I help to maintain a Rails website. It's running JRuby 1.5.5, Rails 2.3.10, on a Solaris Sparc machine. I have a problem related to logging.
To stop our logfiles growing too large and filling the disk, we're using the log-shifting that's built in to the Logger class. In config/environments/production.rb we have:
config.logger = Logger.new(config.log_path, 10, 100.megabyte)
Which should rotate the logfiles when they reach 100 megabytes, and only keep 10 files.
The problem is two-fold: Rails is not rotating the logs properly, and it is keeping open the old log file to write to it -- but what it is writing is just repeated content of a few requests. So if I do ls -l log I see something like this:
-rw-r--r-- 83040892 Oct 4 15:07 production.log
-rw-r--r-- 3303158664 Oct 4 15:07 production.log.0
-rw-r--r-- 104857616 Oct 2 23:13 production.log.1
-rw-r--r-- 104857618 Oct 1 17:12 production.log.2
Note how the most recently cycled log is still open and still being written to (running pfiles confirms that it the Rails server still has three file handles to the log). Note also that it has reached 3 gigabytes in two days, where usually we do 100MB a day. This is because it is full of repeated requests. I can't easily paste it in here, but the log is full of the same 1000 line chunk of requests from 18:50 on Oct 3 (which I believe is the point at which the logs rotated), printed over and over again. From past experience, the log file will keep filling with this repeated content until the disk fills up.
Is log shifting/Rails logging just plain broken? (There's nothing odd about our logfile usage: we don't do any direct logging, it all just comes from the Rails framework.) The obvious next step is to try something like logrotate, but if Rails is refusing to close old log files and is writing junk to them forever, I suspect it won't solve my problem (because the log will never be closed, and thus the disk space never recovered).
The symptom seems to be that one old logfile still keeps getting used, although you successfully rotated the logs.
The cause is most likely that one or more of your Rails instances or threads is still using the old file handle.
The solution is to make sure that all the Rails instances restart completely after logs are rotated, so they all use the new file handle / name.
Use logrotate instead of config.logger to rotate your logs!
I'd suggest to use the UNIX logrotate to rotate your logs, instead of config.logger.
IMHO that's a better solution, more reliable, you have more control over the log rotation, and you can provide some post-rotation commands to restart your Rails processes. (either via logrotate's postrotate or endscript option)
See:
http://www.opencsw.org/packages/logrotate/ (logrotate Package for Solaris)
http://www.thegeekstuff.com/2010/07/logrotate-examples/ (logrotate tutorial with examples)
http://linux.die.net/man/8/logrotate
Can you use Unicorn?
- Unicorn is has built-in support for re-opening all log files in your application via USR1 signal - this allows logrotate to rotate files atomically...
- Unicorn keeps track of and restarts it's workers! You can kill the workers after log rotation and Unicorn will restart them, making sure they use the new log file.
See: https://github.com/blog/517-unicorn (many advantages for Unicorn over Mongrel)
If you're using Mongrel and can't switch to Unicorn:
use logrotate, and restart your Mongrels via the postrotate option.
hope this helps..
I've always used the platform's log rotation mechanism when dealing with the Rails log files. Following the advice from http://www.nullislove.com/2007/09/10/rotating-rails-log-files/ and because I run Passenger from http://overstimulate.com/articles/logrotate-rails-passenger as well.
The first method uses the logrotate copytruncate method of creating the new log file, so processes that still have a handle to it will always write to the current log file.
Other things to check on the server are:
Make sure that none of the gems or plugins have a handle to the Logger inside the ruby context.
Since you're using JRuby ensure that there isn't a stuck/runaway thread somewhere that's trying to fulfill a request but getting stuck logging.
Just like with Passenger, consider restarting the Rails server processes every now and again. I know this is effectively a hack but it might work.
Neil,
I don't know if this works for your particular situation, but I was having a similar problem and I think I just solved it. In my case, I had two symptoms. The first was the same issue as you -- my log rotation was screwy...in particular, the production.log.1 file was kept open and continued logging to is was happening while production.log was also getting logged to. The second symptom was that the log files ownerships and group memberships would keep changing to root. My Rails app is deployed via Capistrano, using a 'deployer' user, so I'd get all sorts of neat errors whenever the app tried to write to a log file that was no longer owned by deployer.
I'm embarrassed to say how long it took me to realize what the cause of both problems was. Somewhere along the way, I updated cron with the app's crontab as root. This must have been when I messing around at the command prompt...if I had just stayed with my deployment recipe via Capistrano, I wouldn't have inadvertently done that. In any case, I finally looked in /var/spool/cron/crontabs and I found two copies of my crontab file...one for deployer and one for root. So, the processes that cron fired off for my app were getting duplicated -- one was run under deployer and a second under root. It was that second one that was screwing things up. Once I deleted root's crontab, all was better.
Some caveats: on my setup, there were no non-app-related tasks in root's crontab, i.e. it was an exact duplicate of deployer's crontab...so deleting it had no side effects for me. Also, my server is running Ubuntu...the path to your crontabs may be different.
Hope that helps.
David
I think you forgot the 's' in megabytes
or instead use something like this
config.logger = Logger.new(config.log_path, 10, 102400)
also check this link it's very helpful
http://railsillustrated.com/logger-tricks.html
Today, for no reason, my rails(2.1.0) application is very slow or even not responding.
It happens intermittently.
So sometimes it works but again it doesn't work.
When it doesn't work, I can't even load 'script/console production'.
I want to know where it's stuck.
How do I load console step-by-step so that I can know which part causes the problem?
Thanks.
Sam
You could be running out of memory on your server and so the machine can't respond to you. If you are running mongrel then get monit onto it to limit the memory it can use and restart where necessary.
If you are using Passenger, try limiting the amount of instances, and if you have already done that look up a script that kills passenger instances when too big via a cron job.
If it's not a memory issue then I'd probably need more information.
If you are able, try temporarily removing all plugins/gems and see if the app will boot. One of those is likely the problem.
The script/console file contains this.
#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'commands/console'
You can find the commands/console file here for Rails 2.1.
I recommend loading up irb and trying to load your app from there.
require 'config/boot'
If that works, then try stepping through the rest of the mentioned commands/console script to find out which part is the problem.
If the boot file won't load, take a look at that config/boot.rb file to see if you can determine which part isn't working. Good luck!
When all else fails, you should be able to use strace to debug what the script/application is actually doing. Note that this should only be one of the last resorts as it produces extremely verbose information (and also mostly limited to I/O actions).
Try using 'strace script/console production' for example