I'm using ruby standard logger, I want rotational daily one, so in my code I have :
Logger.new("#{$ROOT_PATH}/log/errors.log", 'daily')
It is working perfectly, but it created two files errors.log.20130217 and errors.log.20130217.1.
How can I force it to create just one file a day ?
Your code is correct for a long-running application.
What's happening is you're running code more than once on a given day.
The first time you run it, Ruby creates a log file "errors.log".
When the day changes, Ruby renames the file to "errors.log.20130217".
But somehow you ran the code again, perhaps you're running two apps (or processes, or workers, or threads) that use similar code, and your logger saw that the file name "errors.log.20130217" already existed.
Your logger didn't want to clobber that file, but still needed to rename "errors.log" to a date, so the logger instead created a different file name "errors.log.20130217.1"
To solve this, run your code just once.
If you're running multiple apps called "foo" and "bar" then use log file names like "foo-errors.log" and "bar-errors.log". Or if you're using multiple workers, give each worker its own log file name (e.g. by using the worker's process id, or worker pool array index, or however you're keeping track of your workers).
If you really want to solve this using the Ruby logger, you'll need to override the logger #shift_log_period so it doesn't choose a ".1" suffix. You could subclass Logger and write your worn #shift_log_period to detect that there is an existing log file for the date, and if so, use it instead of doing the file rename.
This is the code causing it from the logger:
def shift_log_period(period_end)
postfix = period_end.strftime("%Y%m%d") # YYYYMMDD
age_file = "#{#filename}.#{postfix}"
if FileTest.exist?(age_file)
# try to avoid filename crash caused by Timestamp change.
idx = 0
# .99 can be overridden; avoid too much file search with 'loop do'
while idx < 100
idx += 1
age_file = "#{#filename}.#{postfix}.#{idx}"
break unless FileTest.exist?(age_file)
end
end
#dev.close rescue nil
File.rename("#{#filename}", age_file)
#dev = create_logfile(#filename)
return true
There is no solution (AFAIK) using the Ruby logger, with its built-in rotator, to manage logs written by multiple apps (a.k.a. workers, processes, threads) simultaneously. This is because each of the apps gets it own log file handle.
Alternatively, use any of the good log rotator tools, such as logrotate as suggested by the Tin Man user in the question comments: http://linuxcommand.org/man_pages/logrotate8.html
In general, logrotate will be your best bet IMHO.
Related
I am developing a Rails app for network automation. Part of app consists logic to run operations, part are operations themselves. Operation is simply a ruby class that performs several commands for network device (router, switch etc).
Right now, operation is simply part of Rails app repo. But in order to make development process more agile, I would like to decouple app and operations. I would have 2 repos - one for app and one for operations. App deploy would follow standard procedure, but operation would sync every time something is pushed to master. And what is more important, I don't want to restart app after operations repo update.
So my question is:
How to exclude several classes (or namespaces) from being cashed in production Rails app - I mean every time I call this class it would be reread file from disk. What could be potential dangers of doing so?
Some code example:
# Example operation - I would like to add or modify such classes withou
class FooOperation < BaseOperation
def perform(host)
conn = new_connection(host) # method from BaseOperation
result = conn.execute("foo")
if result =~ /Error/
# retry, its known bug in device foo
conn.execute("foo")
else
conn.exit
return success # method from BaseOperation
end
end
end
# somewhere in admin panel I would do so:
o = Operations.create(name: "Foo", class_name: "Foo")
o.id # => 123 # for next example
# Ruby worker which actually runs an operation
class OperationWorker
def perform(operation_id, host)
operation = Operation.find(operation_id)
# here, everytime I load this I want ruby to search for implementation on filesystem, never cache
klass = operation.class_name.constantize
class.new(host).perform #
end
end
i think you have quite a misunderstanding about how ruby code loading and interpretation works!
the fact that rails reloads classes at development time is kind of a "hack" to let you iterate on the code while the server has already loaded, parsed and executed parts of your application.
in order to do so, it has to implement quite some magic to unload your code and reload parts of it on change.
so if you want to have up-to-date code when executing an "operation" you are probably best of by spawning a new process. this will guarantee that your new code is read and parsed properly when executed with a blank state.
another thing you can do is use load instead of require because it will actually re-read the source on subsequent requests. you have to keep in mind, that subsequent calls to load just add to the already existing code in the ruby VM. so you need to make sure that every change is compatible with the already loaded code.
this could be circumvented by some clever instance_eval tricks, but i'm not sure that is what you want...
So, i'm building a small application using Rails. I need to use XMPP proto for notification of users with a bit of commands to change user status.
Previous version was an application using plain Python(pretty ugly app) with plain SQL request into DB. There was a table "jabber_queue", so in separated script, bot was checking each second for any rows, processing it(sending message), deleting. It was simple and stupid, but it was ok.
Now i see, i need to integrate this bot functionality into Rails(at least to work with RSpec).
This is my few versions of how it can be done:
Use separated async queue solution. For example, Resque. Use separated Ruby script and push events into Resque, pop events in application(not dependent on DB, so easily fit with RSpec tests and test DB). But it makes my application a bit bloated - i need to use second DB with a lot of memory and CPU additional requirements - it will be overhead for my problem. Also, i don't want to support additional "thing" for this application, i know, it can be done a lot easier way.
Use delayed_job(queue solution using current AR DB). But i don't know how to get current AR DB of Rails application in separated script. Anyway, it's a dirty and ugly way.
Launch XMPP bot WITHIN Rails application, as background worker. So worker will get access to "current" AR(in case of testing, to test-DB). But i simply don't know how to do it. I've found a Navvy, but i need to put somewhere at Rails start a string like a "Navvy::Job.enqueue(Cow, :speak)", i don't know where will be best place for this, to start it with RSpec testing and "rails server". Also, there is a BackgrounDRb, but this project is similar to Navvy and inactive too. Using search on stackoverflow, i've found similar problem to mine, but solution leads me to background_job, which can just anything in background, but i still don't know how to get current AR DB access in separated script.
I'm so sorry for this amount of words in my problem, it's just a brainstream. I see some solutions, but i really need advices and some words from more experienced developers.
So, this was solved using yes, third way.
I've created a class for dealing with bot commands and AR models - awesomo.rb. Nothing special, really. I put this in /lib/ of Rails project. Secondly,
Created configuration file for easy setting of password and JID - config/awesomo.yml
development:
xmpp_jid: ...
xmpp_password: ...
I've created a daemon for my bot - awesomo_daemon.rb. Same, in /lib/. This is what it contains:
#preload rails environment
require File.expand_path(File.join(File.dirname(__FILE__),
'..', 'config', 'environment'))
#load xmpp bot class
require 'awesomo_daemon'
#load xmpp bot configuration file for current environment
AWESOMO_CONFIG = YAML.load_file(File.join(File.dirname(__FILE__),
'..', 'config', 'awesomo.yml'))[Rails.env]
#apply configuration before singleton will be initiated
Awesomo.setup AWESOMO_CONFIG['xmpp_jid'], AWESOMO_CONFIG['xmpp_password']
loop {
Awesomo.instance.idle
sleep 1
}
Created daemon script starter- script/awesomo.
#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
Daemons.run "lib/awesomo_daemon.rb", dir_mode: :normal,
dir: File.join(File.dirname(__FILE__), '..', 'tmp', 'pids')
Simply run it with command script/awesomo start.
And I can use any my models within awesomo.rb! Also, for queue, i'm using little model - XMPPJob with fields jtype(for example, "xmpp_message"), body("hey!"), to("john#jabber.com"). Fetching it within awesomo.rb idle function of bot class with limit(5).each do |job| case jtype ....
To post any new job for my "awesomo", i'm using function send_message:
def self.send_message to, body
xmppjob = XmppJob.new :jtype => "xmpp_message", :body => body, :to => to
xmppjob.save
end
Everything works perfectly, except XMPP library(xmpp4r-simple) itself, but I'll rewrite it soon using just xmpp4r.
I have read the documentation on this guide and the class. I wish to create a logger which take logging informations each day and after let's say a week, delete the oldest logging information automatically each time.
logfile = File.open(RAILS_ROOT + '/log/'+ (Date.today << 1).to_s + '_custom.log', 'a') #create log file
logfile.sync = true #automatically flushes data to file
CUSTOM_LOGGER = CustomLogger.new(logfile, 'daily') #constant accessible anywhere
Plus, I wish to create a custom logging, so for instance something that looks like this (format):
class MyLogger < Logger
def format_message(severity, timestamp, progname, msg)
"#{timestamp} : #{msg}\n"
end
end
So basically, I would like to have a better idea where to place everything correctly under which directory. For instance, where MyLogger should be logically placed... (anywhere? A helper? or under app/config/ ?
Is that a valid way to implement this?
I made it by putting everything in config/initializers and creating a file named my_logger.rb. I'm still stuck at deleting/managing log files.
Does the server handle that part with a log rotation ( I know there's something with logrotation from the linux OS)? Or Rails can handle that internally?
Where should MyLogger be logically placed?
Probably put it under /lib. You can then require it from the initializer where you set the custom logger.
How can you periodically delete the oldest logging information?
There are a countless ways you can do this and choosing will be based on your constraints. You haven't spoken much about your constraints, so it's going to be hard to give you the just-right answer. E.g. you could clean up old logs every time you add a new log entry, you could run a cron job, you could install some non-Rails software that does log rotation and other log maintenance, you could use Papertrail, if you use Heroku you could look up https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes.
Remember Rails is designed more to handle requests and respond to them in the context of that request, than to run maintenance outside of the context of receiving a request. You could do maintenance as a side-effect of every format_message request to MyLogger, checking for the oldest logging entry and if you find one older than a week, delete them. You haven't given a constraint why you can't do this in-process, and if you're prototyping something early and portable, then this would get you going fast.
In my rails application, I have a background process runner, model name Worker, that checks for new tasks to run every 10 seconds. This check generates two SQL queries each time - one to look for new jobs, one to delete old completed ones.
The problem with this - the main log file gets spammed for each of those queries.
Can I direct the SQL queries spawned by the Worker model into a separate log file, or at least silence them? Overwriting Worker.logger does not work - it redirects only the messages that explicitly call logger.debug("something").
The simplest and most idiomatic solution
logger.silence do
do_something
end
See Logger#silence
Queries are logged at Adapter level as I demonstrated here.
How do I get the last SQL query performed by ActiveRecord in Ruby on Rails?
You can't change the behavior unless tweaking the Adapter behavior with some really really horrible hacks.
class Worker < ActiveRecord::Base
def run
old_level, self.class.logger.level = self.class.logger.level, Logger::WARN
run_outstanding_jobs
remove_obsolete_jobs
ensure
self.class.logger.level = old_level
end
end
This is a fairly familiar idiom. I've seen it many times, in different situations. Of course, if you didn't know that ActiveRecord::Base.logger can be changed like that, it would have been hard to guess.
One caveat of this solution: this changes the logger level for all of ActiveRecord, ActionController, ActionView, ActionMailer and ActiveResource. This is because there is a single Logger instance shared by all modules.
Running a rails site right now using SQLite3.
About once every 500 requests or so, I get a
ActiveRecord::StatementInvalid (SQLite3::BusyException: database is locked:...
What's the way to fix this that would be minimally invasive to my code?
I'm using SQLLite at the moment because you can store the DB in source control which makes backing up natural and you can push changes out very quickly. However, it's obviously not really set up for concurrent access. I'll migrate over to MySQL tomorrow morning.
You mentioned that this is a Rails site. Rails allows you to set the SQLite retry timeout in your database.yml config file:
production:
adapter: sqlite3
database: db/mysite_prod.sqlite3
timeout: 10000
The timeout value is specified in miliseconds. Increasing it to 10 or 15 seconds should decrease the number of BusyExceptions you see in your log.
This is just a temporary solution, though. If your site needs true concurrency then you will have to migrate to another db engine.
By default, sqlite returns immediatly with a blocked, busy error if the database is busy and locked. You can ask for it to wait and keep trying for a while before giving up. This usually fixes the problem, unless you do have 1000s of threads accessing your db, when I agree sqlite would be inappropriate.
// set SQLite to wait and retry for up to 100ms if database locked
sqlite3_busy_timeout( db, 100 );
All of these things are true, but it doesn't answer the question, which is likely: why does my Rails app occasionally raise a SQLite3::BusyException in production?
#Shalmanese: what is the production hosting environment like? Is it on a shared host? Is the directory that contains the sqlite database on an NFS share? (Likely, on a shared host).
This problem likely has to do with the phenomena of file locking w/ NFS shares and SQLite's lack of concurrency.
If you have this issue but increasing the timeout does not change anything, you might have another concurrency issue with transactions, here is it in summary:
Begin a transaction (aquires a SHARED lock)
Read some data from DB (we are still using the SHARED lock)
Meanwhile, another process starts a transaction and write data (acquiring the RESERVED lock).
Then you try to write, you are now trying to request the RESERVED lock
SQLite raises the SQLITE_BUSY exception immediately (indenpendently of your timeout) because your previous reads may no longer be accurate by the time it can get the RESERVED lock.
One way to fix this is to patch the active_record sqlite adapter to aquire a RESERVED lock directly at the begining of the transaction by padding the :immediate option to the driver. This will decrease performance a bit, but at least all your transactions will honor your timeout and occurs one after the other. Here is how to do this using prepend (Ruby 2.0+) put this in a initializer:
module SqliteTransactionFix
def begin_db_transaction
log('begin immediate transaction', nil) { #connection.transaction(:immediate) }
end
end
module ActiveRecord
module ConnectionAdapters
class SQLiteAdapter < AbstractAdapter
prepend SqliteTransactionFix
end
end
end
Read more here: https://rails.lighthouseapp.com/projects/8994/tickets/5941-sqlite3busyexceptions-are-raised-immediately-in-some-cases-despite-setting-sqlite3_busy_timeout
Just for the record. In one application with Rails 2.3.8 we found out that Rails was ignoring the "timeout" option Rifkin Habsburg suggested.
After some more investigation we found a possibly related bug in Rails dev: http://dev.rubyonrails.org/ticket/8811. And after some more investigation we found the solution (tested with Rails 2.3.8):
Edit this ActiveRecord file: activerecord-2.3.8/lib/active_record/connection_adapters/sqlite_adapter.rb
Replace this:
def begin_db_transaction #:nodoc:
catch_schema_changes { #connection.transaction }
end
with
def begin_db_transaction #:nodoc:
catch_schema_changes { #connection.transaction(:immediate) }
end
And that's all! We haven't noticed a performance drop and now the app supports many more petitions without breaking (it waits for the timeout). Sqlite is nice!
bundle exec rake db:reset
It worked for me it will reset and show the pending migration.
Sqlite can allow other processes to wait until the current one finished.
I use this line to connect when I know I may have multiple processes trying to access the Sqlite DB:
conn = sqlite3.connect('filename', isolation_level = 'exclusive')
According to the Python Sqlite Documentation:
You can control which kind of BEGIN
statements pysqlite implicitly
executes (or none at all) via the
isolation_level parameter to the
connect() call, or via the
isolation_level property of
connections.
I had a similar problem with rake db:migrate. Issue was that the working directory was on a SMB share.
I fixed it by copying the folder over to my local machine.
Most answers are for Rails rather than raw ruby, and OPs question IS for rails, which is fine. :)
So I just want to leave this solution over here should any raw ruby user have this problem, and is not using a yml configuration.
After instancing the connection, you can set it like this:
db = SQLite3::Database.new "#{path_to_your_db}/your_file.db"
db.busy_timeout=(15000) # in ms, meaning it will retry for 15 seconds before it raises an exception.
#This can be any number you want. Default value is 0.
Source: this link
- Open the database
db = sqlite3.open("filename")
-- Ten attempts are made to proceed, if the database is locked
function my_busy_handler(attempts_made)
if attempts_made < 10 then
return true
else
return false
end
end
-- Set the new busy handler
db:set_busy_handler(my_busy_handler)
-- Use the database
db:exec(...)
What table is being accessed when the lock is encountered?
Do you have long-running transactions?
Can you figure out which requests were still being processed when the lock was encountered?
Argh - the bane of my existence over the last week. Sqlite3 locks the db file when any process writes to the database. IE any UPDATE/INSERT type query (also select count(*) for some reason). However, it handles multiple reads just fine.
So, I finally got frustrated enough to write my own thread locking code around the database calls. By ensuring that the application can only have one thread writing to the database at any point, I was able to scale to 1000's of threads.
And yea, its slow as hell. But its also fast enough and correct, which is a nice property to have.
I found a deadlock on sqlite3 ruby extension and fix it here: have a go with it and see if this fixes ur problem.
https://github.com/dxj19831029/sqlite3-ruby
I opened a pull request, no response from them anymore.
Anyway, some busy exception is expected as described in sqlite3 itself.
Be aware with this condition: sqlite busy
The presence of a busy handler does not guarantee that it will be invoked when there is
lock contention. If SQLite determines that invoking the busy handler could result in a
deadlock, it will go ahead and return SQLITE_BUSY or SQLITE_IOERR_BLOCKED instead of
invoking the busy handler. Consider a scenario where one process is holding a read lock
that it is trying to promote to a reserved lock and a second process is holding a reserved
lock that it is trying to promote to an exclusive lock. The first process cannot proceed
because it is blocked by the second and the second process cannot proceed because it is
blocked by the first. If both processes invoke the busy handlers, neither will make any
progress. Therefore, SQLite returns SQLITE_BUSY for the first process, hoping that this
will induce the first process to release its read lock and allow the second process to
proceed.
If you meet this condition, timeout isn't valid anymore. To avoid it, don't put select inside begin/commit. or use exclusive lock for begin/commit.
Hope this helps. :)
this is often a consecutive fault of multiple processes accessing the same database, i.e. if the "allow only one instance" flag was not set in RubyMine
Try running the following, it may help:
ActiveRecord::Base.connection.execute("BEGIN TRANSACTION; END;")
From: Ruby: SQLite3::BusyException: database is locked:
This may clear up the any transaction holding up the system
I believe this happens when a transaction times out. You really should be using a "real" database. Something like Drizzle, or MySQL. Any reason why you prefer SQLite over the two prior options?