Testing has been the thing I've avoided until I realized that with well written tests I can avoid having to enter console and running a string of commands over and over again. The big issue I'm having, however, is that when a test fails, I can't see much data to use to debug with. For example, when I'm testing models I need to see the models attributes, and I need to control when it's been built, created, and saved, and a way to detect these states. With rails fixtures and plain vanilla tests this is annoying to do.
How can I make debugging fixtures and rails tests easier? Is there a way to display more console-like information when a test fails? Which testing tools can I use to debug more easily? After all, debugging the tests themselves is a huge waste of time, and the biggest obstacle to using them in the first place.
Thanks!
Debugging your tests is not very different from debugging your code.
Rails already comes with byebug gem (uncomment it, if necessary):
group :development, :test do
gem 'byebug'
end
Now you are ready to put
debugger
line anywhere in your code.
Byebug supports many additional features, which you can find here: https://github.com/deivid-rodriguez/byebug
Also check https://github.com/deivid-rodriguez/pry-byebug project which provides even better experience for debugging.
Logging helps a lot to investigate and fix issues. Not only in test but also when you deploy it on production. You can log state of your models and other useful info. Also standard rails logging logs SQL that is send to the database so you can easily see what's going on there. SQL logging is on by default in dev and test and looks similar to this:
[2015-09-24 23:19:36 ledger DEBUG]: [1m[36m (0.2ms)[0m [1mBEGIN[0m
[2015-09-24 23:19:36 ledger DEBUG]: [1m[35mSQL (2.1ms)[0m INSERT INTO "projections_tags" ("ledger_id", "tag_id", "name", "authorized_user_ids") VALUES ($1, $2, $3, $4) RETURNING "id" [["ledger_id", "ea9a68c5-c7c8-4964-b078-45bfc93d41ef"], ["tag_id", 9], ["name", "test"], ["authorized_user_ids", "{1}"]]
[2015-09-24 23:19:36 ledger DEBUG]: [1m[36m (0.4ms)[0m [1mCOMMIT[0m
I'm using log4r as a logging library since it gives more control over the log output.
I also don't like debugging but in some cases it really helps. In this case byebug is your friend.
And of corse rails console and rails dbconsole
Related
I've been digging around stackoverflow trying to find others who get these prepared statements already exists errors.
In most cases configuring unicorn properly with the after/before fork resolves these issues.
However in my case we are still getting errors as such:
ActiveRecord::StatementInvalid: PG::Error: ERROR: prepared statement "a495" already exists: INSERT INTO "user_logins" ("account_id", "created_at", "ip_address", "user_agent", "user_id") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
This error gets thrown in different areas in our app but always seems to have the same statement number 'a495'.
We are on rails 3.2.17, using postgres and we are on heroku.
I really have no idea why this is happening, but its starting to happen more frequently now.
Any help would be greatly appreciated.
In the rails stack trace this error is being thrown in the .prepare call. I'm confused because its checking for the sql key in the statements collection. If it doesn't exist it prepares the new one....however when trying to prepare it, its throwing the error.
def prepare_statement(sql)
sql_key = sql_key(sql)
unless #statements.key? sql_key
nextkey = #statements.next_key
#connection.prepare nextkey, sql
#statements[sql_key] = nextkey
end
#statements[sql_key]
end
We had the same problem, and did very thorough investigation. We concluded that in our case, this error is caused by Rack::Timeout, that very occasionally interrupts code execution after the new statement has been already created, but before the counter is updated on Rails side. Next prepared statement then tries to use the same name (e.g. a494), and a collision occurred.
My belief is that Rails has not implemented prepared statements correctly. Instead of using the increasing counter (a001, a002, ...), they should have used GUIDs. This way, a race condition described above wouldn't be an issue.
We didn't find a workaround. Improving the performance of an app, and increasing the window for Rack::Timeout, made this problem nearly extinct, but it still happens from time to time.
This is typically not a Postgres issue, but an issue with sharing database connections in something like Unicorn:
Understanding Heroku Postgres Log Statements and Common Errors
Here's my solution for Heroku, which unfortunately is a little involved. On the plus side, though, you don't need to suffer from 100's of error notifications when this error starts happening. All that's needed is that the app/dyno is restarted.
The basic outline of the procedure is that when we detect a ActiveRecord::StatementInvalid exception, with an error message description that contains the words 'prepared statement', we run the heroku restart command using Heroku's platform-api gem.
Put the platform-api gem in your Gemfile, and run bundle install
Set the HEROKU_API_KEY to the correct value. (You can generate a key from your Heroku dashboard). Use heroku config:set HEROKU_API_KEY=whatever-the-value-is.
Set the HEROKU_APP_NAME to the correct value. You can get this information from the heroku CLI, but it's just whatever you called your app.
Add the following to your ApplicationController (/app/controllers/application_controller.rb):
...
class ApplicationController < ActionController::Base
rescue_from ActiveRecord::StatementInvalid do |exception|
# notify your error handler, or send an email, or whatever
# ...
if exception.message =~ /prepared statement/
restart_dyno
end
end
def restart_dyno
heroku = PlatformAPI.connect_oauth(ENV["HEROKU_API_KEY"])
heroku.dyno.restart(ENV["HEROKU_APP_NAME"], "web")
end
end
That's it. Hope this helps.
I'm running a Rails app (v 3.1.10) on a Heroku Cedar stack with Papertrail add-on going crazy because of the size of the logs.
My app is really verbose and the logs are getting huge (really huge):
Sometimes because I serialize a lots of data in one field and that makes a huge SQL request. In my model I have many:
serialize :a_game_data, Hash
serialize :another_game_data, Hash
serialize :a_big_set_of_game_data, Hash
[...]
Thanks to my AS3 Flash app working with bigs sets of json...
Sometimes because there's a lots of partials to render:
Rendered shared/_flash_message.html.erb (0.1ms)
Rendered shared/_header_cart_info.html.erb (2.7ms)
Rendered layouts/_header.html.erb (19.4ms)
[...]
It's not the big issue here, but I've added this case too because Jamiew handle it, see below...
Sometimes because there's lots of sql queries on the same page:
User Load (2.2ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1
Course Load (5.3ms) SELECT "courses".* FROM "courses" WHERE (id = '1' OR pass_token = NULL)
Session Load (1.3ms) SELECT "sessions".* FROM "sessions" WHERE "sessions"."id" = 1 LIMIT 1
Training Load (1.3ms) SELECT "trainings".* FROM "trainings" WHERE "trainings"."id" = 1 LIMIT 1
[...]
It's a big (too) complex App we've got here... yeah...
Sometimes because there's a lots of params:
Parameters: {"_myapp_session"=>"BkkiJTBhYWI1MUVlaVdtbE9Eb1Y2I5BjsAVEkiEF9jc3JmX3Rva2VlYVWZyM2I0dEZaR1YwNXFjZhZTQ1uBjsARkkiUkiD3Nlc3Npb25faWQGOgZFRhcmRlbi51c2yN1poVm8vdWo3YTlrdUZzVTA9BjsARkkiH3dAh7CMTQ0Yzc4ZDJmYzg5ZjZjOGQ5NVyLmFkbWluX3VzZXIua2V5BjsAVFsISSIOQWRtaW5Vc2VyBjsARlsGaQZJIiIkMmEkMTAkcmgvQ2Rwc0lrYzFEbGJFRG9jMnZvdQY7AFRJIhl3YXJkZW4udXNlci51c2VyLmtleQY7AFRbCEkiCVVzZXIGOwBGWwZpBkkiIiQyYSQxMCRBUFBST2w0aWYxQmhHUVd0b0V5TjFPBjsAVA==--e4b53a73f6b622cfe7550b2ee12678712e2973c7", "authenticity_token"=>"EeiWmlODoYXUfr3b4tFZGV05qr7ZhVo/uj7a9kuFsU0=", "utf8"=>"✓", "locale"=>"fr", "id"=>"1", "a"=>1, "a"=>1, "a"=>1, "a"=>1, "a"=>1, "a"=>1, [...] Hey! You've reach the end of the line but it's not the end of the parameters...}
The AS3 Flash app send big json data to the controller...
I didn't mention the (in)famous "Assets pipeline logging problem" because now I'm using the quiet_assets gem to handle this:
https://github.com/evrone/quiet_assets
So... what did I try?
1: Dennis Reimann's middleware solution:
http://dennisreimann.de/blog/silencing-the-rails-log-on-a-per-action-basis/
2: Spagalocco's gem (inspired by solution #1):
https://github.com/spagalloco/silencer
3: jamiew's monkeypatches (inspired by solution #1 + a bonus):
https://gist.github.com/1558325
Nothing is really working as expected but it's getting close.
I would rather use a method in my ApplicationController like this:
def custom_logging(opts={}, show_logs=true)
disable_logging unless show_logs
remove_sql_requests_from_logs if opts[:remove_sql_requests]
remove_rendered_from_logs if opts[:remove_rendered]
remove_params_from_logs if opts[:remove_params]
[...]
end
...and call it in any controller method: custom_logging({:remove_sql_requests=>1, :remove_rendered=>1})
You got the idea.
So, is there any good resource online to handle this?
Many thanks for your advices...
I"m the author of the silencer gem mentioned above. Are you looking to filter logging in general or for a particular action? The silencer gem handles the latter problem. While you can certainly use it in different ways, it's mostly intended for particular actions.
It sounds like what you are looking for less verbose logging. I would recommend you take a look at lograge. I use that in production in most of my Rails apps and have found it to be quite useful.
If you need something more specialized, you may want to look at implementing your own LogSubscriber which is essentially the lograge solution.
Set your log level in the Heroku enviroment
View your current log level:
heroku config
You most likely have "Info", which is just a lot of noise
Change it to warn or error
heroku config:add LOG_LEVEL=WARN
Also, when viewing the logs, only specify the "app" server
heroku logs --source app
I personally, append --tail to see the logs live.
heroku logs --source app --tail
I have built a quite big rails application. Now I have a lot of queries on some pages. I'd really like the rails development log to show what line in my code that made the query request so I easy could remove some.
This is what I want.
[called from ProjectsController.rb line 32]CACHE (0.0ms) SELECT "firms".* FROM "firms" WHERE "firms"."subdomain" = 'testing' LIMIT 1
Is there a tag to put in the config/environments/development.rb file
This would be great!
config.log_tags = [:code_line_call]
A more lightweight solution is Active Record Query Tracer
Which does exactly what you are looking for.
I did not find a way to get the code line in the log.
The solution was Miniprofiler
It is a great tool for finding sql requests in your code
Ryan Bates made a railscast about it
railscasts 368-miniprofiler
I have a strange problem, when I run my (only / first) cucumber test, part of which creates a new entry in my Countries table using:
Factory.create(:country)
the models don't get committed to my database (MySql 5) & my test fails as the view tries to load this data. Here is a snippet from my test.log
[4;36;1mSQL (0.1ms)[0m [0;1mSAVEPOINT active_record_1[0m
[4;35;1mCountry Create (0.1ms)[0m [0mINSERT INTO `countries` (`name`, `country_code`, `currency_code`) VALUES('Ireland', 'IE', 'EUR')[0m
[4;36;1mSQL (0.1ms)[0m [0;1mRELEASE SAVEPOINT active_record_1[0m
[4;35;1mSQL (0.1ms)[0m [0mSAVEPOINT active_record_1[0m
However when I load up the rails console and run exactly the same command i.e. Factory.create(:country), the records get committed to the database. Here is the output from test.log
[4;36;1mSQL (3.5ms)[0m [0;1mBEGIN[0m
[4;35;1mCountry Create (0.2ms)[0m [0mINSERT INTO `countries` (`name`, `country_code`, `currency_code`) VALUES('Ireland', 'IE', 'EUR')[0m
[4;36;1mSQL (1.1ms)[0m [0;1mCOMMIT[0m
From env.rb
Cucumber::Rails::World.use_transactional_fixtures = false
Any advise is very much appreciated, I've spent the last two days trying to figure this out but had no success.
The second line of what you posted from your test.log file shows the record being inserted. I see that you've turned off transactional fixtures, which is fine.
Are you using the database_cleaner gem by chance? If so, it's going to essentially roll your database back to its original state after the tests are done. This means that, unless you pause your tests while they're running, and after data is inserted, you'll never see it in the DB because it gets removed after the test suite is run.
I don't know that this is what's causing the issue at the root of the thing, but it would definitely explain why, when you run just that one command from the console, that you see it in the DB, but you don't after you run your tests. The test data is supposed to be removed after the test suite executes, so your tests have a fresh, consistent starting point for each run. It's an important part of making sure that your tests run in the same environment every time, and therefore can be counted upon to give you reliable results.
I'm using pry, and ActiveRecord queries from the console always print out their corresponding SQL statements. I don't want this behavior. Googling around I only see that this must be explicitly achieved by setting the ActiveRecord logger to standard out.
Is this the default behavior of pry or is it the result of something I set that I forgot about? And how can I stop it?
Try looking for a .irbrc file in the project root or your home directory. You may see this or similar:
ActiveRecord::Base.logger = Logger.new(STDOUT)
Sometimes this ends up in another script that gets included in .irbrc.
YMMV, but I really like having SQL logged to the console. To each his own...
If you are using Rails 3.1+ this is now the default behaviour. Check here Disable Rails SQL logging in console