Any way to keep each request separate in the Rails log? - ruby-on-rails

I'd like to have the logs from each request be grouped together and easy to tell apart from another request's logs, eg
**********************
Started GET "/" for 67.205.67.76 at 2013-09-15 00:05:15 -0700
Processing by RootController#index as HTML
Rendered root/_index.html.erb within layouts/application (7.0ms)
Rendered layouts/_fonts_hack.html.erb (0.0ms)
Rendered layouts/_ie_version_vars.html.erb (0.0ms)
Rendered topbars/_logged_out_topbar.html.erb (2.0ms)
Rendered layouts/_old_browser_warnings.html.erb (0.0ms)
Completed 200 OK in 27ms (Views: 26.0ms | ActiveRecord: 0.0ms)
**********************
Started GET "/" for 67.205.67.76 at 2013-09-15 00:05:15 -0700
Processing by RootController#index as HTML
Rendered root/_index.html.erb within layouts/application (7.0ms)
Rendered layouts/_fonts_hack.html.erb (0.0ms)
Rendered layouts/_ie_version_vars.html.erb (0.0ms)
Rendered topbars/_logged_out_topbar.html.erb (2.0ms)
Rendered layouts/_old_browser_warnings.html.erb (0.0ms)
Completed 200 OK in 27ms (Views: 26.0ms | ActiveRecord: 0.0ms)
***********************
Is something like this possible? sometimes in my logs currently its kind of hard to visually see where one request begins and another ends, and sometimes logs from different requests get interspersed.

A better option would be to use the request uuid
config.log_tags = [:uuid]

Doesnt seem like it from what I can tell, but next best thing: add Pid and Thread object_id tags to each log, then search for and highlight a particular tag once you get to the start of a request, and then all the highlighted lines will be the ones in your request.
config.log_tags = [ lambda {|r| "#{Process.pid}##{Thread.current.object_id}" } ]

Related

Frequent unexpected requests logged by Rails app in EC2 - from Amazon public IP addresses - Why?

I'm running a rails app that writes files to Amazon S3. Now I have uniqorn running as server and see all lots of request coming in from different amazon AWS IP-addresses.
Here is a part of my log file:
Started GET "/" for 54.245.168.11 at 2016-10-02 18:30:50 +0000
Processing by StaticPagesController#home as */*
Rendered static_pages/home.html.erb within layouts/application (0.2ms)
Rendered layouts/_shim.html.erb (0.1ms)
Rendered layouts/_header.html.erb (0.3ms)
Completed 200 OK in 3ms (Views: 3.0ms | ActiveRecord: 0.0ms)
Started GET "/" for 54.252.254.235 at 2016-10-02 18:30:51 +0000
Processing by StaticPagesController#home as */*
Rendered static_pages/home.html.erb within layouts/application (0.2ms)
Rendered layouts/_shim.html.erb (0.0ms)
Rendered layouts/_header.html.erb (0.3ms)
Completed 200 OK in 3ms (Views: 3.0ms | ActiveRecord: 0.0ms)
Started GET "/" for 54.232.40.107 at 2016-10-02 18:30:52 +0000
Processing by StaticPagesController#home as */*
Rendered static_pages/home.html.erb within layouts/application (0.7ms)
Rendered layouts/_shim.html.erb (0.2ms)
Rendered layouts/_header.html.erb (1.8ms)
Completed 200 OK in 12ms (Views: 10.6ms | ActiveRecord: 0.0ms)
Started GET "/" for 54.252.79.139 at 2016-10-02 18:30:52 +0000
Processing by StaticPagesController#home as */*
Rendered static_pages/home.html.erb within layouts/application (0.7ms)
Rendered layouts/_shim.html.erb (0.3ms)
Rendered layouts/_header.html.erb (1.5ms)
Completed 200 OK in 13ms (Views: 11.6ms | ActiveRecord: 0.0ms)
Started GET "/" for 54.250.253.203 at 2016-10-02 18:30:52 +0000
Processing by StaticPagesController#home as */*
Rendered static_pages/home.html.erb within layouts/application (0.5ms)
Rendered layouts/_shim.html.erb (0.1ms)
Rendered layouts/_header.html.erb (0.3ms)
Completed 200 OK in 4ms (Views: 3.2ms | ActiveRecord: 0.0ms)
Started GET "/" for 54.243.31.203 at 2016-10-02 18:30:54 +0000
Processing by StaticPagesController#home as */*
Rendered static_pages/home.html.erb within layouts/application (0.5ms)
Rendered layouts/_shim.html.erb (0.0ms)
Rendered layouts/_header.html.erb (0.3ms)
Completed 200 OK in 4ms (Views: 3.5ms | ActiveRecord: 0.0ms)
I can't find any info on this online. My application isn't live yet, so nobody looks at it. Why is AWS doing this?
These are Route 53 Health Checks, which are a different type of health check than the health checks configured in Elastic Load Balancer (ELB), and different from the EC2 instance health and reachability checks that EC2 automatically performs on every EC2 instance.
Take a look at the published AWS IP Address Ranges, and you'll find that these match up to the address blocks specifically assigned to Route 53 Health Checks, quite nicely.
For example, this request...
Started GET "/" for 54.232.40.107 at 2016-10-02 18:30:52 +0000
...matches this entry...
{
"ip_prefix": "54.232.40.64/26",
"region": "sa-east-1",
"service": "ROUTE53_HEALTHCHECKS"
},
...because 54.232.40.64/26 is the CIDR spec for the IP address block from 54.232.40.64 through 54.232.40.127.
If you configured a Route 53 health check against your stack, then you probably have your answer, at this point.
Route 53 Health Checks are often used to drive the logic of complex DNS configurations, such as in cases where there are multiple candidate servers geographically distributed, and you want to control the answer returned by DNS based on the health of the servers... but they can also be used for monitoring latency and application performance (measuring TCP connect time and time-to-first-byte, as well as availability), tied into CloudWatch alarms, and used to drive scaling and other logic. If you configured anything like this, the question may be solved.
But if you didn't, this could be a case of another AWS user who has configured health checks against an IP address they were formerly using, or has configured the checks against your IP address or hostname by simple mistake.
If that is the case, you'll need to inspect the request headers from the incoming request. You'll find something like this:
User-Agent: Amazon Route 53 Health Check Service; ref:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; report http://amzn.to/zzzzzzz
With these values in hand, along with the other headers and any relevant log entries, you can visit the "report" URL provided in the headers you captured, (it's similar to the one I've shown above, captured from my logs, but sanitized since it seems to be generic, but I'm not certain -- the link in my logs redirects me here) and submit a request for AWS to work with you to to eliminate the unwanted health checks.
But first, try this...
Take the value immediately after "ref:" and create a URL from it, using this template:
https://console.aws.amazon.com/route53/healthchecks/home#/details/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
(Replace xxxx with your ref:, excluding the actual string ref:).
If the health check is from your own AWS account, and you're logged into the console, and have permission to view it, then this crafted link should bring the health check right up for you to view. If you get an error that you are "not authorized to access this resource," then the health check is not from your AWS account, or you aren't a privileged user on the AWS account and thus may lack the ability to view it.

RefineryCMS image thumbnails not showing up

I've got a RefineryCMS 3 application and in the backend the thumbnails for uploaded images are not showing up.
If I run the app in production mode on my development machine then the thumbnails show up. On the production machine the rails app is ran through passenger & apache and it fails.
I have copied my development copy to the production server for a test and ran the refinery project in development mode and I still get the same issue
rails s -b 192.168.200.215
Here is the standard output when I try and load the images area in refinery admin;
Started GET "/refinery/images" for 192.168.200.4 at 2016-03-01 15:32:25 +1100
Processing by Refinery::Admin::ImagesController#index as HTML
Parameters: {"locale"=>:en}
Refinery::Authentication::Devise::User Load (0.2ms) SELECT "refinery_authentication_devise_users".* FROM "refinery_authentication_devise_users" WHERE "refinery_authentication_devise_users"."id" = ? ORDER BY "refinery_authentication_devise_users"."id" ASC LIMIT 1 [["id", 1]]
Refinery::Authentication::Devise::Role Load (0.1ms) SELECT "refinery_authentication_devise_roles".* FROM "refinery_authentication_devise_roles" INNER JOIN "refinery_authentication_devise_roles_users" ON "refinery_authentication_devise_roles"."id" = "refinery_authentication_devise_roles_users"."role_id" WHERE "refinery_authentication_devise_roles_users"."user_id" = ? [["user_id", 1]]
Rendered /var/lib/gems/2.1.0/gems/refinerycms-acts-as-indexed-2.0.1/app/views/refinery/acts_as_indexed/admin/_search_header.html.erb (0.2ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/admin/_search_header.html.erb (4.4ms)
(0.2ms) SELECT COUNT(*) FROM "refinery_images"
CACHE (0.0ms) SELECT COUNT(*) FROM "refinery_images"
Refinery::Image Load (0.3ms) SELECT "refinery_images".* FROM "refinery_images" ORDER BY updated_at DESC LIMIT 20 OFFSET 0
Refinery::Image::Translation Load (0.3ms) SELECT "refinery_image_translations".* FROM "refinery_image_translations" WHERE "refinery_image_translations"."refinery_image_id" IN (2)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-images-3.0.1/app/views/refinery/admin/images/_grid_view.html.erb (10.3ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-images-3.0.1/app/views/refinery/admin/images/_images.html.erb (14.5ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-images-3.0.1/app/views/refinery/admin/images/_records.html.erb (33.7ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-acts-as-indexed-2.0.1/app/views/refinery/acts_as_indexed/admin/_search.html.erb (2.2ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/admin/_search.html.erb (4.8ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-images-3.0.1/app/views/refinery/admin/images/_actions.html.erb (17.0ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-images-3.0.1/app/views/refinery/admin/images/index.html.erb within layouts/refinery/admin (61.1ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/_html_tag.html.erb (0.2ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/admin/_javascripts.html.erb (159.9ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/admin/_head.html.erb (232.9ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/_site_bar.html.erb (3.4ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/admin/_menu_item.html.erb (28.4ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/admin/_menu.html.erb (31.9ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/_no_script.html.erb (0.3ms)
Rendered /var/lib/gems/2.1.0/gems/refinerycms-core-3.0.1/app/views/refinery/_message.html.erb (0.1ms)
Completed 200 OK in 375ms (Views: 362.0ms | ActiveRecord: 1.1ms)
Started GET "/system/images/W1siZiIsIjIwMTYvMDMvMDEvN3lmN3ZpNDRtZV9hYWEuanBnIl0sWyJwIiwidGh1bWIiLCIxMzV4MTM1I2MiXV0/aaa.jpg?sha=7a0325c4ef12ef3d" for 192.168.200.4 at 2016-03-01 15:32:27 +1100
When it tries to get the /system/images/.... it fails in the web browser console. If I try and access that directly with the following
http://192.168.200.215:3000/system/images/W1siZiIsIjIwMTYvMDMvMDEvN3lmN3ZpNDRtZV9hYWEuanBnIl0sWyJwIiwidGh1bWIiLCIxMzV4MTM1I2MiXV0/aaa.jpg
I get the following error in the browser
Internal Server Error
The production server didn't have imagemagick installed.
sudo apt-get install imagemagick
This fixed it.

Rails. First request slow, times out (40sec), the next ones are faster (1.5 sec). No custom caching, how can that be?

I moved an app from the bamboo stack to cedar and the server is behaving wierdly, it times out on the first request, taking 40sec according to the logs, while the next, same request, takes 1.5 sec. You would think the dyno slept, but I've got 2-3 dynos (using hirefire), and 3 workers.
rails 3.2
ruby 1.9.3
heroku cedar
heroku postgres crane
MongoDB, MongoHQ large
mongo_mapper 0.11.1
thin
no caching
It's an app with low user interaction, as it's mostly a huge background worker tool. The clients using it are few, but important, I obviously can't have the app timing out every morning or after lunch. You'd hate it too.
➜ my_app git:(master) ✗ heroku logs -t -p web --remote production
first request after some inactivity 40880ms
Started GET "/my_model" for xx.160.183.xxx at 2013-10-31 15:19:34 +0000
Processing by MyModelsController#index as HTML
User Load (6.8ms) SELECT "users".* FROM "users" WHERE "users"."id" = 110237164 LIMIT 1
MyModel Load (12.0ms) SELECT "my_model".* FROM "my_model" ORDER BY name ASC
Rendered my_model/_my_model_statuses.html.erb (3515.6ms)
Rendered my_model/_my_model_statuses.html.erb (3514.8ms)
Rendered my_model/_my_model_statuses.html.erb (3210.6ms)
Rendered my_model/_my_model_statuses.html.erb (3968.4ms)
Rendered my_model/_my_model_statuses.html.erb (2652.0ms)
Rendered my_model/_my_model_statuses.html.erb (2253.5ms)
Rendered my_model/_my_model_statuses.html.erb (1339.7ms)
Rendered my_model/_my_model_statuses.html.erb (1333.0ms)
Rendered my_model/_my_model_statuses.html.erb (2901.9ms)
Rendered my_model/_my_model_statuses.html.erb (636.3ms)
Rendered my_model/_my_model_statuses.html.erb (1598.6ms)
Rendered my_model/_my_model_statuses.html.erb (10.1ms)
Rendered my_model/_my_model_statuses.html.erb (909.3ms)
Rendered my_model/_my_model_statuses.html.erb (911.5ms)
Rendered my_model/_my_model_statuses.html.erb (1435.4ms)
Rendered my_model/_my_model_statuses.html.erb (1043.9ms)
Rendered my_model/_my_model_statuses.html.erb (1712.3ms)
Rendered my_model/_my_model_statuses.html.erb (793.8ms)
Rendered my_model/_my_model_statuses.html.erb (290.2ms)
Rendered my_model/_my_model_statuses.html.erb (1142.0ms)
Rendered my_model/_my_model_statuses.html.erb (342.6ms)
Rendered my_model/_my_model_statuses.html.erb (376.0ms)
Rendered my_model/_my_model_statuses.html.erb (2161.8ms)
Rendered my_model/_my_model_statuses.html.erb (1194.9ms)
Rendered my_model/_my_model_statuses.html.erb (1293.4ms)
Rendered my_model/_my_model_overview.html.erb (40695.0ms)
Rendered my_model/index.html.erb within layouts/application (40721.5ms)
Rendered shared/_menu.html.erb (1.0ms)
Rendered shared/_usermenu.html.erb (2.3ms)
Rendered shared/_header.html.erb (27.1ms)
Rendered shared/_messages.html.erb (0.1ms)
Rendered feedbacks/_new.html.erb (15.8ms)
Completed 200 OK in 40880ms (Views: 40856.2ms | ActiveRecord: 18.8ms)
It's a table, where each row is a partial, extracted so that I could see what took time.
Here it is after the timeout. 1398ms
Started GET "/my_models" for xxx.160.183.xxx at 2013-10-31 15:21:18 +0000
Processing by MyModelsController#index as HTML
User Load (2.4ms) SELECT "users".* FROM "users" WHERE "users"."id" = 110237164 LIMIT 1
MyModel Load (3.2ms) SELECT "my_models".* FROM "my_models" ORDER BY name ASC
Rendered my_models/_my_model_statuses.html.erb (87.7ms)
Rendered my_models/_my_model_statuses.html.erb (25.5ms)
Rendered my_models/_my_model_statuses.html.erb (35.7ms)
Rendered my_models/_my_model_statuses.html.erb (107.8ms)
Rendered my_models/_my_model_statuses.html.erb (163.9ms)
Rendered my_models/_my_model_statuses.html.erb (26.2ms)
Rendered my_models/_my_model_statuses.html.erb (26.3ms)
Rendered my_models/_my_model_statuses.html.erb (19.3ms)
Rendered my_models/_my_model_statuses.html.erb (121.4ms)
Rendered my_models/_my_model_statuses.html.erb (62.6ms)
Rendered my_models/_my_model_statuses.html.erb (94.9ms)
Rendered my_models/_my_model_statuses.html.erb (8.2ms)
Rendered my_models/_my_model_statuses.html.erb (19.1ms)
Rendered my_models/_my_model_statuses.html.erb (15.8ms)
Rendered my_models/_my_model_statuses.html.erb (59.9ms)
Rendered my_models/_my_model_statuses.html.erb (20.6ms)
Rendered my_models/_my_model_statuses.html.erb (78.8ms)
Rendered my_models/_my_model_statuses.html.erb (25.3ms)
Rendered my_models/_my_model_statuses.html.erb (13.3ms)
Rendered my_models/_my_model_statuses.html.erb (130.1ms)
Rendered my_models/_my_model_statuses.html.erb (9.7ms)
Rendered my_models/_my_model_statuses.html.erb (11.1ms)
Rendered my_models/_my_model_statuses.html.erb (92.0ms)
Rendered my_models/_my_model_statuses.html.erb (51.6ms)
Rendered my_models/_my_model_statuses.html.erb (23.1ms)
Rendered my_models/_my_models_overview.html.erb (1385.7ms)
Rendered my_models/index.html.erb within layouts/application (1386.3ms)
Rendered shared/_menu.html.erb (0.3ms)
Rendered shared/_usermenu.html.erb (0.6ms)
Rendered shared/_header.html.erb (1.5ms)
Rendered shared/_messages.html.erb (0.0ms)
Rendered feedbacks/_new.html.erb (1.5ms)
Completed 200 OK in 1398ms (Views: 1388.8ms | ActiveRecord: 5.5ms)
I'm not using any explicit cache, just plain Rails ActiveRecord to populate the table. Locally it's not like this at all. Heroku support recommends concurrency and caching, which is for a high-traffic app. This is a low-traffic app, a tool. They haven't commented on the bamboo vs cedar topic so I'm reaching out here in case anybody learns things I'm not aware of.
In both cases DB < 20 ms, so this is all CPU stuff, rendering templates.
It's how heroku handles resources. It's called dyno-idling and they released this about it last year: https://devcenter.heroku.com/articles/dynos#dyno-idling.
The heorku way to fix it is to pay for an extra dyno, then it will run all the time. But you can also use ping services to ping your site. Some will do it every hour, some every fifteen minutes and some once a minute. That will keep your dyno active and your thin server always "awake"
Yeah... I'll share instead of deleting the question. Hopefully my mistakes will help somebody get to the bottom of this faster.
The problem was with a mongodb query. Not sleeping dynos or services. Simply my own code.
It as Very confusing as MongoHQ has a "Slow queries" page that reported "no slow queries". I'd like to applaud Matt McCay for the excellent support and respectful approach, despite this being a (yet another?) "user error". He showed me the mongodb logging that showed this:
Tue Nov 5 22:59:46.722 [conn2335303] query myapp-production.day_reports query: { $query: { my_model_id: 2147259060, some_unindexed_field: { $gt: 0 } }, $orderby: { some_index_timestamp: -1 } } ntoreturn:1 ntoskip:0 nscanned:775 scanAndOrder:1 keyUpdates:0 locks(micros) r:7474265 nreturned:0 reslen:20 7474ms
Tue Nov 5 22:59:49.439 [conn2335303] query myapp-production.day_reports query: { $query: { my_model_id: 2147259071, some_unindexed_field: { $gt: 0 } }, $orderby: { some_index_timestamp: -1 } } ntoreturn:1 ntoskip:0 nscanned:502 scanAndOrder:1 keyUpdates:0 locks(micros) r:2570470 nreturned:0 reslen:20 2570ms
Show just two of many (30) in the same query. Note the 7474ms and 2570ms
I wonder why these wouldn't be listed as "slow queries" in MongoHQ... Let this be a warning to not let yourself be fooled or blinded by that.
Solution: In my case, I solved this by denormalizing, I saved the calculated value on my model on each write. I could have used indexes too, it would have been really fast, but in this case the models I read are in PostgreSQL and with the denormalization I wouldn't need to connect to MongoDB at all - on this particular request.
Why was it so much faster on the following queries?
The answer is found here: http://docs.mongodb.org/manual/faq/fundamentals/#does-mongodb-handle-caching
MongoDB saves/caches the last fetched data, in RAM.
How should I have troubleshooted this, ideally?
Isolate and eliminate. I should have plugged in and out both heroku postgresql and mongohq to rule them out. The "slow first time, fast after" made me fixate on server issues, I should know better.
Study the logs more thoroughly.

406 error when template and partial exist as well as when they don't exist

I'm getting a 406 error and it's rendering a blank white page, even though the log makes it look like it's rendering my shared/404 page. I can even remove the layout and 404 page and same error. Any ideas?
Started GET "/" for 127.0.0.1 at 2012-02-19 22:26:56 -0800
Processing by MinisitesController#show as HTML
Account Load (0.4ms) SELECT "accounts".* FROM "accounts" WHERE "accounts"."full_domain" = 'test.eg.local' AND (accounts.deleted_at IS NULL) LIMIT 1
Rendered shared/404.html.haml (0.3ms)
Completed 406 Not Acceptable in 81ms (Views: 12.3ms | ActiveRecord: 2.2ms)

ActionView errors when pushing to Heroku

I'm building my first webapp, following the Michael Hartl Rails Tutorial. After finishing Chapter 10, I find that my application works locally but does not work when I push it to Heroku. The development log output reads:
Started GET "/" for 98.154.183.51 at Thu Jan 06 01:07:15 -0800 2011
Processing by PagesController#home as HTML
Rendered layouts/_header.html.erb (28.3ms)
Rendered pages/home.html.erb within layouts/application (30.9ms)
Completed in 35ms
User Load (29.6ms) SELECT "users".* FROM "users" WHERE ("users"."id" = 101) LIMIT 1
CACHE (0.0ms) SELECT "users".* FROM "users" WHERE ("users"."id" = 101) LIMIT 1
Rendered layouts/_header.html.erb (149.3ms)
CACHE (0.0ms) SELECT "users".* FROM "users" WHERE ("users"."id" = 101) LIMIT 1
Rendered layouts/_sidebar.html.erb (8.1ms)
Rendered layouts/_footer.html.erb (0.6ms)
Rendered sessions/new.html.erb within layouts/application (342.4ms)
Completed 200 OK in 421ms (Views: 390.1ms | ActiveRecord: 29.9ms)
Whereas the heroku production log reads:
Started GET "/" for 98.154.183.51 at Thu Jan 06 01:07:15 -0800 2011
Processing by PagesController#home as HTML
Rendered layouts/_header.html.erb (28.3ms)
Rendered pages/home.html.erb within layouts/application (30.9ms)
Completed in 35ms
ActionView::Template::Error (PGError: ERROR: invalid input syntax for integer: "nil"
: SELECT "users".* FROM "users" WHERE ("users"."id" = 'nil') LIMIT 1):
I'm not sure why the Heroku-deployed app looks for a user_id of "nil", whereas my local app looks for one of "101." This is my first app, so I'm not sure where to even begin the troubleshooting. I've tried pushing my local db to the Heroku one via heroku rake db:push, but I still get the same errors. Any advice? Thanks in advance for your help.
Make sure you
Migrated your database on Heroku with heroku rake db:migrate
Have a valid user in your database (you could e.g. create one with the Rails console on Heroku)
It seems your code is trying to load a User instance when you have none in your database. If your app works once you have some users in your database, fix your code to act appropriately if no users are present.
You might want to run you rake db:seed also if you have setup some initial values.

Resources