I'm utterly stumped as to why I'm not able to see the Rails controller outputs in my development log. I've spent days beating my head against a wall trying to figure this out and I'm not sure what else to try.
Setup: Rails 5.2.3 app running ruby 2.6.3 via docker-compose.
It started with me not being able to see my app logs when running docker logs <container-name>. However, I soon realized that I was able to see the output from puma starting and a shell script that ran rake tasks that the issue might be with rails.
To help assist with finding the issue:
Tore down and rebuilt the docker environment, several times
Stopped writing via STDOUT in favor of logs/development.log
Disabled lograge and elastic-apm, just in case
Reverted my development.rb config back to what's generated with a rails new
Followed the suggestions here
However, when running the rails console via docker exec -it <container-name>:
Running Rails.logger.level returns 2 which is warn, despite the default logging level being dev
I'm able to see log output when running Rails.logger.warn 'foo'
After setting Rails.logger.level = 0 I'm able to see output when running Rails.logger.debug 'foo'
I tried setting the value explicitly as config.log_level = :debug in development.rb yet it still set itself to the warn level.
However, I'm still not able to see any logs when navigating the application. Any thoughts?
Ugh. I feel like the biggest schmuck but I've figured out the issue.
I went back though source-control to see what has changed recently. In addition to the elastic-apm gem, I also added the Unleash gem.
I went to check out it's configuration and it looks like following their recommenced configuration causes logging to break. The line that was specifically causing offense was in the unleash initializer setting config.logger = Rails.logger
Related
Ruby 2.3.6, Rails 4.2.6, Stackdriver 0.15.0
Following the directions at https://cloud.google.com/logging/docs/setup/ruby, I've added the stackdriver gem to Gemfile. From what I can tell, I shouldn't need to do anything else.
However, I'm not seeing any log messages from within Rails showing up in Stackdriver. I've even tried execing a shell in the container, running bundle exec rails console, and explicitly logging with Rails.logger.error "this is a log message".
config.google_cloud.use_logging = true and config.log_level = :debug, FWIW.
UPDATE: I took a closer look at the logger instance with pp Rails.logger, and it contains this:
#resource=
#<Google::Cloud::Logging::Resource:0x0000558b075d1f50
#labels={:cluster_name=>"onehq-cluster", :namespace_id=>"default"},
#type="container">,
My container is running in the staging namespace, which would explain why I'm not seeing log messages. When I look in the Stackdriver log viewer in the default namespace, lo, there are logs. Unfortunately, sending all messages to default won't do me any good, because I have apps running in two namespaces, and I need to be able to tell them apart.
I tried adding additional configuration:
config.google_cloud.logging.monitored_resource.type = "container"
config.google_cloud.logging.monitored_resource.labels = { cluster_name: "my-cluster", namespace_id: "staging" }
And now Rails.logger.error "this is a log message" doesn't show up anywhere.
UPDATE 2: After much web searching, I managed to turn up https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2025 which suggests that the namespace to which log messages are sent is determined by the value of an environment variable, and that said env var is not being set automatically in GKE containers. (It has supposedly been resolved by https://github.com/GoogleCloudPlatform/google-cloud-ruby/pull/2071 but I don't think that's live yet, at least not in my cluster.) I'll try forcing the env var in my deployment file, and if that works I'll convert the last part of this to an answer.
After much web searching, I managed to turn up https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2025 which suggests that the namespace to which log messages are sent is determined by the value of an environment variable, and that said env var is not being set automatically in GKE containers. (It has supposedly been resolved by https://github.com/GoogleCloudPlatform/google-cloud-ruby/pull/2071 but I don't think that's live yet, at least not in my cluster.) I forced the env var in my deployment file, and IIRC that worked. (I no longer have access to that cluster, so I can't verify that.)
Google-Cloud-Env gem has released this morning the version (1.0.2) you may update your bundle.
When I tail the logs and hit my app, I have heroku[router] telling me I received a 500, and the app shows the usual "We're sorry, but something went wrong" message... but there are no additional details from Rails in the logs to tell me what is causing the 500.
I've read a couple of other threads that asked about this but they mostly end with people suggesting to do a rake:db migrate which fixed the error but there's no further discussion about the logging unfortunately. (And yes I'm sure there's no pending migrations 😄)
I have the rails_12factor gem installed
I've tried setting config.log_level = :debug in production.rb
I've tried creating a config variable of LOG_LEVEL set to DEBUG (which should supersede the config setting)
I've set $stdout.sync = true in config.ru as suggested here
What am I missing?
As an aside, have tried re-producing my actual problem locally with a backup of the database being used with Heroku but I can't reproduce it, so I'm finally forced to get to the bottom of this logging problem.
I first just got this to work by manually adding:
config.logger = Logger.new(STDOUT)
to production.rb but that obscured the real issue...
RACK_ENV and RAILS_ENV were both set to staging while I had the gem loading in the :production set... So although Heroku told me it was using the rails_12factor gem when the instance started, it wasn't really (I assumed it wouldn't be listed there if it wasn't a production environment, wrong assumption!)
staging.rb was loading production.rb which is why setting STDOUT there worked, thus confusing the issue further.
Thanks for the replies, and hope this helps someone else in future - go check your config variables and/or change where the gem is being loaded! :)
Debugging errors on heorku can get difficult most of the time.
For people facing a similar problem I suggest using PaperTrail heroku add-on for logging, It is free upto a certain limit.
It is realtime and provides with searching of logs, events hooks, mailing alerts etc. facilities. You will never have trouble debugging errors on heroku.
You can simply install it using the heroku cli command.
heroku addons:create papertrail
My rails app running in development environment stopped logging all of a sudden and I am not able to figure out why.
I tried logging into a new file by doing
config.logger = Logger.new('log/temp.log')
config.log_level = :debug
But still no luck. The new file temp.log was created but nothing is logged in the file. The thing is this happens on my development server running nginx (I run my rails app using "rails s -d" on this server). The exact same files, when I run on my local machine (my own computer), logging works fine.
So I feel the reason logging is not working is because of something specific to the server, but then I didn't do anything much on the server (e.g. I didn't install new gems, etc.) Logging has been working fine until few days ago.
When I go to rails console
rails c
> Rails.logger.debug "hello"
=> true
I do get "hello" logged into 'log/temp.log' specified above in config file.
I think permission on log directory or file is ok. What else could be wrong?
I believe it's a locking issue which you might be able to solve after removing the call to the logger which causes this.
I ran into this issue with Redmine 1.x,
I found newrelic_rpm entry in the production.log, saying it didn't run, and 1 line of a Redmine plugin init.
After removing both, newrelic_rpm from the environment.rb (config.gem line), and the plugin logger init message, the logging facility appears to be restored and log entries are appearing again.
I'm having a real tough time diagnosing a 500 error from my application running in production. I've had it working before, but after re-deploying via Capastrano I am unable to get it going.
Here are the facts:
The server is setup with nginx + passenger, and I'm using
PostgreSQL.
Static assets are working properly, as in I'm able to access them just fine in a browser.
I can access the rails console via RAILS_ENV=production bundle exec rails console and perform Active Record actions (like
retrieving data from the db).
Within console, I can run app.get("/"), which returns a 500 error as well (after first showing the query that was run to load
the model).
The production.log file is never written to. I've set permissions 777 on it just for the hell of it. I've also set the log level to
:debug with nothing to show for it.
The nginx log (which passenger also uses) shows no indication of errors, it just notifies about cache misses.
Because nothing of use is being logged, I have no idea what to do here. I've tried setting full permission on the entire app directory with no help. Restarted the server multiple times, nothing. The database is there and rails can clearly communicate with it. I'm not sure what I did to get it to run the first time around. I just don't know why rails isn't outputting anything to the log.
Okay, I figured this out. The app ran fine in development mode, so I knew something production-specific was screwing it up. I went into config/environments/production.rb and changes these settings:
# Full error reports are disabled and caching is turned on
config.consider_all_requests_local = false # changed from true
config.action_controller.perform_caching = true # changed from false
And then after restarting passenger, rails showed me the error w/ stacktrace. Turns out I was forgetting to precompile the asset pipeline!
Things to check
1) Are you sure you are running in production environment?
Check to see if any entries are in the development.log file
2) Set up your app to email you when a 500 error occurs with a full stack trace. I use the Exception Notifier gem but there are plenty of other solutions for this.
3) When checking your app in the console are you sure you are starting the console in production mode? It is possible that the app is not starting up at all and you just forgot to set the production param thereby thinking that the app runs fine when it doesn't.
4) Are you getting an nginx 500 error or the Rails 500 error? If nginx then it is likely that your app is not starting at all and highly unlikely that you will get any rails error in your log file. The assets are static files and navigating to them proves nothing other than that they exist.
5) Are you sure you are checking the right folder on the server? Sounds really stupid but capistrano could be deploying the app to a folder that is different to the folder that nginx is looking for for your app so double check both the folder capistrano is deploying to and the folder that nginx is looking for are the same.
Just a suggestion, I would use puma rather than passenger. It's awesome with nginx.
My problem is passenger's log file (error.log) has nothing. Then it's a rotation log issue. Run
passenger-config reopen-logs
solved my problem. More.
Have you tried running in development mode to see if the error reports itself?
I wrote a demo HelloWorld Rails app and tested it with WEBrick (it doesn't even use a DB, it's just a controller which prints "hello world"). Then I tried to deploy it to a local Apache powered with Passenger. In fact this test is just to get Passenger working (it's my first deploy on Apache). Now I'm not even sure that Passenger works, but I don't get any error on the Apache side.
When I fire http://rails.test/ the browser shows the Rails 500 error page - so I assume that Passenger works. I want to investigate the logs, but it happens that production.log is empty! I don't think it's a permission problem, because if I delete the file, it is recreated when I reload the page. I tried to change the log level in conf/environments/production.rb, tried to manually write to log file with Rails console production and
Rails.logger.error('asdf')
it returns true but nothing gets written to production.log. The path (obtained per Rails.logger.inspect) is correct, and I remark that the file is recreated if I manually remove it. How can I know what's going on?
(I already checked the Apache logs, plus I set the highest debug level for Passenger but it seems a Rails problem, so is not logged by the server)
Assuming you're running Rails 3.2.1, this is a bug. It was patched in 3.2.2.
If you can't upgrade to 3.2.2 for any reason, this comment on GitHub has a workaround:
# config/initializers/patch_rails_production_logging.rb
Rails.logger.instance_variable_get(:#logger).instance_variable_get(:#log_dest).sync = true if Rails.logger
Setting this works on Rails 3.2.11:
Rails.logger = ActiveSupport::BufferedLogger.new(Rails.root.join("log","production.log"))