I am running a Rails 5.1.5 app with webpacker 3.2.2, postgres 9.4, apache2, and passenger. I am not sure what other info is relevant here but can edit as necessary.
My app works locally in development. After I deploy to the server, it will work but eventually gives a 500 error on certain pages from 2 different model/controller actions.
Here is the access log that does not give much info:
71.143.443.562 - - [28/Feb/2018:19:15:11 -0700] "GET /posts/jewelry-appraisals-got-even-easier HTTP/1.1" 500 497 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML,$
I am not sure why and do not know how to reproduce the error (or where to find it). Any ideas? Thank you!
Update -
Here is the apache2 error.log
[ N 2018-02-28 15:56:07.0526 1243/T3 age/Cor/CoreMain.cpp:917 ]: Checking whether to disconnect long-running connections for process 1416, application /var/www/sellmyjewelry/current (productio$
here is my rails production.log:
I, [2018-03-01T02:59:04.227988 #6241] INFO -- : [a45553e8-7b89-45ad-ba05-dade585c17d4] Completed 401 Unauthorized in 2ms (ActiveRecord: 0.4ms)
F, [2018-03-01T02:59:04.228965 #6241] FATAL -- : [a45553e8-7b89-45ad-ba05-dade585c17d4]
F, [2018-03-01T02:59:04.229058 #6241] FATAL -- : [a45553e8-7b89-45ad-ba05-dade585c17d4] NoMethodError (To respond to a custom format, register it as a MIME type first: http://guides.rubyonrails.org$
F, [2018-03-01T02:59:04.229093 #6241] FATAL -- : [a45553e8-7b89-45ad-ba05-dade585c17d4]
F, [2018-03-01T02:59:04.229149 #6241] FATAL -- : [a45553e8-7b89-45ad-ba05-dade585c17d4] actionpack (5.1.5) lib/abstract_controller/collector.rb:26:in `method_missing
'
Update #2
I made it so all the errors are local. I get a "Mime-Type" error on pages that use the method .html_safe on longer pieces of content. I added mime types in the initialize file but still get the error.
Here is a screenshot of the error I get:
Thanks for the help.
Related
I have a Rails 5.2.3 application with Ruby 2.4.5.
I found a weird issue that the request info are logged twice in stdout.
Here is the log config in config/environments/product.rb
config.logger = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
config.log_tags = [ lambda {|req| "#{req.cookie_jar["_session_id"]}" }, :remote_ip, :uuid ]
Supposedly it will tag every log with the remote ip and request uuid, and it does for most of logs, except there is a weird additional log for the request. In following example, the last line is a duplicated log for the request without the tag:
[INFO] [2019-10-28 06:11:45 UTC] [127.0.0.1] [f6de1900-a7e5-4486-8b73-7095d0cacb35] Started GET "/api/v1/nodes?pageSize=20&pageNumber=1" for 127.0.0.1 at 2019-10-28 06:11:45 +0000
[INFO] [2019-10-28 06:11:45 UTC] [127.0.0.1] [f6de1900-a7e5-4486-8b73-7095d0cacb35] Processing by Api::V1::NodesController#index as XML
...
...
[INFO] [2019-10-28 06:20:02 UTC] [127.0.0.1] [993e0db4-3995-41ef-851a-bfea1bc25781] Completed 200 OK in 1084ms (Views: 341.9ms | ActiveRecord: 150.6ms)
127.0.0.1 - - [28/Oct/2019:06:20:02 +0000] "GET /api/v1/nodes?pageSize=20&pageNumber=1 HTTP/1.1" 200 - 1.1347
I checked configurations, there is no any other logger configured.
The consequence is that there is a timer in client side to check notification status by every 5 seconds. I added a log silencer to avoid log such request:
# config/application.rb
config.middleware.insert_before Rails::Rack::Logger, LogSilencer, silenced: /notification_messages/
# lib/log_silencer.rb
class LogSilencer
def initialize(app, opts = {})
#app = app
#silenced = opts.delete(:silenced)
end
def call(env)
if #silenced.match(env['PATH_INFO'])
Rails.logger.silence do
#app.call(env)
end
else
#app.call(env)
end
end
end
It does avoid logging from the tagged logger, but the duplicated request logs are still there, then the stdout is full of this request
127.0.0.1 - - [28/Oct/2019:07:04:21 +0000] "GET /api/v1/notification_messages/to_notify HTTP/1.1" 200 - 0.0118
127.0.0.1 - - [28/Oct/2019:07:04:26 +0000] "GET /api/v1/notification_messages/to_notify HTTP/1.1" 200 - 0.0140
127.0.0.1 - - [28/Oct/2019:07:04:31 +0000] "GET /api/v1/notification_messages/to_notify HTTP/1.1" 200 - 0.0137
127.0.0.1 - - [28/Oct/2019:07:04:36 +0000] "GET /api/v1/notification_messages/to_notify HTTP/1.1" 200 - 0.0149
...
Spent a whole day try to find out who generated this log, but no any clue...
Would like to ask for help on how to turn off this log so make the log data clear...
Thanks!
Seems the logs come from the web server used when up the Rails app.
When use WEBrick, the log is like
=> Booting WEBrick
=> Rails 5.2.3 application starting in development on http://localhost:4000
=> Run `rails server -h` for more startup options
[2019-10-30 07:01:13] INFO WEBrick 1.3.1
[2019-10-30 07:01:13] INFO ruby 2.4.5 (2018-10-18) [x86_64-linux]
[2019-10-30 07:01:13] INFO WEBrick::HTTPServer#start: pid=5812 port=4000
127.0.0.1 - - [30/Oct/2019:07:01:17 UTC] "GET /test HTTP/1.1" 304 0
- -> /test
When use unicorn, the log is like:
I, [2019-10-30T07:02:47.626962 #5956] INFO -- : Refreshing Gem list
I, [2019-10-30T07:02:48.289373 #5956] INFO -- : listening on addr=0.0.0.0:4000 fd=17
I, [2019-10-30T07:02:48.393598 #5956] INFO -- : master process ready
I, [2019-10-30T07:02:48.394853 #5965] INFO -- : worker=0 ready
I, [2019-10-30T07:02:48.399878 #5968] INFO -- : worker=1 ready
127.0.0.1 - - [30/Oct/2019:07:02:52 +0000] "GET /test HTTP/1.1" 304 - 0.0724
I can find that the request log format is a bit different (the time zone info, the time consumed etc).
I use Unicorn, its logger by default uses stderr, but seems the logger configuration does not work
logger Logger.new("#{rails_root}/log/unicorn.log")
so I have to set the stderr path
stderr_path "#{rails_root}/log/unicorn.stderr.log
Then the request logs are in the unicorn.stderr.log file, and STDOUT are the rails app logs.
But still dont know how to turn it off since it is a kind of duplicated and useless log...
I have enabled my GitLab server to use the GitLab registry which is hosted on another server. It was installed from source.
The GitLab admin console indicates that the registry is enabled and active, however when I try to access the registry a 500 error is thrown.
The production logs show the following
Started GET "/root/Test/container_registry" for 10.188.144.64 at 2016-12-01 11:42:00 +0000
Processing by Projects::ContainerRegistryController#index as HTML
Parameters: {"namespace_id"=>"root", "project_id"=>"Test"}
Completed 500 Internal Server Error in 61ms (ActiveRecord: 4.0ms)
TypeError (no implicit conversion of nil into String):
lib/json_web_token/rsa_token.rb:20:in `read'
lib/json_web_token/rsa_token.rb:20:in `key_data'
lib/json_web_token/rsa_token.rb:24:in `key'
lib/json_web_token/rsa_token.rb:28:in `public_key'
lib/json_web_token/rsa_token.rb:33:in `kid'
lib/json_web_token/rsa_token.rb:12:in `encoded'
app/services/auth/container_registry_authentication_service.rb:30:in `full_access_token'
app/models/project.rb:463:in `container_registry_repository'
app/controllers/projects/container_registry_controller.rb:28:in `container_registry_repository'
app/controllers/projects/container_registry_controller.rb:8:in `index'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
Any advice or guidance with fixing this would be welcome!
Thanks in advance.
I'm making an expensive call to my Heroku Rails server. After 13-15 seconds the console in my browser reports a Service Unavailable 503 error. However when I check my heroku logs, it reports:
Completed 200 OK in 45592ms (Views: 220.3ms | ActiveRecord: 33457.5ms)
Other times the heroku logs report a memory exceeded quota. Here is an example of that below.
2015-06-11T15:17:20.238285+00:00 app[web.1]: Completed 200 OK in 81881ms (Views: 201.6ms | ActiveRecord: 18021.2ms)
2015-06-11T15:17:33.482930+00:00 heroku[web.1]: Process running mem=841M(164.4%)
2015-06-11T15:17:33.482930+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-06-11T15:17:53.147570+00:00 heroku[web.1]: Process running mem=841M(164.4%)
2015-06-11T15:17:53.147679+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-06-11T15:17:59.751540+00:00 app[web.1]: E, [2015-06-11T15:17:59.695813 #3] ERROR -- : worker=2 PID:13 timeout (121s > 120s), killing
2015-06-11T15:17:59.916750+00:00 app[web.1]: E, [2015-06-11T15:17:59.906435 #3] ERROR -- : reaped #<Process::Status: pid 13 SIGKILL (signal 9)> worker=2
2015-06-11T15:18:02.487428+00:00 app[web.1]: I, [2015-06-11T15:18:02.427293 #16] INFO -- : worker=2 ready
Why is it reporting a Completed 200 when the console is reporting a 503?
there are 2 different things here:
your app
the heroku load balancer
In this case the load balancer see that the request is taking too long and sends you the 503. In the back the service processes the request and returns completes with a 200.
See:
https://devcenter.heroku.com/articles/limits
https://devcenter.heroku.com/articles/request-timeout
Heroku does not gives us much information for error-code = H10. Simple put, something is going wrong with your application code / configuration. To see what's going wrong, run heroku run rails console and you will be able to see error details which will be help you in resolving the error. No need to use logs. heroku run rails console is a big help
This happened with me when I used Node. I put a timeout for 30 seconds and I used to get HTTP:503. I then came to know that it's the timeout that caused the issue. I changed the timeout to <30 seconds and it is working fine. Maybe it's because of the loadbalancer that Heroku uses.
Thanks.
The behaviour is: the browser hangs keep loading without any response back.
Please see the following log.
I, [2014-07-16T14:14:09.766062 #11713] INFO -- : Started GET "/somepage" for xxx.xxx.xxx.xxx at 2014-07-16 14:14:09 +0800
I, [2014-07-16T14:14:40.872411 #31111] INFO -- : Started GET "/" for yyy.yyy.yyy.yyy at 2014-07-16 14:14:40 +0800
I, [2014-07-16T14:14:40.874968 #31111] INFO -- : Processing by HomeController#index as */*
I, [2014-07-16T14:14:40.893189 #31111] INFO -- : Rendered ...
I, [2014-07-16T14:14:40.901553 #31111] INFO -- : Completed 200 OK in 26ms (Views: 22.5ms | ActiveRecord: 2.6ms)
... more other requests handled normally
I, [2014-07-16T14:29:35.395493 #11713] INFO -- : Processing by HomeController#somepage as HTML
I, [2014-07-16T14:29:35.403614 #11713] INFO -- : Rendered home/somepage.html.erb (1.5ms)
I, [2014-07-16T14:29:35.403762 #11713] INFO -- : Completed 200 OK in 8ms (Views: 5.0ms | ActiveRecord: 0.5ms)
In between the "Started GET ..." and "Processing by" logs, the rails app can handle other requests normally. However, the 2 logs (should be from the same request) have around 15mins gap.
Server: Apache with passenger
Database: Mysql using mysql2 adapter
I completely lost and have no clues what is the root cause....
I faced this similar kind of issue where the browser kept on loading with no logs in the log file.And what i found out was that we have mistakenly written find.all in one of our model as during testing it only had 1000 records as compared to production with more than 1 million records.so changing it to batch records/pagination resolved it.
...........Hope this might help you.
Using carrier wave to upload a pdf document via CarrierWave. Then using RMagick to save the first page of the pdf as a thumbnail.jpg. I have this working correctly in my development environment (Mac OS X 10.9), however its failing in production (Nginx + Unicorn on Ubuntu 12.04 LTS).
It appears that the upload completes successfully, based on watching the upload percentage in the browser status bar. However, something errors in the processing, and I get Rails error page. The problem is, there is nothing in my production.log.
I get the following in unicorn.log:
I, [2013-12-12T01:22:55.232431 #2935] INFO -- : worker=1 ready
I, [2013-12-12T01:22:55.246992 #2932] INFO -- : worker=0 ready
E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing
E, [2013-12-12T01:26:00.676250 #2929] ERROR -- : reaped #<Process::Status: pid 2935 SIGKILL (signal 9)> worker=1
I, [2013-12-12T01:26:00.676382 #2929] INFO -- : worker=1 spawning...
I, [2013-12-12T01:26:00.681087 #3502] INFO -- : worker=1 spawned pid=3502
I, [2013-12-12T01:26:00.681399 #3502] INFO -- : Refreshing Gem list
SECURITY WARNING: No secret option provided to Rack::Session::Cookie.
This poses a security threat. It is strongly recommended that you
provide a secret to prevent exploits that may be possible from crafted
cookies. This will not be supported in future versions of Rack, and
future versions will even invalidate your existing user cookies.
Called from: /home/deployer/apps/______/shared/bundle/ruby/1.9.1/gems/actionpack-3.2.9/lib/action_dispatch/middleware/session/abstract_store.rb:28:in `initialize'.
And this in my nginx error.log
2013/12/12 01:26:00 [error] 2597#0: *9 upstream prematurely closed connection while reading response header from upstream, client: ###.###.###.###, server: ___.org, request: "POST /admin/explores HTTP/1.1", upstream: "http://unix:/tmp/unicorn.sock:/admin/explores", host: "___.org", referrer: "https://___.org/admin/explores/new"
Not sure how to get a better error description of what's going on. Would appreciate any help.
Thanks!
Ruby 1.9.3 + Rails 3.2.9
The line E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing from your unicorn.log tells you the problem: you have a timeout. Increase unicorn's timeout to 60 seconds by setting timeout 60 in your unicorn.rb. More information: http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-timeout