Unknown slow process of requests - ruby-on-rails

The behaviour is: the browser hangs keep loading without any response back.
Please see the following log.
I, [2014-07-16T14:14:09.766062 #11713] INFO -- : Started GET "/somepage" for xxx.xxx.xxx.xxx at 2014-07-16 14:14:09 +0800
I, [2014-07-16T14:14:40.872411 #31111] INFO -- : Started GET "/" for yyy.yyy.yyy.yyy at 2014-07-16 14:14:40 +0800
I, [2014-07-16T14:14:40.874968 #31111] INFO -- : Processing by HomeController#index as */*
I, [2014-07-16T14:14:40.893189 #31111] INFO -- : Rendered ...
I, [2014-07-16T14:14:40.901553 #31111] INFO -- : Completed 200 OK in 26ms (Views: 22.5ms | ActiveRecord: 2.6ms)
... more other requests handled normally
I, [2014-07-16T14:29:35.395493 #11713] INFO -- : Processing by HomeController#somepage as HTML
I, [2014-07-16T14:29:35.403614 #11713] INFO -- : Rendered home/somepage.html.erb (1.5ms)
I, [2014-07-16T14:29:35.403762 #11713] INFO -- : Completed 200 OK in 8ms (Views: 5.0ms | ActiveRecord: 0.5ms)
In between the "Started GET ..." and "Processing by" logs, the rails app can handle other requests normally. However, the 2 logs (should be from the same request) have around 15mins gap.
Server: Apache with passenger
Database: Mysql using mysql2 adapter
I completely lost and have no clues what is the root cause....

I faced this similar kind of issue where the browser kept on loading with no logs in the log file.And what i found out was that we have mistakenly written find.all in one of our model as during testing it only had 1000 records as compared to production with more than 1 million records.so changing it to batch records/pagination resolved it.
...........Hope this might help you.

Related

Redmine very slow on first page load after some time of inactivity

Every morning, the first person that tries to access Redmine at my work is waiting almost one minute for redmine to load the page. After that, the rest of the team have no problem and almost every page are loading in less than 200ms.
Here is the log between the last "good" loading and the first "bad" loading of this morning:
App 103 output: I, [2021-04-16T00:19:24.965247 #103] INFO -- : Completed 200 OK in 829ms (Views: 553.2ms | ActiveRecord: 1.8ms)
App 103 output: I, [2021-04-16T03:09:12.511395 #103] INFO -- : Started GET "/login?back_url=https%3A%2F%2Fmyapp.com%2F" for WWW.XXX.YYY.ZZZ at 2021-04-16 03:09:12 +0000
App 103 output: I, [2021-04-16T03:09:12.532015 #103] INFO -- : Processing by AccountController#login as HTML
App 103 output: I, [2021-04-16T03:09:12.532084 #103] INFO -- : Parameters: {"back_url"=>"https://myapp.com/"}
App 103 output: I, [2021-04-16T03:09:12.694146 #103] INFO -- : Redirected to https://myapp.com/login?back_url=https%3A%2F%2Fmyapp.com%2Flogin%3Fback_url%3Dhttps%253A%252F%252Fmyapp.com%252F
App 103 output: I, [2021-04-16T03:09:12.694280 #103] INFO -- : Filter chain halted as :session_expiration rendered or redirected
App 103 output: I, [2021-04-16T03:09:12.694473 #103] INFO -- : Completed 302 Found in 162ms (ActiveRecord: 2.7ms)
App 103 output: I, [2021-04-16T03:09:13.201746 #103] INFO -- : Started GET "/login?back_url=https%3A%2F%2Fmyapp.com%2Flogin%3Fback_url%3Dhttps%253A%252F%252Fmyapp.com%252F" for WWW.XXX.YYY.ZZZ at 2021-04-16 03:09:13 +0000
App 103 output: I, [2021-04-16T03:09:13.205997 #103] INFO -- : Processing by AccountController#login as HTML
App 103 output: I, [2021-04-16T03:09:13.206088 #103] INFO -- : Parameters: {"back_url"=>"https://myapp.com/login?back_url=https%3A%2F%2Fmyapp.com%2F"}
App 103 output: I, [2021-04-16T03:09:13.283815 #103] INFO -- : Current user: anonymous
App 103 output: I, [2021-04-16T03:09:13.300562 #103] INFO -- : Rendering account/login.html.erb within layouts/base
App 103 output: I, [2021-04-16T03:09:13.304521 #103] INFO -- : Rendered account/login.html.erb within layouts/base (3.8ms)
App 103 output: I, [2021-04-16T03:09:13.526917 #103] INFO -- : Completed 200 OK in 321ms (Views: 242.1ms | ActiveRecord: 50.2ms)
App 103 output: I, [2021-04-16T03:09:15.396901 #103] INFO -- : Started POST "/login" for WWW.XXX.YYY.ZZZ at 2021-04-16 03:09:15 +0000
App 103 output: I, [2021-04-16T03:09:15.400666 #103] INFO -- : Processing by AccountController#login as HTML
App 103 output: I, [2021-04-16T03:09:15.400741 #103] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"*********************************", "username"=>"myuser", "password"=>"[FILTERED]", "login"=>"Connexion"}
App 103 output: I, [2021-04-16T03:09:15.461825 #103] INFO -- : Current user: anonymous
App 103 output: I, [2021-04-16T03:09:16.117981 #103] INFO -- : Successful authentication for 'myuser' from WWW.XXX.YYY.ZZZ at 2021-04-16 03:09:16 UTC
App 103 output: I, [2021-04-16T03:09:16.454431 #103] INFO -- : Redirected to https://myapp.com/my/page
App 103 output: I, [2021-04-16T03:09:16.454677 #103] INFO -- : Completed 302 Found in 1054ms (ActiveRecord: 778.5ms)
App 103 output: I, [2021-04-16T03:09:16.929227 #103] INFO -- : Started GET "/my/page" for WWW.XXX.YYY.ZZZ at 2021-04-16 03:09:16 +0000
App 103 output: I, [2021-04-16T03:09:17.650222 #103] INFO -- : Processing by MyController#page as HTML
App 103 output: I, [2021-04-16T03:09:18.625577 #103] INFO -- : Current user: myuser (id=208)
App 103 output: I, [2021-04-16T03:09:18.923542 #103] INFO -- : Rendering my/page.html.erb within layouts/base
App 103 output: I, [2021-04-16T03:09:35.622052 #103] INFO -- : Rendered queries/_columns.html.erb (509.9ms)
App 103 output: I, [2021-04-16T03:09:39.698459 #103] INFO -- : Rendered issues/_list.html.erb (3990.1ms)
App 103 output: I, [2021-04-16T03:09:39.736960 #103] INFO -- : Rendered my/blocks/_issues.erb (5549.7ms)
App 103 output: I, [2021-04-16T03:09:44.899092 #103] INFO -- : Rendered queries/_columns.html.erb (1643.8ms)
App 103 output: I, [2021-04-16T03:09:48.195990 #103] INFO -- : Rendered issues/_list.html.erb (3114.5ms)
App 103 output: I, [2021-04-16T03:09:48.238683 #103] INFO -- : Rendered my/blocks/_issues.erb (5564.4ms)
App 103 output: I, [2021-04-16T03:09:48.271494 #103] INFO -- : Rendered my/page.html.erb within layouts/base (29347.8ms)
App 103 output: I, [2021-04-16T03:10:17.568436 #103] INFO -- : Completed 200 OK in 59918ms (Views: 57311.4ms | ActiveRecord: 2235.3ms)
App 103 output: I, [2021-04-16T03:11:32.073474 #103] INFO -- : Started GET "/projects/myapp-v2?jump=my" for WWW.XXX.YYY.ZZZ at 2021-04-16 03:11:32 +0000
App 103 output: I, [2021-04-16T03:11:32.075240 #103] INFO -- : Processing by ProjectsController#show as HTML
App 103 output: I, [2021-04-16T03:11:32.075285 #103] INFO -- : Parameters: {"jump"=>"my", "id"=>"myapp-v2"}
App 103 output: I, [2021-04-16T03:11:32.083287 #103] INFO -- : Current user: myuser (id=208)
App 103 output: I, [2021-04-16T03:11:34.636170 #103] INFO -- : Rendering projects/show.html.erb within layouts/base
App 103 output: I, [2021-04-16T03:11:36.682491 #103] INFO -- : Rendered projects/_members_box.html.erb (157.3ms)
App 103 output: I, [2021-04-16T03:11:36.684234 #103] INFO -- : Rendered projects/show.html.erb within layouts/base (2048.0ms)
App 103 output: I, [2021-04-16T03:11:37.452800 #103] INFO -- : Completed 200 OK in 5377ms (Views: 2762.9ms | ActiveRecord: 620.0ms)
At 2021-04-16T03:10:17.568436 you can see a 200 OK response in 59918ms.
I run Remdine docker with a docker-compose file. Redmine is connecting to the host MySQL server and uses the host mail server too (no problem with that).
Here is the redmine-docker-compose.yml file content:
version: '3.1'
services:
redmine:
image: redmine:4.2-passenger
user: "$UID:$GID"
logging:
driver: "journald"
options:
tag: "redmine-docker"
mode: "non-blocking"
max-buffer-size: "4m"
ports:
- 8792:3000
environment:
REDMINE_DB_MYSQL: 172.17.0.1
REDMINE_DB_DATABASE: redmine
REDMINE_DB_PORT: 3306
REDMINE_DB_USERNAME: redmine
REDMINE_DB_PASSWORD: ***************
REDMINE_DB_ENCODING: utf8
REDMINE_SECRET_KEY_BASE: *************************
volumes:
- ./redmine/files:/usr/src/redmine/files
- ./redmine/themes/PurpleMine2-master:/usr/src/redmine/public/themes/PurpleMine2-master
- ./redmine/config/configuration.yml:/usr/src/redmine/config/configuration.yml
- ./redmine/repos:/usr/src/redmine/repos
- ./redmine/etc/passwd:/etc/passwd:ro
- ./redmine/usr/src/redmine/Passengerfile.json:/usr/src/redmine/Passengerfile.json
networks:
default:
external:
name: br-redmine-docker
Here is the configuration.yml content:
production:
email_delivery:
delivery_method: :smtp
smtp_settings:
address: 172.17.0.1
port: 25
enable_starttls_auto: false
And the Passengerfile.json content:
{
"max_pool_size": 15,
"pool_idle_time": 60,
"min_instances": 4
}
Note that I created the Passengerfile.json because of this exact same problem but it did not resolve it at all.
Here is the content of passwd file (it is just a copy of the file present in the docker image with one more line at the end):
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
redmine:x:999:999::/home/redmine:/bin/sh
hostuser:x:998:996::/home/hostuser:/bin/sh
And I created a .env file with this:
UID=998
GID=996
hostuser is a user that exists on my host (it's name is redmine on the host but I only needed to assign the right UID and GID in order to keep ownership of the redmine files to this user).
I also checked the wait_timeout MySQL variable that closes connections after a period of inactivity but it is on the default value which is 8 hours and my problem appears within approximately 4 hours so it can't be the cause.
I also wanted to disable tini in case it was that but I can't find any way to do that.
Any help is appreciated.
Thank you.
As a workaround, I ended up with writing a cron task that keep redmine awake with the curl command launched every hour. Here is a sample of my crontab:
0 */1 * * * /usr/bin/curl "https://my.redmine.com/login?keep-redmine-awake-cron" >/dev/null 2>&1
The URL parameter ?keep-redmine-awake-cron is only here to identify these requests in the web server logs.
The output of the command is ignored with >/dev/null 2>&1 because I don't want cron to send an email containing this useless output.
Though, I was not able to spot the real cause of those long requests after some inactivity.

Omniauth and Devise started to fail out of the blue, this morning, when authenticating with Facebook?

Everything was fine and then:
I, [2017-03-28T02:54:18.998055 #663] INFO -- : Completed 200 OK in 39ms (Views: 14.3ms | ActiveRecord: 8.5ms)
I, [2017-03-28T02:54:26.225944 #663] INFO -- : Started GET "/users/auth/facebook" for 176.109.217.212 at 2017-03-28 02:54:26 -0400
I, [2017-03-28T02:54:26.411915 #663] INFO -- : Started GET "/users/auth/facebook/callback?code=CODE and STATE" for 176.109.217.212 at 2017-03-28 02:54:26 -0400
I, [2017-03-28T02:54:26.566419 #663] INFO -- : Processing by Users::OmniauthCallbacksController#failure as HTML
I, [2017-03-28T02:54:26.566667 #663] INFO -- : Parameters: {"code"=>"CODE", "state"=>"STATE"}
I, [2017-03-28T02:54:26.567608 #663] INFO -- : Redirected to https://example.com/users/sign_in
All Facebook logins seem to be down, however the error is hard to catch. I display login failed "credentials invalid" to my users but it's not their fault. How can I tell them facebook's to blame.
Facebook made a migration yesterday during the night. You have to use a version of omniauth-facebook >= 4.0.0 to make it work again (source https://github.com/mkdynamic/omniauth-facebook/issues/278)
Update your Gemfile with gem 'omniauth-facebook', '4.0.0', then run bundle update and everything should go back to normal.

Migrating from heroku to dokku (digital ocean)

I'm switching everything from heroku off and moving to digital ocean (cheaper and a static ip address!!)
I have searchbox on heroku. And i'll be needing something like this on digital ocean.
Currently i'm hitting an error making a request (still setup for heroku?) That looks like this
I, [2016-06-22T13:18:41.226034 #275] INFO -- : Started GET "/events_autocomplete?query=test" for 212.250.145.155 at 2016-06-22 13:18:41 +0000
I, [2016-06-22T13:18:41.253478 #275] INFO -- : Processing by EventsController#autocomplete as JSON
I, [2016-06-22T13:18:41.253665 #275] INFO -- : Parameters: {"query"=>"test"}
I, [2016-06-22T13:18:41.264460 #275] INFO -- : Completed 500 Internal Server Error in 11ms (Searchkick: 5.7ms | ActiveRecord: 0.0ms)
F, [2016-06-22T13:18:41.266812 #275] FATAL -- :
Faraday::ConnectionFailed (Connection refused - connect(2) for "localhost" port 9200):
app/controllers/events_controller.rb:61:in `autocomplete'
I, [2016-06-22T13:18:41.319310 #275] INFO -- : Started GET "/search_events?utf8=%E2%9C%93&search=test&commit=Search" for 212.250.145.155 at 2016-06-22 13:18:41 +0000
I, [2016-06-22T13:18:41.321272 #275] INFO -- : Processing by EventsController#search as HTML
I, [2016-06-22T13:18:41.321365 #275] INFO -- : Parameters: {"utf8"=>"✓", "search"=>"test", "commit"=>"Search"}
I, [2016-06-22T13:18:41.326730 #275] INFO -- : Completed 500 Internal Server Error in 5ms (ActiveRecord: 0.0ms)
F, [2016-06-22T13:18:41.328614 #275] FATAL -- :
Faraday::ConnectionFailed (Connection refused - connect(2) for "localhost" port 9200):
app/controllers/events_controller.rb:83:in `search'
I'm very new at this so any help would be much appreciated
Thanks
Sam
Edit
def search
#events = Event.where("event_type != 'festival' AND complete = true").page(params[:page]).per(10).search params[:search], suggest: true, misspellings: { distance: 1 }, order: { date: :asc, eventname: :asc }, match: :word_start
if #events.results.any?
render 'events/results'
else
render 'events/noresults'
end
end
You can use the official dokku-elasticsearch plugin to setup an ES instance and link it to your application. Once you do, ELASTICSEARCH_URL will be set for your app and as long as your code respects that, you should be good to go.

ActiveRecord::StatementInvalid (Could not find table)

I've got a rails application on DigitalOcean. I'm using the Comfortable Mexican Sofa CMS. I added the gem and ran rails generate comfy:cms and rake db:migrate but when I navigate to example.com/admin, I get a 500 error and my application isn't able to find the necessary tables. Here's the error I get in the logs.
I, [2014-06-24T21:51:18.397255 #18789] INFO -- : Started GET "/" for 173.10.20.190 at 2014-06-24 21:51:18 +0000
I, [2014-06-24T21:51:18.425183 #18789] INFO -- : Processing by HomeController#index as HTML
I, [2014-06-24T21:51:18.442366 #18789] INFO -- : Rendered home/index.html.erb within layouts/application (1.2ms)
I, [2014-06-24T21:51:18.445116 #18789] INFO -- : Completed 200 OK in 20ms (Views: 9.5ms | ActiveRecord: 0.0ms)
I, [2014-06-24T21:51:18.599085 #18786] INFO -- : Started GET "/assets/rails.png" for 173.10.20.190 at 2014-06-24 21:51:18 +0000
I, [2014-06-24T21:51:23.516755 #18789] INFO -- : Started GET "/admin" for 173.10.20.190 at 2014-06-24 21:51:23 +0000
I, [2014-06-24T21:51:23.522184 #18789] INFO -- : Processing by Comfy::Admin::Cms::BaseController#jump as HTML
I, [2014-06-24T21:51:23.526053 #18789] INFO -- : Completed 500 Internal Server Error in 4ms
F, [2014-06-24T21:51:23.532357 #18789] FATAL -- :
ActiveRecord::StatementInvalid (Could not find table 'comfy_cms_sites'):
What seems strange to me is that when I run rails c and ActiveRecord::Base.connection.tables, the tables show up and 'comfy_cms_sites' is there.
Make sure you've run your migrations on your production db with
rake db:migrate RAILS_ENV=production

Empty HTML in production

Rails is returning blank HTML when running in production.
The tests I describe bellow are done with the following line: wget http://localhost:3000/login -O here.txt ; cat here.txt. I expect see some HTML being output after executing it.
On my server, I can see the HTML being rendered when I start rails in development mode, using the command rails s. I can see the following in the log:
Started GET "/login" for 127.0.0.1 at 2013-08-21 18:30:32 +0000
Processing by UserSessionsController#new as */*
Rendered user_sessions/_login.html.haml (59.5ms)
Rendered user_sessions/new.html.haml within layouts/application (62.8ms)
Compiled bootstrap-datepicker.css (0ms) (pid 8050)
Compiled application.css (4ms) (pid 8050)
Compiled bootstrap.min.css (0ms) (pid 8050)
# lots of Compiled lines
Rendered layouts/_navigation.html.haml (3.2ms)
Rendered layouts/_messages.html.haml (1.0ms)
Completed 200 OK in 1404ms (Views: 1395.6ms | ActiveRecord: 0.8ms)
When starting my application with RAILS_ENV=production rails s, doing the same request, I got the following on the log:
Started GET "/login" for 127.0.0.1 at 2013-08-21 18:30:52 +0000
Processing by UserSessionsController#new as */*
Rendered user_sessions/_login.html.haml (57.8ms)
Rendered user_sessions/new.html.haml within layouts/application (60.9ms)
Completed 200 OK in 69ms (ActiveRecord: 0.9ms)
But no HTML is output. What is happening? Why this is not working the same way that it does on development?
I use rails 3.2.13, and in this test it runs with WEBrick. I also am using RVM if that matters.
Update: Using the content of config/environments/development.rb on config/environments/production.rb does not fix the problem
The problem is that I was rescuing an exception on ApplicationController and not handling it properly by a logical error. The exception was not raised on development mode.

Resources