I have a Rails app that works perfectly in development. I have deployed to the server and the app loads and correctly shows its landing and about pages. However, when I go to a page with Action Cable, the server blocks. The last request out from the web page is Action Cable related, and further page actions (form submission, and even reloads) and not responded to.
I have isolated it to the server as attempting to load the site's main page from another browser does not work. Closing the tab where the request to Action Cable was first made results in the other page loading immediately (and all of the pending requests from the first page suddenly being processed).
I do not see any errors in production.log, but here is the part leading up to the hang:
I, [2018-01-24T21:12:38.601260 #9840] INFO -- : [86b3fcc1-cebd-4d70-99ff-b6f9f147bc00] Rendered collection of skill_checks/_skill_check.html.erb [5 times] (16.7ms)
I, [2018-01-24T21:12:38.601352 #9840] INFO -- : [86b3fcc1-cebd-4d70-99ff-b6f9f147bc00] Rendered rooms/show.html.erb within layouts/application (26.6ms)
I, [2018-01-24T21:12:38.602314 #9840] INFO -- : [86b3fcc1-cebd-4d70-99ff-b6f9f147bc00] Completed 200 OK in 32ms (Views: 26.7ms | ActiveRecord: 1.8ms)
I, [2018-01-24T21:12:38.740061 #9840] INFO -- : [5a06a7d8-a4a5-4ab4-8287-377dfa3447fe] Started GET "/cable" for 65.99.102.74 at 2018-01-24 21:12:38 +0000
I, [2018-01-24T21:12:38.740860 #9840] INFO -- : [5a06a7d8-a4a5-4ab4-8287-377dfa3447fe] Started GET "/cable/" [WebSocket] for 65.99.102.74 at 2018-01-24 21:12:38 +0000
I, [2018-01-24T21:12:38.740970 #9840] INFO -- : [5a06a7d8-a4a5-4ab4-8287-377dfa3447fe] Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: upgrade, HTTP_UPGRADE: websocket)
I, [2018-01-24T21:12:38.794498 #9840] INFO -- : RoomsChannel is transmitting the subscription confirmation
I, [2018-01-24T21:12:38.795245 #9840] INFO -- : RoomsChannel is streaming from room_helloworld
Once I close the offending tab, load continues (in this case, of a javascript request that was triggered on page reload) as follows:
I, [2018-01-24T21:15:55.109087 #9840] INFO -- : Finished "/cable/" [WebSocket] for 65.99.102.74 at 2018-01-24 21:15:55 +0000
I, [2018-01-24T21:15:55.109342 #9840] INFO -- : RoomsChannel stopped streaming from room_helloworld
I, [2018-01-24T21:15:55.110922 #9840] INFO -- : [2cf5cc53-2f44-42a9-be30-496bb80646a0] Started GET "/rooms/helloworld?update=true" for 65.99.102.74 at 2018-01-24 21:15:55 +0000
I, [2018-01-24T21:15:55.112841 #9840] INFO -- : [2cf5cc53-2f44-42a9-be30-496bb80646a0] Processing by RoomsController#show as HTML
I've truncated the remainder; there were no failures or anything else of interest in there.
I do have a redis server running on the same server and have put its port information into config/cable.yml under the production: section.
I believe that Rails can see it as I haven't seen any errors. Additionally, to verify this I connected to redis through redis-cli and subscribed to a channel, and then through the Rails console sent a broadcast message to that channel and verified receipt.
Update: I've tried a few different builds to isolate the issue. Here's what I now know:
I removed my only call to ActionCable.server.broadcast, so that isn't the source of the block.
I added javascript console logs to connected, disconnected, and received. The client does successfully subscribe.
I tried a build where I removed the App.cable.subscriptions.create from the client. While there is no websocket connection in this case (as you would expect), the server does not block/hang.
Given the above, I am certain this has to do with ActionCable, though I'm not sure if it has anything to do with my redis configuration.
This turned out the be the behaviour you get if you haven't set the following lines inside of your nginx conf file (inside of the server section):
location /cable {
passenger_app_group_name your_app_websocket;
passenger_force_max_concurrent_requests_per_process 0;
}
where, if your app name is "chat", the first line would read:
passenger_app_group_name chat_action_cable
Which, frankly, I'm not sure where is defined, but it may be part of the Rails magic.
Related
I have a rails server and this is a snip from the server log
Processing CustomController#my_action (for 10.0.2.2 at 2018-05-11 08:38:53) [GET]
Parameters: {"id"=>"12345", "action"=>"my_action", "controller"=>"custom"}
...
all the other stuff that gets logged
...
Completed in 251ms (View: 181, DB: 60) | 200 OK [https://myserver.com/custom/my_action/12345]
...
...
Processing CustomController#my_action (for 10.0.2.2 at 2018-05-11 08:38:55) [GET]
Parameters: {"id"=>"12345", "r"=>"876646", "h"=>"16", "m"=>"16", "s"=>"51", "send_image"=>"1", "gears"=>"0", "cookie"=>"1", "gt_ms"=>"1174", "uid"=>"dummyemail#myserver.com", "realp"=>"0", "java"=>"0", "qt"=>"0", "urlref"=>"https://myserver.com/reporting/report_builder/", "controller"=>"custom", "action_name"=>"", "idsite"=>"", "rec"=>"1", "_idts"=>"1525949211", "_idn"=>"1", "_refts"=>"0", "res"=>"1280x720", "_id"=>"f2fe432581e2e376", "wma"=>"0", "fla"=>"0", "_idvc"=>"1", "_viewts"=>"1525949211", "dir"=>"0", "action"=>"my_action", "url"=>"https://myserver.com/custom/my_action/12345", "pdf"=>"1", "ag"=>"0"}
...
all other stuff
Completed in 227ms (View: 177, DB: 39) | 200 OK [https://myserver.com/custom/my_action/12345?action_name=&idsite=&rec=1&r=705226&h=16&m=16&s=52&url=https%3A%2F%2Fmyserver.com%2Fcustom%2Fmy_action%2F2408&urlref=https%3A%2F%2Fmyserver.com%2Freporting%2Freport_builder%2F&uid=ctl-support%40myserver.com&_id=f2fe432581e2e376&_idts=1525949212&_idvc=0&_idn=1&_refts=0&_viewts=&send_image=1&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=0&java=0&gears=0&ag=0&cookie=1&res=1280x720>_ms=757]
The problem with the above log is, I see duplicate server calls made for a single action.
I see this only when piwik tracking script is loaded. However, if I monitor the network tab in the browser, I could see only one call to the application server(myserver.com) and one call to the piwik server (mypiwikserver.com).
What might be the cause of this and how to fix this ?
I have Rails application running on Thin with multithreading. There is a bunch of polling requests and sometimes, when running integration tests concurrent I have about 1-2% of missing flash failures. In my production.log I have something like the following in case of failure:
I, [2017-08-21T23:31:51.280455 #2146] INFO -- : Redirected to http://192.168.102.1/projects/599bcfd4875aa40862863a5b
I, [2017-08-21T23:31:51.282501 #2146] INFO -- : Rendered text template (0.1ms)
I, [2017-08-21T23:31:51.283151 #2146] INFO -- : Completed 200 OK in 3062ms (Views: 1.4ms)
I, [2017-08-21T23:31:51.309004 #23906] INFO -- : Rendered projects/_tests_block.html.haml (24.0ms)
I, [2017-08-21T23:31:51.309392 #23906] INFO -- : Rendered projects/show.js.haml within layouts/application (29.9ms)
I, [2017-08-21T23:31:51.318025 #23906] INFO -- : Completed 200 OK in 124ms (Views: 48.4ms)
I, [2017-08-21T23:31:51.318956 #2146] INFO -- : Started GET "/projects/599bcfd4875aa40862863a5b" for 192.168.102.2 at 2017-08-21 23:31:51 -0700
I, [2017-08-21T23:31:51.321849 #2146] INFO -- : Processing by ProjectsController#show as HTML
I, [2017-08-21T23:31:51.321994 #2146] INFO -- : Parameters: {"id"=>"599bcfd4875aa40862863a5b"}
My redirect call looks like redirect_to #project, notice: success_message
Between redirect and start of getting show page I have the finish of polling request from another page. In this case when show page is rendered, there is no flash[:notice].
I tried to keep and clear flash explicitly, so there is before_filter { flash.keep } in ApplicationController and flash.clear in my notification partial.
I also tried to log flashes before every action and in this concurrent case no another action steals flash, it's just missing.
Is this a multithreading problem? Or maybe I should use or implement flash with kind of "destination"? So that should be thread-safe too anyway.
PS Ruby 2.3.4, Rails 4.2.7, Thin 1.5.1
I've all of a sudden started getting this error when creating users in my app. Emails were working before I left for vacation, now I come back and this is what my tech support gives me! Ugh.
So, I make a user, and I get redirected to my admin panel with the following error in a flash notice: getaddrinfo: name or service not known. The user isn't created.
Looking at the logs, it looks like everything worked fine:
I, [2016-04-12T08:01:52.089647 #11555] INFO -- : Started POST "/admin/user/new" for 72.238.202.193 at 2016-04-12 08:01:52 -0500
I, [2016-04-12T08:01:52.092114 #11555] INFO -- : Processing by RailsAdmin::MainController#new as HTML
I, [2016-04-12T08:01:52.092259 #11555] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"vxhuTwXhQo6nYrskQcYH9W56Ej95LgzEbs8cnkjXQI4=", "user"=>{"company_id"=>"35", "username"=>"myuser#thedomain.com", "first_name"=>"Test", "last_name"=>"User", "full_name"=>"Test User", "time_zone"=>"Central Time (US & Canada)", "email"=>"myuser#thedomain.com", "phone_number"=>"", "password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "reset_password_sent_at"=>"[FILTERED]", "remember_created_at"=>"", "sign_in_count"=>"0", "current_sign_in_at"=>"", "last_sign_in_at"=>"", "current_sign_in_ip"=>"", "last_sign_in_ip"=>"", "authentication_token"=>"", "enable_notifications"=>"1", "confirmation_token"=>"MBrmpsD6Wtf1VBrhtDyc", "confirmed_at"=>"", "confirmation_sent_at"=>"April 12, 2016 10:00", "unconfirmed_email"=>"myuser#thedomain.com", "terms_accepted"=>"1", "cancel_subscription"=>"0", "on_trial"=>"0", "is_account_owner"=>"1", "role_ids"=>["", "", "2"]}, "return_to"=>"https://www.myserver.com/admin/user?sort=created_at&sort_reverse=false", "_save"=>"", "model_name"=>"user"}
I, [2016-04-12T08:01:52.212064 #11555] INFO -- : Rendered devise/mailer/confirmation_instructions.html.erb (1.6ms)
I, [2016-04-12T08:01:52.340343 #11555] INFO -- :
Sent mail to myuser#thedomain.com (9.3ms)
I've filtered the email address but, rest assured, it's a valid email.
I'm using EC2 to host my application, so I can't edit /etc/resolv.conf since any changes will be overridden.
I've run nslookup to make sure the server can find the domain MX records, which it does using the nameserver specified in /etc/resolv.conf.
What else can I do to troubleshoot this issue?
I used a recommendation I ran across on another blog to use Ruby's version of Resolv instead of the libc library with Ubuntu. So, I added the following line to an initializer: require resolv-replace.rb.
Now my error is a little more descriptive: Hostname not known: smtp.sendgrid.net, so I checked config/environments/production.rb and it turns out I had a trailing space at the end of my host line. Removed it and all is working now!
I'm using Devise + Omniauth (actually omniauth-foursquare, the foursquare OAuth2 strategy) in my Rails app.
Upon signing in, I keep being redirected to my app (/users/sign_in#=) with the following error:
Could not authenticate you from Foursquare because "Csrf detected"
Any insights?
This is what I see last in the console:
I, [2014-10-19T19:43:38.947771 #2] INFO -- omniauth: (foursquare) Request phase initiated.
Started GET "/users/auth/foursquare" for 142.255.113... at 2014-10-19 19:43:38 +0000
Parameters: {"code"=>"2MZGM413...", "state"=>"1ba1cec3beb4..."}
Processing by Devise::SessionsController#new as HTML
Started GET "/users/auth/foursquare/callback?code=2MZGM413...&state=1ba1cec3beb4..." for 142.255.113... at 2014-10
I, [2014-10-19T19:43:39.378477 #2] INFO -- omniauth: (foursquare) Callback phase initiated.
E, [2014-10-19T19:43:39.378660 #2] ERROR -- omniauth: (foursquare) Authentication failure! csrf_detected: OmniAuth::Strategies::OAuth2::CallbackError, csrf_detected | CSRF detected
As far as I can tell, my troubles started when I received a "cookie overflow" erorr and switched to session_storing: (session_store.rb)
Rails.application.config.session_store :active_record_store, key: '_APPNAME_session', domain: 'DOMAIN.com'
Thanks!
If you inspect the source for your app, you'll see that the generated HTML includes the CSRF meta tags in the HEAD element (i.e. csrf-token and csrf-param). Most likely, one of your layouts/views is calling the csrf_meta_tags method. Try removing it to see if it works, and then consider creating a separate layout without it for forms that won't need it. Other people with more experience on frontend rails could chime in here with best practices.
Here's a snippet of my production.log:
Started GET "/product/514034/754240" for XX.XX.202.138 at 2012-06-21 11:52:28 -0700
Started GET "/product/614409/666897" for XX.XX.228.38 at 2012-06-21 11:52:28 -0700
Processing by ProductsController#show as HTML
Parameters: {"category_id"=>"514034", "product_id"=>"754240"}
Processing by ProductsController#show as HTML
Parameters: {"category_id"=>"614409", "product_id"=>"666897"}
Logged in 2940659 via auth cookie
Logged in 585210 via auth cookie
[e3e3fc56bb6bd137741b269ee397683c] [2940659] Read fragment views/global-caches/header (0.7ms)
[e3e3fc56bb6bd137741b269ee397683c] [2940659] Rendered shared/_email_form.html.haml (0.7ms)
[d81bb986be5acc0277c0c9e11b414249] [585210] Read fragment views/global-caches/sharebar-message (0.7ms)
[d81bb986be5acc0277c0c9e11b414249] [585210] Rendered shared/_email_form.html.haml (0.7ms)
...
As you can see, it's logging two concurrent sessions of two different users simultaneously to the same log file. This makes it impossible to parse my logs and determine, for example, the time it took to generate each kind of page, because the entries are not in the expected order of:
Started GET "/URL/BLAH" for IP at DATE
... stuff...
Completed 200 OK in XXms (ActiveRecord: YY.Yms)
Instead I get an unpredictable interleaved log like this:
Started GET "/URL/BLAH" for IP at DATE
Started GET "/URL/BLAH" for IP at DATE
... stuff...
Completed 200 OK in XXms (ActiveRecord: YY.Yms)
...stuff...
Completed 200 OK in XXms (ActiveRecord: YY.Yms)
So it's impossible to match the "completeds" with the "Started."
What I'd like is a way to have each child process write to its own log or something. Or if it's possible a way to write the each pageview's log atomically, but that might be impossible or difficult or hurt performance.
Rails 3.2 provides nice option config.log_tag
You can add to your production.rb:
config.log_tags = [ lambda { Time.now.to_i }]
So each line in your logs will be prepended by numbers. Example:
[1351867173] Started GET "/" for 127.0.0.1 at 2012-11-02 16:39:33 +0200
[1351867173] Processing by RecipesController#main as HTML
Logs are still shuffled, but now we can normalize, order them.
sort -f -s -k1.1,1.11 production.log | sed 's/^.............//' > sorted_production.log
(Sorter by first symbols (by timestamp) and remove timestamp by sed)
Now logs are easy to analyze.
In addition there is fix on related issue https://github.com/rails/rails/pull/7317 in rails 3.2.9
So keep this in mind.
Sorry for bad English... )