I have been struggling to get my Rails app deployed correctly for a while now, and have decided it is finally time to consult the community for some help. I have read just about every stackoverflow post on this issues, including the following, with no luck:
Rails 5 ActionCable fails to upgrade to WebSocket on Elastic Beanstalk -> From this post I ensured I was using an Application Load Balancer
ActionCable on AWS: Error during WebSocket handshake: Unexpected response code: 404 -> Configured an nginx proxy, with no change
Problem Description
I am using the following setup:
Ruby 2.7.5
Rails 6.1.0
GraphQL
React Frontend (separate repo)
Elastic Beanstalk
Ruby 2.7 running on 64bit Amazon Linux 2/3.4.1
Application Load Balancer
Postgres ActionCable adapter
My application is deployed to AWS Elasticbeanstalk and all requests to /graphql are successful. However, when attempting to connect to /cable I get this error in my browser console:
WebSocket connection to 'wss://api.redacted.io/cable' failed:
When checking the Elastic Beanstalk logs I see:
/var/app/containerfiles/logs/production.log:
I, [2022-02-20T19:35:25.849990 #32761] INFO -- : [8e1d3e86-81cc-4708-89d3-ebad56470f8f] Started GET "/cable" for <redacted IP> at 2022-02-20 19:35:25 +0000
I, [2022-02-20T19:35:25.850342 #32761] INFO -- : [8e1d3e86-81cc-4708-89d3-ebad56470f8f] Started GET "/cable/"[non-WebSocket] for <redacted IP> at 2022-02-20 19:35:25 +0000
E, [2022-02-20T19:35:25.850384 #32761] ERROR -- : [8e1d3e86-81cc-4708-89d3-ebad56470f8f] Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: close, HTTP_UPGRADE: )
I, [2022-02-20T19:35:25.850419 #32761] INFO -- : [8e1d3e86-81cc-4708-89d3-ebad56470f8f] Finished "/cable/"[non-WebSocket] for <redacted IP> at 2022-02-20 19:35:25 +0000
Potentially Relevant Files
cable.yml:
development:
adapter: postgresql
test:
adapter: test
production:
adapter: postgresql
production.rb:
...
config.action_cable.url = 'wss://api.redacted.io/cable'
config.action_cable.allowed_request_origins = ['redacted.io', 'http://redacted.io', 'https://redacted.io']
...
.ebextensions/nginx_proxy.config:
files:
"/etc/nginx/conf.d/websockets.conf" :
content: |
upstream backend {
server unix:///var/run/puma/my_app.sock;
}
server_names_hash_bucket_size 128;
server {
listen 80;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
server_name redacted.elasticbeanstalk.com;
# prevents 502 bad gateway error
large_client_header_buffers 8 32k;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
# prevents 502 bad gateway error
proxy_buffers 8 32k;
proxy_buffer_size 64k;
proxy_pass http://backend;
proxy_redirect off;
location /assets {
root /var/app/current/public;
}
# enables WS support
location /cable {
proxy_pass http://backend/cable;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
container_commands:
01restart_nginx:
command: "nginx -t && service nginx restart"
After posting on reddit, I was able to fix my issue by:
Removing my .ebextensions/nginx_proxy.config file.
Creating a new file, .platform/nginx/conf.d/elasticbeanstalk/websocket.conf with the contents:
location /cable {
proxy_pass http://my_app/cable;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Related
I'm trying to use Hotwire for Rails, it works perfectly with my development environment, but having issues with websocket connection after deploy to our UAT (staging) environment.
Stack
Ruby: 3.0.0
Rails: 6.0.3.5
Nginx: 1.14.1
Passenger: Phusion Passenger 6.0.4
nginx.conf on my Gateway server
upstream myapp {
server myapp.uat.mycompany.com.au:80;
}
server {
server_name myapp.uat.mycompany.com.au;
location / {
proxy_pass http://myapp;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /cable {
proxy_pass http://active_migration;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
listen 443 ssl; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/myapp.uat.mycompany.com.au/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/myapp.uat.mycompany.com.au/privkey.pem; # managed by Certbot
}
server {
if ($host = myapp.uat.mycompany.com.au) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name myapp.uat.mycompany.com.au;
listen 80;
return 404; # managed by Certbot
}
nginx.conf on my app server
server {
server_name myapp.uat.mycompany.com.au;
listen 80;
underscores_in_headers on;
passenger_enabled on;
passenger_app_env uat;
root /var/www/myapp/current/public;
location /cable {
passenger_app_group_name myapp_action_cable;
passenger_force_max_concurrent_requests_per_process 0;
}
}
cable.yml
development:
adapter: redis
url: <%= ENV.fetch("ACTION_CABLE_URL") { "redis://localhost:6379/8" } %>
channel_prefix: myapp
test:
adapter: test
uat:
adapter: redis
url: <%= ENV.fetch("ACTION_CABLE_URL") { "redis://localhost:6379/8" } %>
channel_prefix: myapp
production:
adapter: redis
url: <%= ENV.fetch("ACTION_CABLE_URL") { "redis://localhost:6379/8" } %>
channel_prefix: myapp
uat.rb (environment configuration file)
config.action_cable.url = 'wss://myapp.uat.mycompany.com.au/cable'
config.action_cable.allowed_request_origins = ['https://myapp.uat.mycompany.com.au', 'http://myapp.uat.mycompany.com.au']
Errors displayed in Chrome console
connection.js:70 WebSocket connection to 'wss://myapp.uat.mycompany.com.au/cable' failed: WebSocket is closed before the connection is established.
Errors in uat.log
I, [2021-10-29T15:13:33.876173 #668] INFO -- : [25f60292-8a89-4ce5-9f35-fb3ad56bef00] Started GET "/cable" for 10.1.32.123 at 2021-10-29 15:13:33 +1100
I, [2021-10-29T15:13:33.876890 #668] INFO -- : [25f60292-8a89-4ce5-9f35-fb3ad56bef00] Started GET "/cable/" [WebSocket] for 10.1.32.123 at 2021-10-29 15:13:33 +1100
I, [2021-10-29T15:13:33.876966 #668] INFO -- : [25f60292-8a89-4ce5-9f35-fb3ad56bef00] Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: upgrade, HTTP_UPGRADE: websocket)
E, [2021-10-29T15:13:33.877255 #668] ERROR -- : [25f60292-8a89-4ce5-9f35-fb3ad56bef00] WebSocket error occurred: wrong number of arguments (given 2, expected 1)
E, [2021-10-29T15:13:33.878046 #668] ERROR -- : WebSocket error occurred: wrong number of arguments (given 2, expected 1)
E, [2021-10-29T15:13:34.234885 #668] ERROR -- : WebSocket error occurred: wrong number of arguments (given 2, expected 1)
E, [2021-10-29T15:13:36.397514 #20846] ERROR -- : WebSocket error occurred: wrong number of arguments (given 2, expected 1)
Note: the above WebSocket error keeps happening ...
I googled around and found someone had the similar problem, so tried with the solution of
commenting out the config.action_cable.url in the uat.rb environment file and put action_cable_meta_tag in the application.html.slim file, but it didn't work.
I've spent lots of time with it, could someone shed some light on this please?
It works fine in my production environment. The only difference I can see is that the passenger version is Phusion Passenger 6.0.4 in my production env.
Will try to upgrade my staging passenger version
Could not connect websocket using Action Cable in Rails 5.1. HTTP server is Unicorn on nginx and adapter is Redis.
Rails configuration is the following.
# config/environments/production.rb
config.action_cable.disable_request_forgery_protection = true
nginx configuration is the following.
upstream unicorn {
server unix:/rails/current/tmp/sockets/unicorn.sock;
}
server {
listen 80;
charset utf-8;
server_name sub.example.com;
root /rails/current/public;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://unicorn;
break;
}
}
location /cable {
proxy_pass http://unicorn;
proxy_http_version 1.1;
proxy_set_header Upgrade "websocket";
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
}
...
}
Errors in console of web browser are the following.
WebSocket connection to 'wss://sub.example.com/cable' failed: Error during WebSocket handshake: Unexpected response code: 404
Errors in Rails are the following.
[ERROR] Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: close, HTTP_UPGRADE: )
[INFO] Finished "/cable/"[non-WebSocket] for xxx.xxx.xxx.xxx at 2017-xx-xx
A strangest thing is a error in Rails does not have HTTP_UPGRADE value. But HTTP request headers of web browser include Upgrade key and websocket value. Also setting "websocket" for proxy header in nginx configurations.
What should I do?
On the back of that writeup, I just realized the universal solution is
a simple change to the config/secrets.yml file to reference the
ENV["PORT"] setting
...
# Be sure to restart your server when you modify this file.
development:
secret_key_base: 231bf79489c63f8c8facd7...
action_cable_url : http://localhost:<%= ENV["PORT"] %>
test:
secret_key_base: 1ab8adbcf8410aebb...
action_cable_url : http://localhost:<%= ENV["PORT"] %>
# Do not keep production secrets in the repository,
# instead read values from the environment.
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
action_cable_url : <%= ENV["SERVER_PORT"] %>
.
You can also have this in your dot file:
export PORT=3000
source
I am using websocket-rails gem for handling messaging service for my application.
Tech Stack: Rails 4.2.5, Ruby 2.3.0p0, Passenger as app-server(Also tried with Puma), Thin as websocket server on port 3001, ngnix as web server and
Websocket patch of gem 'websocket-rails', github: 'moaa/websocket-rails', branch: 'sync_fixes'
When an client hits the websocket server, it actively triggers the server event of client_connected and I can see on websocket_rails_server.log the message that I print, i.e.
"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
"+++++++++++ WebsocketRails Client CONNECTED ++++++++++++++"
"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
But, on client end it abruptly ends with response:
WebSocket connection to 'ws://beta.server.com/websocket' failed: Error during WebSocket handshake: Unexpected response code: 504
I have tried with every solution on Github issues as well as Stackoverflow related question, but no help yet.
My nginx config is as follows(if that's needed anyway):
server {
listen 80;
server_name beta.server.com;
root /var/www/server-rails/current/public;
proxy_cache_bypass 1;
proxy_no_cache 1;
location /websocket {
proxy_pass http://localhost:3001/websocket;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
add_header Access-Control-Allow-Origin *;
proxy_set_header Upgrade websocket;
proxy_set_header Connection upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
passenger_enabled on;
passenger_sticky_sessions on;
passenger_app_env staging;
}
I'm trying to introduce chatting to my Rails app. For this purpose I used gem private_pub and it works perfectly in development mode.
In production I was using Apache + Passenger, but I couldn't configure Faye with it, so I changed Apache to Nginx. My main app is still on Apache server, and this demo on Nginx with port 8080 (just for test).
I'm able to connect to faye.js by entering http://chat.mysite.com:8080/faye.js, but the connection from app throws an error (browser console).
WebSocket connection to 'ws://localhost:9292/faye' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
After this error, another error appears every 5 seconds.
faye.js:2 GET http://localhost:9292/faye?message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fhands…22%2C%22callback-polling%22%5D%2C%22id%22%3A%221%22%7D%5D&jsonp=jsonp2 net::ERR_CONNECTION_REFUSED
My private_pub.yml
production:
server: "http://localhost:9292/faye"
secret_token: "mysecret"
signature_expiration: 3600 # one hour
My private_pub.ru
require "bundler/setup"
require "yaml"
require "faye"
require "private_pub"
Faye::WebSocket.load_adapter('thin')
PrivatePub.load_config(File.expand_path("../config/private_pub.yml", __FILE__), ENV["RAILS_ENV"] || "development")
run PrivatePub.faye_app
My nginx site.conf
server {
listen 8080;
server_name www.chat.mysite.com;
passenger_enabled on;
passenger_app_env production;
root /var/www/mysite/public;
location ^~ /faye {
proxy_pass http://127.0.0.1:9292;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_buffering off;
proxy_redirect off;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
#proxy_set_header X-Forwarded-Proto https;
proxy_cache_bypass $http_pragma $http_authorization;
proxy_no_cache $http_pragma $http_authorization;
break;
}
}
If I change private_pub.yml to http://localhost:9292/faye/faye, I saw error like "can't load resource /faye/faye.js".
How should I change my Nginx conf or app yml to resolve websocket error?
I see private_pub is quite similar to ActionCable in its design. Before you go too far down the road, you may want read my blog post on "ActionAcable - The good and bad parts" as it addresses suitable use cases when a system like private_pub is good, and when it's not.
If of course you are aware of the shortcomings already, then good luck!
I tried to configure my private_pub.yml as #niceman said. Now all is working good.
production:
server: "http://my-ip:8080/faye"
Trying to deploy super simple rails 5 application to AWS using beanstalk.
Everything works fine except actioncable. Browser can't connect to the cable server.
WebSocket connection to 'ws://prod.3x52xijcqx.us-west-1.elasticbeanstalk.com/cable' failed: Error during WebSocket handshake: Unexpected response code: 404
I found similar questions at stackoverflow but all of them were pointing to configure nginx using this template, however it didn't help me.
I customized nginx config using this script which I've put inside .ebextensions
files:
"/etc/nginx/conf.d/websockets.conf":
content: |
upstream backend {
server unix:///var/run/puma/my_app.sock;
}
server_names_hash_bucket_size 128;
server {
listen 80;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
server_name prod.3x52xijcqx.us-west-1.elasticbeanstalk.com;
# prevents 502 bad gateway error
large_client_header_buffers 8 32k;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
# prevents 502 bad gateway error
proxy_buffers 8 32k;
proxy_buffer_size 64k;
proxy_pass http://backend;
proxy_redirect off;
location /assets {
root /var/app/current/public;
}
# enables WS support
location /cable {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
container_commands:
01_restart_nginx:
command: "service nginx restart"
Full app source is here