Elastic Beanstalk Redis Fail, Webapp Unresponsive - ruby-on-rails

Question

Can't get past sidekiq errors.
Trying to migrate from Heroku to AWS EB. I have a rails app running rails 4.2.0, ruby 2.3 on a linux machine, but keep running into issues. The webapp won't load - it simply times out over and over.
INFO: Running in ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
INFO: See LICENSE and the LGPL-3.0 for licensing details.
INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
INFO: Booting Sidekiq 3.5.4 with redis options {:url=>nil}
ERROR: heartbeat: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
Redis keeps telling me its url is nil despite what seems to be a solid setup. (It works on another app I managed to get running with the same configuration. I also found the ERROR MISCONF notice to be troublesome too, but the Redis URL isn't even setting. Further, both are in the same security group
This is my config/sidekiq.rb:
rails_root = Rails.root || File.dirname(__FILE__) + '/../..'
rails_env = Rails.env || 'development'
redis_config = YAML.load_file(rails_root.to_s + '/config/redis.yml')
redis_config.merge! redis_config.fetch(Rails.env, {})
redis_config.symbolize_keys!
Sidekiq.configure_server do |config|
config.redis = { url: "redis://#{ENV['REDIS_HOST']}:#{redis_config[:port]}/12" }
end
Sidekiq.configure_client do |config|
config.redis = { url: "redis://#{ENV['REDIS_HOST']}:#{redis_config[:port]}/12" }
end
And my config/redis.yml:
development:
host: localhost
port: 6379
test:
host: localhost
port: 6379
production:
host: ENV['REDIS_HOST']
port: 6379
My applicatoin.yml:
REDIS_HOST: project-name-001.random-token.0001.use1.cache.amazonaws.com
Here's the setup_swap.config, sidekiq.config, and nginx.config.
I've also seen this issue, but I assume it's unrelated. Perhaps I'm mistaken? If irrelevant, will address in another post.
Starting nginx: nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
What could it be?
Is there anything important I'm missing?
Edit: Add nginx.conf
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
index index.html index.htm;
server {
listen 80 ;
listen [::]:80 ;
server_name localhost;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
}
# redirect server error pages to the static page /40x.html
#
error_page 404 /404.html;
location = /40x.html {
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# Settings for a TLS enabled server.
#
# server {
# listen 443 ssl http2 ;
# listen [::]:443 ssl http2 ;
# server_name _;
# root /usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/server.crt";
# ssl_certificate_key "/etc/pki/nginx/private/server.key";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# ssl_ciphers <redacted>;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /40x.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
}
Updated response: I updated nginx.conf to read: include /etc/nginx/conf.d/webapp_healthd.conf; but still got the following:
[root] service nginx restart
nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
Stopping nginx: [ OK ]
Starting nginx: nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
[ OK ]
And also, the following persists:
ERROR: heartbeat: MISCONF Redis is configured to save RDB snapshots,
but is currently not able to persist on disk. Commands that may modify
the data set are disabled. Please check Redis logs for details about
the error.
Update 2 removed duplicate references to localhost port 80 and nginx stopped complaining, but I still get the Heartbeat MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. error.
Output from Sidekiq.redis(&:info):
{
"redis_version"=>"3.2.8",
"redis_git_sha1"=>"00000000",
"redis_git_dirty"=>"0",
"redis_build_id"=>"12e5c8be08dc4d3",
"redis_mode"=>"standalone",
"os"=>"Linux 4.4.51-40.60.amzn1.x86_64 x86_64",
"arch_bits"=>"64",
"multiplexing_api"=>"epoll",
"gcc_version"=>"4.8.3",
"process_id"=>"24835",
"run_id"=>"83a8de8b50f482a4e271228435b2f0c8e3fa5b5c",
"tcp_port"=>"6379",
"uptime_in_seconds"=>"341217",
"uptime_in_days"=>"3",
"hz"=>"10",
"lru_clock"=>"1108155",
"executable"=>"/usr/local/bin/redis-server",
"config_file"=>"/etc/redis/redis.conf",
"connected_clients"=>"2",
"client_longest_output_list"=>"0",
"client_biggest_input_buf"=>"0",
"blocked_clients"=>"0",
"used_memory"=>"842664",
"used_memory_human"=>"822.91K",
"used_memory_rss"=>"3801088",
"used_memory_rss_human"=>"3.62M",
"used_memory_peak"=>"924360",
"used_memory_peak_human"=>"902.70K",
"total_system_memory"=>"1043574784",
"total_system_memory_human"=>"995.23M",
"used_memory_lua"=>"37888",
"used_memory_lua_human"=>"37.00K",
"maxmemory"=>"0",
"maxmemory_human"=>"0B",
"maxmemory_policy"=>"noeviction",
"mem_fragmentation_ratio"=>"4.51",
"mem_allocator"=>"jemalloc-4.0.3",
"loading"=>"0",
"rdb_changes_since_last_save"=>"177",
"rdb_bgsave_in_progress"=>"0",
"rdb_last_save_time"=>"1493941570",
"rdb_last_bgsave_status"=>"err",
"rdb_last_bgsave_time_sec"=>"0",
"rdb_current_bgsave_time_sec"=>"-1",
"aof_enabled"=>"0",
"aof_rewrite_in_progress"=>"0",
"aof_rewrite_scheduled"=>"0",
"aof_last_rewrite_time_sec"=>"-1",
"aof_current_rewrite_time_sec"=>"-1",
"aof_last_bgrewrite_status"=>"ok",
"aof_last_write_status"=>"ok",
"total_connections_received"=>"17",
"total_commands_processed"=>"141824",
"instantaneous_ops_per_sec"=>"0",
"total_net_input_bytes"=>"39981126",
"total_net_output_bytes"=>"72119284",
"instantaneous_input_kbps"=>"0.00",
"instantaneous_output_kbps"=>"0.00",
"rejected_connections"=>"0",
"sync_full"=>"0",
"sync_partial_ok"=>"0",
"sync_partial_err"=>"0",
"expired_keys"=>"3",
"evicted_keys"=>"0",
"keyspace_hits"=>"14",
"keyspace_misses"=>"533",
"pubsub_channels"=>"0",
"pubsub_patterns"=>"0",
"latest_fork_usec"=>"160",
"migrate_cached_sockets"=>"0",
"role"=>"master",
"connected_slaves"=>"0",
"master_repl_offset"=>"0",
"repl_backlog_active"=>"0",
"repl_backlog_size"=>"1048576",
"repl_backlog_first_byte_offset"=>"0",
"repl_backlog_histlen"=>"0",
"used_cpu_sys"=>"167.52",
"used_cpu_user"=>"46.03",
"used_cpu_sys_children"=>"0.00",
"used_cpu_user_children"=>"0.00",
"cluster_enabled"=>"0",
"db0"=>"keys=1,expires=0,avg_ttl=0"
}
Interestingly, I can't find my redis logs to investigate further. In my redis.conf, all I see is this.
# Specify the log file name. Also the empty string can be used to force
# Redis to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile ""
I've even run find / -path /sys -prune -o -path /proc -prune -o -name *redis* and don't see ANY log files. (╯°□°)╯︵ ┻━┻
What's also strange is that production.log is simply not getting written to, check the permissions: rw-r--r-- 1 webapp webapp 0 May 8 20:01 production.log

Answer 1

Please share your /etc/nginx/nginx.conf, I guess nginx.conf include other servers conf files in conf.d folder, check for the line include /etc/nginx/conf.d/*.conf; in your nginx.conf, if so it might load the file twice or other default file with the same server name, you can change it to include /etc/nginx/conf.d/webapp_healthd.conf or what ever name you want, but before check what is the file on the machine.
Also Check out the /etc/nginx/sites-enabled/ directory if there is any temp file such as ~default or .save. check it with ls -lah, delete them, restart nginx and check for errors or do it via ebextensions and deploy again.
UPDATE
Try to remove from nginx.confall the section of server { ... }, make sure to include inside http your file /etc/nginx/conf.d/webapp_healthd.conf, there you already have server listen 80; and localhost..
nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/webapp_healthd.conf;
index index.html index.htm;
}
003_nginx.config
files:
"/etc/nginx/conf.d/webapp_healthd.conf" :
mode: "000755"
owner: root
group: root
content: |
upstream my_app {
server unix:///var/run/puma/my_app.sock;
}
log_format healthd '$msec"$uri"'
'$status"$request_time"$upstream_response_time"'
'$http_x_forwarded_for';
server {
listen 80;
server_name _ localhost; # need to listen to localhost for worker tier
root /var/app/current/public;
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}
access_log /var/log/nginx/access.log main;
access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
try_files $uri/index.html $uri #my_app;
location #my_app {
proxy_pass http://my_app; # match the name of upstream directive which is defined above
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /assets {
alias /var/app/current/public/assets;
gzip_static on;
gzip on;
expires max;
add_header Cache-Control public;
}
# redirect server error pages to the static page /40x.html
#
error_page 404 /404.html;
location = /40x.html {
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
"/opt/elasticbeanstalk/hooks/appdeploy/post/03_restart_nginx.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
rm /etc/nginx/conf.d/webapp_healthd.conf.bak
rm /etc/nginx/conf.d/custom.conf
service nginx restart