I have a ruby on rails app running on elastic beanstalk and I wanted to upload some large files - possibly around 5gb.
To do this, I added a config file on .ebextensios/nginx/01_upload_file_size.config with the following content:
files:
"/etc/nginx/conf.d/proxy.conf" :
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 20G;
After I deploy the code to EB, I restart the nginx server using the command sudo service nginx reload. This seem to work for a while.
Uploading large files the next day gives me 'Connection is reset' error. The log file log/nginx/error.logs tells me error client intended to send too large body: 24084848 bytes
I have no idea why this occurs. Seems like the config file is ignored after a short term or maybe reset but I can't see any reference of this happening in the documentation. Note that when I SSH into the eb environment again and restart nginx again, I can upload large files without a problem.
After looking after everything, I saw these events on my EB console.
Added instance [i-076127f714faac566] to your environment.
Removed instance [i-0c51791325b54873c] from your environment.
I also notice that the IP address of the host changes when the config resets.
I think that when the instances were automatically added and removed from EB, it didn't apply the config file or didn't restart the nginx server like I did manually via SSH.
So the question is: How do I make sure that the client_max_body_type is always set to 20G, even after instance is removed and re added? Or, how to make the config persistent so I don't have to manually restart the nginx server?
I think you have two questions here - why is EB replacing your instance, and how can you automate the restart of nginx.
Answering the first question will take a bit of research on your part, but I suspect it may be the default CloudWatch alarm that kills instances when network traffic drops below a certain threshold.
The second question should be fairly straightforward; following the documentation, you should be able to add a section to 01_upload_file_size.config that automatically restarts nginx during the deployment process:
container_commands:
01_restart_nginx:
command: "service nginx reload"
I would also check to make sure that the /etc/nginx/conf.d/proxy.conf file is actually being created - I don't know if folders under .ebextensions are supported. You might need to move your config file to .ebextensions/01_upload_file_size.config.
Related
I am having a very difficult time trying to launch a sample Rails 6 application to Elastic Beanstalk. For context, I am following these instructions
ADD RDS to Ruby Application
ADD an RDS to Beanstalk
I have followed these instructions to a tee and am still unable to connect to the rds database that I have provisioned. I keep receiving the following error:
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Whenever I try to run RAILS_ENV=production rails db:migrate or any other rake task, I keep getting that error.
On my AWS console, under Configuration and Software, I have the following environment variables:
Also in my database.yml file I have the rds configured variables listed as such.
production:
adapter: postgresql
database: <%= ENV['RDS_DB_NAME'] %>
username: <%= ENV['RDS_USERNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
port: <%= ENV['RDS_PORT'] %>
I have mapped my values as instructed in the documentation and am certain that they are correct.
Finally, I have sshed into my beanstalk provisioned ec2 instance and have executed the following command:
psql -U username -p 5432 -h examplehost.rds.amazonaws.com -d ebdb
provided the password and am able to connect. I am really at my wits end, I've spent too much time trying to diagnose this and am running out of ideas. I don't know where to look too next for ideas on how to trouble shoot this. I've read so many stack overflow questions and blogs that my head is spinning. If anyone has any ideas on how to resolve this, I would greatly appreciate it.
---Update----
I have created a new environment variable on the elastic beanstalk console.
ENV['DATABASE_URL'] = postgres://YourUserName:YourPassword#YourHostname:5432/YourDatabaseName
I made the necessary configurations, uploaded my .zip file and the connection to the database failed.
---- UPDATE-----
printenv does not show the varialbes provided by beanstalk, however this command does sudo /opt/elasticbeanstalk/bin/get-config environment.
My first advice is that, in my opinion, it is a much better option to create an Amazon RDS on their own, and not tied to Beanstalk.
As the AWS documentation indicates (emphasis mine):
AWS Elastic Beanstalk provides support for running Amazon Relational Database Service (Amazon RDS) instances in your Elastic Beanstalk environment. To learn about that, see Adding a database to your Elastic Beanstalk environment. This works great for development and testing environments. However, it isn't ideal for a production environment because it ties the lifecycle of the database instance to the lifecycle of your application's environment.
And:
To decouple your database instance from your environment, you can run a database instance in Amazon RDS and configure your application to connect to it on launch. This enables you to connect multiple environments to a database, terminate an environment without affecting the database, and perform seamless updates with blue-green deployments.
In my opinion, even for testing or development, it is always advisable to configure a small database instance and give your application the ability of define the most appropriate mechanism for connecting to your database.
The only downside is that you will probably need to configure a VPC, although it should not be actually a problem and, in ay case, it is worth value.
If for any reason you need to use the Beanstalk provisioned RDS database perhaps you have some workarounds to your problem (it should be a workaround because your configuration looks fine - please, only, verify that the database configuration is defined for the right Beanstalk environment).
For instance, one thing you can try is to store the database connection configuration in a S3 bucket, as also suggested in the AWS documentation. The idea is basically create some configuration file with the necessary connectivity information, store it in S3, and read that configuration in your application, i.e., process that file, in order to initialize your database.
But maybe you can try another approach.
Please, consider this SO question, and the answer from Jon McAuliffe and others. As indicated, Beanstalk will provide your application with environment variables, but maybe this variables will not be exposed as shell variables, they will be exposed to your application in different ways depending on the runtime the application needs to be executed on.
In the case of Ruby, you are accessing these variables in the correct way but, for any reason, your program is not having access to that information.
This probably also explains why printenv does not print any if your variables but the get-config script does.
But maybe you can take advantage of the fact that get-config provides you the right information and, either define this variables in your ENV by executing the get-config script for every RDS* key, perhaps in your environment.rb - please, be aware that I programmed in Ruby when I was a student but there is a long time since that, do the task in the file you consider appropriate - or using .ebextensions and a custom configuration file. You can find several examples here.
For instance, consider the following (copy and paste, with minor modifications of this example configuration):
commands:
01_update_env:
command: "/tmp/update_environment_variables.sh"
files:
"/tmp/update_environment_variables.sh":
mode: "000755"
content : |
#!/bin/bash
RDS_HOSTNAME=$(/opt/elasticbeanstalk/bin/get-config environment -k RDS_HOSTNAME)
if [ -z "$RDS_HOSTNAME" ]; then
echo "Could not determine RDS hostname"
exit 1
fi
echo "RDS hostname $RDS_HOSTNAME..."
# Just export the variable at OS level, or make it visible to
# the rails env in some other way
export RDS_HOSTNAME=$RDS_HOSTNAME
# Process the rest of the variables...
# Probably we should create a list and iterate through it
A similar approach could be the one exposed in this stackoverrun question, but restricted to the container that Beanstalk will use to encapsulate your app. AFAIK, the container should receive as env variables the different RDS* ones corresponding to the database configuration.
Dan, be aware that I have not tested these solutions, they are only ideas: please, be careful with that, I do not want to cause any damage to your system.
I found an answer for this problem with a mysql server that might still help you. Basically, even though I followed all your steps, could see my envars using sudo /opt/elasticbeanstalk/bin/get-config environment and could connect directly to my database with the mysql command, I was still getting the following error:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) (Mysql2::Error::ConnectionError)
The solution turned out to be the fact that Elastic Beanstalk was not connecting my envars to my bundle exec rails console command in the eb ssh instance access. I solved the issue by prepending all of the required envars explicitly to any rails commands I ran from within the eb ssh instance access. So for example, in order to run rails console, I had to run the following:
RAILS_MASTER_KEY=xxxxxxx RAILS_ENV=production RDS_HOSTNAME=xxxxxxx RDS_PASSWORD=xxxxxxx RDS_USERNAME=xxxxxxx RDS_DB_NAME=xxxxxxx AWS_REGION=xxxxxxx AWS_BUCKET=xxxxxxx bundle exec rails c
Replace the xxxxxxxs above with the values from the corresponding variables in your EB > Configuration > Software tab, and you should be able to connect to the remote database and run migrations, rake tasks and other database-reliant functions.
For Linux2 instances I was having the same issue and just noticed that the env variables I set in the config just didn't exist for su that I had set myself to -- if I remain the default login after eb ssh env prints everything I expected
edit: sorry -- env printing of variables on linux 2 instance enabled by
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
so what I did was find where those env variables were being exported for default user shell, which was /etc/profile.d/sh.local as noted in the above aws knowledge center link and just source that file when I needed to start the rails console as su
I'm having troubles with capistrano and sidekiq monit.
I setup a user for capistrano and everything was going smoothly until I installed Sidekiq.
My problem is when I try to execute cap staging sidekiq:monit:config (sidekiq:monit:start has the same permission problem).
Everytime I've tried, it "freezes" because it asks for the password.
Then I tryed to set sidekiq_monit_use_sudo to false. It's ok, it doesn't use sudo, but then it doesn't have permission to copy the /tmp/monit.conf into /etc/monit/conf.d/ folder.
It's the first time I'm setting up a server and I'm kinda lost here =|
Maybe try to config the sidekiq monit manually?
I'm using ruby 2.5 and these gems:
capistrano 3.10
capistrano-sidekiq 1.0
rails 5.1
Also I have the :pty config set to true as I don't feel comfortable not using a password.
Thank you!
You have a couple of options, of which I'll define two, the right one, and the easy/bad one.
Local User Monit
My personal usage of Monit is on a shared server on which I do not have root access. So I run Monit itself as a non-root user.
In order to do this, I compiled Monit with its prefix as $HOME/apps, so that the config files are in $HOME/apps/etc. This avoids the sudo issue. If you have access to the package manager and installed Monit that way, you can run monit as your user with the -c param to define where it should look for configuration files:
monit -c $HOME/config/monitrc
In order to get Capistrano to recognize the local monit, you will need some extra parameters in config/deploy.rb:
#set :monit_bin, '/usr/bin/monit' # Use this if you compile monit yourself.
set :sidekiq_monit_conf_dir, '/home/myuser/config/monit.d' # Feel free to customize.
set :sidekiq_monit_use_sudo, false
In the monitrc file you have defined with the -c option, you will need to make sure whatever folder you define in :sidekiq_monit_conf_dir is pulled in via includes:
include /home/myuser/config/monit.d/*.conf
Since I don't have an init system, I have Cron start Monit every 30 minutes, which is a noop if it is already running:
# Restart monit if it dies
*/30 * * * * $HOME/apps/bin/monit > /dev/null
If you have root access, you can improve upon this by having an init script (or systemd unit file) start Monit as your local user.
Bad option: give your user access to the conf dir
You can edit /etc/monit/monitrc to include your local user config directory as above. Similarly, you can allow your user to write to /etc/monit/conf.d. The major downside of these solutions is that you are now allowing your non-root user to create files which will be executes as root, opening a privilege escalation vulnerability. If your user ever got compromised, you certainly don't want an easy way for the attacker to get to root.
I include this option mostly because it's commonly considered, and should be avoided in the vast majority of cases (such as whenever you care about security). However, this might be useful in occasional rare cases (such as when you have a short term server for internal use only behind a firewall with only trusted users, and you need to set it up in a hurry).
I have deployed my Rails app on AWS Elasticbeanstalk and the app allows the user
to upload videos so in my nginx.conf file which is in etc/nginx directory in EC2
Instance I have to add a line in server part to avoid the error "Nginx: 413 Request Entity Too Large Error":
client_max_body_size 999M;
and then service nginx reload
But after uploading 10-15 videos this file gets reset and then I have to again add
this line and the IP of EC2 Instance also changes,
Can someone Please help me fix this issue.
It sounds like your instance is being terminated and rebuilt. Can you check the "Events" page for your environment and see if that's happening? That should be the only reason the filesystem is reset and the IP address is changing. I suspect an auto-scaling rule is being triggered.
As to why the file is being reset, have you ever noticed the following warning that gets displayed every time you connect via eb ssh:
This EC2 instance is managed by AWS Elastic Beanstalk. Changes made via
SSH WILL BE LOST if the instance is replaced by auto-scaling.
At least with the Ruby+Passenger platform, EB generates the nginx configuration files from templates that are found at /opt/elasticbeanstalk/support/conf - you should see files called nginx_config.erb and nginx_config_healthd.erb.
You can use EB extensions (described in the documentation) to modify those files during a deploy, which will ensure that your changes persist through auto-scaling. Your extension might look something like this:
.ebextensions/01-edit-nginx-conf.config
container_commands:
01backup_nginx_config:
command: "cp -n /opt/elasticbeanstalk/support/conf/nginx_config.erb /opt/elasticbeanstalk/support/conf/nginx_config.erb.original"
02edit_nginx_config:
command: "sh -c \"sed '/passenger_ruby/ i \
\ client_max_body_size 999M;' /opt/elasticbeanstalk/support/conf/nginx_config.erb.original > /opt/elasticbeanstalk/support/conf/nginx_config.erb\""
What is the best method to setup persistant storage for a Rails/Dokku app? The Dokku docs dont seem to say anything about the subject. When used Google to search the docs site the only thing it returned was the dokku-volume-plugin, which I've tried the without success.
I can create a volume for my app:
dokku volume:add myapp /public
but nothing gets written to the volume.
Is this the current(2015) best way to setup persistant storage with Dokku? If it is, am I missing something?
I use dokku-volume-plugin without any problems. Here's how it works.
The dokku volumes:add myapp /app/uploads/ command adds a volume that will be persisted on the host for the files stored inside your app's /app/uploads/ directory. If your app tries to write into that directory, it would instead by written on the host. The files are actually stored in the folder /home/dokku/.o_volume/.
From what I can tell, the only difference with your command and my command is the trailing slash. dokku volume:add myapp /public/ should fix your issues.
Alternatively, you could try an Amazon S3 based solution.
For the archives, so that nobody walks down the wrong path:
The current (2016, dokku > 0.5) path changed. I used #mixxorz approach in the past with success, but as of now the built-in plugin storage seems to take over the stage:
(... ssh dokku#host || dokku ...) storage:mount <app> /var/lib/dokku/data/storage:/app/public/uploads
Its well documented at http://dokku.viewdocs.io/dokku/dokku-storage/ .
The concepts stay the same.
I have two jobs that are queued simulataneously and one worker runs them in succession. Both jobs copy some files from the builds/ directory in the root of my Rails project and place them into a temporary folder.
The first job always succeeds, never have a problem - it doesn't matter which job runs first either. The first one will work.
The second one receives this error when trying to copy the files:
No such file or directory - /Users/apps/Sites/my-site/releases/20130829065128/builds/foo
That releases folder is two weeks old and should not still be on the server. It is empty, housing only a public/uploads directory and nothing else. I have killed all of my workers and restarted them multiple times, and have redeployed the Rails app multiple times. When I delete that releases directory, it makes it again.
I don't know what to do at this point. Why would this worker always create/look in this old releases directory? Why would only the second worker do this? I am getting the path by using:
Rails.root.join('builds') - Rails.root is apparently a 2 week old capistrano release? I should also mention this only happens in the production environment. What can I do
?
Rescue is not being restarted (stopped and started) on deployments which is causing old versions of the code to be run. Each worker continues to service the queue resulting in strange errors or behaviors.
Based on the path name it looks like you are using Capistrano for deploying.
Are you using the capistrano-resque gem? If not, you should give that a look.
I had exactly the same problem and here is how I solved it:
In my case the problem was how capistrano is handling the PID-files, which specify which workers currently exist. These files are normally stored in tmp/pids/. You need to tell capistrano NOT to store them in each release folder, but in shared/tmp/pids/. Otherwise resque does not know which workers are currently running, after you make a new deployment. It looks into the new release's pids-folder and finds no file. Therefore it assumes that no workers exist, which need to be shut down. Resque just creates new workers. And all the other workers still exist, but you cannot see them in the Resque-Dashboard. You can only see them, if you check the processes on the server.
Here is what you need to do:
Add the following lines in your deploy.rb (btw, I am using Capistrano 3.5)
append :linked_dirs, ".bundle", "tmp/pids"
set :resque_pid_path, -> { File.join(shared_path, 'tmp', 'pids') }
On the server, run htop in the terminal to start htop and then press T, to see all the processes which are currently running. It is easy to spot all those resque-worker-processes. You can also see the release-folder's name attached to them.
You need to kill all worker-processes by hand. Get out of htop and type the following command to kill all resque-processes (I like to have it completely clean):
sudo kill -9 `ps aux | grep [r]esque | grep -v grep | cut -c 10-16`
Now you can make a new deploy. You also need to start the resque-scheduler again.
I hope that helps.