I know this is a very common question but I feel like I need a specific answer to help find out where I am going wrong...
Loaded whenever gem to manage cron jobs - it works fine in development but when I loaded the app into AWS I can't seem to it work...
When I SSH into the instance I can run crontab -l and it lists the whenever tasks but it just doesn't seem to actually execute them. I also cant find any log files to read into why it's not firing.
This is what I pulled from the eb activity log..
+++ GEM_ROOT=/opt/rubies/ruby-2.3.6/lib/ruby/gems/2.3.0
++ (( 0 != 0 ))
+ cd /var/app/current
+ su -c 'bundle exec whenever --update-cron'
[write] crontab file updated
+ su -c 'crontab -l'
# Begin Whenever generated tasks for:
/var/app/current/config/schedule.rb at: 2018-01-10 06:08:24 +0000
0 * * * * /bin/bash -l -c 'cd /var/app/current && bundle exec
bin/rails runner -e production '\''Trendi.home'\'' >>
app/views/pages/cron.html.erb 2>&1'
# End Whenever generated tasks for:
/var/app/current/config/schedule.rb at: 2018-01-10 06:08:24 +0000
[2018-01-10T06:08:24.705Z] INFO [15603] - [Application update app-
a3a0-180109_230627#16/AppDeployStage1/AppDeployPostHook] : Completed
activity. Result:
Successfully execute hooks in directory
/opt/elasticbeanstalk/hooks/appdeploy/post.
This is my config file from ebextensions folder
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/01_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Using similar syntax as the appdeploy pre hooks that is
managed by AWS
set -xe
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container
-k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container
-k support_dir)
EB_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container
-k app_deploy_dir)
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
cd $EB_DEPLOY_DIR
su -c "bundle exec whenever --update-cron"
su -c "crontab -l"
My Schedule.rb
ENV['RAILS_ENV'] = "production"
set :output, "app/views/pages/cron.html.erb"
every 1.hour, at: ":00"do # 1.minute 1.day 1.week 1.month 1.year
is also supported
runner "Trendi.home", :environment => 'production'
end
And my task that is stored at /lib/
module Trendi
def self.home
#exectuted task code here
end
You could try a simpler command in your crontab and change it to fire every minute. E.g.
/bin/echo "test" >> /home/username/testcron.log
That way you can quickly rule out if it's the cronjob that's the culprit. Keep in mind to use full paths to each command. So in your case you might want to change the "bundle" command to use the full path. You can find the path with the "which" command.
Also, are you sure you are correctly escaping here?
-e production '\''Trendi.home'\''
Wouldn't this be more adequate?
-e production "Trendi.home"
I am trying to figure out how to run rake db:migrate on my ECS service but only on one machine after deployment.
Anyone has experience with that?
Thanks
You may do it via Amazon ECS one-off task.
Build a docker image with rake db migrate as "CMD" in your docker file.
Create a task definition. You may choose one task per host while creating the task-definition and desired task number as "1".
Run a one-off ECS task inside your cluster. Make sure to make it outside service. Once It completed the task then the container will stop automatically.
You can write a script to do this before your deployment. After that, you can define your other tasks as usual.
You can also refer to the container lifecycle in Amazon ECS here: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_life_cycle.html. However, this is the default behavior of the docker.
Let me know if it works for you.
I built a custom shell script to run when my docker containers start ( CMD command in docker ):
#!/bin/sh
web_env=${WEB_ENV:-1}
rails_env=${RAILS_ENV:-staging}
rails_host=${HOST:-'https://www.example.com'}
echo "*****************RAILS_ENV is $RAILS_ENV default to $rails_env"
echo "***************** WEB_ENV is $WEB_ENV default to $web_env"
######## Rails migration ################################################
echo "Start rails migration"
echo "cd /home/app/example && bundle exec rake db:migrate RAILS_ENV=$rails_env"
cd /home/app/example
bundle exec rake db:migrate RAILS_ENV=$rails_env
echo "Complete migration"
if [ "$web_env" = "1" ]; then
######## Generate webapp.conf##########################################
web_app=/etc/nginx/sites-enabled/webapp.conf
replace_rails_env="s~{{rails_env}}~${rails_env}~g"
replace_rails_host="s~{{rails_host}}~${rails_host}~g"
# sed: -i may not be used with stdin in MacOsX
# Edit files in-place, saving backups with the specified extension.
# If a zero-length extension is given, no backup will be saved.
# we use -i.back as backup file for linux and
# In Macosx require the backup to be specified.
sed -i.back -e $replace_rails_env -e $replace_rails_host $web_app
rm "${web_app}.back" # remove webapp.conf.back cause nginx to fail.
# sed -i.back $replace_rails_host $web_app
# sed -i.back $replace_rails_server_name $web_app
######## Enable Web app ################################################
echo "Web app: enable nginx + passenger: rm -f /etc/service/nginx/down"
rm -f /etc/service/nginx/down
else
######## Create Daemon for background process ##########################
echo "Sidekiq service enable: /etc/service/sidekiq/run "
mkdir /etc/service/sidekiq
touch /etc/service/sidekiq/run
chmod +x /etc/service/sidekiq/run
echo "#!/bin/sh" > /etc/service/sidekiq/run
echo "cd /home/app/example && bundle exec sidekiq -e ${rails_env}" >> /etc/service/sidekiq/run
fi
echo ######## Custom Service setup properly"
What I did was to build a docker image to be run as a web server ( Nginx + passenger) or Sidekiq background process. The script will decide whether it is a web or Sidekiq via ENV variable WEB_ENV and rails migration will always get executed.
This way I can be sure the migration always up to date. I think this will work perfectly for a single task.
I am using a Passenger docker that has been designed very easy to customize but if you use another rails app server you can learn from the docker design of Passenger to apply to your own docker design.
For example, you can try something like:
In your Dockerfile:
CMD ["/start.sh"]
Then you create a start.sh where you put the commands which you want to execute:
start.sh
#! /usr/bin/env bash
echo "Migrating the database..."
rake db:migrate
I'm running a Rails 4.2 app on Elastic Beanstalk, and need to set log permissions and create the /tmp/uploads folder (plus permissions) after deploy.
I was running two ebextensions scripts to do this, but on some occasions they would fail because the folder /var/app/current/ didn't yet exist.
I'm presuming this is because the permissions and/or folders should be created on /app/ondeck/ first so that EB can copy the contents over to /var/app/current/, but I'm interested to see if there's a recommended and more foolproof approach to doing this?
For reference, my two ebextension scripts were:
commands:
01_set_log_permissions:
command: "chmod 755 /var/app/current/log/*"
and
commands:
01_create_uploads_folder:
command: "mkdir -p /var/app/current/tmp/uploads/"
02_set_folder_permission:
command: "chmod 755 /var/app/current/tmp/uploads/"
Thanks,
Dan
you should probably use files tag and not command:
commands:
create_post_dir:
command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/99_make_changes.sh":
mode: "000777"
content: |
#!/bin/bash
mkdir -p /var/app/current/tmp/uploads/
chmod 755 /var/app/current/tmp/uploads/
it will be triggered after app deploy finished
I've used the below stpes:
Create a folder .ebextensions
Creta a file .config
Move .config to .ebextensions
Edit the file .config, it must have the below sintaxe
commands:
command1:
command: mkdir /opt/jenkins
command2:
command: chmod 644 /opt/jenkins
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-commands
* Pay Attention *
You cannot run again command1 "mkdir /opt/jenkins", you will have a error, so you must do a test before.
What about using Container Commands?
http://docs.aws.amazon.com/ko_kr/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-container-commands
You can use the container_commands key to execute commands for your container. The commands in container_commands are processed in alphabetical order by name. They run after the application and web server have been set up and the application version file has been extracted, but before the application version is deployed. They also have access to environment variables such as your AWS security credentials.
Container commands are run from the staging directory, where your source code is extracted prior to being deployed to the application server. Any changes you make to your source code in the staging directory with a container command will be included when the source is deployed to its final location.
container_commands:
01_set_log_permissions:
command: "chmod 755 log/*"
and
container_commands:
01_create_uploads_folder:
command: "mkdir -p tmp/uploads/"
02_set_folder_permission:
command: "chmod 755 tmp/uploads/"
I have a Rails 4.2.1 app, using Ruby 2.2. I'm trying to use the Whenever Gem to update cron tasks on my Elastic Beanstalk from my code base. I've followed a few resources from AWS where you can add files to the .ebextensions folder and use EB's post deployment hook via shell files. Here are a couple resources:
https://forums.aws.amazon.com/thread.jspa?threadID=137136
http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job
http://blog.endpoint.com/2015/01/elastic-beanstalk-whenever.html
Following the blog articles, I added the file below, uncommented the gitignore line about files in the .ebextensions folder, and deployed my app. Unfortunatley, I have been able to see any changes. I've checked the log files (log/eb-tools.log, log/cron, etc.), and grepped all the log files for the shell file I create, whenever, and more. No luck though.
commands:
create-post-dir:
command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/99_update_cron.sh"
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Using similar syntax as the appdeploy pre hooks that is managed by AWS
# Loading environment data
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_APP_PIDS_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
# Setting up correct environment and ruby version so that bundle can load all gems
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
# Now we can do the actual restart of the worker. Make sure to have double quotes when using env vars in the command.
su -c "cd $EB_APP_CURRENT_DIR; bundle exec whenever --update-cron --set='environment=$RACK_ENV'" - $EB_APP_USER
How can I make sure this shell file is getting called? Can I test it without a new deployment each time? Also, I'm open to other options if the Whenever gem is not the best option. I mainly want to be able to have my cron tasks managed in code and under version control.
Thanks in advance!
UPDATE:
I had a type on the .ebextensions folder, which was causing it to not be added. After that was fixed, I was able to read error messages, and create a cron script that updated the crontab using the Whenever gem.
I was able to figure this out after fixing a typo in my .ebextensions folder name. Afterwards, the scripts got compiled, and log messages started to appear. After reading the log messages, I came up with the following config script (.ebextensions/01_cron.config):
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/01_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Using similar syntax as the appdeploy pre hooks that is managed by AWS
set -xe
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
EB_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
cd $EB_DEPLOY_DIR
su -c "bundle exec whenever --update-cron"
su -c "crontab -l"
I have set up a RoR environement on AWS' elastic beanstalk. I am able to ssh into my EC2 instance.
My home directory is /home/ec2-user, which is effectively empty.
If I move up a directory, there is also a /home/webapp directory that i do not have access to.
Is there a way to run a rake command or rails console on my elastic beanstalk instance?
If I type rails console I get Usage: rails new APP_PATH [options]
If I type RAILS_ENV=production bundle exec rails console, I get "Could not locate Gemfile"
For rails, jump to /var/app/current then as #juanpastas said, run RAILS_ENV=production bundle exec rails c
Don't know why, but since EBS runs everything as root, this worked for me:
sudo su
bundle exec rails c production
None of these solutions mentioned here worked for me, so I cooked up a little script that I put in script/aws-console.
You can run it from the /var/app/current directory as root:
eb ssh
cd /var/app/current
sudo script/aws-console
My script can be found as a Gist here.
None of the other answers worked for me so I went looking - this is working for me now on an elastic beanstalk 64bit amazon linux 2016.03 V2.1.2 ruby 2.2 (puma) stack
cd /var/app/current
sudo su
rake rails:update:bin
bundle exec rails console
that returns me the expected console
Loading production environment (Rails 4.2.6)
irb(main):001:0>
For Ruby 2.7:
if you don't need environment variables:
BUNDLE_PATH=/var/app/current/vendor/bundle/ bundle exec rails c
It looks like environment variables are not loaded automatically anymore, which might prevent rails console from starting.
I solved it by creating this .ebextensions file:
# Simply call `sudo /var/app/scripts/rails_c`
commands:
create_script_dir:
command: "mkdir -p /var/app/scripts"
ignoreErrors: true
files:
"/var/app/scripts/export_envvars":
mode: "000755"
owner: root
group: root
content: |
#!/opt/elasticbeanstalk/.rbenv/shims/ruby
if __FILE__ == $0
require 'json'
env_file = '/var/app/scripts/envvars'
env_vars = env_vars = JSON.parse(`/opt/elasticbeanstalk/bin/get-config environment`)
str = ''
env_vars.each do |key, value|
new_key = key.gsub(/\s/, '_')
str << "export #{new_key}=\"#{value}\"\n"
end
File.open(env_file, 'w') { |f| f.write(str) }
end
"/var/app/scripts/rails_c":
mode: "000755"
owner: root
group: root
content: |
. ~/.bashrc
/var/app/scripts/export_envvars
. /var/app/scripts/envvars
cd /var/app/current
/opt/elasticbeanstalk/.rbenv/shims/bundle exec rails c
Create a .ebextension file named setvars.config and add those lines to it
commands:
setvars:
command: /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "export \(.key)=\"\(.value)\""' > /etc/profile.d/sh.local
packages:
yum:
jq: []
Then deploy your code again it should work.
reference: https://aws.amazon.com/ar/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
For Ruby 2.7:
As someone said, if you don't need env vars, run the following
BUNDLE_PATH=/var/app/current/vendor/bundle/ bundle exec rails c
However, if you need ENV, I recommend doing this as per AWS doc:
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-linux2/
tl;dr
On Amazon Linux 2, all environment properties are centralised into a single file called /opt/elasticbeanstalk/deployment/env. No user can access these outside the app. So, they recommend to add some hook scripts after deploy to basically create a copy.
#!/bin/bash
#Create a copy of the environment variable file.
cp /opt/elasticbeanstalk/deployment/env /opt/elasticbeanstalk/deployment/custom_env_var
#Set permissions to the custom_env_var file so this file can be accessed by any user on the instance. You can restrict permissions as per your requirements.
chmod 644 /opt/elasticbeanstalk/deployment/custom_env_var
#Remove duplicate files upon deployment.
rm -f /opt/elasticbeanstalk/deployment/*.bak
If because of some reason you don't want to run as root, do the following to pass env vars from root into new user environment:
sudo -u <user> -E env "PATH=$PATH" bash -c 'cd /var/app/current/ && <wtv you want to run>'
I like to create an eb_console file at the root of my rails app, then chmod u+x it. It contains the following:
ssh -t ec2-user#YOUR_EC2_STATION.compute.amazonaws.com 'cd /var/app/current && bin/rails c'
This way, I just have to run:
./eb_console
like I would have run heroku run bundle exec rails c.
#!/bin/sh
shell_join () {
ruby -r shellwords -e 'puts Shellwords.join(ARGV)' "$#"
}
command_str () {
printf 'set -e; . /etc/profile.d/eb_envvars.sh; . /etc/profile.d/use-app-ruby.sh; set -x; exec %s\n' "$(shell_join "$#")"
}
exec sudo su webapp -c "$(command_str "$#")"
Put above file somewhere in your source code, deploy, eb ssh into the eb instance, cd /var/app/current, and then execute path/to/above/script bin/rails whatever argumeents you usually use.
Reason why I have written above script is:
When using sudo, it drops some environment variables which might actually be needed for your rails app; so manually load the profiles which the Elastic Beanstalk platform provides.
Current Beanstalk ruby platform assumes you run rails application on user webapp, a non-login-able user, so it would be wise to run your command in this user.
For the latest ruby version, please use the following command:
BUNDLE_PATH=/opt/rubies/ruby-2.6.3/lib/ruby/gems/2.6.0/ bundle exec rails c production
Running it with sudo is not needed.
add an eb extension shortcut:
# .ebextensions/irb.config
files:
"/home/ec2-user/irb":
mode: "000777"
owner: root
group: root
content: |
sudo su - -c 'cd /var/app/current; bundle exec rails c'
then:
$ eb ssh
$ ./irb
irb(main):001:0>
None of these were working for me, including the aws-console script. I finally ended up creating a script directory in /var/app/current and then creating a rails file in that directory as outline by this answer on another SO question.
eb ssh myEnv
cd /var/app/current
sudo mkdir script
sudo vim script/rails
Add this to file and save:
echo #!/usr/bin/env ruby
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
APP_PATH = File.expand_path('../../config/application', __FILE__)
require File.expand_path('../../config/boot', __FILE__)
require 'rails/commands'
Then make it executable and run it:
sudo chmod +x script/rails
sudo script/rails console
And it worked.
You have to find the folder with your Gemfile :p.
To do that, I would take a look in you web server config there should be a config that tells you where your app directory is.
Maybe you know where your app is.
But in case you don't know, I would give a try to:
grep -i your_app_name /etc/apache/*
grep -i your_app_name /etc/apache/sites-enabled/*
To search files containing your_app_name in Apache config.
Or if you are using nginx, replace apache above by nginx.
after you find application folder, cd into it and run RAILS_ENV=production bundle exec rails c.
Making sure that your application is configured to run in production in Apache or nginx configuration.