I'm moving a rails app from Heroku to a linux server and deploying it using Caprover. It's an app very dependent on background jobs, which I run with sidekiq.
I've managed to make it work by running both the rails server (bundle exec rails server -b 0.0.0.0 -p80 &) and sidekiq (bundle exec sidekiq &) from a script that launches both in the CMD of the Dockerfile.
But I guess it would be much better (separation of concerns) if the rails server was in one Docker container and sidekiq in another one. But I can't figure out how to connect them. How do I tell my rails app that sidekiq lives in another container?
Because I use Caprover I'm limited to Dockerfiles to deploy my images, so I can't use docker-compose.
Is there a way to tell rails that it should use a certain sidekiq found in a certain Docker container? Caprover uses Docker swarm if that is of any help.
Am I thinking about this the wrong way?
My setup, currently, is as follows:
1 Docker container with rails server + sidekiq
1 Docker container with the postgres DB
1 Docker container with the Redis DB
My desired setup would be:
1 Docker container with rails server
1 Docker container with sidekiq
1 Docker container with postgres DB
1 Docker container with Redis DB
Is that even possible with my current limitations?
My rails + sidekiq Dockerfile is as follows:
FROM ruby:2.6.4-alpine
#
RUN apk update && apk add nodejs yarn postgresql-client postgresql-dev tzdata build-base ffmpeg
RUN apk add --no-cache --upgrade bash
RUN mkdir /myapp
WORKDIR /myapp
COPY Gemfile /myapp/Gemfile
COPY Gemfile.lock /myapp/Gemfile.lock
RUN bundle install --deployment --without development test
COPY . /myapp
RUN yarn
RUN bundle exec rake yarn:install
# Set production environment
ENV RAILS_ENV production
ENV RAILS_SERVE_STATIC_FILES true
# Assets, to fix missing secret key issue during building
RUN SECRET_KEY_BASE=dumb bundle exec rails assets:precompile
# Add a script to be executed every time the container starts.
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 80
COPY start_rails_and_sidekiq.sh /myapp/start_rails_and_sidekiq.sh
RUN chmod +x /myapp/start_rails_and_sidekiq.sh
# Start the main process.
WORKDIR /myapp
CMD ./start_rails_and_sidekiq.sh
the start_rails_and_sidekiq.sh looks like this:
#!/bin/bash
# Start the first process
bundle exec rails server -b 0.0.0.0 -p80 &
status=$?
if [ $status -ne 0 ]; then
echo "Failed to start Rails server: $status"
exit $status
fi
# Start the second process
bundle exec sidekiq &
status=$?
if [ $status -ne 0 ]; then
echo "Failed to start Sidekiq: $status"
exit $status
fi
# Naive check runs checks once a minute to see if either of the processes exited.
# This illustrates part of the heavy lifting you need to do if you want to run
# more than one service in a container. The container exits with an error
# if it detects that either of the processes has exited.
# Otherwise it loops forever, waking up every 60 seconds
while sleep 60; do
ps aux |grep puma |grep -q -v grep
PROCESS_1_STATUS=$?
ps aux |grep sidekiq |grep -q -v grep
PROCESS_2_STATUS=$?
# If the greps above find anything, they exit with 0 status
# If they are not both 0, then something is wrong
if [ $PROCESS_1_STATUS -ne 0 -o $PROCESS_2_STATUS -ne 0 ]; then
echo "One of the processes has already exited."
exit 1
fi
done
I'm totally lost!
Thanks in advance!
Method 1
According to CapRover docs, it seems that it is possible to run Docker Compose on CapRover, but I haven't tried it myself (yet).
Method 2
Although this CapRover example is for a different web app, the Internal Access principle is the same:
You can simply add a srv-captain-- prefix to the name of the container if you want to access it from another container.
However, isn't this method how you told your Rails web app where to find the PostgreSQL DB container? Or are you accessing it through an external subdomain name?
I've got a Python Travis CI build and a Python unit test executes attempts to change the group of a file on the filesystem. The file was previously created by the unit test, so the user executing the test owns the file.
I'm able to start a sub-shell in which I can run chgrp commands (per the Travis guidelines), but unfortunately, this screws up the virtualenv set up for my specific Python version (and who knows what else).
How to replicate (in Travis CI script):
language: python
sudo: true
python:
- "3.4"
- "3.5"
before_install:
- sudo apt-get -qq update
- sudo gpasswd -a $USER fuse
script:
- touch testfile
- chgrp fuse testfile | echo 0 # this does not work - bad
- sudo -E su $USER -c "chgrp fuse testfile" # the sudo / su wrapper is required per Travis instructions, see link above - good
- python --version # reports 3.4 or 3.5 - good
- sudo -E su $USER -c "python --version" # always reports 2.7 - bad
- sudo -E su $USER -c "python --version" # always reports 3.2 - bad
As I've commented in the block above, running a command which attempts to change the group of the testfile (which is what my unit test code is doing) only works when wrapped with sudo -E su $USER -c.
Unfortunately, when I do this, I lose the ability to access python 3.4 and 3.5 in those script phases (which I've specified above) in the virtualenv that Travis has set up for me.
Any idea how I can achieve both of my goals? (allowing chgrp from the travis non-root user while simultaneously not mucking with the virtualenv or the python on the path?
When you create a new group, you have to log out and log in again to be able to use chgrp.
Using sudo is a way around this behavior. Since you're already using it for groupadd and usermod, I suggest changing the last line to sudo chgrp newtravisgroup newfile.
You can also use su to create a new login shell where newtravisgroup will be available but using sudo as mentioned above is the simplest way.
Edit:
When you use su PATH is reset. That's the reason python reverts back to the system python. You can activate the virtualenv again before running your test.
sudo -E su $USER -c "source $VIRTUAL_ENV/bin/activate; python --version"
I have set up a RoR environement on AWS' elastic beanstalk. I am able to ssh into my EC2 instance.
My home directory is /home/ec2-user, which is effectively empty.
If I move up a directory, there is also a /home/webapp directory that i do not have access to.
Is there a way to run a rake command or rails console on my elastic beanstalk instance?
If I type rails console I get Usage: rails new APP_PATH [options]
If I type RAILS_ENV=production bundle exec rails console, I get "Could not locate Gemfile"
For rails, jump to /var/app/current then as #juanpastas said, run RAILS_ENV=production bundle exec rails c
Don't know why, but since EBS runs everything as root, this worked for me:
sudo su
bundle exec rails c production
None of these solutions mentioned here worked for me, so I cooked up a little script that I put in script/aws-console.
You can run it from the /var/app/current directory as root:
eb ssh
cd /var/app/current
sudo script/aws-console
My script can be found as a Gist here.
None of the other answers worked for me so I went looking - this is working for me now on an elastic beanstalk 64bit amazon linux 2016.03 V2.1.2 ruby 2.2 (puma) stack
cd /var/app/current
sudo su
rake rails:update:bin
bundle exec rails console
that returns me the expected console
Loading production environment (Rails 4.2.6)
irb(main):001:0>
For Ruby 2.7:
if you don't need environment variables:
BUNDLE_PATH=/var/app/current/vendor/bundle/ bundle exec rails c
It looks like environment variables are not loaded automatically anymore, which might prevent rails console from starting.
I solved it by creating this .ebextensions file:
# Simply call `sudo /var/app/scripts/rails_c`
commands:
create_script_dir:
command: "mkdir -p /var/app/scripts"
ignoreErrors: true
files:
"/var/app/scripts/export_envvars":
mode: "000755"
owner: root
group: root
content: |
#!/opt/elasticbeanstalk/.rbenv/shims/ruby
if __FILE__ == $0
require 'json'
env_file = '/var/app/scripts/envvars'
env_vars = env_vars = JSON.parse(`/opt/elasticbeanstalk/bin/get-config environment`)
str = ''
env_vars.each do |key, value|
new_key = key.gsub(/\s/, '_')
str << "export #{new_key}=\"#{value}\"\n"
end
File.open(env_file, 'w') { |f| f.write(str) }
end
"/var/app/scripts/rails_c":
mode: "000755"
owner: root
group: root
content: |
. ~/.bashrc
/var/app/scripts/export_envvars
. /var/app/scripts/envvars
cd /var/app/current
/opt/elasticbeanstalk/.rbenv/shims/bundle exec rails c
Create a .ebextension file named setvars.config and add those lines to it
commands:
setvars:
command: /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "export \(.key)=\"\(.value)\""' > /etc/profile.d/sh.local
packages:
yum:
jq: []
Then deploy your code again it should work.
reference: https://aws.amazon.com/ar/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
For Ruby 2.7:
As someone said, if you don't need env vars, run the following
BUNDLE_PATH=/var/app/current/vendor/bundle/ bundle exec rails c
However, if you need ENV, I recommend doing this as per AWS doc:
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-linux2/
tl;dr
On Amazon Linux 2, all environment properties are centralised into a single file called /opt/elasticbeanstalk/deployment/env. No user can access these outside the app. So, they recommend to add some hook scripts after deploy to basically create a copy.
#!/bin/bash
#Create a copy of the environment variable file.
cp /opt/elasticbeanstalk/deployment/env /opt/elasticbeanstalk/deployment/custom_env_var
#Set permissions to the custom_env_var file so this file can be accessed by any user on the instance. You can restrict permissions as per your requirements.
chmod 644 /opt/elasticbeanstalk/deployment/custom_env_var
#Remove duplicate files upon deployment.
rm -f /opt/elasticbeanstalk/deployment/*.bak
If because of some reason you don't want to run as root, do the following to pass env vars from root into new user environment:
sudo -u <user> -E env "PATH=$PATH" bash -c 'cd /var/app/current/ && <wtv you want to run>'
I like to create an eb_console file at the root of my rails app, then chmod u+x it. It contains the following:
ssh -t ec2-user#YOUR_EC2_STATION.compute.amazonaws.com 'cd /var/app/current && bin/rails c'
This way, I just have to run:
./eb_console
like I would have run heroku run bundle exec rails c.
#!/bin/sh
shell_join () {
ruby -r shellwords -e 'puts Shellwords.join(ARGV)' "$#"
}
command_str () {
printf 'set -e; . /etc/profile.d/eb_envvars.sh; . /etc/profile.d/use-app-ruby.sh; set -x; exec %s\n' "$(shell_join "$#")"
}
exec sudo su webapp -c "$(command_str "$#")"
Put above file somewhere in your source code, deploy, eb ssh into the eb instance, cd /var/app/current, and then execute path/to/above/script bin/rails whatever argumeents you usually use.
Reason why I have written above script is:
When using sudo, it drops some environment variables which might actually be needed for your rails app; so manually load the profiles which the Elastic Beanstalk platform provides.
Current Beanstalk ruby platform assumes you run rails application on user webapp, a non-login-able user, so it would be wise to run your command in this user.
For the latest ruby version, please use the following command:
BUNDLE_PATH=/opt/rubies/ruby-2.6.3/lib/ruby/gems/2.6.0/ bundle exec rails c production
Running it with sudo is not needed.
add an eb extension shortcut:
# .ebextensions/irb.config
files:
"/home/ec2-user/irb":
mode: "000777"
owner: root
group: root
content: |
sudo su - -c 'cd /var/app/current; bundle exec rails c'
then:
$ eb ssh
$ ./irb
irb(main):001:0>
None of these were working for me, including the aws-console script. I finally ended up creating a script directory in /var/app/current and then creating a rails file in that directory as outline by this answer on another SO question.
eb ssh myEnv
cd /var/app/current
sudo mkdir script
sudo vim script/rails
Add this to file and save:
echo #!/usr/bin/env ruby
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
APP_PATH = File.expand_path('../../config/application', __FILE__)
require File.expand_path('../../config/boot', __FILE__)
require 'rails/commands'
Then make it executable and run it:
sudo chmod +x script/rails
sudo script/rails console
And it worked.
You have to find the folder with your Gemfile :p.
To do that, I would take a look in you web server config there should be a config that tells you where your app directory is.
Maybe you know where your app is.
But in case you don't know, I would give a try to:
grep -i your_app_name /etc/apache/*
grep -i your_app_name /etc/apache/sites-enabled/*
To search files containing your_app_name in Apache config.
Or if you are using nginx, replace apache above by nginx.
after you find application folder, cd into it and run RAILS_ENV=production bundle exec rails c.
Making sure that your application is configured to run in production in Apache or nginx configuration.
I'm deploying my Rails App servers with Chef. Have just swapped to RVM from a source install of Ruby (because I was having issues with my deploy user).
Now I have my deploy sorted, assets compiled and bundler's installed all my gems.
The problem I have is supervising Unicorn with Runit..
RVM is not installed as root user - only as my deploy user has it, as follows:
$ rvm list
rvm rubies
=* ruby-2.0.0-p247 [ x86_64 ]
I can manually start Unicorn successfully from my deploy user. However, it won't start as part of runit.
My run file looks like this. I have also tried the solution in this SO question unsuccessfully..
#!/bin/bash
cd /var/www/html/deploy/production/current
exec 2>&1
exec chpst -u deploy:deploy /home/deploy/.rvm/gems/ruby-2.0.0-p247/bin/unicorn -E production -c config/unicorn_production.rb
If I run it manually, I get this error:
/usr/bin/env: ruby_noexec_wrapper: No such file or directory
I created a small script (gist here) which does run as root. However, if I call this from runit, I see the workers start but I get two processes for runit and I can't stop or restart the service:
Output of ps:
1001 29062 1 0 00:08 ? 00:00:00 unicorn master -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
1001 29065 29062 9 00:08 ? 00:00:12 unicorn worker[0] -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
root 29076 920 0 00:08 ? 00:00:00 su - deploy -c cd /var/www/html/deploy/production/current; export GEM_HOME=/home/deploy/.rvm/gems/ruby-2.0.0-p247; /home/deploy/.rvm/gems/ruby-2.0.0-p247/bin/unicorn -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
1001 29083 29076 0 00:08 ? 00:00:00 -su -c cd /var/www/html/deploy/production/current; export GEM_HOME=/home/deploy/.rvm/gems/ruby-2.0.0-p247; /home/deploy/.rvm/gems/ruby-2.0.0-p247/bin/unicorn -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
What should I do here? Move back to monit which worked nicely?
your run file is doing it wrong, you are using the binary without setting the environment, for that purpose you should use wrappers:
rvm wrapper ruby-2.0.0-p247 --no-links unicorn
To simplify the script use alias so it does not need to be changed when you decide which ruby should be used:
rvm alias create my_app_unicorn ruby-2.0.0-p247
And change the script to:
#!/bin/bash
cd /var/www/html/deploy/production/current
exec 2>&1
exec chpst -u deploy:deploy /home/deploy/.rvm/wrappers/my_app_unicorn/unicorn -E production -c config/unicorn_production.rb
This will ensure proper environment is used for execution of unicorn and any time you want change ruby used to run it just crate alias to a new ruby.
I'm using the init.d script provided (the init.d script from the sidekiq github repo), but I am on an ubuntu system with RVM installed system wide.
I cannot seem to figure out how to cd into my app directory and issue the command without there being some complaining in the log and nothing actually starting.
Question: What should the startup command for sidekiq look like in my init.d script when I am using RVM? My user is named ubuntu. Currently I have this in my init.d script:
START_CMD="$BUNDLE exec $SIDEKIQ"
# where bundle is /usr/local/rvm/gems/ruby-1.9.3-p385/bin/bundle
# and sidekiq is sidekiq
# I've also tried with the following args: -e $APP_ENV -P $PID_FILE -C $APP_CONFIG/sidekiq.yml -d -L $LOG_FILE"
RETVAL=0
start() {
status
if [ $? -eq 1 ]; then
[ `id -u` == '0' ] || (echo "$SIDEKIQ runs as root only .."; exit 5)
[ -d $APP_DIR ] || (echo "$APP_DIR not found!.. Exiting"; exit 6)
cd $APP_DIR
echo "Starting $SIDEKIQ message processor .. "
echo "in dir `pwd`"
su - ubuntu -c "$START_CMD >> $LOG_FILE 2>&1 &"
RETVAL=$?
#Sleeping for 8 seconds for process to be precisely visible in process table - See status ()
sleep 8
[ $RETVAL -eq 0 ] && touch $LOCK_FILE
return $RETVAL
else
echo "$SIDEKIQ message processor is already running .. "
fi
}
My sidekiq.log gives me this error:Could not locate Gemfile. However, I print the working directory and I am most definitely in my app's current directory, according to the echo pwd, at the time this command is executed.
When I take out the su - ubuntu -c [command here], I get this error:
/usr/bin/env: ruby_noexec_wrapper: No such file or directory
My solution is to just start the process manually. When I manually cd into my app directory and issue this command:
bundle exec sidekiq -d -L log/sidekiq.log -P tmp/pids/sidekiq.pid
things go as planned, and then
sudo /etc/init.d/sidekiq status
tells me things are up and running.
Also, sudo /etc/init.d/sidekiq stop and status work as expected.
I wrote a blog post a few months ago on my experience writing an init.d script for Sidekiq, however I was using rbenv rather than RVM.
https://cdyer.co.uk/blog/init-script-for-sidekiq-with-rbenv/
I think you should be able to use something almost identical except for modifying the username and app dir variables.
use wrappers:
BUNDLER=/usr/local/rvm/wrappers/ruby-1.9.3-p385/bundle
in case the bundler wrapper is not available generate it with:
rvm wrapper ruby-1.9.3-p385 --no-links bundle # OR:
rvm wrapper ruby-1.9.3-p385 --no-links --all
you can use aliases to make it easier:
rvm alias create my_app 1.9.3-p385
and then use it like this:
BUNDLER=/usr/local/rvm/wrappers/my_app/bundle
this way you will not have to change the script when application ruby changes - just update the alias, there is a good description/integration for this in rvm-capistrano => https://github.com/wayneeseguin/rvm-capistrano/#create-application-alias-and-wrappers