Trying to setup whenever cron jobs on my app that runs on a docker setup with redis and sidekiq in separate containers, but haven't been able to set them to fire on a time schedule, manually it works fine.
schedule.rb file
env :PATH, ENV['PATH']
set :output, "log/cron.log"
every 1.minutes do
runner "SampleJob.perform_async"
end
Test job I am trying to run:
class SampleJob
include Sidekiq::Worker
def perform(*args)
p 'Job is running'
end
end
my Dockerfile
FROM ruby:2.7.1
ENV LANG C.UTF-8
ENV NODE_VERSION 12
ENV NODE_ENV production
ENV INSTALL_PATH /home/app/app/current
RUN curl -sL https://deb.nodesource.com/setup_$NODE_VERSION.x | bash -
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qq
RUN apt-get install -y --no-install-recommends nodejs postgresql-client yarn build-essential vim
RUN apt-get install -y cron
RUN mkdir -p $INSTALL_PATH
WORKDIR $INSTALL_PATH
COPY Gemfile Gemfile.lock ./
RUN gem install bundler
RUN bundle install
COPY . $INSTALL_PATH
RUN rm -rf tmp
RUN useradd -Ms /bin/bash api -u 1001
RUN chown -R api:api /home/app /usr/local/bundle
USER root
RUN touch /home/app/app/current/log/cron.log
RUN bundle exec whenever --update-crontab --set environment='development'
CMD cron && bundle exec puma
RUN service cron start
EXPOSE 3100
CMD rails server -p 3100 -b 0.0.0.0
running crontab -l inside my app's container gives me:
# Begin Whenever generated tasks for: /home/app/limpar-api/current/config/schedule.rb at: 2021-05-05 17:42:43 +0000
PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/b
.
. (many path variables)
.
* * * * * /bin/bash -l -c 'cd /home/app/limpar-api/current && bundle exec bin/rails runner -e development '\''SampleJob.perform_async'\'' >> log/cron.log 2>&1'
# End Whenever generated tasks for: /home/app/limpar-api/current/config/schedule.rb at: 2021-05-05 17:42:43 +0000
Any step I might be missing from the mix?
Related
I have a Dockerfile with some commands I would like to use conditionally:
FROM + image_name (I have a M1 chip MacOS so I need to add --platform=linux/amd64 to it but I want to deploy in a AWS EC2 linux instance that doesn't need it)
On production I would like to run my project with nginx so I want the Dockerfile to end with this RUN mkdir -p tmp/sockets. But for testing, I have no need of the nginx so I would like my Dockerfile to end with this
# Expose port
EXPOSE 3000
# Start rails
CMD ["rails", "server", "-b", "0.0.0.0"]
I thought of using the multi stage dockerfile to solve the FROM image problem but the Dockerfile resulting is quite lengthy since it is basically the same except for the FROM image part.
For the nginx part I wanted to use a shell script but I am not sure how to write the exposing port and final command to start rails.
These are the files:
run_dockerfile.sh
#!/bin/bash
if [ ${RUN_DOCKERFILE} = "PROD" ]; then
mkdir -p tmp/sockets
else
????
fi
My Dockerfilelooks like this:
# Start from the official ruby image
# To run Dockerfile with arm64 architecture (M1 chip MacOS for example)
FROM --platform=linux/amd64 ruby:2.6.6 AS ARM64
# Set environment
ARG BUILD_DEVELOPMENT
# if --build-arg BUILD_DEVELOPMENT=1, set RAILS_ENV to 'development' or set to null otherwise.
ENV RAILS_ENV=${BUILD_DEVELOPMENT:+development}
# if RAILS_ENV is null, set it to 'production' (or leave as is otherwise).
ENV RAILS_ENV=${RAILS_ENV:-production}
# Update and install JS & DB
RUN apt-get update -qq && apt-get install -y nodejs postgresql-client
# Create a directory for the application and use it
RUN mkdir /myapp
WORKDIR /myapp
# Gemfile and lock file need to be present, they'll be overwritten immediately
COPY Gemfile /myapp/Gemfile
COPY Gemfile.lock /myapp/Gemfile.lock
# Install gem dependencies
RUN gem install bundler:2.2.32
RUN bundle install
RUN curl https://deb.nodesource.com/setup_12.x | bash
ADD https://dl.yarnpkg.com/debian/pubkey.gpg /tmp/yarn-pubkey.gpg
RUN apt-key add /tmp/yarn-pubkey.gpg && rm /tmp/yarn-pubkey.gpg
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qq && apt-get install -y yarn && apt-get install -y npm
RUN yarn add bootstrap
COPY . /myapp
# So that webpacker compiles
RUN yarn config set ignore-engines true
RUN rm -rf bin/webpack*
RUN rails webpacker:install
RUN bundle exec rails webpacker:compile
RUN bundle exec rake assets:precompile
# This script runs every time the container is created, necessary for rails
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
# Run run_dockerfile.sh
COPY run_dockerfile.sh run_dockerfile.sh
RUN chmod u+x run_dockerfile.sh && ./run_dockerfile.sh
##################################################
# Start from the official ruby image
# To run Dockerfile without arm64 architecture
FROM ruby:2.6.6 AS AMD64
# Set environment
ARG BUILD_DEVELOPMENT
# if --build-arg BUILD_DEVELOPMENT=1, set RAILS_ENV to 'development' or set to null otherwise.
ENV RAILS_ENV=${BUILD_DEVELOPMENT:+development}
# if RAILS_ENV is null, set it to 'production' (or leave as is otherwise).
ENV RAILS_ENV=${RAILS_ENV:-production}
# Update and install JS & DB
RUN apt-get update -qq && apt-get install -y nodejs postgresql-client
# Create a directory for the application and use it
RUN mkdir /myapp
WORKDIR /myapp
# Gemfile and lock file need to be present, they'll be overwritten immediately
COPY Gemfile /myapp/Gemfile
COPY Gemfile.lock /myapp/Gemfile.lock
# Install gem dependencies
RUN gem install bundler:2.2.32
RUN bundle install
RUN curl https://deb.nodesource.com/setup_12.x | bash
ADD https://dl.yarnpkg.com/debian/pubkey.gpg /tmp/yarn-pubkey.gpg
RUN apt-key add /tmp/yarn-pubkey.gpg && rm /tmp/yarn-pubkey.gpg
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qq && apt-get install -y yarn && apt-get install -y npm
RUN yarn add bootstrap
COPY . /myapp
# So that webpacker compiles
RUN yarn config set ignore-engines true
RUN rm -rf bin/webpack*
RUN rails webpacker:install
RUN bundle exec rails webpacker:compile
RUN bundle exec rake assets:precompile
# This script runs every time the container is created, necessary for rails
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
# Run run_dockerfile.sh
COPY run_dockerfile.sh run_dockerfile.sh
RUN chmod u+x run_dockerfile.sh && ./run_dockerfile.sh
Is there any way I could do the .sh or are there any recommendations on the proper way to do it? Thank you!
From the way you've described the problem, you don't really need very many special cases at all.
The one important detail is that it's very easy to override the image's CMD when you run a container. If you have two Compose files, for example, you can just set the service's command:
# docker-compose.yml
version: '3.8'
services:
myapp:
image: registry.example.com/myapp:${MYAPP_TAG:-latest}
ports: ['3000:80']
# docker-compose.override.yml
# for developer use
version: '3.8'
services:
myapp:
build: .
command: rails server -b 0.0.0.0 -p 80
The other variations you list shouldn't matter. You should get consistent results if you build your image FROM --platform=linux/amd64 on an x86-64 host, explicitly specifying the native platform; RUN mkdir a directory you won't use is harmless. The one inconsistency seems to be the container port, but you can explicitly tell rails server which port to use so it matches. I'd use the same image in all environments.
FROM --platform=linux/amd64 ruby:2.6.6 # even on an Intel/AMD host system
...
RUN mkdir tmp/sockets # even if it's unused
CMD ["nginx", "-g", "daemon off;"] # can be overridden when the container runs
I am running a docker image and whenever I make any new changes inside the app folder such as in controllers or views the changes are not relfected in browser. Everytime I need to create a new image inorder to reflect the changes.
Dockerfile
FROM ruby:3.0
USER root
RUN apt-get update && apt-get install -y build-essential libpq-dev nodejs
RUN rm -rf .idea \
rm -rf .gitignore
ENV LC_ALL=en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US.UTF-8
RUN mkdir -p /app
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN gem install bundler && bundle install --jobs 20 --retry 5
COPY . ./
# Adding a non root user
RUN groupadd --system non-root-user
RUN useradd -u 1001 --system non-root-user -g non-root-user -d /home/non-root-user -m -s /bin/bash
RUN chown -R non-root-user /app
RUN chmod -R 777 /app
# Switching to non-root user 'non-root-user'
USER non-root-user
# Exposing port 3010
EXPOSE 3010
# Default entry point (i.e. will be appened before every command being executed)
ENTRYPOINT ["sh", "./config/docker/startup.sh"]
startup.sh
#! /bin/sh
echo "Creating tmp/pids..."
mkdir -p tmp/pids/
# Start Application
echo "Starting up puma(app server)..."
bundle exec puma -p 3010 -C ./config/puma.rb
I have updated the below configuration in my development.rb still no success
config.file_watcher = ActiveSupport::FileUpdateChecker
Ruby - 3.0.0
Rails - 6.1
I am running the container with the below command
docker run -it -p 3010:3010 -e RAILS_ENV=development my_image
We are running sphinx through supervisord inside docker. We are trying to enable autorestart, but it does work. We are trying to verify this by manually killing the searchd process.
Below is the configuration
[program:sphinx]
command=searchd --pidfile --config "config/sphinx.conf"
autostart=true
autorestart=unexpected
startsecs=5
startretries=3
exitcodes=0
We also attempted other configuration
program:sphinx]
command= rake ts:stop
command= rake ts:configure
command= rake ts:start
autostart=true
exitcodes=0,2
autorestart=unexpected
Are we missing something?
Dockerfile
FROM ruby:2.3
RUN apt-get update
#Install Sphinx
RUN wget -P /tmp http://sphinxsearch.com/files/sphinx-2.3.2-beta.tar.gz
RUN mkdir /opt/sphinx_src
RUN tar -xzvf /tmp/sphinx-2.3.2-beta.tar.gz -C /opt/sphinx_src
WORKDIR /opt/sphinx_src/sphinx-2.3.2-beta
RUN ./configure --with-pgsql --with-mysql
RUN make
RUN make install
RUN apt-get install -q -y supervisor cron
ADD supervisor-cron.conf /etc/supervisor/conf.d/cron.conf
RUN service supervisor stop
RUN apt-get install -y net-tools telnet
WORKDIR /opt/app
ADD start.sh /usr/local/sbin/start
RUN chmod 755 /usr/local/sbin/start
EXPOSE 9312
CMD ["/usr/local/sbin/start"]
start.sh
#!/bin/sh
export set BUNDLE_GEMFILE=Gemfile
cd /opt/app
bundle install
echo "About to perform Sphinx Start"
cp /opt/app/supervisor-sphinx.conf /etc/supervisor/conf.d/sphinx.conf
supervisord -n
My crontasks from schedule.rb doesn't work on docker container, but crontab -l result already contains this lines:
# Begin Whenever generated tasks for: /app/config/schedule.rb
45 19 * * * /bin/bash -l -c 'bundle exec rake stats:cleanup'
45 19 * * * /bin/bash -l -c 'bundle exec rake stats:count'
0 5 * * * /bin/bash -l -c 'bundle exec rake stats:history'
# End Whenever generated tasks for: /app/config/schedule.rb
I can run this commands manually in container and it works. It seems like cron doesn't start.
Dockerfile:
FROM ruby:2.4.0-slim
RUN apt-get update
RUN apt-get install -qq -y --no-install-recommends build-essential libpq-dev cron postgresql-client
RUN cp /usr/share/zoneinfo/Europe/Moscow /etc/localtime
ENV LANG C.UTF-8
ENV RAILS_ENV production
ENV INSTALL_PATH /app
RUN mkdir $INSTALL_PATH
RUN touch /log/cron.log
ADD Gemfile Gemfile.lock ./
WORKDIR $INSTALL_PATH
RUN bundle install --binstubs --without development test
COPY . .
RUN bundle exec whenever --update-crontab
RUN service cron start
ENTRYPOINT ["bundle", "exec", "puma"]
In a Dockerfile, the RUN command is only execute when building the image.
If you want to start cron when you start your container, you should run cron in CMD. I modified your Dockerfile by removing RUN service cron start and changing your ENTRYPOINT.
FROM ruby:2.4.0-slim
RUN apt-get update
RUN apt-get install -qq -y --no-install-recommends build-essential libpq-dev cron postgresql-client
RUN cp /usr/share/zoneinfo/Europe/Moscow /etc/localtime
ENV LANG C.UTF-8
ENV RAILS_ENV production
ENV INSTALL_PATH /app
RUN mkdir $INSTALL_PATH
RUN touch /log/cron.log
ADD Gemfile Gemfile.lock ./
WORKDIR $INSTALL_PATH
RUN bundle install --binstubs --without development test
COPY . .
RUN bundle exec whenever --update-crontab
CMD cron && bundle exec puma
It's a best practice to reduce the number of layer an image have, for example, you should always combine RUN apt-get update with apt-get install in the same RUN statement, and clean apt files after: rm -rf /var/lib/apt/lists/*
FROM ruby:2.4.0-slim
RUN apt-get update && \
apt-get install -qq -y --no-install-recommends build-essential libpq-dev cron postgresql-client \
rm -rf /var/lib/apt/lists/* && \
cp /usr/share/zoneinfo/Europe/Moscow /etc/localtime
ENV LANG C.UTF-8
ENV RAILS_ENV production
ENV INSTALL_PATH /app
RUN mkdir $INSTALL_PATH && \
touch /log/cron.log
ADD Gemfile Gemfile.lock ./
WORKDIR $INSTALL_PATH
RUN bundle install --binstubs --without development test
COPY . .
RUN bundle exec whenever --update-crontab
CMD cron && bundle exec puma
The following Dockerfile it's supposed to start sidekiq when the container it's started:
FROM phusion/baseimage:0.9.18
ENV RUBY_MAJOR "2.0"
ENV RUBY_VERSION "2.0.0-p643"
ENV RUBYGEMS_VERSION "2.5.2"
ENV BUNDLER_VERSION "1.11.2"
ENV NODE_VERSION "5.5.0"
ENV BOWER_VERSION "1.7.6"
ENV APT_PACKAGES " \
git imagemagick \
gcc g++ make patch binutils libc6-dev \
libjemalloc-dev libffi-dev libssl-dev libyaml-dev zlib1g-dev libgmp-dev \
libxml2-dev libxslt1-dev libpq-dev libreadline-dev libsqlite3-dev htop \
"
ENV APT_REMOVE_PACKAGES "openssh-server"
RUN apt-get update && apt-get -y dist-upgrade
RUN apt-get install -y $APT_PACKAGES
RUN apt-get remove --purge -y $APT_REMOVE_PACKAGES
RUN apt-get autoremove --purge -y
WORKDIR /tmp
RUN curl -o ruby.tgz \
"https://cache.ruby-lang.org/pub/ruby/${RUBY_MAJOR}/ruby-${RUBY_VERSION}.tar.gz" && \
tar -xzf ruby.tgz && \
cd ruby-${RUBY_VERSION} && \
./configure --enable-shared --with-jemalloc --disable-install-doc && \
make -j4 && \
make install && \
rm /usr/local/lib/libruby-static.a
ENV GEM_SPEC_CACHE "/tmp/gemspec"
RUN echo 'gem: --no-document' > $HOME/.gemrc
RUN gem update --system ${RUBYGEMS_VERSION}
RUN gem install -v ${BUNDLER_VERSION} bundler
RUN curl https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.gz \
|tar -xz -C /usr --strip-components=1
RUN npm install bower#${BOWER_VERSION} -g
RUN rm /etc/my_init.d/00_regen_ssh_host_keys.sh
RUN rm -r /etc/service/sshd
RUN useradd -m app
RUN mkdir /home/app/webapp && chown app:app -R /home/app
RUN rm -rf /tmp/* /var/tmp/* /var/lib/apt /var/lib/dpkg /usr/share/man /usr/share/doc
WORKDIR /home/app/webapp
# Copy the Gemfile as well as the Gemfile.lock and install
# the RubyGems. This is a separate step so the dependencies
# will be cached unless changes to one of those two files
# are made.
COPY Gemfile Gemfile.lock ./
RUN gem install bundler && bundle install --jobs 20 --retry 5 --without development test
# Copy the main application.
COPY . ./
COPY .cron/daily_run_app /etc/cron.daily/
RUN chmod a+x /etc/cron.daily/daily_run_app
# Precompile Rails assets
RUN bundle exec rake assets:precompile
EXPOSE 3000
COPY .conf_files/sidekiq.sh /etc/service/sidekiq/run
COPY .conf_files/appserver.sh /etc/service/appserver/run
# Start puma
RUN chmod a+x /etc/service/appserver/run
ENTRYPOINT /etc/service/appserver/run
#ENTRYPOINT bundle exec puma -C config/puma.rb
# Start sidekiq
#ENTRYPOINT bundle exec sidekiq -L log/sidekiq.log
#RUN chmod a+x /etc/service/sidekiq/run
#ENTRYPOINT /etc/service/sidekiq/run
The script /etc/service/appserver/run:
#!/bin/bash
SIDEKIQ_THREADS=${SIDEKIQ_THREADS:-16}
cd /home/app/webapp
exec chpst -u app bundle exec puma -C config/puma.rb \
2>&1 |logger -t appserver
SIDEKIQ_CONFIG="/home/app/webapp/config/sidekiq.yml"
exec chpst -u root bundle exec sidekiq -t 5 -c $SIDEKIQ_THREADS -C $SIDEKIQ_CONFIG \
2>&1 |logger -t sidekiq
You can enquequed jobs because you can see that in the sidekiq dasboard, but they are not going to be processed because sidekiq it's not started, but it should be started with the Dockerfile and Script that are shown above. So when you type manually:
bundle exec sidekiq
Right here sidekiq it's started and the jobs now could be processed. what could be happening?