I have two questions:
How can I add a heroku worker just before running a delayed job and remove it after it finishes?
Is my cron.rake ok?
cron.rake:
desc "This task is called by the Heroku cron add-on"
task :cron => :environment do
puts "requesting homepage to refresh cache"
uri = URI.parse('http://something.com')
Net::HTTP.get(uri)
puts "end requesting homepage"
puts "start sending daily mail"
User.notified_today.each do |user|
Delayed::Job.enqueue UserMailer.daily_mail(user).deliver
end
puts "end sending daily mail"
end
I use collectiveidea delayed_job.
I've had good success with HireFire.
Easy setup:
Add gem 'hirefire' to your Gemfile
Create Rails.root/config/initializers/hirefire.rb with the config information.
To add remove/remove workers, hook into your ORM's after :create / after :destroy
With DataMapper on Heroku, I did it like this (You must set the ENV vars yourself)
MAX_CONCURRENT_WORKERS = 5
if ENV["HEROKU_APP"]
Delayed::Job.after :create do
workers_needed = [Delayed::Job.count, MAX_CONCURRENT_WORKERS].min
client = Heroku::Client.new(ENV['HEROKU_USERNAME'], ENV['HEROKU_PASSWORD'])
client.set_workers(ENV['HEROKU_APP'], workers_needed)
puts "- Initialized Heroku workers for ZipDecoder"
end
Delayed::Job.after :destroy do
workers_needed = [Delayed::Job.count, MAX_CONCURRENT_WORKERS].min
client = Heroku::Client.new(ENV['HEROKU_USERNAME'], ENV['HEROKU_PASSWORD'])
client.set_workers(ENV['HEROKU_APP'], workers_needed)
puts "- Cleaned Up a Delayed Job for ZipDecoder ---------------------------------"
end
end
You maybe can use an "autoscale" plugin like workless or heroku-autoscale.
About the cron I don't see any problem on it...
Related
What is the best way to test something that requires background jobs with Cucumber? I need to run DelayedJob and Sneakers workers in background while tests are running.
You can run any application in the background:
#pid = Process.spawn "C:/Apps/whatever.exe"
Process.detach(#pid)
And even kill it after tests are done:
Process.kill('KILL', #pid) unless #pid.nil?
You can create your own step definition in features/step_definitions/whatever_steps.rb (hopefully with a better name)
When /^I wait for background jobs to complete$/ do
Delayed::Worker.new.work_off
end
That can be extended for any other scripts you'd like to run with that step. Then in the test, it goes something like:
Then I should see the text "..."
When I wait for background jobs to complete
And I refresh the page
Then I should see the text "..."
If anyone has similar problem I ended up writing this (thanks to Square blog post):
require "timeout"
class CucumberExternalWorker
attr_accessor :worker_pid, :start_command
def initialize(start_command)
raise ArgumentError, "start_command was expected" if start_command.nil?
self.start_command = start_command
end
def start
puts "Trying to start #{start_command}..."
self.worker_pid = fork do
start_child
end
at_exit do
stop_child
end
end
private
def start_child
exec({ "RAILS_ENV" => Rails.env }, start_command)
end
def stop_child
puts "Trying to stop #{start_command}, pid: #{worker_pid}"
# send TERM and wait for exit
Process.kill("TERM", worker_pid)
begin
Timeout.timeout(10) do
Process.waitpid(worker_pid)
puts "Process #{start_command} stopped successfully"
end
rescue Timeout::Error
# Kill process if could not exit in 10 seconds
puts "Sending KILL signal to #{start_command}, pid: #{worker_pid}"
Process.kill("KILL", worker_pid)
end
end
end
This can be called as following (added it to env.rb for cucumber):
# start delayed job
$delayed_job_worker = CucumberExternalWorker.new("rake jobs:work")
$delayed_job_worker.start
I have a rake file that is being called by a job scheduler. The file outputs the desc but I am not able to log anything else to the console. What am I missing?
inbox.rake
namespace :inbox do
desc 'Check inbox for new app builds'
task process_inbox: :environment do
puts "my task is working"
end
end
Similar to Heroku logs, you need STDOUT to see the outputs. Could be as simple as
my_logger = Logger.new(STDOUT)
my_logger.info "work or die"
puts sends the text to STDOUT, which is different when you run rake from the terminal versus invoking from another ruby process.
Where do you expect to see this text?
Try manually printing to console.
namespace :inbox do
desc 'Check inbox for new app builds'
task process_inbox: :environment do
Rails.logger.info "my task is working"
end
end
You're not using --quiet / --silent are you?
I've got the mailman gem integrated into my rails project. It fetches emails from gmail successfully. In my app there is a model Message for my emails. The emails are properly saved as Message model.
The problem is that the emails are saved multiple times sometimes and I can't recognize a pattern. Some emails are saved once, some two times and some are saved three times.
But I can't find the failure in my code.
Here is my mailman_server script:
script/mailman_server
#!/usr/bin/env ruby
# encoding: UTF-8
require "rubygems"
require "bundler/setup"
require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'mailman'
Mailman.config.ignore_stdin = true
#Mailman.config.logger = Logger.new File.expand_path("../../log/mailman_#{Rails.env}.log", __FILE__)
if Rails.env == 'test'
Mailman.config.maildir = File.expand_path("../../tmp/test_maildir", __FILE__)
else
Mailman.config.logger = Logger.new File.expand_path("../../log/mailman_#{Rails.env}.log", __FILE__)
Mailman.config.poll_interval = 15
Mailman.config.imap = {
server: 'imap.gmail.com',
port: 993, # usually 995, 993 for gmail
ssl: true,
username: 'my#email.com',
password: 'my_password'
}
end
Mailman::Application.run do
default do
begin
Message.receive_message(message)
rescue Exception => e
Mailman.logger.error "Exception occurred while receiving message:\n#{message}"
Mailman.logger.error [e, *e.backtrace].join("\n")
end
end
end
The email is processed inside my Message class:
def self.receive_message(message)
if message.from.first == "my#email.com"
Message.save_bcc_mail(message)
else
Message.save_incoming_mail(message)
end
end
def self.save_incoming_mail(message)
part_to_use = message.html_part || message.text_part || message
if Kontakt.where(:email => message.from.first).empty?
encoding = part_to_use.content_type_parameters['charset']
Message.create topic: message.subject, message: part_to_use.body.decoded.force_encoding(encoding).encode('UTF-8'), communication_partner: message.from.first, inbound: true, time: message.date
else
encoding = part_to_use.content_type_parameters['charset']
Message.create topic: message.subject, message: part_to_use.body.decoded.force_encoding(encoding).encode('UTF-8'), communication_partner: message.from.first, inbound: true, time: message.date, messageable_type: 'Company', messageable_id: Kontakt.where(:email => message.from.first).first.year.id
end
end
def self.save_bcc_mail(message)
part_to_use = message.html_part || message.text_part || message
if Kontakt.where(:email => message.to.first).empty?
encoding = part_to_use.content_type_parameters['charset']
Message.create topic: message.subject, message: part_to_use.body.decoded.force_encoding(encoding).encode('UTF-8'), communication_partner: message.to.first, inbound: false, time: message.date
else
encoding = part_to_use.content_type_parameters['charset']
Message.create topic: message.subject, message: part_to_use.body.decoded.force_encoding(encoding).encode('UTF-8'), communication_partner: message.to.first, inbound: false, time: message.date, messageable_type: 'Company', messageable_id: Kontakt.where(:email => message.to.first).first.year.id
end
end
I have daemonized the mailman_server with this script:
script/mailman_daemon
#!/usr/bin/env ruby
require 'rubygems'
require "bundler/setup"
require 'daemons'
Daemons.run('script/mailman_server')
I deploy with capistrano.
This are the parts which are responsible for stopping, starting and restarting my mailman_server:
script/deploy.rb
set :rails_env, "production" #added for delayed job
after "deploy:stop", "delayed_job:stop"
after "deploy:start", "delayed_job:start"
after "deploy:restart", "delayed_job:restart"
after "deploy:stop", "mailman:stop"
after "deploy:start", "mailman:start"
after "deploy:restart", "mailman:restart"
namespace :deploy do
desc "mailman script ausfuehrbar machen"
task :mailman_executable, :roles => :app do
run "chmod +x #{current_path}/script/mailman_server"
end
desc "mailman daemon ausfuehrbar machen"
task :mailman_daemon_executable, :roles => :app do
run "chmod +x #{current_path}/script/mailman_daemon"
end
end
namespace :mailman do
desc "Mailman::Start"
task :start, :roles => [:app] do
run "cd #{current_path};RAILS_ENV=#{fetch(:rails_env)} bundle exec script/mailman_daemon start"
end
desc "Mailman::Stop"
task :stop, :roles => [:app] do
run "cd #{current_path};RAILS_ENV=#{fetch(:rails_env)} bundle exec script/mailman_daemon stop"
end
desc "Mailman::Restart"
task :restart, :roles => [:app] do
mailman.stop
mailman.start
end
end
Could it be that multiple instances of the mailman server are started during my deploy at nearly the same time and then each instance polls nearly at the same time? The second and third instance pools before the first instance marks the email as read and polls and processes the email as well?
Update 30.01.
I had set the polling intervall to 60 seconds. but that changes nothing.
I checked the folder where the mailman pid file is stored. there is only one mailman pid file. So there is definitely only one mailman server running. I checked the logfile and can see, that the messages are fetched multiple times:
Mailman v0.7.0 started
IMAP receiver enabled (my#email.com).
Polling enabled. Checking every 60 seconds.
Got new message from 'my.other#email.com' with subject 'Test nr 0'.
Got new message from 'my.other#email.com' with subject 'Test nr 1'.
Got new message from 'my.other#email.com' with subject 'test nr 2'.
Got new message from 'my.other#email.com' with subject 'test nr 2'.
Got new message from 'my.other#email.com' with subject 'test nr 3'.
Got new message from 'my.other#email.com' with subject 'test nr 4'.
Got new message from 'my.other#email.com' with subject 'test nr 4'.
Got new message from 'my.other#email.com' with subject 'test nr 4'.
So that seems to me, that the problem is definitely in my mailman server code.
Update 31.1.
Seems to me, that is has something to do with my production machine. when I'm testing this in development with the exact same configuration (changed my local database from sqlite to mysql this morning to test it) as on the production machine I don't get duplicates. Probably is everything ok with my code, but there is a problem with the production machine. Will ask my hoster if they could see a solution for this. To fix this I will go with Ariejan'S suggestion.
The solution:
I found the problem. I deploy to a machine where the tmp directory is a shared one between all releases. I forgot to define the path where the pid file of the mailman_daemon should be saved. So it was saved in the script directory instead of the /tmp/pids directory. Because of this the old mailman_daemon could not be stopped after a new deploy. That had led to an army of working mailman_daemons which were polling my mailaccount... After killing all these processes all went well! No more duplicates!
This may be some concurrency/timing issue. E.g. new mails are imported before the ones currently processing have been saved.
Edit: Just noticed you have Mailman.config.poll_interval set to 15. This means it will check for new messages every 15 seconds. Try increasing this value to the default 60 seconds. Regardless of this setting, it might be a good idea to add the deduplication code I mentioned below.
My tip would be to also store the message_id from each email, so you can easily spot duplicates.
Instead of:
Message.create(...)
do:
# This makes sure you have the latest pulled version.
message = Message.find_or_create(message_id: message.message_id)
message.update_attributes(...)
# This makes sure you only import it once, then ignore further duplicates.
if !Message.where(message_id: message.message_id).exists?
Message.create(...)
end
For more info on message_id: http://rdoc.info/github/mikel/mail/Mail/Message#message_id-instance_method
Remember that email and imap are not meant to be consistent data stores like you'd expect Postgres or Mysql to be. Hope this helps you sort out the duplicate mails.
I found the problem. I deploy to a machine where the tmp directory is a shared one between all releases. I forgot to define the path where the pid file of the mailman_daemon should be saved. So it was saved in the script directory instead of the /tmp/pids directory. Because of this the old mailman_daemon could not be stopped after a new deploy. That had led to an army of working mailman_daemons which were polling my mailaccount... After killing all these processes all went well! No more duplicates!
I use http caching on Heroku:
def homepage
response.headers['Cache-Control'] = 'public, max-age=86340'
...
end
I also have added Heroku's free cron addon:
desc "This task is called by the Heroku cron add-on"
task :cron => :environment do
if Time.now.hour == 0 # run at midnight
# I want to request a page here
end
end
Could you tell me what should I put inside this file in order to request a page?
A similar question has been asked at How to force fragment cache on rails from cron schedule?
They don't provide an example in the answer.
require 'uri'
require 'net/http'
desc "This task is called by the Heroku cron add-on"
task :cron => :environment do
if Time.now.hour == 0 # run at midnight
uri = URI.parse('http://my-app.heroku.com/page')
Net::HTTP.get(uri)
end
end
That should do it.
What exactly would be the best way to go about using a cron task to send daily e-mails of updates to all users on my network? The e-mail would be made up of different information from multiple models.
I want to do something like "1 new friend requests : name ..." from the request model and user model and "There are 3 upcoming events from your friends: event name hosted by name..." from the event and user model.
I realize this is a common task but I didn't see much information on it, so any general tips about doing something like this would be greatly appreciated!
Side note: I will be using the Heroku daily cron plug-in to accomplish this if that matters (although I don't think it should).
I usually just write a rake task and add it to CRON.
The rake task will look like this:
namespace :notifications do
desc "Sends notifications"
task :send => :environment do
MyModel.all_users_to_notify.each do |u|
MyMailer.notification(u).deliver
end
end
end
And your crontab should look like this:
RAILS_ENV=production
HOME=/path/to/your/rails/app
PATH=/path/to/ruby/binaries
30 17 * * * rake notifications:send
If anyone else is looking for this answer, I recommend using the whenever gem for cron tasks. This will keep you from having to write a rake task, as well as giving you a nicer cron syntax:
app/mailers/my_mailer.rb:
class MyMailer < ActionMailer::Base
def my_email
...
end
end
config/schedule.rb
job_type :runner, "cd :path && rvm 2.0.0 do bundle exec script/rails runner -e :environment ':task' :output"
every 1.days, at: "7:00 am", roles: [:app] do
runner "MyMailer.my_email.deliver"
end