Two Minute Spates

Updated 2008/09/24 for latest version of backgroundrb 1.0.4

The other day on our staging server, I noticed that the BackgrounDRb queue worker had died. As it turned out, the queue worker had died over 3 months ago!!

There was no cause for alarm as the staging server isn’t critical but this did start me to worrying. We needed to implement a monitoring solution which not only verified that BackgrounDRb was running but also that particular workers were running.

As we had just implemented god monitoring with a custom condition for another issue, its a slam dunk to do the same again. (Thanks to Jesse Newland and his god tutorial at AtlRUG.)

Here’s the configuration file that got it done for us.

#run on command line with 'god -c backgroundrb.god -D'
RAILS_ROOT = '/var/www/rails/rollbook/current'

#load required rails and backgroundrb files
require File.dirname(__FILE__) + '/../boot'
require File.dirname(__FILE__) + '/../environment'
require 'erb'
$LOAD_PATH << "#{RAILS_ROOT}/vendor/plugins/backgroundrb/lib"
require "#{RAILS_ROOT}/vendor/plugins/backgroundrb/lib/backgroundrb.rb"

#create custom condition for checking that QueryProcessingWorker is running
MiddleMan = BackgrounDRb::ClusterConnection.new
module God
  module Conditions
    class Backgroundrb < PollCondition
      def initialize; super; end
      def valid?; true; end 

      def test
        begin
          queue_worker = MiddleMan.all_worker_info.values.flatten.select { |w| :queue_processing_worker == w[:worker] }
          queue_worker.empty?
        rescue #if all_worker_info raises exception, then bdrb isn't running and we were unable to connect
          true
        end
      end
    end
  end
end

God.watch do |w|
  w.name      = 'backgroundrb'
  w.interval  = 1.minute
  w.restart   = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production stop && #{RAILS_ROOT}/script/backgroundrb -e production start"
  w.stop      = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production stop"
  w.start     = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production start"
  w.grace     = 1.minute
  w.pid_file  = "#{RAILS_ROOT}/tmp/pids/backgroundrb_11000.pid"

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.running  = false
    end
  end 

  w.restart_if do |restart|
    restart.condition(:backgroundrb) do |c|
      #just restart it
    end
  end
end 

In the select call on line 21, you can modify the condition to access :job_key or :status as well. Obviously, you need to modify RAILS_ROOT for your situation.

If you have suggestions for improvement or questions, hit me up in the comments. Enjoy!