Monitoring BackgrounDRb workers with God

Updated 2008/09/24 for latest version of backgroundrb 1.0.4

The other day on our staging server, I noticed that the BackgrounDRb queue worker had died. As it turned out, the queue worker had died over 3 months ago!!

There was no cause for alarm as the staging server isn’t critical but this did start me to worrying. We needed to implement a monitoring solution which not only verified that BackgrounDRb was running but also that particular workers were running.

As we had just implemented god monitoring with a custom condition for another issue, its a slam dunk to do the same again. (Thanks to Jesse Newland and his god tutorial at AtlRUG.)

Here’s the configuration file that got it done for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

#run on command line with 'god -c backgroundrb.god -D' 
RAILS_ROOT = '/var/www/rails/rollbook/current'
 
#load required rails and backgroundrb files
require File.dirname(__FILE__) + '/../boot'
require File.dirname(__FILE__) + '/../environment'
require 'erb'
$LOAD_PATH << "#{RAILS_ROOT}/vendor/plugins/backgroundrb/lib"
require "#{RAILS_ROOT}/vendor/plugins/backgroundrb/lib/backgroundrb.rb"
 
#create custom condition for checking that QueryProcessingWorker is running
MiddleMan = BackgrounDRb::ClusterConnection.new
module God 
  module Conditions
    class Backgroundrb < PollCondition
      def initialize; super; end 
      def valid?; true; end 
 
      def test
        begin
          queue_worker = MiddleMan.all_worker_info.values.flatten.select { |w| :queue_processing_worker == w[:worker] }
          queue_worker.empty?
        rescue #if all_worker_info raises exception, then bdrb isn't running and we were unable to connect
          true
        end 
      end 
    end 
  end 
end
 
God.watch do |w| 
  w.name      = 'backgroundrb'
  w.interval  = 1.minute
  w.restart   = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production stop && #{RAILS_ROOT}/script/backgroundrb -e production start"
  w.stop      = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production stop"
  w.start     = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production start"
  w.grace     = 1.minute
  w.pid_file  = "#{RAILS_ROOT}/tmp/pids/backgroundrb_11000.pid"
 
  w.start_if do |start|
    start.condition(:process_running) do |c| 
      c.running  = false
    end 
  end 
 
  w.restart_if do |restart|
    restart.condition(:backgroundrb) do |c| 
      #just restart it
    end 
  end 
end

In the select call on line 21, you can modify the condition to access :job_key or :status as well. Obviously, you need to modify RAILS_ROOT for your situation.

If you have suggestions for improvement or questions, hit me up in the comments. Enjoy!

Posted Thursday, August 14th, 2008 under backgroundrb, god, intellum, monitoring, rails, ruby.

3 comments

  1. Jesse Newland says:
    August 15, 2008 at 12:12 am

    Nice work, Jonathan! Glad you found the tutorial useful. Backgroundrb is notoriously hard to monitor, and this solution is right on. If you wouldn’t mind, please fork my god_examples project on github and add this. Thanks!

  2. curi says:
    February 23, 2009 at 4:51 pm

    You don’t need to load the boot and environment files. Replace that with:

    require ‘activerecord’

    It’ll save a bunch of memory by not loading your whole app.

  3. Jonathan Wallace says:
    February 27, 2009 at 4:41 pm

    Thanks curi. I’ll be updating my post with your suggestion and a few other updates here shortly.

Leave a Reply