How to: running a stand alone worker

Sometimes, holding your worker in a file in some folder called workers or some miserable place under lib/workers might not be enough.

Sometimes you want to give your worker its own repository, or project. You want it to be maintained as a product or by a separate team. And you want it to be able to be modular, with a nice and clean project layout.

In this case, here's how I would do it, and how to make it Sneakers-enabled

Project Layout

Similar to a gem with a binary, here is an example

$ bundle gem myworker -b
$ cd myworker; tree .
.
├── Gemfile
├── LICENSE.txt
├── README.md
├── Rakefile
├── bin
│   └── myworker
├── lib
│   ├── myworker
│   │   └── version.rb
│   └── myworker.rb
└── myworker.gemspec

We'll get rid of the gemspec though, for convenience. Then we'll add a Procfile.

$ tree .
.
├── Gemfile
├── LICENSE.txt
├── Procfile
├── README.md
├── Rakefile
├── bin
│   └── myworker
└── lib
    ├── myworker
    │   └── version.rb
    └── myworker.rb

Baking Sneakers In

Let's set up a Gemfile

# Gemfile
source 'https://rubygems.org'

gem "statsd-ruby"    # using statsd with Sneakers::Metrics
gem "foreman"        # for an easy deployment story with Upstart
gem "rake"
gem 'sneakers'

group :test, :development do
  gem "minitest"
  gem "rr"
end

And a Procfile

# Procfile
myworker: bin/myworker

As you can see most of the magic will be in the bin shim.

Running Sneakers Programmatically

Setup, configuration and running should be in your binary - bin/myworker:

#!/usr/bin/env ruby
require 'bundler/setup'
root = File.expand_path('../lib', File.dirname(__FILE__))
$: << root
require 'myworker'
require 'sneakers/runner'
require 'logger'
require 'statsd-ruby'


statsd = Statsd.new(ENV['STATSD_HOST'], 9125)

Sneakers.configure(:amqp => ENV['AMQP_URL'], :daemonize => false, :log => STDOUT, :metrics => Sneakers::Metrics::StatsdMetrics.new(statsd))
Sneakers.logger.level = Logger::INFO

r = Sneakers::Runner.new([ Myworker ])
r.run

Note that we configure logging to STDOUT, and we are not daemonizing (which means we're losing auto-scaling but whatever; when a worker is temporarily down due to restart we just have more messages in the queue!).

This mode is useful for running under Foreman, which when exporting to init.d or my favorite - upstart, will take care of log aggregation, daemonizing, and supervision.

Foreman and Beyond

All is left now is to say foreman start, or export via forman's awesome export feature to upstart/init.d or what-have-you.

You can now maintain your worker separately and get a good warm feeling about it. Since this is an independent project, first thing that comes into my mind is testing. It's a piece of cake; see Testing Your Worker.