Skip to content

Why i built it

Jump to bottom Edit New page
Dotan J. Nahum edited this page Jan 10, 2014 · 7 revisions

Truth is, I've already built a background processing framework specifically on JRuby, called frenzy_bunnies.

It uses the top-notch Java RabbitMQ driver (wrapped by the JRuby library HotBunnies), and it drills down to "bare-metal" Java for performance which made it rock-solid in production for the last 1.5 years, processing millions of I/O and computationally intensive jobs. All under negligible CPU and memory usage (~200MB when most busy).

It's been running in production, requires zero-maintenance. If you're on JRuby, you can use it and stop here :)

CRuby, Performance, and High-Availability

Then came a project where I needed really good velocity, to appeal to non-JRuby/Java devs, and to have really good performance. This time, I also didn't have the luxury of finding replacements in JRuby-land for gems using c-extensions should I bump into them.

I really, really, wanted to use Sidekiq. Or any wheel that was already invented. But I was getting held back at 600req/s on an EC2-Large (a microbenchmark with empty jobs) and I felt a good starting point would be to check off that limit.

Nor could I allow the single-point-of-failure that being Redis, which isn't really suitable for a highly-available background processing framework - I couldn't lose messages (it's worth mentioning that Redis is often my go-to swiss-army knife and you'll have to pry it from my dying corpse).

Sneakers

Compared to Sidekiq, per my use case, I needed

  • A great-performing framework limited only by broker speed - at least 1000req/s acknowledged and persisted on EC2-Large (Sneakers does more than that), and
  • That would use all cores
  • A highly available processing framework (here we have same guarantees as RabbitMQ offers, which is great)
  • A familiar DSL/API that also supports advanced messaging semantics such as reject, requeue, acknowledge, etc, and
  • That would not expose the whole guts of AMQP at me, but just-enough from it.

And irrelevant of the comparison to Sidekiq or any other background processing framework I needed

  • It should use a ruby that doesn't care about content of gems and can run C-extensions. MRI.
  • A production-ready package that holds all of these together allowing me to be as lazy as possible
  • Metrics and logging baked in
  • Convenient deployment, maintenance and supervision story

...

Add a custom sidebar
Clone this wiki locally