We had a few important things in mind when desiging this system:

• Performance
• Scalibility
• Reliability

The first item is obvious. The system needs to be fast and relatively resource friendly. For this reason, we chose the Starling queue system from Twitter.

The second item, scalibility, means that the system should:

• Be distributed. Many machines submitting tasks, and many machines processing these tasks.
• Handle a large number of requests. Each photo upload spawns resize operations for frequently used sizes. Each resize spawns off additional S3 uploads, for example.
• Support different priority tasks.

The third item requires the system to handle failures, retry common errors, and notify us in case of major failures. Also, the system needs to surive the server dying and coming back and temporary network errors.

With the help of Starling, and our previous experience with a database-backed message queue system, I think we’ve achieved our goals. We look forward to the community’s feedback.

Requirements

sudo gem install starling
sudo gem install daemons


Installation


script/plugin install http://kabqueue.googlecode.com/svn/trunk/kabqueue


    class ActionViewTemplateHandler < ActionView::TemplateHandler # :nodoc:
      def render(template, local_assigns)
        @markaby_template = Template.new(template)
        @markaby_template.render(@view.assigns.merge(local_assigns), @view)
      end
    end

Now, let’s get rid of the whole portion on top (from module ActionView to end). That’s it!