q4m

= q4m

Interface to pragmatically setup a queue job for Q4M!

= So... What's Q4M

Q4M is a open-sourced MySQL Engine. that attempts to take away some logic from your application by encapsulating it inside of mysql.

The documentation is kinda bad, but the magic that it does in the background is sweeeeeet!

You basically store all of your background jobs inside a MySQL table. that table's engine should be a queue (q4m) engine!

Exampe:

drop table if exists shell_jobs, ruby_jobs;
create table shell_jobs (command text) engine=queue;
create table ruby_jobs (command text) engine=queue;

That engine will work under some kind of transaction. When you connect to the database the engine will give you a list of all the pending jobs. When you start a transaction you'll receive ONE job to work with. In the mean time the whole table gets locked.. And if you do a 'select *' all results will return emtpy!

Don't panic!

This is great, you can never access two jobs at the same time. if another worker/machine connects they'll get a list of jobs except the one is wrapped in the transaction!

So that means that the locking only happens on the connection that is perfoming the transaction but any other workers will get the full list of jobs except the ones that are being processed by other workers.

When the first transaction gets terminated the engine will automatically unlock the table and delete the record.

This is great, because you can have any number of workers working concurrently without making a mess!

Meaning that a job can never be called in twice by anyone, and that you can grow horizontally, the limit is the sky!

What's next?

The job is done here!

Go do something more interesting like clustering! ;)

Seek & destroy, Seek & destroy!

= So what exactly does this gem do?

Gives you an interface to quickly setup how specificly a job is supposed to run!

Basically, you define a class create an method called execute, which receives a job argument. truly a MySQL record in the form of an array, what you do from there is up to you!

if something goes wrong just call the queue_abort method and everything gets rolled back!

= Installation

kazu@utopia:~$ gem install q4m

= Simple example

require 'rubygems'
require 'q4m'

# Specify how shell jobs will run

class ShellJob < ::Q4m::Queue::Job

  def execute job
    queue_abort unless system(job.first)
  end

end

host = 'localhost'
user = 'root'
pass = 'secret'
database = 'my_db'

# Instantiate and run a job!
mysql = Mysql.new host, user, pass, database
job = ShellJob.new mysql, Logger.new('q4m.log')

job.run

= Quick tip

Note that: The ShellJob class will try to connect to the 'shell_jobs' q4m table by default, if you want to use a different table override the table_name method and specify one of your liking!

class ShellJob < ::Q4m::Queue::Job

 def execute job
    queue_abort unless system(job.first)
 end

  def table_name
    'my_cool_table'
  end

end

= Run on a specific environment like rails?

Would you like to have an entire rails-app as your worker? but dont't want to load the whole env with rake tasks everytime?

That would be sweet right? and you can even reuse/test that code.

No problem!

You can leave only-one instance of rails running and tell it to pull jobs from the queue whenever it can?

Yes!!

But let's keep it flexibe so let's run two kinds of jobs, :ruby & :shell

module Background

  # Specify a shell job. isn't this short & sweet?
  class ShellJob < ::Q4m::Queue::Job

    def execute job
      queue_abort unless system(job.first)
    end

  end

  # Now do the same thing for ruby
  # and let's encapsulate the whole thing, and say that all ruby
  # code that goes into the queue will be methods and they must
  # always return booleans, if true complete the job successfully!
  # rollback the transaction otherwise!
  class RubyJob < ::Q4m::Queue::Job

    def execute job
      queue_abort unless (eval(job.first) == true)
    end

  end

  # This hack will mimic a very simple deamon that will run in the
  # background stay alert for any new jobs that get appended to
  # the queue
  class Deamon

    def self.run type = :ruby
      log   = Rails.root + 'log' + 'q4m.log'
      path  = Rails.root + 'config' + 'database.yml'
      conf  = YAML::load_file(path)['queues']
      mysql = Mysql.new conf['host'], conf['username'], conf['password'], conf['database']
      job   = if type == :ruby
        Background::RubyJob.new mysql, Logger.new(log)
      else
        Background::ShellJob.new mysql, Logger.new(log)
      end
      if File.exist? '/tmp/run_queue.txt'
        job.run
        sleep 1
        print '.'
        mysql.close
        run
      end
    end

  end

end

# Now let's create a model that will help us create jobs for ruby
class RubyJob < ActiveRecord::Base

  establish_connection :queues

  def self.append command
    create! :command => command
  end

end

# And for shell
# this can be encapsulated into a parent class but whatever!
class ShellJob < ActiveRecord::Base

  establish_connection :queues

  def self.append command
    create! :command => command
  end

end

# take it one step farther and build a helper, after all we are
# going to be working with strings!
class String

  def jobify type = :ruby
    return RubyJob.append self if type == :ruby
    return ShellJob.append self if type == :shell
    nil
  end

end

= How to actually run it (on rails)!

# With the code above you can do something like this!

$ touch /tmp/run_queue.txt

# Run the deamon in a rails console!
>> Background::Deamon.run

# Let's create some background jobs on another console..

# Create a shell job, send it to the queue for later processing!
'RAILS_ENV=production rake -T'.jobify :shell

# Do the same but with ruby!
# NOTE: method inside should return boolean!!
'User.create_thumbnails(1)'.jobify

# Things should start running on the background, look at your logs!

# To stop the deamon cleanly just delete the file '/tmp/run_queue.txt'

= This is great!

Basically any machine running rails with this code, can pull jobs and do some hard work for you! even if they are not part of your cluster!

The only requirement is that they can access the mysql-database.

and no, you don't need to install that engine locally!

= Extensibility!

Go farther just create a new job class and run whatever you like!

= Q4M lib?

This is how I installed it.

wget http://q4m.kazuhooku.com/dist/old/mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar.gz
unzip *.gz
tar -xvf mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar
cd q4m-0.8.9-linux-x86_64
sudo cp libqueue_engine.so /usr/lib/mysql/plugin
sudo chmod 777 /usr/lib/mysql/plugin/libqueue_engine.so
mysql -u root -p -f mysql < support-files/install.sql

You might wanna go to:

http://q4m.github.com/

And see tha latest on how to get it installed.

just thought of posting it, so you'll get an idea.

Yeah, but what's your distro?

it's right here!

deploy@staging> cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04 LTS"