messenger_pigeon

MessengerPigeon

A means of getting data from one location to another.

The goal of MessengerPigeon is to provide a highly-configurable and adaptable animal to take record-based data from any number of sources, and copy that data to any number of destinations. A pigeon may modify, drop or add to your data while in transit, but should only do these things when you ask it nicely.

MessengerPigeon is under heavy development.

Installation

gem install messenger_pigeon

Usage

Set up a ~/.messenger-pigeon.rc configuration file (as below) then:

$ messenger-pigeon

An alternate configuration file can be specified like so

$ messenger-pigeon -c my-other-configuration-file

Finally the names of pigeons can be given to let only those out. If no pigeons are listed, all are let-loose.

$ messenger-pigeon Gemima

Configuration

Example Configuration demonstrating some features:

• CSV source to an OrgMode target
• Setting the data format for a target
• Globbed file selection
• Source hooks
• Data Filtering
• Pre and Post-filter data transformations

~/.messenger-pigeon.rc:

{
  target: {
    'OrgMode' => {
      type: 'OrgMode',
      options: {
        file: 'clocks.org',
        heading_selector: '%{project}',
        level_separator: ' - ',
        data_format: proc do |d|
          clock = '  CLOCK: [%{date} %{start_time}]--[%{date} %{end_time}] =>  %{duration}'
          clock % d
        end
      }
    }
  },
  source: {
    'TimeTracker' => {
      type: 'CSV',
      options: {
        file_glob: 'test.csv',
        on_complete: :archive
      }
    }
  },
  pigeon: {
    'Gemima' => {
      source: 'TimeTracker',
      filters: {
         project: "Work"
      },
      transforms: {
        pre_filter: {
          date: proc { |a| Date.parse(a).strftime('%Y-%m-%d') },
          duration: proc do |a|
            r = format('%.2f', a.to_i + (a.to_f - a.to_i) / 100 * 60)
            r.sub(/\./, ':')
          end
        },
        post_filter: {}
        logger: proc do |d|
          puts format('[Gemima] Logging %shrs on %s', d[:duration], d[:date])
        end
      },
      target: 'OrgMode'
    }
  }
}

Modules

The following modules are available:

• CSV (source only)
• SQL
• Console (target only)
• Redmine
• OrgMode (target only)

CSV

Source

...
source: {
  'TimeTracker' => {
    type: 'CSV',
    options: {
      file_glob: 'test.csv',
      on_complete: :archive
    }
  }
}
...

Target

Not yet implemented

SQL

Uses Sequel to interface to many different types of databases.

See the Sequel documentation for listing of supported databases and their connection string formats.

Source

...
source: {
  'Mydatabase' => {
    type: 'SQL',
    options: {
      connection_string: 'sqlite:/',
      query: "SELECT * FROM artists"
    }
  }
}
...

Target

...
target: {
  'Mydatabase' => {
    type: 'SQL',
    options: {
      connection_string: 'sqlite:/',
      table: 'artists'
    }
  }
}
...

Console

Target-only.

Target

...
target: {
  'Console => {
    type: 'Console',
    options: {}
   }
}
...

OrgMode

Source

Not yet implemented

Target

...
target: {
  'OrgMode' => {
    type: 'OrgMode',
    options: {
      file: 'clocks.org',
      heading_selector: '%{project}',
      level_separator: ' - ',
      data_format: proc do |d|
        clock = '  CLOCK: [%{date} %{start_time}]--[%{date} %{end_time}] =>  %{duration}'
        clock % d
      end
    }
  }
}
...

Redmine

Configuration should be self-explanatory except for perhaps the 'resource' option. This is required option that corresponds to a capitalised, camel-cased and singular equivalent to the resources listed here:

http://www.redmine.org/projects/redmine/wiki/Rest_api

E.g., 'time_entries' becomes 'TimeEntry'

Source

'mode' is either ':all', or ':specific'.

Example for :all to get all issues in Project 2.

...
source: {
  'Redmine' => {
    type: 'Redmine',
    options: {
      resource: "Issue",
      site: 'https://url.to.redmine.org',
      user: 'username',
      password: 's3kr1t'
      mode: :all,
      params: {
        status_id: '*',
        project_id: '2'
      }
    }
  }
},
...

Example for :specific to get the issue with ID = 30

...
source: {
  'Redmine' => {
    type: 'Redmine',
    options: {
      resource: "Issue",
      site: 'https://url.to.redmine.org',
      user: 'username',
      password: 's3kr1t'
      mode: :specific,
      key: 30
    }
  }
},
...

Target

Note: The generators section of the configuration creates keys in the hash. This should give a reasonable overview of what is required. See the description of keys required on the particular resource page for more information.

...
target: {
  'Redmine' => {
    type: 'Redmine',
    options: {
      resource: "TimeEntry",
      site: 'https://url.to.redmine.org',
      user: 'username',
      password: 's3kr1t'
    }
  },
},
pigeon: {
  'Gemima' => {
    filters: {
      issue_id: proc { |a| !a.nil? },
      activity_id: proc { |a| !a.nil? }
    },
    generators: {
      activity_id: proc { |a| a[:activity_id] || '9' },
      issue_id: proc do |a|
        if a[:description]
          m = a[:description].match(/\#(?<id>[0-9]+)/)
          m[:id]
        end
      end,
      hours: proc { |a| a[:duration] },
      spent_on: proc { |a| a[:date] },
      comments: proc do |a|
        a[:description].sub(/\#[0-9]+ */, '') if a[:description]
      end
    },
    transforms: {
      pre_filter: {
        date: proc { |a| Date.parse(a).strftime('%Y-%m-%d') }
      },
      post_filter: {}
    },
    source: 'MySource',
    target: 'Redmine'
  }
}
...

And more?

Suggestions / contributions of more modules are welcome