cloudd

cloudd

node.js cloud engine Apart from all the buzz words, key aspect of this software is to specify set of tasks that are interdependent in a config file (similar to DAG input on condor) and run them.

Basics

• Task - is the smallest unit of execution (preferrable should be an idempotent task)
• Job - collection of tasks with dependencies (acyclic)

Features

• job execution engine
• Task could be a javascript unit or executable that would be exec'ed
• Command line to check the status of jobs, tasks and current running processes
• Command line Submit of jobs
• instantiate jobs cron style
• Good unit test coverage

Upcoming

• Failure of tasks to be handled
• Job load balancing among cloudd hosts
• Persistence of state
• clustering among cloudd hosts

Example

• Lets see how to define a Hello My World Job
  • Job Hello My World comprise of three tasks
  • Hello Task
  • My Task
  • World Task
  • Dependencies
  • My would be run only after Hello
  • World would be run only after My
• Sample yaml config file would look like (see in examples/hellomyworld.yaml)

name: Hello World
description:
  Sample 'hello my world' job
jobs:
  my:
    executable: echo "my"
  hello:
    executable: echo "hello"
  world:
    executable: echo "world"
dependencies:
  - my-world:
     parent: [my]
     child: [world]
  - hello-my:
     parent: [hello]
     child: [my]

To try this example

Have two terminals, one for the server and another one for submitting the job

# terminal 1
cloudd server

# terminal 2
cloudd submit HW examples/hellomyworld.yaml

Usage

To run the server

cloudd server

Command line utilities

# to show current running tasks
cloudd ps

# to show total tasks that are ready to run (ordered list)
cloudd tasks

# to show total jobs in the system (non-completed)
cloudd jobs

# to submit a new job
cloudd submit job-id sample-job-file.yaml

# to submit a cron job (say every 5 seconds)
cloudd cron "*/5 * * * * *" job-id sample-job-file.yaml

Config file

• id's for the job and task cannot have '#'
• input is a field that would be passed to task