org-converge

-- mode: org; mode: auto-fill; --

#+STARTUP: showeverything

• Org Converge

  [[https://secure.travis-ci.org/wallyqs/org-converge.png?branch=master]]

** Description

A framework to create documented reproducible runs using [[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]], borrowing several ideas of what is possible to do with tools like =chef-solo=, =rake=, =foreman= and =capistrano=.

** Install

: gem install org-converge

** Motivation

[[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]] has proven to be flexible enough to write [[http://www.jstatsoft.org/v46/i03][reproducible research papers]], then I believe that configuring and setting up a server for example, is something that could also be done using the same approach, given that /converging/ the configuration is a kind of run that one ought to be able to reproduce.

Taking the original Emacs implementation as reference, Org Converge uses the [[https://github.com/wallyqs/org-ruby][Ruby implementation of the Org mode parser]] to implement and enhance the functionality from Org Babel by adding helpers to define the properties of a run while taking advantage of what is already there in the Ruby ecosystem.

** Usage examples

Org Converge supports the following kind of runs below:

*** Parallel runs

Each one of the code blocks is run on an independent process.
This is akin to having a =Procfile= based application, where
one of the runner command would be to start a web application
and the other one to start the a worker processes.

In the following example, we are defining 2 code blocks, one 
that would be run using Ruby and one more that would be run in Python.
Notice that the Python block has =:procs= set to 2, meaning that 
it would spawn 2 processes for this.

#+begin_src sh ,#+TITLE: Sample parallel run 　 ,#+name: infinite-worker-in-ruby ,#+begin_src ruby $stdout.sync = true loop { puts "working!"; sleep 1; } ,#+end_src 　 ,#+name: infinite-worker-in-python ,#+begin_src python :procs 2 import sys import time

while True: print "working too" sys.stdout.flush() time.sleep(1) ,#+end_src #+end_src

The above example can be run with the following:

: org-run procfile-example.org

Sample output of the run:

#+begin_src sh [2014-06-07T18:05:48 +0900] infinite-worker-in-ruby -- started with pid 19648 [2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- started with pid 19649 [2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- started with pid 19650 [2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- working too [2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- working too [2014-06-07T18:05:48 +0900] infinite-worker-in-ruby -- working! [2014-06-07T18:05:49 +0900] infinite-worker-in-python:1 -- working too [2014-06-07T18:05:49 +0900] infinite-worker-in-python:2 -- working too #+end_src

*** Sequential runs

In case the code blocks form part of a runlist that should be 
ran sequentially, it is possible to do this by specifying the
~runmode=sequential~ option.

#+begin_src sh ,#+TITLE: Sample sequential run ,#+runmode: sequential 　 ,#+name: first ,#+begin_src sh echo "first" ,#+end_src 　 ,#+name: second ,#+begin_src sh echo "second" ,#+end_src 　 ,#+name: third ,#+begin_src sh echo "third" ,#+end_src 　 ,#+name: fourth ,#+begin_src sh echo "fourth" ,#+end_src #+end_src

In order to specify that this is to be run sequentially, we set the runmode option in the command line:

: org-run runlist-example.org --runmode=sequential

Another way of specifying this is via the Org mode file itself:

#+begin_src org ,#+TITLE: Defining the runmode as an in buffer setting ,#+runmode: sequential #+end_src

Sample output:

#+begin_src sh [2014-06-07T18:10:33 +0900] first -- started with pid 19845 [2014-06-07T18:10:33 +0900] first -- first [2014-06-07T18:10:33 +0900] first -- exited with code 0 [2014-06-07T18:10:33 +0900] second -- started with pid 19846 [2014-06-07T18:10:33 +0900] second -- second [2014-06-07T18:10:33 +0900] second -- exited with code 0 [2014-06-07T18:10:33 +0900] third -- started with pid 19847 [2014-06-07T18:10:33 +0900] third -- third [2014-06-07T18:10:33 +0900] third -- exited with code 0 [2014-06-07T18:10:33 +0900] fourth -- started with pid 19848 [2014-06-07T18:10:33 +0900] fourth -- fourth [2014-06-07T18:10:33 +0900] fourth -- exited with code 0 #+end_src

*** Configuration management runs

For example, using Org Babel tangling functionality we can spread
config files on a server by writing the following on a ~server.org~ file...

#+begin_src sh 　 Configuration for a component that shoul be run in multitenant mode: 　 ,#+begin_src yaml :tangle /etc/component.yml multitenant: false status_port: 10004 ,#+end_src #+end_src

Then run:

: sudo org-tangle server.org

Note: The above tangle command is a reimplementation of the tangling functionality from Org mode using Ruby. If you are interested in tangling from the command line without losing functionality and have available a local Emacs Org mode install, the following project can be useful: https://github.com/ngirard/org-noweb

*** Idempotent runs

A run can have idempotency checks (similar to how the execute resource from [[http://docs.opscode.com/resource_execute.html][Chef]] works).

An example of this, would be when installing packages.  In this example,
we want to install the =build-essential= package once, and skip it in following runs:

#+begin_src sh ,** Installing the dependencies 　 Need the following so that bundle install can compile the native extensions correctly. 　　 ,#+name: build-essential-installed ,#+begin_src sh dpkg -l | grep build-essential ,#+end_src 　 　 ,#+name: build_essentials ,#+begin_src sh :unless build-essential-installed apt-get install build-essential -y ,#+end_src 　　 ,#+name: bundle_install ,#+begin_src sh cd project_path bundle install ,#+end_src #+end_src

Furthermore,since we are using Org mode syntax, it is possible to reuse this setup file by including it into another Org file:

#+begin_src sh ,#+TITLE: Another setup 　 Include the code blocks from the server into this: 　　 ,#+include: "server.org" 　 ,#+name: install_org_mode ,#+begin_src sh apt-get install org-mode -y ,#+end_src #+end_src #+end_src

Since this a run that involves converging into a state, it would be run sequentially with idempotency checks applied:

: sudo org-converge setup.org

*** Dependencies based runs

In this type of runs we use the =:after= and =:before=
header arguments to specify the prerequisites for a code block to run,
similar to some of the functioality provided by tools like =rake= 
(Behind the scenes, these arguments create =Rake= tasks)

In order for this kind of run to work, it has to be specified
what is the task that we are converging to by using 
the =#+final_task:= in buffer setting:

#+begin_src sh ,#+TITLE: Linked tasks example ,#+runmode: tasks ,#+final_task: final 　 ,#+name: second ,#+begin_src sh :after first for i in seq 5 10; do echo $i >> out.log done ,#+end_src 　 ,#+name: first ,#+begin_src ruby 5.times { |n| File.open("out.log", "a") {|f| f.puts n } } ,#+end_src 　 ,#+name: final ,#+begin_src python :after second :results output print "Wrapping up with Python in the end" f = open('out.log', 'a') f.write('11') f.close() ,#+end_src 　 ,#+name: prologue ,#+begin_src sh :before first :results output echo "init" > out.log ,#+end_src #+end_src

: org-run chained-example.org --runmode=chained

Instead of using =--runmode= options, it is also possible to just declare in buffer
that the Org file should be run chained mode.

#+begin_src org
,#+TITLE:   Defining the runmode as an in buffer setting 
,#+runmode: chained
#+end_src

Sample output:

#+begin_src sh [2014-06-07T18:14:25 +0900] Running final task: final [2014-06-07T18:14:25 +0900] prologue -- started with pid 20035 [2014-06-07T18:14:25 +0900] prologue -- exited with code 0 [2014-06-07T18:14:25 +0900] first -- started with pid 20036 [2014-06-07T18:14:26 +0900] first -- exited with code 0 [2014-06-07T18:14:26 +0900] second -- started with pid 20038 [2014-06-07T18:14:26 +0900] second -- exited with code 0 [2014-06-07T18:14:26 +0900] final -- started with pid 20040 [2014-06-07T18:14:26 +0900] final -- Wrapping up with Python in the end [2014-06-07T18:14:26 +0900] final -- exited with code 0 #+end_src

*** Remote runs

For any of the cases above, it is also possible to specify
whether the code blocks should be run remotely on another node.
This is done by using =:dir= in the code block header argument.

#+begin_src sh ,#+sshidentityfile: vagrant/keys/vagrant ,#+name: remote-bash-code-block 　 ,#+begin_src sh :results output :dir /vagrant@127.0.0.1#2222:/tmp random_number=$RANDOM for i in seq 1 10; do echo "[$random_number] Running script is $0 being run from pwd"; done ,#+end_src #+end_src

Note that in order for the above to work, it is also needed to set identity to be used by ssh.

*** Asserted runs

In case the Org mode file has a results block which represents the expected result, 
there is an ~org-spec~ command which can be useful to check whether there was a change
that no longer makes the results from the Org file valid. Example:

#+begin_src sh ,#+TITLE: Expected results example 　　 ,#+name: hello ,#+begin_src ruby :results output 10.times do puts "hola" end ,#+end_src 　　 ,#+RESULTS: hello ,#+begin_example hola hola hola hola hola hola hola hola hola hola ,#+end_example #+end_src

We can be able to verify whether this is still correct by running:

: org-spec test.org

#+begin_src sh Checking results from 'hello' code block: OK #+end_src

As an example, let's say that the behavior of the original code block changed, and now says hello 5 times instead. In that case the output would be as follows:

#+begin_src diff Checking results from 'hello' code block: DIFF @@ -1,11 +1,6 @@ -hola -hola -hola -hola -hola -hola -hola -hola -hola -hola +hello +hello +hello +hello +hello #+end_src

** Contributing

The project is still in very early development and a proof of concept at this moment. But if you feel that it is interesting enough, please create a ticket to start the discussion.