live_ast

= LiveAST

== Summary

Live abstract syntax trees of methods and procs.

== Synopsis

require 'live_ast'

class Greet def default "hello" end end

ASTs of methods

p Greet.instance_method(:default).to_ast

=> s(:defn, :default, s(:args), s(:scope, s(:block, s(:str, "hello"))))

ASTs of lambdas, procs, blocks

f = lambda { "foo" } p f.to_ast

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil, s(:str, "foo"))

def query(&block) p block.to_ast # => s(:iter, s(:call, nil, :query, s(:arglist)), nil, s(:str, "bar")) end

query do "bar" end

ASTs from dynamic code -- fully integrated version

require 'live_ast/full'

f = eval "lambda { 'dynamic1' }" p f.to_ast

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil, s(:str, "dynamic1"))

eval "def g ; 'dynamic2' ; end" p method(:g).to_ast

=> s(:defn, :g, s(:args), s(:scope, s(:block, s(:str, "dynamic2"))))

ASTs from dynamic code -- pure ruby version

u = ast_eval "lambda { 'dynamic3' }", binding p u.to_ast

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil, s(:str, "dynamic3"))

ast_eval "def v ; 'dynamic4' ; end", binding p method(:v).to_ast

=> s(:defn, :v, s(:args), s(:scope, s(:block, s(:str, "dynamic4"))))

== Install

% gem install live_ast

Or from inside an unpacked .tgz download, rake install / rake uninstall.

== Description

LiveAST enables a program to find the ASTs of objects created by dynamically generated code. It may be used in a strictly noninvasive manner, where no standard classes or methods are modified, or it may be transparently integrated into Ruby. The default setting is in between.

RubyParser is the default parsing engine. To replace it with Ripper, gem install live_ast_ripper and then require 'live_ast_ripper'. A simple plug-in interface allows LiveAST to work with any parser.

The advantage of RubyParser is that it gives the traditional ParseTree sexps used by tools such as ruby2ruby. The disadvantage is that the newer Ruby 1.9 syntax is not supported. However the 1.8 syntax restriction only applies to files from which ASTs are actually requested. All other files may use 1.9 syntax.

LiveAST is thread-safe.

Ruby 1.9.2 or higher is required.

== Links

• Home: http://quix.github.com/live_ast
• Feature Requests, Bug Reports: http://github.com/quix/live_ast/issues
• Manual Download: http://github.com/quix/live_ast/archives/master
• Repository: http://github.com/quix/live_ast

== +to_ruby+

When the default parser is active,

require 'live_ast/to_ruby'

will define the +to_ruby+ method for the +Method+, +UnboundMethod+, and +Proc+ classes. These methods are one-liners which pass the extracted ASTs to ruby2ruby.

require 'live_ast/to_ruby'

p lambda { |x, y| x + y }.to_ruby # => "lambda { |x, y| (x + y) }"

class A def f "A#f" end end

p A.instance_method(:f).to_ruby # => "def f\n "A#f"\nend"

In general, +to_ruby+ will hook into the unparser provided by the parser plug-in, if one is found.

== Pure Ruby and +ast_eval+

An essential feature of require 'live_ast' is that it is implemented in pure ruby. However since pure ruby is not powerful enough to replace +eval+, in this case +ast_eval+ must be used instead of +eval+ for AST-accessible objects. +ast_eval+ has the same semantics as +eval+ except that the binding argument is required.

require 'live_ast'

u = ast_eval "lambda { 'dynamic3' }", binding p u.to_ast

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil, s(:str, "dynamic3"))

== Full Integration

In order for LiveAST to be transparent to the user, +eval+ must be replaced. This is accomplished with the help of the +boc+ gem (http://quix.github.com/boc).

To replace +eval+,

% gem install boc

and then

require 'live_ast/full'

The new AST-electrified +eval+, +instance_eval+, +module_eval+, +class_eval+, and Binding#eval all pass RubySpec (http://rubyspec.org) with the minor exception of backtraces sometimes not matching that of the original +eval+ (see the "Backtraces" section below for details).

require 'live_ast/full'

f = eval "lambda { 'dynamic1' }" p f.to_ast

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil, s(:str, "dynamic1"))

Since LiveAST itself is pure ruby, any new platforms supported by +boc+ should work with live_ast/full.

== Limitations

A method or block definition must not share a line with other methods or blocks in order for its AST to be accessible.

require 'live_ast'

class A def f ; end ; def g ; end end A.instance_method(:f).to_ast # => raises LiveAST::MultipleDefinitionsOnSameLineError

a = lambda { } ; b = lambda { } a.to_ast # => raises LiveAST::MultipleDefinitionsOnSameLineError

Code given to the -e command-line switch is not AST-accessible.

Evaled code appearing before require 'live_ast/full' is not AST-accessible.

In some circumstances +ast_eval+ and the replaced +eval+ will not give the same backtrace as the original +eval+ (next section).

---

== Technical Issues

You can probably skip these next sections. Goodbye.

== Backtraces

+ast_eval+ is meant to be compatible with +eval+. For instance the first line of +ast_eval+'s backtrace should be identical to that of +eval+:

require 'live_ast'

ast_eval %{ raise "boom" }, binding

=> test.rb:3:in `': boom (RuntimeError)

Let's make a slight change,

require 'live_ast'

f = ast_eval %{ lambda { raise "boom" } }, binding f.call

=> test.rb|ast@a:3:in `block in ': boom (RuntimeError)

What the heck is '|ast@a' doing there? LiveAST's implementation has just been exposed: each source input is assigned a unique key which enables a Ruby object to find its own definition.

In the first case above, +ast_eval+ has removed the key from the exception backtrace. But in the second case there is no opportunity to remove it since +ast_eval+ has already returned.

If you find this to be problem---for example if you cannot add a filter for the jump-to-location feature in your editor---then +raise+ may be redefined to strip these tokens,

require 'live_ast' require 'live_ast/replace_raise'

f = ast_eval %{ lambda { raise "boom" } }, binding f.call

=> test.rb:4:in `block in ': boom (RuntimeError)

However this only applies to a +raise+ call originating from Ruby code. An exception from within a native method will likely still contain the token in its backtrace (e.g., in MRI the exception raised by 1/0 comes from C).

== Replacing the Parser

Despite its name, LiveAST knows nothing about ASTs. It merely reports what it finds in the line-to-AST hash returned by the parser's +parse+ method. Replacing the parser class is therefore easy: the only specification is that the +parse+ instance method return such a hash.

To override the default parser with your own,

LiveAST.parser = YourParser

To test it, provide some examples of what the ASTs look like in YourParser::Test. See the +live_ast_ruby_parser+ gem for reference.

== Noninvasive Mode

For safety purposes, require 'live_ast' performs the invasive act of redefining +load+ (but not +require+); otherwise bad things can happen to the unwary. The addition of +to_ast+ to a few standard Ruby classes is also a meddlesome move.

To avoid these modifications,

require 'live_ast/base'

will provide the essentials of LiveAST but will not touch core classes or methods.

To select features individually,

require 'live_ast/to_ast' # define to_ast for Method, UnboundMethod, Proc require 'live_ast/to_ruby' # define to_ruby for Method, UnboundMethod, Proc require 'live_ast/ast_eval' # define Kernel#ast_eval require 'live_ast/ast_load' # define Kernel#ast_load (mentioned below) require 'live_ast/replace_load' # redefine Kernel#load

== Noninvasive Interface

The following alternative interface is available.

require 'live_ast/base'

class A def f "A#f" end end

p LiveAST.ast(A.instance_method(:f))

=> s(:defn, :f, s(:args), s(:scope, s(:block, s(:str, "A#f"))))

p LiveAST.ast(lambda { })

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil)

f = LiveAST.eval("lambda { }", binding)

p LiveAST.ast(f)

=> s(:iter, s(:call, nil, :lambda, s(:arglist)), nil)

ast_eval # => raises NameError

== Reloading Files In Noninvasive Mode

Use +ast_load+ or (equivalently) LiveAST.load when reloading an AST-aware file.

require 'live_ast/ast_load' require 'live_ast/to_ast'

require "foo" Foo.instance_method(:bar).to_ast # caches AST

... the bar method is changed in foo.rb ...

ast_load "foo.rb" p Foo.instance_method(:bar).to_ast # => updated AST

Note if +load+ is called instead of +ast_load+ then the last line will give the old AST,

load "foo.rb" # oops! forgot to use ast_load p Foo.instance_method(:bar).to_ast # => stale AST

Realize that foo.rb may be referenced by an unknown number of methods and blocks. If the original foo.rb source were dumped in favor of the modified foo.rb, then an unknown number of those references would be invalidated (and some may even point to the wrong AST).

This is the reason for the caching that results in the stale AST above. It should now be clear why the default behavior of require 'live_ast' is to redefine +load+: doing so prevents this problem entirely. On the other hand if it is fully known where files are being reloaded (if at all) then there's no need for paranoia; the noninvasive option may be the most appropriate.

== The Source/AST Cache

+ast_eval+ and +load+ cache all incoming code, while required files are cached on a need-to-know basis. When an AST is requested, the corresponding source file is parsed and discarded, leaving behind method and block ASTs. +to_ast+ removes an AST from the cache and attaches it to the appropriate object (a Proc or Module).

Ignored, unextracted ASTs will therefore linger in the cache. Since sexps are generally small there is little need for concern unless one is continually evaling/reloading and failing to extract the sexps. Nevertheless it is possible that old ASTs will eventually need to be garbage collected. To flush the cache,

(1) Check that +to_ast+ has been called on all objects whose ASTs are desired.

(2) Call LiveAST.flush_cache.

Calling +to_ast+ prevents the object's AST from being flushed (since it grafts the AST onto the object).

ASTs of procs and methods whose sources lie in required files will never be flushed. However a method redefined via +ast_eval+ or +load+ is susceptible to +flush_cache+ even when its original definition pointed to a required file.

== About +require+

No measures have been taken to detect manipulations of $LOADED_FEATURES which would cause +require+ to load the same file twice. Though +require+ could be replaced in similar fashion to +load+---heading off problems arising from such "raw" reloads---the overhead would seem inappropriate in relation to the rarity of this case.

Therefore the working assumption is that +require+ will load a file only once. Furthermore, if a file has not been reloaded then it is assumed that the file is unmodified between the moment it is required and the moment the first AST is pulled from it.

== Author

• James M. Lawrence < [e-mail] >

== License

Copyright (c) 2011 James M. Lawrence. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.