Decorated Syntax Tree
The dst
package enables manipulation of a Go syntax tree with high fidelity. Decorations (e.g.
comments and line spacing) remain attached to the correct nodes as the tree is modified.
Where does go/ast
break?
The go/ast
package wasn't created with source manipulation as an intended use-case. Comments are
stored by their byte offset instead of attached to nodes, so re-arranging nodes breaks the output.
See this Go issue for more information.
Consider this example where we want to reverse the order of the two statements. As you can see the
comments don't remain attached to the correct nodes:
code := `package a
func main(){
var a int // foo
var b string // bar
}
`
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "", code, parser.ParseComments)
if err != nil {
panic(err)
}
list := f.Decls[0].(*ast.FuncDecl).Body.List
list[0], list[1] = list[1], list[0]
if err := format.Node(os.Stdout, fset, f); err != nil {
panic(err)
}
Here's the same example using dst
:
code := `package a
func main(){
var a int // foo
var b string // bar
}
`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
list := f.Decls[0].(*dst.FuncDecl).Body.List
list[0], list[1] = list[1], list[0]
if err := decorator.Print(f); err != nil {
panic(err)
}
Usage
Parsing a source file to dst
and printing the results after modification can be accomplished with
several Parse
and Print
convenience functions in the decorator
package.
For more fine-grained control you can use Decorator
to convert from ast
to dst
, and Restorer
to convert back again.
Comments are added at decoration attachment points. See here
for a full list of these points, along with demonstration code of where they are rendered in the
output.
The decoration attachment points have convenience functions Append
, Prepend
, Replace
, Clear
and All
to accomplish common tasks. Use the full text of your comment including the //
or /**/
markers. When adding a line comment, a newline is automatically rendered.
code := `package main
func main() {
println("Hello World!")
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
call := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)
call.Decs.Start.Append("// you can add comments at the start...")
call.Decs.Fun.Append("/* ...in the middle... */")
call.Decs.End.Append("// or at the end.")
if err := decorator.Print(f); err != nil {
panic(err)
}
Spacing
The Before
property marks the node as having a line space (new line or empty line) before the node.
These spaces are rendered before any decorations attached to the Start
decoration point. The After
property is similar but rendered after the node (and after any End
decorations).
code := `package main
func main() {
println(a, b, c)
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
call := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)
call.Decs.Before = dst.EmptyLine
call.Decs.After = dst.EmptyLine
for _, v := range call.Args {
v := v.(*dst.Ident)
v.Decs.Before = dst.NewLine
v.Decs.After = dst.NewLine
}
if err := decorator.Print(f); err != nil {
panic(err)
}
Decorations
The common decoration properties (Start
, End
, Before
and After
) occur on all nodes, and can be
accessed with the Decorations()
method on the Node
interface:
code := `package main
func main() {
var i int
i++
println(i)
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
list := f.Decls[0].(*dst.FuncDecl).Body.List
list[0].Decorations().Before = dst.EmptyLine
list[0].Decorations().End.Append("// the Decorations method allows access to the common")
list[1].Decorations().End.Append("// decoration properties (Before, Start, End and After)")
list[2].Decorations().End.Append("// for all nodes.")
list[2].Decorations().After = dst.EmptyLine
if err := decorator.Print(f); err != nil {
panic(err)
}
dstutil.Decorations
While debugging, it is often useful to have a list of all decorations attached to a node. The
dstutil package provides a helper function Decorations
which
returns a list of the attachment points and all decorations for any node:
code := `package main
// main comment
// is multi line
func main() {
if true {
// foo
println( /* foo inline */ "foo")
} else if false {
println /* bar inline */ ("bar")
// bar after
} else {
// empty block
}
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
dst.Inspect(f, func(node dst.Node) bool {
if node == nil {
return false
}
before, after, points := dstutil.Decorations(node)
var info string
if before != dst.None {
info += fmt.Sprintf("- Before: %s\n", before)
}
for _, point := range points {
if len(point.Decs) == 0 {
continue
}
info += fmt.Sprintf("- %s: [", point.Name)
for i, dec := range point.Decs {
if i > 0 {
info += ", "
}
info += fmt.Sprintf("%q", dec)
}
info += "]\n"
}
if after != dst.None {
info += fmt.Sprintf("- After: %s\n", after)
}
if info != "" {
fmt.Printf("%T\n%s\n", node, info)
}
return true
})
Newlines
The Before
and After
properties cover the majority of cases, but occasionally a newline needs to
be rendered inside a node. Simply add a \n
decoration to accomplish this.
Clone
Re-using an existing node elsewhere in the tree will panic when the tree is restored to ast
. Instead,
use the Clone
function to make a deep copy of the node before re-use:
code := `package main
var i /* a */ int`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
cloned := dst.Clone(f.Decls[0]).(*dst.GenDecl)
cloned.Decs.Before = dst.NewLine
cloned.Specs[0].(*dst.ValueSpec).Names[0].Name = "j"
cloned.Specs[0].(*dst.ValueSpec).Names[0].Decs.End.Replace("/* b */")
f.Decls = append(f.Decls, cloned)
if err := decorator.Print(f); err != nil {
panic(err)
}
Apply
The dstutil package is a fork of golang.org/x/tools/go/ast/astutil
,
and provides the Apply
function with similar semantics.
Imports
The decorator can automatically manage the import
block, which is a non-trivial task.
Use NewDecoratorWithImports
and NewRestorerWithImports
to create an import aware decorator / restorer.
During decoration, remote identifiers are normalised - *ast.SelectorExpr
nodes that represent
qualified identifiers are replaced with *dst.Ident
nodes with the Path
field set to the path of
the imported package.
When adding a qualified identifier node, there is no need to use *dst.SelectorExpr
- just add a
*dst.Ident
and set Path
to the imported package path. The restorer will wrap it in a
*ast.SelectorExpr
where appropriate when converting back to ast, and also update the import
block.
To enable import management, the decorator must be able to resolve the imported package for
selector expressions and identifiers, and the restorer must be able to resolve the name of a
package given it's path. Several implementations for these resolvers are provided, and the best
method will depend on the environment. See below for more details.
Load
The Load convenience function uses
go/packages
to load packages and decorate all loaded ast files, with import management enabled:
dir, err := tempDir(map[string]string{
"go.mod": "module root",
"main.go": "package main \n\n func main() {}",
})
defer os.RemoveAll(dir)
if err != nil {
panic(err)
}
pkgs, err := decorator.Load(&packages.Config{Dir: dir, Mode: packages.LoadSyntax}, "root")
if err != nil {
panic(err)
}
p := pkgs[0]
f := p.Syntax[0]
b := f.Decls[0].(*dst.FuncDecl).Body
b.List = append(b.List, &dst.ExprStmt{
X: &dst.CallExpr{
Fun: &dst.Ident{Path: "fmt", Name: "Println"},
Args: []dst.Expr{
&dst.BasicLit{Kind: token.STRING, Value: strconv.Quote("Hello, World!")},
},
},
})
r := decorator.NewRestorerWithImports("root", gopackages.New(dir))
if err := r.Print(p.Syntax[0]); err != nil {
panic(err)
}
Mappings
The decorator exposes Dst.Nodes
and Ast.Nodes
which map between ast.Node
and dst.Node
. This
enables systems that refer to ast
nodes (such as go/types
) to be used:
code := `package main
func main() {
var i int
i++
println(i)
}`
fset := token.NewFileSet()
astFile, err := parser.ParseFile(fset, "", code, parser.ParseComments)
if err != nil {
panic(err)
}
typesInfo := types.Info{
Defs: make(map[*ast.Ident]types.Object),
Uses: make(map[*ast.Ident]types.Object),
}
conf := &types.Config{}
if _, err := conf.Check("", fset, []*ast.File{astFile}, &typesInfo); err != nil {
panic(err)
}
dec := decorator.NewDecorator(fset)
f, err := dec.DecorateFile(astFile)
if err != nil {
panic(err)
}
dstDef := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.DeclStmt).Decl.(*dst.GenDecl).Specs[0].(*dst.ValueSpec).Names[0]
astDef := dec.Ast.Nodes[dstDef].(*ast.Ident)
obj := typesInfo.Defs[astDef]
var astUses []*ast.Ident
for id, ob := range typesInfo.Uses {
if ob != obj {
continue
}
astUses = append(astUses, id)
}
var dstUses []*dst.Ident
for _, id := range astUses {
dstUses = append(dstUses, dec.Dst.Nodes[id].(*dst.Ident))
}
dstDef.Name = "foo"
for _, id := range dstUses {
id.Name = "foo"
}
if err := decorator.Print(f); err != nil {
panic(err)
}
Resolvers
There are two separate interfaces defined by the resolver package
which allow the decorator and restorer to automatically manage the imports block.
The decorator uses a DecoratorResolver
which resolves the package path of any *ast.Ident
. This is
complicated by dot-import syntax (see below).
The restorer uses a RestorerResolver
which resolves the name of any package given the path. This
is complicated by vendoring and Go modules.
When Resolver
is set on Decorator
or Restorer
, the Path
property must be set to the local
package path.
Several implementations of both interfaces that are suitable for different environments are
provided:
DecoratorResolver
gotypes
The gotypes
package provides a DecoratorResolver
with full dot-import compatibility. However it requires full
export data for all imported packages, so the Uses
map from go/types.Info
is required. There
are several methods of generating go/types.Info
. Using golang.org/x/tools/go/packages.Load
is
recommended for full Go modules compatibility. See the decorator.Load
convenience function to automate this.
goast
The goast package
provides a simplified DecoratorResolver
that only needs to scan a single ast file. This is unable
to resolve identifiers from dot-imported packages, so will panic if a dot-import is encountered in
the import block. It uses the provided RestorerResolver
to resolve the names of all imported
packages. If no RestorerResolver
is provided, the guess implementation is used.
RestorerResolver
gopackages
The gopackages
package provides a RestorerResolver
with full compatibility with Go modules. It uses
golang.org/x/tools/go/packages
to load the package data. This may be very slow, and uses the go
command line tool to query package data, so may not be compatible with some environments.
gobuild
The gobuild
package provides an alternative RestorerResolver
that uses the legacy go/build
system to load
the imported package data. This may be needed in some circumstances and provides better performance
than go/packages
. However, this is not Go modules aware.
guess and simple
The guess and
simple packages
provide simple RestorerResolver
implementations that may be useful in certain circumstances, or
where performance is critical. simple
resolves paths only if they occur in a provided map.
guess
guesses the package name based on the last part of the path.
Example
Here's an example of supplying resolvers for the decorator and restorer:
code := `package main
import "fmt"
func main() {
fmt.Println("a")
}`
dec := decorator.NewDecoratorWithImports(token.NewFileSet(), "main", goast.New())
f, err := dec.Parse(code)
if err != nil {
panic(err)
}
f.Decls[1].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr).Args = []dst.Expr{
&dst.CallExpr{
Fun: &dst.Ident{Name: "A", Path: "foo.bar/baz"},
},
}
res := decorator.NewRestorerWithImports("main", guess.New())
if err := res.Print(f); err != nil {
panic(err)
}
Alias
To control the alias of imports, use a FileRestorer
:
code := `package main
import "fmt"
func main() {
fmt.Println("a")
}`
dec := decorator.NewDecoratorWithImports(token.NewFileSet(), "main", goast.New())
f, err := dec.Parse(code)
if err != nil {
panic(err)
}
res := decorator.NewRestorerWithImports("main", guess.New())
fr := res.FileRestorer()
fr.Alias["fmt"] = "fmt1"
if err := fr.Print(f); err != nil {
panic(err)
}
Details
For more information on exactly how the imports block is managed, read through the test
cases.
Dot-imports
Consider this file...
package main
import (
. "a"
)
func main() {
B()
C()
}
B
and C
could be local identifiers from a different file in this package,
or from the imported package a
. If only one is from a
and it is removed, we should remove the
import when we restore to ast
. Thus the resolver needs to be able to resolve the package using
the full info from go/types
.
Status
This package is well tested and used in many projects. The API should be considered stable going forward.
Chat?
Feel free to create an issue or chat in the
#dst Gophers Slack channel.