@d1ll0n/solc-typed-ast

solc-typed-ast

A TypeScript package providing a normalized typed Solidity AST along with the utilities necessary to generate the AST (from Solc) and traverse/manipulate it.

Features

• Various Solidity compiler versions support, starting from 0.4.13.
• Various compiler selection strategies, including compiler auto-detection via pragma solidity directives in Solidity sources with possible fallback options.
• Interaction with AST nodes, using type definitions.
• Compensation of differences between legacy and compact Solidity AST.
• Referenced AST nodes linking.
• Easy tree traversal with siblings, parent/children relations, recursive walks and node selection via custom predicates.
• XPath AST traversal (based on jSel package).
• Generating the source code from the AST.

Installation

Package could be installed globally via following command:

npm install -g solc-typed-ast

Also it can be installed as the dependency:

npm install --save solc-typed-ast

Usage

Easy compiling

The package introduces easy and universal compiler invocation. Starting with Solidity 0.4.13, source compilation could be done universally:

import { CompileFailedError, CompileResult, compileSol } from "solc-typed-ast";

let result: CompileResult;

try {
    result = await compileSol("sample.sol", "auto", []);
} catch (e) {
    if (e instanceof CompileFailedError) {
        console.error("Compile errors encountered:");

        for (const failure of e.failures) {
            console.error(`Solc ${failure.compilerVersion}:`);

            for (const error of failure.errors) {
                console.error(error);
            }
        }
    } else {
        console.error(e.message);
    }
}

console.log(result);

The second argument with the "auto" value specifies a compiler selection strategy. If "auto" is specified and source code contains valid pragma solidity directive, then compiler version will be automatically picked from it. If compile process will not succedd, the execution will fall back to "compiler guessing": trying to compile source with a few different versions of the new and old Solidity compilers. The other option would be to specify a concrete supported compiler version string, like "0.7.0" for example. There is also a support for various compiler selection strategies, including used-defined custom ones (CompilerVersionSelectionStrategy interface implementations).

Used compilers

Package supports switching between native binary Solc compilers and its WASM versions. The CLI option --compiler-kind and kind argument of compile*() functions family may be used for that purpose. Compilers are downloaded on-demand to the directory .compiler_cache at the package installation directory (by default). The compiler cache location may be customized by setting SOL_AST_COMPILER_CACHE environment variable to a custom path. For example:

SOL_AST_COMPILER_CACHE=~/.compiler_cache sol-ast-compile sample.sol --compiler-kind native --tree

or

export SOL_AST_COMPILER_CACHE=~/.compiler_cache

sol-ast-compile sample.sol --compiler-kind native --tree

Invalidation of downloaded compler cache

If there is a need to invalidate the downloaded compiler cache, then follow next steps:

1. Locate .compiler_cache directory:

sol-ast-compile --locate-compiler-cache

1. Manually remove list.json files to invalidate list of available compilers, remove certain compilers, or remove entire directory. Missing pieces will be downloaded again.

Typed universal AST

After the source is compiled and original compiler has provided the raw AST, the ASTReader could be used to read the typed universal AST:

import { ASTReader } from "solc-typed-ast";

const reader = new ASTReader();
const sourceUnits = reader.read(result.data);

console.log("Used compiler version: " + result.compilerVersion);
console.log(sourceUnits[0].print());

The typed universal AST has following benefits:

• It is universal between legacy and compact AST. Solc changed AST format between releases, so dependent packages were either forced to migrate from one raw AST version to another or to build adaptive logic (such as implemented in this package).
• It has TypeScript types and proper AST node classes hierarchy.
• It has built-in tree traversal routines, like walk(), getChildrenBySelector(), getParents(), getClosestParentBySelector() and so on.

Converting an AST back to source

One of the goals of each AST is to provide a way for programmatic modification. To do that, you could modify properties of the typed universal AST nodes. Then use the ASTWriter to write modified AST back to source code:

import {
    ASTWriter,
    DefaultASTWriterMapping,
    LatestCompilerVersion,
    PrettyFormatter
} from "solc-typed-ast";

const formatter = new PrettyFormatter(4, 0);
const writer = new ASTWriter(
    DefaultASTWriterMapping,
    formatter,
    result.compilerVersion ? result.compilerVersion : LatestCompilerVersion
);

for (const sourceUnit of sourceUnits) {
    console.log("// " + sourceUnit.absolutePath);
    console.log(writer.write(sourceUnit));
}

CLI tool

Package bundles a sol-ast-compile CLI tool to provide help with development process. It is able to compile the Solidity source and output short AST structure with following:

sol-ast-compile sample.sol --tree

Use --help to see all available features.

Project overview

The project have following directory structure:

├── .compiler_cache             # Cache of downloaded compilers (by default, if not configured by SOL_AST_COMPILER_CACHE).
├── coverage                    # Test coverage report, produced by "npm test" command.
├── dist                        # Generated JavaScript sources for package distribution (produced by "npm run build" and published by "npm publish" commands).
├── docs                        # Project documentation and API reference, produced by "npm run docs:render" or "npm run docs:refresh" commands.
├── src                         # Original TypeScript sources.
│   ├── ast                     # AST-related definitions and logic:
│   │   ├── implementation      #   - Implemented universal AST nodes:
│   │   │   ├── declaration     #       - declarations or definitions;
│   │   │   ├── expression      #       - expressions;
│   │   │   ├── meta            #       - directives, units, specifiers and other information nodes;
│   │   │   ├── statement       #       - statements;
│   │   │   └── type            #       - type-related nodes.
│   │   ├── legacy              #   - Solc legacy raw AST processors, that are producing arguments for constrcuting universal AST nodes.
│   │   ├── modern              #   - Solc modern (or compact) raw AST processors, that are producing arguments for constrcuting universal AST nodes.
│   │   ├── postprocessing      #   - AST postprocessors to apply additional logic (fixes and discovery) during tree finalization process.
│   │   └── writing             #   - Components to convert universal AST nodes back to Solidity source code.
│   ├── bin                     # Executable files, that are shipped with the package and deployed via "npm install" or "npm link" commands.
│   ├── compile                 # Compile-related definitions and logic.
│   ├── misc                    # Miscellaneous functionality and utility modules.
│   └── types                   # Solc AST typeString parser and AST.
└── test                        # Tests:
    ├── integration             #   - Integration test suites.
    ├── samples                 #   - Solidity and compiler ourput JSON samples for the tests.
    └── unit                    #   - Unit test suites.

A key points for better understanding:

• The ASTNode is generic implementation of universal AST node and a base class for all concrete node implementations, except Yul nodes.
• The ASTReader takes raw Solc AST (legacy or modern) and produces universal AST.
• The ASTContext provides node-to-node dynamic reference resolution map. The example of such reference would be a vReferencedDeclaration property of the Identifier node.
• The LegacyNodeProcessor class, ModernNodeProcessor class and their descendant classes are raw-to-universal AST conversion bridge.
• The compiling-related logic (located in src/compile/*) should not have imports from or any bindings to the universal AST (src/ast/*). Compile-related utils should be standalone as much as possible.

Development installation

Prerequisites

We suggest to use latest NodeJS LTS release lts/gallium (v16.14.0) and associated version of NPM. If there is a need to run different NodeJS versions, consider using NVM or similar tool, that is available for your platform.

Clone and build

Clone repository, install and link:

git clone https://github.com/ConsenSys/solc-typed-ast.git
cd solc-typed-ast/
npm install
npm link

Prior to running tests the compiler pre-downloading script docker/download.sh may be used to setup local compiler cache:

docker/download.sh 'linux-amd64' '.compiler_cache'  # platform-dependent native compiler builds
docker/download.sh 'wasm' '.compiler_cache'         # cross-platform WASM compiler builds

Supported platforms are listed here: https://github.com/ethereum/solc-bin

Project documentation and API reference

The project documentation is contained in the docs/ directory. It could be built via following command:

npm run docs:refresh

It is also published here: https://consensys.github.io/solc-typed-ast/

The list of AST node types

The list of known AST node types can be found here.