buildgraph

Build Graph

Installing

Install with pip install buildgraph

Import with from buildgraph import BaseStep, buildgraph

Introduction

Build Graph provides a set of tools to run build steps in order of their dependencies.

Build graphs can be constructed by hand, or you can let the library construct the graph for you.

Here's an example of a graph:

@buildgraph()
def getGraph(prod=True):
    RunTest("api")
    RunTest("client")
    package = BuildPackage(prod)
    if prod:
        UploadPackage(package)
    else:
        WritePackageToFile(package, "packages/")


if __name__ == "__main__":
    test_graph = getGraph(False, config={"code_root": "src"})
    test_graph.printExecutionOrder()
    test_graph.run()

In the following examples, we'll be using this step definition:

class Adder(BaseStep):
    """
    Returns its input plus 1
    """
    def execute(self, n):
        new = n + 1
        print(new)
        return new

Manual construction

Defining steps

Steps are defined by constructing a step definition and binding the required arguments.

# This will create a single 'Adder' step with input 5
a = Adder(5)

Step arguments can be other steps:

# This will provide the output from step a as input to step b
a = Adder(0).alias("a")  # Set an alias to identify the steps
b = Adder(a).alias("b")

To run the steps, we pick the last step in the graph and call its run method.

...
result = b.run()
print(result)  # 2

A step from anywhere in the graph can be run, but only that step's dependencies will be executed.

print(a.run())  # 1 - Step b won't be run

Side dependencies

Sometimes you'll need to run a step a before step b, but a's output won't be used by b.

class Printer(BaseStep):
    def execute(self, msg):
        print(msg)

p = Printer("Hi")
a = Adder(0).alias("a")
b = Adder(a).alias("b").after(p)  # This ensures b will run after p
b.run()

The after(*steps) method specified steps that must be run first. If multiple steps are provided it doesn't enforce an ordering between those steps.

Detatched steps

If a step is defined but not listed as a dependency it won't be run:

a = Adder(0).alias("a")
b = Adder(1).alias("b")
b.run()  # This won't run a

You can check which steps will be run with the getExecutionOrder and printExecutionOrder methods.

Circular dependencies

Buildgraph will check for loops in the graph before running it and will raise an exception if one is detected.

Automatic construction

The @buildgraph decorator builds a graph where every node is reachable from the final node.

@buildgraph()
def addergraph():
    a = Adder(0)
    b = Adder(b)
    c = Adder(c)

addergraph.run()  # This will run all 3 steps

If the steps don't have dependencies the execution order isn't guaranteed, but the steps that are defined first will be run first unless another dependency enforces a different order.

Returning from a graph

Graphs can return results from a step too.

@buildgraph()
def addergraph():
    a = Adder(0)
    b = Adder(a)
    return b

result = addergraph().run() 
print(result)  # 2

Parameterised graphs

Graphs can take input which will be used to construct it.

@buildgraph()
def loopinggraph(loops):
    a = Adder(0)
    for i in range(loops-1):
        a = Adder(a)
    return a

looponce = loopinggraph(1)
looponce.run()  # 1

loopmany = loopinggraph(5)
loopmany.run()  # 5

Graphs which take no config or parameters can be run without explicitly building the graph first:

@buildgraph()
def simpleGraph():
    return Adder(0)

simpleGraph.run()  # simpleGraph is a config-less graph
simpleGraph().run()  # These two lines are equivalent


@buildgraph()
def configGraph(n):
    return Adder(n)

configGraph(2).run()  # Graphs with config must be built by calling it as a function

Nested Graphs

Graphs can be nested:

@buildgraph()
def getInnerGraph(p):
    print(f"Building inner graph with input {p}")
    return AppendAndPrint(p)

@buildgraph()
def getOuterGraph(p):
    print(f"Building outer graph with input {p}")
    inner1 = getInnerGraph(p, config={"namespace": "inner1"})
    inner2 = getInnerGraph(inner1, config={"namespace": "inner2"})
    outer = AppendAndPrint(proxy)
    return outer

Unpacking Return Types

Graphs can return multiple values which can be unpacked:

@buildgraph()
def graph():
    a = Adder(1)
    b = Adder(a)
    return a, b

first, second = graph.run()

Extending steps

All steps must inherit from BaseStep and implement an execute method.

You can see example steps from buildgraph/steps.py. These steps can also be imported and used in code.

Steps can take variable positional and keyword arguments:

class VarStep(BaseStep):
    def execute(self, *args, x=0, **kwargs):
        total = sum(args) + x + sum(kwargs.values())
        print(total)

VarStep(1, 2, 3, x=4, y=5, z=6).run()

Shared Config

Steps can receive a config object before running that other steps can share.

class ConfigStep(BaseStep):
    def configure(self, config):
        self.username = config['username']

    def execute(self):
        print(f"My name is {self.username}")

@buildgraph()
def getGraph():
    ConfigStep()
    ConfigStep()

graph = getGraph(config = {"username": "bob"})
graph.run()  # Both steps will print 'bob'

Exception Handling

Exceptions thrown inside steps will be caught, printed and the re-raised inside a StepFailedException object alongwith the step and the arguments passed the the execute function.

After handling an exception execution of further steps will stop.

Type checking

Buildgraph will perform type checking when the graph is built if the execute method has type annotations on its parameters.

Configuring buildgraph

By default buildgraph prints coloured output. You can disable this with buildgraph.setColor(False).

Examples

See the scripts in examples/ for examples for more complex graphs.