restate-ts

Restate

Restate is an experimental Typescript framework for building backends using state machines. With Restate, you define all database models as state machines which can only be modified through state transitions. The logic for the transitions are defined in code, and it's also possible to run code asynchronously in response to state transitions, for example to trigger new transitions, send emails or make API requests. This enables more complex business logic to be expressed by connecting together simpler state machines.

The point of Restate is to help build systems which are:

• Debuggable: All state transitions are tracked, making it easy to trace how a database object ended up in its current state and what triggered its transitions (an admin interface is in the works).
• Understandable: All business logic is encoded in state transitions and consumers, making it easy to understand the full behavior of your system. Writing decoupled code also becomes easier with consumers.
• Reliable: Consumers are automatically retried on failure and change data capture is used to ensure no transitions are missed.

Does that sound interesting? Then keep reading for a walkthrough of a sample project!

Getting started

Installation

To get started with Restate, we are going to create a standard Node project and install Restate:

$ mkdir my-first-restate-project && cd my-first-restate-project
$ npm init
$ npm install --save restate-ts

For this example, we are going to be using Express to build our API, so we need to install that as well:

$ npm install --save express

Restate has a built in development tool with auto-reloading, start it and keep it running in the background as you code:

$ npx restate

You'll see a warning message saying that no project definition was found, but don't worry about that, we'll create one soon!

Defining models and transitions

Database models in Reshape are defined in a custom file type, .rst, and stored in the restate/ folder of your project. Every database model is a state machine and hence we need to define the possible states and transitions between those states.

For this project, we are going to model a very simple application that tracks orders. Orders start out as created and are then paid by the customer. Once the order has been paid, we want to book a delivery with our carrier. To model this in Restate, let's create a new file called restate/Order.rst with the following contents:

model Order {
  // All models have an autogenerated `id` field with a prefix to make them easily identifiable
  // In this case, they will look something like: "order_01gqjyp438r30j3g28jt78cx23"
  prefix "order"

  // The common fields defined here will be available across all states
  field amount: Int

  state Created {}

  state Paid {
    field paymentReference: String
  }

  // States can inherit other state's fields, so in this case `DeliveryBooked` will have `amount` and `paymentReference` fields as well
  state DeliveryBooked: Paid {
    field trackingNumber: String
  }

  // `Create` doesn't have any starting states and is hence an initializing transition.
  // It will be used to create new orders.
  transition Create: Created {
    field amount: Int
  }

  // `Pay` is triggered when payment is received for the order
  transition Pay: Created -> Paid {
    field paymentReference: String
  }

  // `BookDelivery` is triggered when an order has been sent and we are ready to book delivery
  transition BookDelivery: Paid -> DeliveryBooked {}
}

Generating the Restate client

Once we have defined our models, the dev session you have running will automatically generate types for your models as well as a client to interact with them. All of this can be imported directly from the restate-ts module.

The starting point of any Restate project is the project definition, which lives in src/restate.ts. The definition we export from that file defines how our models' transitions are handled. Let's start with some placeholder values, create a src/restate.ts file with the following code:

import { RestateProject, RestateClient, Order } from "restate-ts";

const project: RestateProject = {
  async main(restate: RestateClient) {
    // main is the entrypoint for your project. Here you could for example start a web server and use
    // `restate` to create and transition objects.
  },

  transitions: {
    // We need to provide implementations for all transitions
    order: {
      async create(restate: RestateClient, transition: Order.Create) {
        throw new Error("Create transition not implemented");
      },

      async pay(
        restate: RestateClient,
        order: Order.Created,
        transition: Order.Pay
      ) {
        throw new Error("Pay transition not implemented");
      },

      async bookDelivery(
        restate: RestateClient,
        order: Order.Paid,
        transition: Order.BookDelivery
      ) {
        throw new Error("BookDelivery transition not implemented");
      },
    },
  },
};

// The definition should be the default export
export default project;

Creating orders

Before we can create orders, we need to actually implement the Create transition in src/restate.ts:

const project: RestateProject = {
  // ...
  transitions: {
    order: {
      async create(restate: RestateClient, transition: Order.Create) {
        // We should return the shape of the object after the transition has been applied
        // As this is an initializing transition, it will result in a new object being created
        return {
          state: Order.State.Created,
          // amount is passed through the transition and saved to the object
          amount: transition.data.amount,
        };
      },
    },
  },
};

To interact with our backend, we are going to create a simple HTTP API using express. Restate is fully API agnostic though so you can interact with it however you want; REST, GraphQL, SOAP, anything goes! Let's start a simple web server from the main function in src/restate.ts with a single endpoint to create a new order:

import express from "express";
import { RestateProject, RestateClient, Order } from "restate-ts";

const project: RestateProject = {
  async main(restate: RestateClient) {
    const app = express();

    app.post("/orders", async (req, res) => {
      // Get amount from query parameter
      const amount = parseInt(req.query.amount);

      // Trigger `Create` transition to create a new order object
      const [order] = await restate.order.transition.create({
        data: {
          amount,
        },
      });

      // Respond with our new order object in JSON format
      res.json(order);
    });

    app.listen(3000, () => {
      console.log("API server started!");
    });
  },
};

The dev session should automatically reload and you should see "API server started!" in the output. Let's test it!

$ curl -X POST "localhost:3000/orders?amount=100"
{
  "id": "order_01gqjyp438r30j3g28jt78cx23",
  "state": "created",
  "amount": 100
}

It works and we get a nice order back! Here you see both the amount field, which we specified in Order.rst, but also id and state. These are fields which are automatically added for all models.

Querying orders

Being able to create data wouldn't do much good if we can't get it back, which Restate handles using queries. We'll add the following code to our main function to introduce a new endpoint for getting all orders:

app.get("/orders", async (req, res) => {
  // Get all orders from the database
  const orders = await restate.order.findAll();

  res.json(orders);
});

And if we try that, we unsurprisingly get back:

$ curl localhost:3000/orders
[
  {
    "id": "order_01gqjyp438r30j3g28jt78cx23",
    "state": "created",
    "amount": 100
  }
]

Transitioning orders when paid

Now we are getting to the nice parts. We've created our order and the next step is to update it to the paid state once we receive a payment. The first step is to add a very simple implementation for the Pay transition:

const project: RestateProject = {
  // ...
  transitions: {
    order: {
      async pay(
        restate: RestateClient,
        order: Order.Created,
        transition: Order.Pay
      ) {
        return {
          // The spread operator is a convenient way of avoiding having to specify all fields again
          ...order,
          state: Order.State.Paid,
          paymentReference: transition.data.paymentReference,
        };
      },
    },
  },
};

For this example, let's say our payment provider will send us a webhook when an order is paid for. To handle that, we'll need another endpoint which should trigger the Pay transition for an order:

app.post("/webhook/order_paid/:orderId", async (req, res) => {
  // Get payment reference from query parameters
  const reference = req.query.reference;

  // Trigger the `Pay` transition for the order, which returns the updated object
  const [order] = await restate.order.transition.pay({
    object: req.params.orderId,
    data: {
      // The `Pay` transition requires us to pass the payment reference
      paymentReference: req.query.reference,
    },
  });

  // Respond with the updated object
  res.json(order);
});

If we were to simulate a webhook request from our payment provider, we get back an order in the expected state and with the passed reference saved to a new field (remember to replace the order ID with the one you got in the last request):

$ curl -X POST "localhost:3000/webhook/order_paid/order_01gqjyp438r30j3g28jt78cx23?reference=abc123"
{
  "id": "order_01gqjyp438r30j3g28jt78cx23",
  "state": "paid",
  "amount": 100,
  "paymentReference": "abc123"
}

Asynchronously booking deliveries

For the final part of this example, we want to book a delivery when an order is paid, with an imagined API call to our shipping carrier. Let's start by implementing the final bookDelivery transition for this:

const project: RestateProject = {
  // ...
  transitions: {
    order: {
      async bookDelivery(
        restate: RestateClient,
        order: Order.Paid,
        transition: Order.BookDelivery
      ) {
        // This is where we'd call the shipping carriers API and get a tracking number back, but for the sake
        // of the example, we'll use a static value
        const trackingNumber = "123456789";

        return {
          ...order,
          state: Order.State.DeliveryBooked,
          trackingNumber,
        };
      },
    },
  },
};

What we could do is simply trigger this transition right in our payment webhook, but our shipping carrier's API is really slow and unreliable, so we don't want to bog down the webhook handler with that. Preferably we want to perform the delivery booking asynchronously! This is where one of Restate's central features come in: consumers.

Consumers let's us write code that runs asynchronously in response to transitions. This lets us improve reliability, performance and code quality through decoupling. Like most everything in Restate, consumers are defined in src/restate.ts. In our case, we want to trigger the BookDelivery transition when the Pay transition has completed, so let's add a consumer for that:

const project: RestateProject = {
  // ...
  consumers: [
    Order.createConsumer({
      // Every consumer should have a unique name
      name: "BookDeliveryAfterOrderPaid",

      // We can tell our consumer to only trigger on specific transitions
      transition: Order.Transition.Pay,

      async handler(
        restate: RestateClient,
        order: Order.Any,
        transition: Order.Pay
      ) {
        // You might notice that `order` has type `Order.Any` rather than `Order.Paid`.
        // It's possible that the object changed since the consumer was queued but we'll always
        // get the latest version in here. Because consumers are asynchronous, this is something
        // we must take into consideration.
        if (order.state != Order.State.Paid) {
          return;
        }

        // Trigger `BookDelivery` transition, which will take a little while but that is completely fine!
        await restate.order.transition.bookDelivery({
          object: order,
        });
      },
    }),
  ],
  // ...
};

If you now mark a payment as paid using the webhook endpoint, you should soon after see that the order has been updated again:

$ curl localhost:3000/orders
[
  {
    "id": "order_01gqjyp438r30j3g28jt78cx23",
    "state": "deliveryBooked",
    "amount": 100,
    "paymentReference": "abc123",
    "trackingNumber": "123456789"
  }
]

That's it for the introduction! Keep reading to learn more about the different features of Restate.

Model definitions

IDs and prefixes

Every Restate model has an implicit field, id, which stores an autogenerated identifier. All IDs are prefixed with a string unique to the model, which makes it easier to identify what an ID is for. Here's an example of defining an Order model with prefix order. Objects of this model will automatically get IDs that look like: order_01gqjyp438r30j3g28jt78cx23.

model Order {
  prefix "order"
}

Fields

All Restate models have two implicit fields: id and state, which store an autogenerated ID and the current state respectively. When defining a model, it's also possible to add custom fields. Fields can be defined top-level, in which case they will be part of all states, or only on specific states.

model User {
  field name: String

  state Verified {
    field age: Int
  }
}

Every field has a data type and is by default non-nullable. If you want to make a field nullable, wrap the type in an Optional:

model User {
  field name: Optional[String]
}

Restate supports the following data types:

Data type	Description	Typescript equivalent
String	Variable-length string	string
Int	Integer which may be negative	number
Decimal	Decimal number	number
Bool	Boolean, either true or false	boolean
Optional[Type]	Nullable version of another type	Type | null

Client

The Restate client is used to create, transition and query objects. In the following examples, we'll be working with a model definition that looks like this:

model Order {
  prefix "order"

  field amount: Int

  state Created {}
  state Paid {}

  transition Create: Created {
    field amount: Int
  }

  transition Pay: Created -> Paid {}
}

Transitions

The client can be used to trigger initializing transitions, which create new objects. The transition call will return the new object after the transition has been applied.

const [order] = await restate.order.transition.create({
  data: {
    amount: 100,
  },
});

For regular transitions, one must also specify which object to apply the transition to by passing an object ID or a full object.

const [paidOrder] = await restate.order.transition.pay({
  object: "order_01gqjyp438r30j3g28jt78cx23",
});

If passing a full object, the types will ensure it's in the correct state for the transition to apply:

const order: Order.Paid = {
  // ...
};

const [paidOrder] = await restate.order.transition.pay({
  // This will trigger a type error because the `Pay` transition can
  // only be applied to orders in state `Created`
  object: order,
});

Transition calls will also return the full transition object if needed:

const [paidOrder, transition] = await restate.order.transition.pay({
  object: "order_01gqjyp438r30j3g28jt78cx23",
});

console.log(transition.id);
// tsn_01gqjyp438r30j3g28jt78cx23
// Transition IDs have prefix "tsn"

It's of course also possible to get a transition by ID or all transitions for an object:

const [paidOrder, transition] = await restate.order.transition.pay({
  object: "order_01gqjyp438r30j3g28jt78cx23",
});

// Find a single transition by ID
const transitionById = await restate.order.getTransition(transition.id);

// Find all transitions for an object (starting with the latest one)
const allTransitions = await restate.order.getObjectTransitions(paidOrder);

For debugging purposes, it's possible to add a free text note to a transition. This field is designed to be human readable and should not be relied upon by your code:

const [paidOrder] = await restate.order.transition.pay({
  order: "order_01gqjyp438r30j3g28jt78cx23",
  note: "Payment manually verified",
});

Queries

There are different kinds of queries depending on how many results you expect back. To find a single object by ID, you can do:

const order: Order.Any | null = await restate.order.findOne({
  where: {
    id: "order_01gqjyp438r30j3g28jt78cx23",
  },
});

Similarly, it's possible to filter by all fields on a model and to find many objects:

const orders: Order.Any[] = await restate.order.findAll({
  where: {
    amount: 100,
  },
});

When querying by state, the resulting object will have the expected type:

const orders: Order.Created[] = await restate.order.findAll({
  where: {
    state: Order.State.Created,
  },
});

If you want an error to be thrown if no object could be found, use findOneOrThrow:

const order: Order.Any = await restate.order.findOneOrThrow({
  where: {
    id: "order_01gqjyp438r30j3g28jt78cx23",
  },
});

You can also limit the number of objects you want to fetch:

const orders: Order.Created[] = await restate.order.findAll({
  where: {
    state: Order.State.Created,
  },
  limit: 10,
});

Testing

Restate has built-in support for testing with a real database. In your test cases, import the project definition from src/restate.ts and pass it to setupTestClient to create a new Restate client for testing. This client will automatically configure an in-memory SQLite database and will run any consumers synchronously when transitions are triggered.

Here's an example in Jest, but any test framework will work:

import { test, expect, beforeEach } from "@jest/globals";
import { Order, RestateClient, setupTestClient } from "restate-ts";

// Import project definition from "restate.ts"
import project from "./restate";

let restate: RestateClient;

beforeEach(async () => {
  // Create a new test client for each test run
  restate = await setupTestClient(project);
});

test("delivery is booked when order is paid", async () => {
  // Create order
  const order = await restate.order.transition.create({
    data: {
      amount: 100,
    },
  });

  // Trigger `Pay` transition on order
  await restate.order.transition.pay({
    object: order,
    data: {
      paymentReference: "abc123",
    },
  });

  // The `BookDeliveryAfterOrderPaid` consumer should have been triggered when the order was paid
  // and transitioned it into `DeliveryBooked`. With the test client, consumers are run synchronously.
  const updatedOrder = await restate.order.findOneOrThrow({
    where: {
      id: order.id,
    },
  });
  expect(user.state).toBe(Order.State.DeliveryBooked);
  expect(user.trackingNumber).toBe("123456789");
});

Config

If you want to configure Restate, create a restate.config.json file in the root of your project. In your config file, you can specify settings based on environment and the environment will be based on the NODE_ENV environemnt variable. When running restate dev, the default environment will be development. For all other commands, it will default to production.

In your config file, you can configure what database to use. Restate supports both Postgres and SQLite, where we recommend using Postgres in production and SQLite during development and testing. Below is an annotated example of a config file, showing what settings exist and the defaults:

{
  "database": {
    "type": "postgres",
    "connection_string": "postgres://postgres:@localhost:5432/postgres"
  },

  // The settings in here will only be used in the development environment
  "development": {
    "database": {
      "type": "sqlite",
      "connection_string": "restate.sqlite"
    }
  }
}

Commands

restate dev

Starts an auto-reloading dev server for your project. It will automatically generate a client and run both your main function and a worker to handle consumers.

restate main

Starts the main function as defined in your project definition.

restate worker

Starts a worker which handles running consumers in response to transitions.

restate generate

Regenerates the Restate client and types based on your *.rst files.

restate migrate

Automatically sets up tables for all your models. Runs automatically as part of restate dev.

License

Restate is MIT licensed