rudder-json-template-engine

The Customer Data Platform for Developers

Website · Documentation · Community Slack

---

JSON Template Engine

Overview

Welcome to our JSON Template Engine! This powerful tool simplifies transforming JSON data from one format to another, making it easier to manage and maintain complex integrations.

Why JSON Template Engine?

As an integration platform supporting over 200 integrations, we understand the challenges of maintaining and optimizing these connections. Traditionally, we used native JavaScript code for data transformation, which required significant effort and maintenance. While JSONata offered a more efficient way to manipulate JSON data, we still encountered performance bottlenecks due to its parsing and interpretation overhead.

Our Solution

To address these challenges, we've developed our own JSON Transformation Engine. This engine generates optimized JavaScript code from transformation templates, reducing runtime overhead and significantly improving performance.

Key Features

• Efficiency: Our engine generates JavaScript code that minimizes parsing and interpretation overhead, ensuring faster execution.

• Extensibility: Easily add new transformation templates to meet your specific integration needs.

• Simplicity: Write concise transformation templates that are easy to understand and maintain.

Implementation

This library generates a javascript function code from the template and then uses the function to evaluate the JSON data. It outputs the javascript code in the following stages:

1. Lexing (Tokenization)
2. Parsing (AST Creation)
3. Translation (Code generation)

  flowchart TD;
      A[Code] --> B[Convert code to tokens];
      B --> C[Parse tokens to create Expressions];
      C --> D[Combine expressions to create statements];
      D --> E[Combine statements to create AST];
      E --> F[Translate AST to JS code]

Engine class abstracts the above steps and provides a convenient way to use the json templates to evaluate the inputs.

Features

Template is a set of statements and result the last statement is the output of the template.

Variables

const a = 1
let b = a + 2
a + b

Refer this example for more details.

Basic Expressions

Conditions

a > b ? a : c

Refer this example for more details.

Comparisons

(a === b || c > d)

Refer this example for more details.

Math Operations

10 - 2 + (2 * 10)

Refer this example for more details.

Logical operations

false || true 

Refer this example for more details.

Input and Bindings

Input refers to the JSON document we would like to process using a template. Bindings refer to additional data or functions we would provide to process the data efficiently.

Example:

• Template: "Hello " + (.name ?? $.defaultName)
• Evaluation: engine.evaluate({name: 'World'}, {defaultName: 'World'});
• {name: 'World'} is input.
  • ^.name refers to "name" property of the input. We can also use .name to refer the same. ^ always refers to the root of the input and . refers to current context. Refer this example for more details.
• {defaultName: 'World'} is bindings.
  • $.defaultName refers to "defaultName" property of the bindings. Refer this example for more details.

Arrays

let arr = [1, 2, 3, 4]
let a = arr[1, 2] // [2, 3]
let b = arr[0:2] // [1, 2]
let c = arr[-2:] // [3, 4]

Refer this example for more details.

Objects

let key = "some key"
// { "a": 1, "b": 2, "c": 3, "some key": 4 }
let obj = {a: 1, b: 2, c: 3, [key]: 4 } 
let a = obj["a"] // 1
let b = obj.a // 1
let c = obj{["a", "b"]} // { "a": 1, "b": 2}
let d = obj{~["a", "b"]} // { "c": 3, "some key": 4}

Refer this example for more details.

Functions

Normal functions

let fn = function(arg1, arg2){
  arg1 + arg2
}

The result of the last statement of function will be returned as result of the function. We can also use rest params (...args).

Lambda/Short functions

let fn = array.map(lambda 2 * ?0);

This function gets converted to:

let fn = array.map(function(args) {
  2 * args[0]
})

Lambda functions are short to express the intention and it is convenient sometimes.

Async functions

let fn = async function(arg1, arg2){
  const result = await doSomethingAsync(arg1, arg2)
  doSomethingSync(result)
}

Note: When we want to use async functions then we need to create template engine using JsonTemplateEngine.create. If you create a template this way then it will be created as an async function so we can await anywhere in the template.

let result = await doSomething(.a, .b)

Refer this example for more details.

Paths

Paths are used to access properties in input, bindings and variables.

Simple Paths

Simple paths support limited path features and get translated as direct property access statements in the generate javascript code. a.b.c gets translated to a?.b?.c so they are very fast compared to Rich paths. Simple paths are ideal when we know the object structure.

Supported features:

• Simple Selectors
• Single Index Filters Refer this example for more details.

Rich Paths

Rich paths gets converted complex code to support different variations in the data.

If we use this rich path~r a.b.c then it automatically handles following variations.

• [{"a": { "b": [{"c": 2}]}}]
• {"a": { "b": [{"c": 2}]}}
• {"a": [{ "b": [{"c": 2}]}]} Refer this example for more details.

Simple selectors

let x = a.b.c;
let y = a."some key".c

Refer this example for more details.

Wildcard selectors

a.*.c // selects c from any direct property of a

Refer this example for more details.

Descendent selectors

// selects c from any child property of a
// a.b.c, a.b1.b2.c or a.b1.b2.b3.c
let x = a..c; 
let y = a.."some key";

Refer this example for more details.

Single Index or Property Filters

let x = a[0].c; 
let y = a[-1].c;  // selects last element from array
let z = a["some key"].c

Refer this example for more details.

Multi Indexes or Properties Filters

let x = a[0, 2, 5].c; 
let y = a["some key1", "some key2"].c;

Refer this example for more details.

Range filters

let x = a[2:5].c; 
let y = a[:-2].c; 
let z = a[2:].c; 

Object Property Filters

let x = obj{["a", "b"]};  // selects a and b
let y = obj{~["a", "b"]}; // selects all properties except a and b

Refer this example for more details.

Conditional or Object Filters

let x = obj{.a > 1};

Refer this example for more details.

Block expressions

let x = obj.({
  a: .a + 1,
  b: .b + 2
});
let x = obj.([.a+1, .b+2]);

Refer this example for more details.

Context Variables

.orders@order#idx.products.({
    name: .name,
    price: .price,
    orderNum: idx,
    orderId: order.id
})

Use context variables: @order and #idx, we can combine properties of orders and products together. Refer this example for more details.

Path Options

We can mention defaultPathType while creating engine instance.

// For using simple path as default path type
// a.b.c will be treated as simple path
JsonTemplateEngine.create(`a.b.c`, {defaultPathType: PathType.SIMPLE});
// For using rich path as default path type
// a.b.c will be treated as rich path
JsonTemplateEngine.create(`a.b.c`, {defaultPathType: PathType.RICH});

We can override the default path option using tags.

// Use ~s to treat a.b.c as simple path
~s a.b.c 
// Use ~r to treat a.b.c as rich path
~r a.b.c

Note: Rich paths are slower compare to the simple paths. Refer this example for more details.

Compile time expressions

Compile time expressions are evaluated during compilation phase using compileTimeBindings option.

// {{$.a.b.c}} gets translated to 1 and
// final translated code will be "let a = 1;"
JsonTemplateEngine.create(`let a = {{$.a.b.c}};`, {
  compileTimeBindings: {
    a: {
      b: {
        c: 1
      }
    }
  }
});

We can use compile time expressions to generate a template and then recompile it as expression. Refer these examples simple compilation and complex compilation for more details.

Comments

Supports both c style single line (//) and block comments (/* .. */). Refer this example for more details.

For more examples, refer Scenarios

Syntax

Getting started

npm install rudder-json-template-engine

const { JsonTemplateEngine } = require('rudder-json-template-engine');
const engine = JsonTemplateEngine.create(`'Hello ' + .name`);
engine.evaluate({name: 'World'}); // => 'Hello World'

Testing

npm test

Contribute

We would love to see you contribute to RudderStack. Get more information on how to contribute here.

License

The RudderStack rudder-json-template-engine is released under the MIT License.

Changelog

0.5.3 (2023-09-07)

Miscellaneous

• update readme docs (140f6a2)