@trayio/cdk-dsl

Connector Development Kit (CDK) DSL

The CDK Domain Specific Language (DSL) is the main component of Tray's CDK, it is used to define all the aspects of a connectors, including the behaviour of its operations.

A CDK connector consists of code written only using the DSL, which is declarative, so it only describes the connector, that description is then interpreted by the runtime to execute a connector's operations.

Documentation

The Tray.io Developer Portal has the most up to date documentation including a quick start guide for building connectors with the CDK

Project Structure

A CDK project is just a regular npm typescript project, preconfigured with all the dependencies, linter rules and compiler options that will be used to build the connector during deployment, so it is not recommended to change those.

Other than the package.json, jest configuration and typescript configuration, a connector will have the following:

• A connector.json file that includes metadata about the connector, such as the name, version, title, etc
• A src directory that has:
  • An Authentication typescript file that contains the type of the auth property of the ctx context object that operations receive together with the input, this type is the same for all operation
  • A test.ctx.json json file that contains a context value (including the auth) that can be used for tests. This file should not be committed to a repository as it will have sensitive information such as access tokens
  • One folder per operation

Context and Authentication

The context value (ctx) is received together with inputs in handlers, it contains values that are common to all operations, the main one being the auth property which usually contain things like tokens to identify the user making the request to a third party service.

Not all connectors need authentications, in that case, a never type can be used (with an empty value in the authentication test file), which is what the init command generates by default.

Operations

A connector will have one folder per operation under the src folder, this folder will contain the following files:

• operation.json which is an object that has operation metadata, like the operation name, title (which are mandatory) and the description.
• input.ts which contains the type of the input of the operation
• output.ts which contains the type of the output of the operation
• handler.ts which is where the logic of the operation is
• handler.test.ts which contains the test cases for testing the operation's behaviour defined in the handler

Input & Output

Each operation has an input.ts and output.ts that defines the input and output parameters for the operation. It should export this as a type named <OperationName>Input or <OperationName>Output.

During the deployment of a connector these input/output types are converted to JSON schema which powers how the Tray builder UI renders the input fields. This conversion also happens locally when you run the tray-cdk build command should you wish to inspect the generated JSON schema.

Additional Input configuration

The following sections outline the number of different ways to render your input fields in Tray's builder UI by adding JSdoc annotations to the properties in your input.ts.

Data types

• string: represents string values
• number: represents a number, e.g. 42. If you want to limit an input field to a whole number you can use the JSdoc annotation @TJS-type integer above the number.
• boolean: is for the two values true and false
• array: represented by [], e.g. number[]
• object: represented by object that contains different fields

Here is an example use of the different data types:

export type ExampleOperationInput = {
	str: string;
	num: number;
	/**
	 * @TJS-type integer
	 */
	int: number;
	bool: boolean;
	obj: object;
	arr: object[];
};

Enums (Static dropdown lists)

Enums are rendered in the Tray builder UI as dropdowns. The user will see the enum display names and the enum values are what will be passed into the handler. By default, user friendly enum display names are generated. e.g. an enum of value my-enum-value will be rendered in the UI with the display name My enum value.

export type ExampleOperationInput = {
	type: ActionType;
};

enum ActionType {
	first-option = 'first-option', // User will see an auto generated display name: First option
	second-option = 'second-option', // User will see an auto generated display name: Second option
	third-option = 'third-option', // User will see an auto generated display name: Third option
}

You may want to provide custom enum display names instead, to do so use the JSdoc annotation @enumLabels followed by a comma separated list of strings that matches the order you have defined your enums in.

export type ExampleOperationInput = {
	type: ActionType;
};

/**
 * @enumLabels The First Option, The Second Option, The Third Option
 */
enum ActionType {
	first-option = 'first-option',
	second-option = 'second-option',
	third-option = 'third-option',
}

DDL (Dynamic dropdown lists)

Sometimes you want to provide the user a dropdown list, but you don't know the items to display in the list because they come from another API. For example you may want to display a list of user names in a dropdown and based on the selection send the user ID. DDL's solve this problem. Once you have created an operation for your DDL (see more on this in the Composite Implementation section) you can use this DDL in the input of many other operations.

The JSdoc annotation @lookupOperation specifies what operation to invoke when the user expands the dropdown. @lookupInput is used to describe the JSON payload to send to that DDL operation. Any inputs defined in input.ts can be passed to the DDL operation using tripple braces. In this example we send the workspaceId field value by doing {{{workspaceId}}}. If your DDL operation calls an authenticated endpoint you can pass along the token used in the current operation by setting @lookupAuthRequired true.

export type ExampleOperationInput = {
	workspaceId: string;
	/**
	 * @title user
	 * @lookupOperation list_users_ddl
	 * @lookupInput {"includePrivateChannels": false,"workspaceId": "{{{workspaceId}}}"}
	 * @lookupAuthRequired true
	 */
	userId: string;
};

Reusing types across multiple operations

You can reuse types by importing them and reuse sections of a schema by using TypeScript intersections.

// input.ts
import { ReusableField } from './ReusableField';
import { ReusableSchema } from './ReusableSchema';

export type ExampleOperationInput = SpecificSchema & ReusableSchema;

type SpecificSchema = {
	specificField: string;
	reusableFields: ReusableField;
};

// ReusableField.ts
export type ReusableField = {
	reusableFieldA: number;
	reusableFieldB: string;
};

// ReusableSchema.ts
export type ReusableSchema = {
	reusableSchemaA: number;
	reusableSchemaB: string;
};

Once the imports and intersection are resolved the above example would look like this

export type Input = {
	specificField: string;
	reusableFields: {
		reusableFieldA: number;
		reusableFieldB: string;
	};
	reusableSchemaA: number;
	reusableSchemaB: string;
};

Union types (Supporting multiple types)

If you want to support two or more different object types in your schema you can achieve this using TypeScript unions.

In the example below our input accepts an array of elements. Each element of this array can be either of type image or text. When the user adds an item to the array they will see a dropdown where they can select if this element will be an image or text. The JSdoc annotation @title on the image and text types will be displayed in the dropdown the user sees.

export type ExampleOperationInput = {
	elements: ImageOrText[];
};

type ImageOrText = Image | Text;

/**
 * @title Image
 */
type Image = {
	name: string;
	src: string;
};

/**
 * @title Text
 */
type Text = {
	text: string;
};

Required/Optional fields

By default all input fields are mandatory, you can set any to optional with a ? in your TypeScript type. Mandatory fields get a red * in the Tray builder UI and will warn the user that they must be filled in before attempting to run a workflow.

export type ExampleOperationInput = {
	mandatoryField: string;
	optionalField?: string;
};

Formatting fields

By default properties on your input type render as simple input fields. You can select a more user friendly way to render the field based on your needs by using the JSdoc annotation @format.

The format options available are:

• datetime - renders a date and time picker
• code - renders a button which on click opens a modal with a simple code editor
• text - for longer text inputs, expands when you enter new lines

export type ExampleOperationInput = {
	/**
	 * @format datetime
	 */
	timestamp: string;
	/**
	 * @format code
	 */
	html: string;
	/**
	 * @format text
	 */
	longNotes: string;
};

Default values

If you want to provide a default initial value for a field you can use the JSdoc annotation @default.

export type ExampleOperationInput = {
	/**
	 * @default string default
	 */
	str: string;
	/**
	 * @default 0.1
	 */
	num: number;
	/**
	 * @TJS-type integer
	 * @default 1
	 */
	int: number;
	/**
	 * @default true
	 */
	bool: boolean;
	/**
	 * @default { "default": true }
	 */
	obj: object;
	/**
	 * @default ["default"]
	 */
	arr: object[];
};

Advanced fields

If you have optional fields that are for more advanced use cases you can obscure these into the Tray builder UI's advanced fields section. This can provide a cleaner interface for your operation, but can still be accessed by the user by expanding the advanced fields section. You can add fields here by using the JSdoc annotation @advanced true.

export type ExampleOperationInput = {
	normalField: string;
	/**
	 * @advanced true
	 */
	advancedField?: string;
};

Handler

A handler at its core, describes a function, that takes an ctx value with an auth property described by the authentication type (which is the same for all operations) and it takes an input value described by the input type of the operation

The output of the handler is described by the output type, in case of a success, or it could contain an error if something failed during the execution of the handler or if the third party returned an error response.

This "successful value or failure error" result of running an operation is described by the OperationHandlerResult<T> type, which is a sum/union/or type of OperationHandlerSuccess<T> and OperationHandlerFailure

So, the core of what an operation handler describes can be summarised as a function:

(ctx: OperationHandlerContext<AuthType>, input: InputType) =>
	OperationHandlerResult<OutputType>;

However, when defining a handler, we can also specify things like validation or whether or not the handler is private, and this is where the DSL comes in.

The handler.ts file needs to define a handler using the OperationHandlerSetup.configureHandler() function, which allows for configuring all aspects of the handler by chaining function calls together for all the components of the handler.

The OperationHandlerSetup.configureHandler() function takes a callback that is used to configure the handler, it looks like this:

export const myOperationHandler =
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        /* use the "handler" value to specify what implementation to use, validation, etc */
    );

Validation

Handlers can have input validation (which runs before the handler implementation is executed to validate the input that will be used to run it) and output validation (which runs after the implementation to validate its output).

To add validation just use the handler argument of the callback described in the previous section:

export const myOperationHandler = OperationHandlerSetup.configureHandler<
	AuthType,
	InputType,
	OutputType
>((handler) =>
	handler
		.addInputValidation((validation) =>
			validation
				.condition((ctx, input) => input.id > 0)
				.errorMessage((ctx, input) => `Id ${input.id} is not positive`)
		)
		.addOutputValidation((validation) =>
			validation
				.condition((ctx, input, output) => output.id === input.id)
				.errorMessage(
					(ctx, input, output) => `Output and Input ids don't match`
				)
		)
);

Note that validation is optional, the only thing that is necessary to define a handler is its implementation.

Implementation

The main aspect of a handler is its implementation, which can be HTTP if the operation will make an HTTP call to a third party, or Composite if the operation will combine zero or more operations when it runs, more implementations for other protocols will be added in the future.

A handler can only have one implementation, it describes what the handler does when it receives a request.

HTTP Implementation

A very simple handler that makes an HTTP call can be configured in the following way:

export const myOperationHandler =
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.get('https://someapi.com/someresource/:id')
                        .handleRequest((ctx, input, request) =>
                            request.addPathParameter('id', input.id.toString()).withoutBody()
                        )
                        .handleResponse((ctx, input, response) => response.parseWithBodyAsJson())
               )
    );

The previous handler makes a GET http request, which is defined by the http.get() call, after which a handleRequest() function is chained, whose purpose is to take an auth value, an input value and a request configuration and add the necessary arguments to that request configuration based on what we want the http call to have, the supported methods on the request configuration are:

• addPathParameter(name, value): Will replace a parameter on the path specified as :name in the url as shown in the previous example, the value will be url encoded
• addHeader(name, value): Adds a header to the request
• withBearerToken(token): Adds an Authorization header with a Bearer token
• addQueryString(name, value): Adds a query string to the request, the value will be url encoded
• withBodyAsJson(body): Adds a body to the request that will be sent as json.
• withBodyAsText(body): Adds a body to the request that will be sent as a plain text.
• withoutBody(): Sends a empty body in the request.

A handler with an authenticated POST request would look like this:

export const myOperationHandler =
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.post('https://someapi.com/someresource')
                        .handleRequest((ctx, input, request) =>
                            request.withBearerToken(auth.access_token)
                                   .withBodyAsJson(input)
                        )
                        .handleResponse((ctx, input, response) => response.parseWithBodyAsJson())
               )
    );

The input does not have to match what is sent as the body, if for example the input has other flags that specify how the connector needs to behave and only part of it contains the body, the withBodyAsJson method can be called in the following way:

request.withBodyAsJson({ name: input.name, title: input.title });

So the handlerRequest function can transform the input in any way it needs to before sending the HTTP request and the same is true for the handleResponse, in the previous examples, the handleResponse simply read the response body as json and returned it, but it can be more complex if necessary.

Supported methods on the response configuration are:

• parseWithBodyAsJson(): Returns a the requests response as a JSON
• parseWithBodyAsText((text) => { //logic to transform it to match your output.}): Returns a requests response as a text to the callback and it can be manipulated to match the operations output schema.

Just like with the input type and the request, the type of the json body in the response can be different from the output type.

This is an example of a handler that transforms the response body into the output type:

export const myOperationHandler =
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.post('https://someapi.com/someresource')
                        .handleRequest(...)
                        .handleResponse((response) => {
                            const httpResponseBody = response.parseWithBodyAsJson<{message: string}>()
                            if (httpResponseBody.isSuccess) {
                                const originalMessage = httpResponseBody.value.message
                                const extendedMessage = originalMessage + ' Extension'
                                return OperationHandlerResult.success({ message: extendedMessage })
                            }
                            return httpResponseBody
                        })
               )
    );

Instead of using an if in the previous case, there is also a OperationHandlerResult.map function that can do the same with a callback.

The handler can also return successful responses for some failure cases or viceversa, this is an example of a handler that "recovers" from errors to always return a successful response:

export const myOperationHandler =
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.post('https://someapi.com/someresource')
                        .handleRequest(...)
                        .handleResponse((response) => {
                            const httpResponseBody = response.withBodyAsJson()
                            if (httpResponseBody.isFailure) {
                                return OperationHandlerResult.success({ completed: false })
                            }
                            return OperationHandlerResult.success({ completed: true })
                        })
               )
    );

Composite Implementation

Composite handlers are used to define behaviours by invoking zero or more operations as part of their behaviour.

They can be used to write "helper" connectors (such as those in Tray's builder), DDL operations or complex operations like an "upsert" that combines more granular "read, create and update" operations.

A very simple composite handler, that just concatenates the firstName and lastName arguments it gets from the input into one string:

export const myOperationHandler =
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingComposite(async (ctx, input, invoke) => {
                    const fullName = input.firstName + ' ' + input.lastName
                    return OperationHandlerResult.success({fullName: fullName})
               })
    );

As an example of more complex behaviour, this handler reads a list of products using another operation and converts the result into a simple list of text and value pairs, this is known as a Dynamic Data List (DDL) operation used to help users select values as part of configuring workflows within the tray builder.

To do this, the handler needs to invoke the regular getProducts operation, this is accomplished by using the invoke functions that composite handlers have access to, and passing it a handler reference and an input (no need to pass the auth value as it will be passed automatically), the handler reference passed to the invoke function is the result of the OperationHandlerSetup.configureHandler() function, which is why they are saved into a constant and exported like this:

export const getProductsHandler =
    OperationHandlerSetup.configureHandler<AuthType, GetProductsInput, GetProductsOutput>((handler) =>
        handler.usingHttp(...)
    );

The getProductsHandler constant contains the handler reference, which also has the input and output type information as part of its type, to make sure that when invoked or tested, only valid input and output values can be used.

To invoke the handler it is a simple call to the invoke function that takes the handler reference as an argument, and returns another function that takes the input type of that handler and returns a Promise<OperationHandlerResult<T>> where T is the output type of the handler and OperationHandlerResult contains a failure response if the call failed or a success response with a value of type T if the call was successful:

const productsResult: OperationHandlerResult<GetProductsOutput> = await invoke(
	getProductsHandler
)({ storeId: input.storeId });

The output type of a DDL operation is defined by the DDLOperationOutput<T> type, where T is the type of the values and can be a string or a number, that type has one field called results which is an array of objects of type {text: string, value: T}.

To use it as the output, it is recommended to define a custom type for the operation as usual in output.ts that derives from DDLOperationOutput<T> specifying whether T is of type string or number, like this example output.ts file:

import { DDLOperationOutput } from '@trayio/cdk-dsl/connector/operation/OperationHandler';

export type GetProductListDdlOutput = DDLOperationOutput<number>; //the values of the elements of the DDL are of type number

With that in mind, this is what the DDL handler would look like:

export const getProductsDDLHandler = OperationHandlerSetup.configureHandler<
	AuthType,
	GetProductsDDLInput,
	GetProductsDDLOutput
>((handler) =>
	handler.usingComposite(async (auth, input, invoke) => {
		//invoke the products operation
		const productsResult: OperationHandlerResult<GetProductsOutput> =
			await invoke(getProductsHandler)({ storeId: input.storeId });

		//if the invocation failed, propagate the failure
		if (productsResult.isFailure) {
			return productsResult;
		}

		//productsResult is of type `OperationHandlerSuccess` now, because we handled the failure case above, so we can get the value
		const products = productsResult.value;

		//converts a product list into a list of text (label) and values (identifiers) for the DDL
		const productsDDL = products.map((product) => {
			return {
				text: product.title,
				value: product.id,
			};
		});

		//returns the DDL list in an object with a "result" value to match the GetProductsDDLOutput type
		return OperationHandlerResult.success({
			result: productsDDL,
		});
	})
);

There are several things to note about the handler, both the DDL handler and the getProductsHandler expect a storeId in the input to return a list of products for that store.

The getProductsHandler is invoked using the invoke function that composite handlers receive as an argument, passing the handler reference, and then calling the result as a function passing the input that handler expects, in this case, just an object with a storeId

The result of that invocation is of type Promise<OperationHandlerResult<GetProductsOutput>>, that is, a promise that has a value that is described by the invoked handler's output type, which is wrapped in the result object because it could be a successful invocation or a failure as described in previous sections.

The await keyword unwraps the promise, and we are left with a OperationHandlerResult<GetProductsOutput>, which forces the handler to deal with both the failure case as well as the success case.

There are multiple ways to do this

• Using if or switch statements to narrow down the type as shown in the example
• Using the OperationHandlerResult.getSuccessfulValueOrFail() function which unwraps the value if successful or terminates the function propagating the error if it is not
• Using the OperationHandlerResult.map() function, which takes an OperationHandlerResult<T> as an argument and a function to convert T to another type in case it is successful, propagating the failure if it is not (works in the same way it works for the map function in Array<T>)

Once the handler has access to the product list value, it just needs to convert each element to a {text: string, value: number} pair and return that list in an object with a result property as shown above.

Finally, for DDL operations, we need to add an extra type property in the operation.json file with ddl as the value:

{
	"name": "get_products_ddl",
	"title": "Get Products DDL",
	"description": "Returns a DDL with product ids as values and product titles as labels",
	"type": "ddl"
}

This will categorise this operation as a DDL and will exclude it from the list of visible operations of the connector.

Testing

The CDK DSL has declarative testing functions to test a handler's behaviour, the tests are in the handler.test.ts file within the operation folder, which can have zero, one or many test cases for that given operation.

The OperationHandlerTestSetup.configureHandlerTest() function is used to describe a test, it takes a handler reference and a callback with an object used to configure the test, in a similar way handlers are configured.

This is what a very basic test looks like:

OperationHandlerTestSetup.configureHandlerTest(
	myOperationHandler,
	(handlerTest) =>
		handlerTest
			.usingHandlerContext('test') //will use `test.ctx.json` as the context value which includes authentication values for all test cases
			.nothingBeforeAll()
			.testCase('should do something', (testCase) =>
				testCase
                    .usingHandlerContext('another') //optionally, a test case can define its own context instead of using the default one defined for all tests
					.givenNothing()
					.when(() => /* return an input that matches the input type */)
					.then(({ output }) => {
                        /* output is OperationHandlerResult<T> where T is a value matching the output type */

                        //This will contain a value of type T if the operation was successful or the test will fail if not
                        const successValue = OperationHandlerResult.getSuccessfulValueOrFail(output)

                        // jest-style matchers like "expect" are available here
                        expect(successValue).toEqual(...)
                    })
					.finallyDoNothing()
			)
			.nothingAfterAll()
);

The structure of a test is well defined, and the type safe declarative DSL will enforce that, in particular, there are a number of aspects that would apply to all test cases:

• A default context to use for all test cases
• Run one or more operations (doesn't have to be the one being tested) before all test cases using the beforeAll() function, or don't do anything before all test cases using the nothingBeforeAll() function, these functions can only be used before adding test cases (this is enforced by the type system)
• Add a test case using the testCase() function.
• Run one or more operations (doesn't have to be the one being tested) after all test cases using the afterAll() function, or don't do anything after all test cases using the nothingAfterAll() function, these functions can only be used after defining test cases, and no test cases can be added after this (this is enforced by the type system)

As for the test cases, they use a BDD style Given/When/Then convention, in particular a test case has:

• An optional usingHandlerContext() function at the beginning of the test case to use a different context than the default for all test cases
• A given() function to run one or more operations at the beginning of the test case or givenNothing() to go straight to running the operation under test
• A when() function to create an input value that will be used to run the operation under test, that value needs to match the input type of the operation
• A then() function that gets the output, input, auth and optionally the result of beforeAll() and given() if present, which can be used to do the assertions of the test case using jest-style matchers
• A finally() function to run one or more operations at the end of the test case, usually for cleanup, or finallyDoNothing() to don't do anything else after the assertions.

Both the beforeAll() and given() functions allow to run multiple operations before all test cases or before a single test case, they receive the invoke function as an argument just like composite handlers and they return an object that can contain some of relevant information about the operations that ran if necessary (like ids of things created) in a value that can be accessed by the when(), then(), finally() and afterAll() functions, they receive them as arguments.

As an example, the following test is for an updateProduct operation, it creates a store for all test cases, and a product for every test case to test the update on, using beforeAll and given respectively, and accessing the identities of the created objects from the testContext (the output of beforeAll) and the testCaseContext (the output of given):

OperationHandlerTestSetup.configureHandlerTest(
	updateProductOperationHandler,
	(handlerTest) =>
		handlerTest
			.usingHandlerContext('test')
			.beforeAll<{ storeId: string }>(async (auth, invoke) => {
				//Creates an store that will be used by all tests
				const createdStoreResult = invoke(createStoreHandler)({
					name: 'something',
				});
				return OperationHandlerResult.map(
					createdStoreResult,
					(createdStoreOutput) => {
						storeId: createdStoreOutput.id;
					}
				);
			})
			.testCase('should do something', (testCase) =>
				testCase
					.given<{ productId: string }>((auth, testContext, invoke) => {
						//Creates an product in the store to be used by the test
						const createdProductResult = invoke(createProductHandler)({
							name: 'some product',
							storeId: testContext.storeId,
						});
						return OperationHandlerResult.map(
							createdProductResult,
							(createdProductOutput) => {
								productId: createdProductOutput.id;
							}
						);
					})
					.when((auth, testContext, testCaseContext) => ({
						productId: testCaseContext.productId,
						name: 'updated name',
					}))
					.then(({ output }) => {
						const outputValue =
							OperationHandlerResult.getSuccessfulValueOrFail(output);
						expect(outputValue.name).toEqual('updated name');
					})
					.finally(({ testCaseContext }) => {
						//Deletes the product created for the test
						return invoke(deleteProductHandler)({
							productId: testCaseContext.productId,
						});
					})
			)
			.afterAll(({ testContext }) => {
				//Deletes the store created for all test cases
				return invoke(deleteStoreHandler)({ storeId: testContext.storeId });
			})
);

It is worth noting that while the DSL can be used to write complex functional tests, in practice, a connector test's focus is more about making sure that operations are properly communicating with the underlying implementation instead of testing its functionality, but ultimately it is up to the developer to decide how much and what type of coverage suits a given connector best.