@trayio/cdk-dsl

Connector Development Kit (CDK) DSL

The CDK Domain Specific Language (DSL) is the main component of Tray's CDK, it is used to define all the aspects of a connectors, including the behaviour of its operations.

A CDK connector consists of code written only using the DSL, which is declarative, so it only describes the connector, that description is then interpreted by the runtime to execute a connector's operations.

Project Structure

A CDK project is just a regular npm typescript project, preconfigured with all the dependencies, linter rules and compiler options that will be used to build the connector during deployment, so it is not recommended to change those.

Other than the package.json, jest configuration and typescript configuration, a connector will have the following:

• A connector.json file that includes metadata about the connector, such as the name, version, title, etc
• A src directory that has:
  • An Authentication typescript file that contains the type of the auth property of the ctx context object that operations receive together with the input, this type is the same for all operation
  • A test.ctx.json json file that contains a context value (including the auth) that can be used for tests. This file should not be committed to a repository as it will have sensitive information such as access tokens
  • One folder per operation

Context and Authentication

The context value (ctx) is received together with inputs in handlers, it contains values that are common to all operations, the main one being the auth property which usually contain things like tokens to identify the user making the request to a third party service.

Not all connectors need authentications, in that case, a never type can be used (with an empty value in the authentication test file), which is what the init command generates by default.

Operations

A connector will have one folder per operation under the src folder, this folder will contain the following files:

• operation.json which is an object that has operation metadata, like the operation name, title (which are mandatory) and the description.
• input.ts which contains the type of the input of the operation
• output.ts which contains the type of the output of the operation
• handler.ts which is where the logic of the operation is
• handler.test.ts which contains the test cases for testing the operation's behaviour defined in the handler

Handler

A handler at its core, describes a function, that takes an ctx value with an auth property described by the authentication type (which is the same for all operations) and it takes an input value described by the input type of the operation

The output of the handler is described by the output type, in case of a success, or it could contain an error if something failed during the execution of the handler or if the third party returned an error response.

This "successful value or failure error" result of running an operation is described by the OperationHandlerResult<T> type, which is a sum/union/or type of OperationHandlerSuccess<T> and OperationHandlerFailure

So, the core of what an operation handler describes can be summarised as a function:

(ctx: OperationHandlerContext<AuthType>, input: InputType) => OperationHandlerResult<OutputType>

However, when defining a handler, we can also specify things like validation or whether or not the handler is private, and this is where the DSL comes in.

The handler.ts file needs to define a handler using the OperationHandlerSetup.configureHandler() function, which allows for configuring all aspects of the handler by chaining function calls together for all the components of the handler.

The OperationHandlerSetup.configureHandler() function takes a callback that is used to configure the handler, it looks like this:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        /* use the "handler" value to specify what implementation to use, validation, etc */
    );

Validation

Handlers can have input validation (which runs before the handler implementation is executed to validate the input that will be used to run it) and output validation (which runs after the implementation to validate its output).

To add validation just use the handler argument of the callback described in the previous section:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation((validation => 
                    validation.condition((ctx, input) => input.id > 0)
                              .errorMessage((ctx, input) => `Id ${input.id} is not positive`))
                )
                .addOutputValidation((validation => 
                    validation.condition((ctx, input, output) => output.id === input.id)
                              .errorMessage((ctx, input, output) => `Output and Input ids don't match`))
                )
    );

Note that validation is optional, the only thing that is necessary to define a handler is its implementation.

Implementation

The main aspect of a handler is its implementation, which can be HTTP if the operation will make an HTTP call to a third party, or Composite if the operation will combine zero or more operations when it runs, more implementations for other protocols will be added in the future.

A handler can only have one implementation, it describes what the handler does when it receives a request.

HTTP Implementation

A very simple handler that makes an HTTP call can be configured in the following way:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.get('https://someapi.com/someresource/:id')
                        .handleRequest((ctx, input, request) =>
                            request.addPathParameter('id', input.id.toString())
                        )
                        .handleResponse((response) => response.withBodyAsJson())
               )
    );

The previous handler makes a GET http request, which is defined by the http.get() call, after which a handleRequest() function is chained, whose purpose is to take an auth value, an input value and a request configuration and add the necessary arguments to that request configuration based on what we want the http call to have, the supported methods on the request configuration are:

• addPathParameter(name, value): Will replace a parameter on the path specified as :name in the url as shown in the previous example, the value will be url encoded
• addHeader(name, value): Adds a header to the request
• withBearerToken(token): Adds an Authorization header with a Bearer token
• addQueryString(name, value): Adds a query string to the request, the value will be url encoded
• withBodyAsJson(body): Adds a body to the request that will be sent as json.

A handler with an authenticated POST request would look like this:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.post('https://someapi.com/someresource')
                        .handleRequest((ctx, input, request) =>
                            request.withBearerToken(auth.access_token)
                                   .withBodyAsJson(input)
                        )
                        .handleResponse((response) => response.withBodyAsJson())
               )
    );

The input does not have to match what is sent as the body, if for example the input has other flags that specify how the connector needs to behave and only part of it contains the body, the withBodyAsJson method can be called in the following way:

    request.withBodyAsJson({name: input.name, title: input.title})

So the handlerRequest function can transform the input in any way it needs to before sending the HTTP request and the same is true for the handleResponse, in the previous examples, the handleResponse simply read the response body as json and returned it, but it can be more complex if necessary.

The parseWithBodyAsJson<T>() function on the response argument returns a value of type OperationHandlerResult<T>, which can be a successful response or a failure based on the status code or if something went wrong executing the call.

Just like with the input type and the request, the type of the json body in the response can be different from the output type.

This is an example of a handler that transforms the response body into the output type:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.post('https://someapi.com/someresource')
                        .handleRequest(...)
                        .handleResponse((response) => { 
                            const httpResponseBody = response.parseWithBodyAsJson<{message: string}>()
                            if (httpResponseBody.isSuccess) {
                                const originalMessage = httpResponseBody.value.message
                                const extendedMessage = originalMessage + ' Extension'
                                return OperationHandlerResult.success({ message: extendedMessage })
                            }
                            return httpResponseBody
                        })
               )
    );

Instead of using an if in the previous case, there is also a OperationHandlerResult.map function that can do the same with a callback.

The handler can also return successful responses for some failure cases or viceversa, this is an example of a handler that "recovers" from errors to always return a successful response:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingHttp((http) =>
                    http.post('https://someapi.com/someresource')
                        .handleRequest(...)
                        .handleResponse((response) => { 
                            const httpResponseBody = response.withBodyAsJson()
                            if (httpResponseBody.isFailure) {
                                return OperationHandlerResult.success({ completed: false })
                            }
                            return OperationHandlerResult.success({ completed: true })
                        })
               )
    );

Composite Implementation

Composite handlers are used to define behaviours by invoking zero or more operations as part of their behaviour.

They can be used to write "helper" connectors (such as those in Tray's builder), DDL operations or complex operations like an "upsert" that combines more granular "read, create and update" operations.

A very simple composite handler, that just concatenates the firstName and lastName arguments it gets from the input into one string:

export const myOperationHandler = 
    OperationHandlerSetup.configureHandler<AuthType, InputType, OutputType>((handler) =>
        handler.addInputValidation(...)
               .addOutputValidation(...)
               .usingComposite(async (ctx, input, invoke) => {
                    const fullName = input.firstName + ' ' + input.lastName
                    return OperationHandlerResult.success({fullName: fullName})
               })
    );

As an example of more complex behaviour, this handler reads a list of products using another operation and converts the result into a simple list of {text: string, value: string} pairs, this is known as a Dynamic Data List (DDL) operation used to help users select values as part of configuring workflows within the tray builder.

To do this, the handler needs to invoke the regular getProducts operation, this is accomplished by using the invoke functions that composite handlers have access to, and passing it a handler reference and an input (no need to pass the auth value as it will be passed automatically), the handler reference passed to the invoke function is the result of the OperationHandlerSetup.configureHandler() function, which is why they are saved into a constant and exported like this:

export const getProductsHandler = 
    OperationHandlerSetup.configureHandler<AuthType, GetProductsInput, GetProductsOutput>((handler) =>
        handler.usingHttp(...)
    );

The getProductsHandler constant contains the handler reference, which also has the input and output type information as part of its type, to make sure that when invoked or tested, only valid input and output values can be used.

With that in mind, this is what the DDL handler would look like:

export const getProductsDDLHandler = 
    OperationHandlerSetup.configureHandler<AuthType, GetProductsDDLInput, GetProductsDDLOutput>((handler) =>
        handler.usingComposite(async (auth, input, invoke) => {
                    const productListResult: OperationHandlerResult<GetProductsOutput> = 
                        await invoke(getProductsHandler)({ storeId: input.storeId })
                    return OperationHandlerResult.map(
                        productListResult, 
                        productList => productList.map(product => { text: product.title, value: product.id })
                    )
               })
    );

There are several things to note about the handler, both the DDL handler and the getProductsHandler expect a storeId in the input to return a list of products for that store.

The getProductsHandler is invoked using the invoke function that composite handlers receive as an argument, passing the handler reference, and then calling the result as a function passing the input that handler expects, in this case, just an object with a storeId

The result of that invocation is of type Promise<OperationHandlerResult<GetProductsOutput>>, that is, a promise that has a value that is described by the invoked handler's output type, which is wrapped in the result object because it could be a successful invocation or a failure as described in previous sections.

The await keyword unwraps the promise, and we are left with a OperationHandlerResult<GetProductsOutput>, which forces the handler to deal with both the failure case as well as the success case.

There are multiple ways to do this

• Using if or switch statements to narrow down the type
• Using the OperationHandlerResult.getSuccessfulValueOrFail() function which unwraps the value if successful or terminates the function propagating the error if it is not
• As shown in the example, using the OperationHandlerResult.map() function.

Once the handler has access to the product list value, it just needs to convert each element to a {text: string, value: string} pair.

Testing

The CDK DSL has declarative testing functions to test a handler's behaviour, the tests are in the handler.test.ts file within the operation folder, which can have zero, one or many test cases for that given operation.

The OperationHandlerTestSetup.configureHandlerTest() function is used to describe a test, it takes a handler reference and a callback with an object used to configure the test, in a similar way handlers are configured.

This is what a very basic test looks like:

OperationHandlerTestSetup.configureHandlerTest(
	myOperationHandler,
	(handlerTest) =>
		handlerTest
			.usingHandlerContext('test') //will use `test.ctx.json` as the context value which includes authentication values for all test cases
			.nothingBeforeAll()
			.testCase('should do something', (testCase) =>
				testCase
                    .usingHandlerContext('another') //optionally, a test case can define its own context instead of using the default one defined for all tests
					.givenNothing()
					.when(() => /* return an input that matches the input type */)
					.then(({ output }) => {
                        /* output is OperationHandlerResult<T> where T is a value matching the output type */

                        //This will contain a value of type T if the operation was successful or the test will fail if not
                        const successValue = OperationHandlerResult.getSuccessfulValueOrFail(output)

                        // jest-style matchers like "expect" are available here
                        expect(successValue).toEqual(...)
                    })
					.finallyDoNothing()
			)
			.nothingAfterAll()
);

The structure of a test is well defined, and the type safe declarative DSL will enforce that, in particular, there are a number of aspects that would apply to all test cases:

• A default context to use for all test cases
• Run one or more operations (doesn't have to be the one being tested) before all test cases using the beforeAll() function, or don't do anything before all test cases using the nothingBeforeAll() function, these functions can only be used before adding test cases (this is enforced by the type system)
• Add a test case using the testCase() function.
• Run one or more operations (doesn't have to be the one being tested) after all test cases using the afterAll() function, or don't do anything after all test cases using the nothingAfterAll() function, these functions can only be used after defining test cases, and no test cases can be added after this (this is enforced by the type system)

As for the test cases, they use a BDD style Given/When/Then convention, in particular a test case has:

• An optional usingHandlerContext() function at the beginning of the test case to use a different context than the default for all test cases
• A given() function to run one or more operations at the beginning of the test case or givenNothing() to go straight to running the operation under test
• A when() function to create an input value that will be used to run the operation under test, that value needs to match the input type of the operation
• A then() function that gets the output, input, auth and optionally the result of beforeAll() and given() if present, which can be used to do the assertions of the test case using jest-style matchers
• A finally() function to run one or more operations at the end of the test case, usually for cleanup, or finallyDoNothing() to don't do anything else after the assertions.

Both the beforeAll() and given() functions allow to run multiple operations before all test cases or before a single test case, they receive the invoke function as an argument just like composite handlers and they return an object that can contain some of relevant information about the operations that ran if necessary (like ids of things created) in a value that can be accessed by the when(), then(), finally() and afterAll() functions, they receive them as arguments.

As an example, the following test is for an updateProduct operation, it creates a store for all test cases, and a product for every test case to test the update on, using beforeAll and given respectively, and accessing the identities of the created objects from the testContext (the output of beforeAll) and the testCaseContext (the output of given):

OperationHandlerTestSetup.configureHandlerTest(
	updateProductOperationHandler,
	(handlerTest) =>
		handlerTest
			.usingHandlerContext('test')
			.beforeAll<{storeId: string}>(async (auth, invoke) => {
				//Creates an store that will be used by all tests
				const createdStoreResult = invoke(createStoreHandler)({name: 'something'})
				return OperationHandlerResult.map(
					createdStoreResult, 
					createdStoreOutput => { storeId: createdStoreOutput.id}
				)
			})
			.testCase('should do something', (testCase) =>
				testCase
					.given<{ productId: string}>((auth, testContext, invoke) => {
						//Creates an product in the store to be used by the test
						const createdProductResult = invoke(createProductHandler)({name: 'some product', storeId: testContext.storeId})
						return OperationHandlerResult.map(
							createdProductResult, 
							createdProductOutput => { productId: createdProductOutput.id}
						)
					})
					.when((auth, testContext, testCaseContext) => ({ productId: testCaseContext.productId, name: 'updated name' }))
					.then(({ output }) => {
						const outputValue = OperationHandlerResult.getSuccessfulValueOrFail(output)
						expect(outputValue.name).toEqual('updated name');
					})
					.finally(({ testCaseContext }) => {
						//Deletes the product created for the test
						return invoke(deleteProductHandler)({productId: testCaseContext.productId})
					})
			)
			.afterAll(({ testContext }) => {
				//Deletes the store created for all test cases
				return invoke(deleteStoreHandler)({storeId: testContext.storeId})
			})
);

It is worth noting that while the DSL can be used to write complex functional tests, in practice, a connector test's focus is more about making sure that operations are properly communicating with the underlying implementation instead of testing its functionality, but ultimately it is up to the developer to decide how much and what type of coverage suits a given connector best.