@openstapps/core-tools

@openstapps/core-tools

Tools to convert and validate StAppsCore

What are the tools for?

The StAppsCore Converter is a tool for converting SC-types (TypeScript) into JSON schema files.

JSON schema files are needed for run-time validation of SC-type objects, as this is a tedious task to do using SC-types defined in TypeScript (not possible without additional coding). That said, StAppsCore Converter practically prepares SC-types to be used for object validation (determining whether a JavaScript/JSON object is a valid object of the corresponding SC-type) using StAppsCore Validator.

The StAppsCore Validator is a tool for run-time validation of objects (determining whether a JavaScript/JSON object is a valid object of the corresponding SC-type. It consumes JSON schema files from StAppsCore as the definitions of SC-types against which are validated concrete (actual) objects (as an example SCDish object in the example below).

Installation

Installation of the npm package (using npm install) makes the tool available as an executable with the name openstapps-core-tools.

How to use the converter?

Add @validatable to the Typedoc comment of the types that you want to convert to JSONSchema.

The command openstapps-core-tools can then be called using these arguments:

openstapps-core-tools schema <srcPath> <schemaPath>

where:

• <srcPath> is path to the project (where used *.ts files are, e.g. src/core,
• <schemaPath> is directory to save output files to, e.g. lib/schema.

Complete command with the example arguments is then:

openstapps-core-tools schema src/core lib/schema

Inside of a script in package.json or if the npm package is installed globally, the tool stapps-convert can be called without its local path (node_modules/.bin):

openstapps-core-tools schema src/core lib/schema

How to use the validator?

Using the validator programatically

import {Validator} from '@openstapps/core-tools/lib/validate';
import {SCDish, SCThingType} from '@openstapps/core';
import {ValidatorResult} from 'jsonschema';
import {join} from 'path';

const objectToValidate: SCDish = {
type: SCThingType.Dish,
  // more properties
};

// instantiate a new validator
const validator = new Validator();

// make the validator read the schema files
validator.addSchemas(join('node_modules', '@openstapps', 'core', 'lib', 'schema')).then(() => {
  // validate an object
  const result: ValidatorResult = validator.validate(objectToValidate, 'SCDish');
});

Using validateFiles function

The JSON files passed to the validateFiles method have an added layer. That layer encapsulates the actual JSON data of the object to be verified and adds a property to enable true negative testing.

Your basic JSON object:

{
  "property1": "value1",
  "property2": "value2",
  ...
}

JSON for validateFiles:

{
  "errorNames": [],
  "instance": {
    "property1": "value1",
    "property2": "value2",
    ...
  },
  "schema": "NameOfSchema"
}

Where errorNames holds the string values of the name property of the expected ValidationErrors from JSON Schema. Empty array means no errors are expected.

schema holds the name of the schema to validate the instance against.

How to use validator as a CLI tool (executable)?

The command openstapps-core-tools can then be called using these arguments:

openstapps-core-tools validate <schemaPath> <testPath> [reportPath]

where:

• <schemaPath> is a directory where JSON schema files are, e.g. lib/schema,
• <testPath> is a directory where test files are, e.g. src/test/resources,
• [reportPath] is a file where the HTML report of the validation will be saved to, e.g. report.html (optional argument - if it's not provided no report will be written).

Command with the example arguments is then for example:

openstapps-core-tools validate lib/schema src/test/resources

Inside of a script in package.json or if the npm package is installed globally, the tool openstapps-validate can be called without its local path (node_modules/.bin):

openstapps-core-tools validate lib/schema src/test/resources report.html

Generate documentation for routes

To generate a documentation for the routes use the following command.

openstapps-core-tools routes PATH/TO/CORE/lib PATH/TO/ROUTES.md

Pack definitions and implementations

To pack all the different files into two distribution files - one for definitions/one for implementations - use the following command:

openstapps-core-tools pack

How to use the Elasticsearch Mapping generator

The mapping generator is intended to be used by the backend directly, but it can also be used to generate these files manually.

Generating the mapping files by hand

To generate the mapping files by hand, you need a local copy of the core-tools and the core, both need to be built first. After that you can run

node lib/cli.js mapping path/to/core path/to/destination ignoredTag1,ignoredTag2,ignoredTag3

If you don't pass in any ignored tags, you will likely be prompted with errors, despite the core being absolutely correct. This is because there are some tags that are not relevant to Elasticsearch, but the program has no direct way to tell which ones simply lack an implementation and which ones can be ignored. Currently the ignored tags include minlength, pattern and see.

Generating the mapping directly from another TypeScript program

This is the more easy way, and it gives you direct access to the generated mapping as a (mostly typesafe) object. To use it, call generateTemplate, However you will first need to generate a ProjectReflection of the core you are working with. If you use the core as a dependency, you can for example use

const map = generateTemplate(getProjectReflection(resolve('node_modules', '@openstapps', 'core', 'src')),
      ignoredTags, false);

to generate the mappings. Note that the result object contains both a list of errors in map.errors and the actual mapping in map.template. You can also specify whether you want the generator to show any errors while generating the mappings in the console in the last parameter, true (default) being show errors and false to suppress them. That said it is very easy to replace all type: "MISSING_PREMAP", type: "PARSE_ERROR", type: "TYPE_CONFLICT" with dynamic: true (you can take these exactly like written here and run a replace over the file).

Fixing a generated mapping by hand

If you get errors when generating the mappings, the mappings might not work, however they will still be generated to the programs best efforts. Most small issues can be fixed after the mapping was generated as a temporary solution and without changing anything in the mapper's code. This however requires some understanding of how mappings work.

The output of the program can easily reach 25.000 lines, but you can find errors quickly by searching for MISSING_PREMAP, PARSE_ERROR and TYPE_CONFLICT. When you reach them you can then manually replace them with your code.

As a last resort you can also replace all errors with dynamic types, this should get the mapping working, but it is NOT RECOMMENDED as a fix other than using it locally.

How to use the UML generator

The UML Generator generates PlantUML from the project reflection of the source files. By default it will include externals, which will take considerably longer to execute, you can disable this behaviour via an option. It can help you to visually explore the data model or document a specific part.

You can either use the public PlantUML-server or start your own local instance. To run, restart or stop the container use the scripts provided in the package.json.

Generating from source-files

openstapps-core-tools plantuml PATH/TO/SOURCEFILES http://PLANTUMLSERVER

Executing this command will generate a .svg file in your current working directory.

Multiple options can be set to enhance the diagram. By default all additional information other than the definitions are disabled. You can use:

• --showProperties to show all mandatory attributes of the classes and interfaces.
• --showOptionalProperties to show all mandatory attributes of the classes and interfaces. --showProperties must be set!
• --showInheritedProperties to show all inherited attributes of the classes and interfaces. --showProperties must be set!
• --showEnumValues to show all enumeration and type (enumeration-like) values
• --showInheritance to show the hierarchy of the classes and interfaces. Inherited attributes will only be shown in their parent.
• --showAssociations to show all references of classes and interfaces between one another
• --excludeExternals to exclude external definitions
• --definitions <definitons> to show only specific definitions to reduce the output of the diagram. <definitions> is a comma seperated list of definitions.
• --outputFileName <fileName> for a custom file name, the file extension will be added automatically (.svg). Otherwise a generic file with a timestamp will be generated into the execution directory. If a file with the same name already exists it will be overwritten!

The best way to explore models is to enable --showInheritance and --showAssociations. Start with just one definition in your --definition <definitions>-list, generate the diagram, look at it, add a new definition that you have seen to your command and generate anew.

Examples

Show the class hierarchy of the whole project:

openstapps-core-tools plantuml PATH/TO/SRCDIR http://PLANTUMLSERVER --showInheritance

Show the dish-module:

openstapps-core-tools plantuml ../core http://localhost:8080 --showProperties --showOptionalProperties --showInheritance --showAssociations --showEnumValues --definitions SCDish,SCThingThatCanBeOfferedOffer

Generating from existing file

The plantuml code is persisted inside the generated file at the very bottom. You can tweak the model by using the function to generate UML from a PlantUML-file(simple text file). Extract the code (starting from @startuml to @enduml), edit it manually and execute this function.

openstapps-core-tools plantuml-file /PATH/TO/Project.plantuml http://PLANTUMLSERVER OptionalCustomFileName

Example-File-Content of Project.plantuml

@startuml
interface MyClass{
  myProperty: string
}
@enduml