@stackql/provider-doc-util

provider-doc-util

Command line utility to help developers prepare and submit StackQL provider specs, see StackQL Readme

Installation

NPM

npm i @stackql/provider-doc-util

YARN

yarn add @stackql/provider-doc-util

Setup

from the package directory, run:

npm link

Background

Read this section for background on the StackQL product

The StackQL utility provides a SQL interface to cloud and SaaS providers, mapping a provider to an ORM, transpiling input SQL to provider API requests, and bringing back response data as a SQL based result set. StackQL is capable of DML operations such as `INSERT` and `DELETE` which can be used to provision or de-provision cloud resources, query operations using `SELECT` to collect, analyze, report on asset or configuration data, and lifecycle operations such as starting a VM instance using the `EXEC` command in StackQL.

The StackQL ORM provides a logical model to work with cloud resources similar to the way databases are organized into schemas. This object/resource hierarchy is summarized below:

provider/
├─ service/
│  ├─ resource/
│  │  ├─ fields
│  │  ├─ methods

an example would be:

google/
├─ compute/
│  ├─ instances/
│  │  ├─ id, name, status, ...
│  │  ├─ start, stop, ...

Enabling StackQL to interact with the google provider using SQL semantics, for example:

Provider discovery operations such as..

SHOW RESOURCES IN google.compute;
DESCRIBE google.compute.instances;

Query operations such as..

SELECT status, COUNT(*) as num_instances 
FROM google.compute.instances
WHERE project = 'myproject' and zone = 'us-west-1a'
GROUP BY status;

Provisioning operations such as creating a Compute Engine instance using an INSERT statement or deprovisioning an instance using a DELETE statement.

Read this section for background on the StackQL Provider Registry

StackQL provider interfaces (such as GCP, Okta, GitHub, AWS, Azure, etc) are defined using annotations to the provider API (OpenAPI) specification, these annotations or extensions allow StackQL to map the providers resource to the desired ORM and define routes for SQL verbs such as `SELECT`, `INSERT`, `DELETE`, and `EXEC`.

Usage

provider-doc-util <command> <apiDoc or providerDevDocRoot> <stackqlProviderName> <stackqlProviderVersion> [<OPTIONS>]

Commands

dev

Creates StackQL provider development docs and splits an API into service scoped documents based upon a specified discriminator (optional). Development docs will parse a providers API and map default routes for StackQL query and DML operations, these can be modified or enriched by the developer. For convenience, development docs can be generated in json, yaml, toml or hcl formats. The development docs are then assembled using the build command and then can be tested locally see Test Using a Local Registry. Once you have tested locally you can raise a pull request to stackql/stackql-provider-registry.

build

Assembles StackQL development docs into a registry compatible format, ready to use as a local registry for testing or to raise a pull request to stackql/stackql-provider-registry.

Options

--svcDiscriminator, -s JSONPath expression OR svcName:servicename

The service discriminator option is used to determine how to split a large provider document into smaller, service scoped documents. The option is required for the dev command and ignored otherwise. If you do not wish to spilt the provider API spec, specify svcName:<servicename> for this option which will define one service in the StackQL provider with the name provided in servicename.

Example: -s "$['x-github'].category" would split the given provider API spec into service documents by matching the x-github.category value in each unique operation (combination of a path and an HTTP method) in API doc.

--resDiscriminator, -r JSONPath expression

The resource discriminator option is used to determine how to identify StackQL resources in a given provider API spec.

Example: -r "$['x-github'].subcategory" would identify resources in the given provider API spec by matching the x-github.subcategory value in each unique operation (combination of a path and an HTTP method) in API doc.

--methodkey, -m JSONPath expression

The method key option determines uniquely named operations which are mapped to SQL methods in the StackQL ORM. These methods are then mapped to default routes (SQL query and DML methods) in StackQL, the developer can override or update these mappings in the development docs which are outputted from the dev command.

If supplied it must be a JSONPath expression relative to the operation (http path and verb), if not supplied it will default to operationId in the OpenAPI specification for each operation.

--output, -o directory

The output directory option specifies where to write out the development docs in a dev operation or the production docs in a build operation.

If not supplied it will default to the current working directory

--format, -f yaml | json | toml | hcl

The output format option specifies the desired output for the development docs - the annotations/extensions required for StackQL which the developer can modify or enrich. For convenience multiple serialization formats are available including yaml, json, toml and hcl (the HashiCorp Configuration Language).

--debug, -d

debug flag which can be set for additional print statements.

Example

This example demonstrates creating a StackQL provider for github and testing this provider locally using stackql

Example Project Structure

The following directory structure represents the target after an API dev workspace is set up and the StackQL provider for github is built.

local-registry/
├─ dev/
│  ├─ github/
│  │  ├─ v1/
│  │  │  ├─ provider.yaml
│  │  │  ├─ services/
│  │  │  │  ├─ repos/
│  │  │  │  │  ├─ repos-v1.yaml
│  │  │  │  │  ├─ repos-v1-resources.yaml
│  │  │  │  ├─ .../
├─ ref/
│  ├─ github/
│  │  ├─ api.github.com.yaml
├─ src/
│  ├─ github/
│  │  ├─ v1/
│  │  │  ├─ provider.json
│  │  │  ├─ services/
│  │  │  │  ├─ repos/
│  │  │  │  │  ├─ repos-v1.json

local-registry/ref/github/api.github.com.yaml is the source OpenAPI 3 specification for the github api, this was sourced from GitHub.

The dev directory contains the output of the dev docs generated by provider-doc-util dev,

The src directory contains the output of the StackQL provider interface generated from the dev docs using provider-doc-util build.

Create Working Provider Docs

PowerShell:

cd local-registry
provider-doc-util dev `
./ref/github/api.github.com.yaml `
github `
v1 `
-s "$['x-github'].category" `
-r "$['x-github'].subcategory" `
-o ./dev `
-f toml

Bash:

cd local-registry
provider-doc-util dev \
ref/github/api.github.com.yaml \
github \
v1 \
-s '$["x-github"].category' \
-r '$["x-github"].subcategory' \
-o ./dev \
-f yaml

Build Provider Docs

PowerShell:

cd local-registry
provider-doc-util `
build `
./dev `
github `
v1 `
-o ./src

Bash:

cd local-registry
provider-doc-util \
build \
./dev \
github \
v1 \
-o ./src

Test Using a Local Registry

cd local-registry
PROVIDER_REGISTRY_ROOT_DIR="$(pwd)"
REG_STR='{"url": "file://'${PROVIDER_REGISTRY_ROOT_DIR}'", "localDocRoot": "'${PROVIDER_REGISTRY_ROOT_DIR}'", "verifyConfig": {"nopVerify": true}}'
AUTH_STR='{"github": { "type": "null_auth" }}'
./stackql shell --auth="${AUTH_STR}" --registry="${REG_STR}"