🚀 Socket Launch Week Day 5:Introducing Repository Access Permissions and Custom Roles.Learn more →
Sign In

@traqula/chevrotain

Package Overview
Dependencies
Maintainers
2
Versions
11
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

@traqula/chevrotain - npm Package Compare versions

Comparing version
0.0.23
 to
0.0.24
+12
-8
package.json
{
"name": "@traqula/chevrotain",
"type": "module",
"version": "0.0.23",
"version": "0.0.24",
"description": "Core components of Traqula",

@@ -30,6 +30,6 @@ "lsd:module": true,

"files": [
"dist/**/*.d.ts",
"dist/**/*.js",
"dist/**/*.js.map",
"dist/**/package.json"
"dist/*/lib/**/*.d.ts",
"dist/*/lib/**/*.js",
"dist/*/lib/**/*.js.map",
"dist/cjs/package.json"
],

@@ -40,4 +40,5 @@ "engines": {

"scripts": {
"build": "node \"../../node_modules/typescript/bin/tsc\" -b",
"build:transpile": "node \"../../node_modules/esbuild/bin/esbuild\" --format=cjs --bundle --log-level=error --outfile=dist/cjs/lib/index.js lib/index.ts"
"build": "yarn build:ts && yarn build:transpile",
"build:ts": "node \"../../node_modules/typescript/bin/tsc\" -b",
"build:transpile": "node \"../../node_modules/esbuild/bin/esbuild\" --format=cjs --target=es2022 --bundle --log-level=error --outfile=dist/cjs/lib/index.js lib/index.ts"
},

@@ -47,3 +48,6 @@ "dependencies": {

},
"gitHead": "36ec1aae80524a84f044182811a55ebeef6d9cda"
"devDependencies": {
"esbuild": "^0.25.0"
},
"gitHead": "9865fe1fe302819d46e19c6f3250ceef778f434f"
}
+7
-175
README.md

@@ -1,12 +0,12 @@

# Traqula core package
# Traqula Chevrotain package
Traqula core contains core components of Traqula.
Most importantly, its [lexer builder](./lib/lexer-builder/LexerBuilder.ts), [parser builder](./lib/parser-builder/parserBuilder.ts), and [generator builder](./lib/generator-builder/generatorBuilder.ts).
This library heavily relies on the amazing [Chevrotain package](https://chevrotain.io/docs/).
Knowing the basics of that package will allow you to quickly generate your own grammars.
Exports the [Chevrotain](https://chevrotain.io/docs/) package and tries to bundle a CJS version too.
This is NOT a redistribution of Chevrotain, not should a project outside of Traqula depend on this.
The reason for exporting is so the transpiled versions of the traqula packages use the same version of Chevrotain (instead of copying).
## Installation
```bash
npm install @traqula/core
npm install @traqula/chevrotain
```

@@ -17,171 +17,3 @@

```bash
yarn add @traqula/core
yarn add @traqula/chevrotain
```
## Usage
Each parser contains two steps:
1. a lexer
2. a grammar + abstract syntax tree generation step.
Sometimes grammar definitions and abstract syntax tree generation is split into separate steps.
In this library, we choose to keep the two together when building a parser.
### Lexer Builder
To tackle the first step, a lexer should be created.
This is a system that separates different groups of characters into annotated groups.
In human language for example the sentence 'I eat apples' is lexed into different groups called **tokens** namely `words` and `spaces`:
`I`, ` `, `eat`, ` `, `apples`.
To create a token definition, you use the provided function `createToken` like:
```typescript
const select = createToken({ name: 'Select', pattern: /select/i, label: 'SELECT' });
```
Lexer definitions are then put in a list and when a lexer is build, the lexer will match a string to the [**first token in the list**](https://chevrotain.io/docs/tutorial/step1_lexing.html#creating-the-lexer) that matches.
Note that the order of definitions in the list is thus essential.
We therefore use a [lexer builder](./lib/lexer-builder/LexerBuilder.ts) which allows you to easily:
1. change the order of lexer rules,
2. and create a new lexer staring from an existing one.
Creating a builder is as easy as:
```typescript
const sparql11Tokens = LexerBuilder.create(<const> [select, describe]);
```
A new lexer can be created from an existing one, and altered by calling:
```typescript
const sparql11AdjustTokens = LexerBuilder.create(sparql11Tokens).addBefore(select, BuiltInAdjust);
```
### Parser Builder
The grammar builder is used to link together grammar rules such that they can be converted into a parser.
Grammar rule definitions come in the form of [ParserRule](./lib/parser-builder/ruleDefTypes.ts) objects.
Each `ParserRule` object contains its name and its returnType.
Optionally, it can also contain arguments that should be provided to the SUBRULE calls.
A simple example of a grammar rule is the rule bellow that allows you to parse booleanLiterals.
```typescript
/**
* Parses a boolean literal.
* [[134]](https://www.w3.org/TR/sparql11-query/#rBooleanLiteral)
*/
export const booleanLiteral: ParserRule<'booleanLiteral', LiteralTerm> = <const> {
name: 'booleanLiteral',
impl: ({ CONSUME, OR, context }) => () => OR([
{ ALT: () => context.dataFactory.literal(
CONSUME(l.true_).image.toLowerCase(),
context.dataFactory.namedNode(CommonIRIs.BOOLEAN),
) },
{ ALT: () => context.dataFactory.literal(
CONSUME(l.false_).image.toLowerCase(),
context.dataFactory.namedNode(CommonIRIs.BOOLEAN),
) },
]),
};
```
The `impl` member of `ParserRule` is a function that receives:
1. essential functions to create a grammar rule (capitalized members),
2. a context object that can be used by the rules,
3. a cache object ([WeakMap](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakMap)) that can be used to cache the creation of long lists in the parser, [increasing parser performance](https://chevrotain.io/docs/guide/performance.html#caching-arrays-of-alternatives).
You cannot unpack the context entry in the function definition itself because the parser uses a [recording phase](https://chevrotain.io/docs/guide/internals.html#grammar-recording) to optimize itself. During this phase, the context entry will be undefined, as such, it can only be accessed within the `ACTION` function.
The result of an `impl` call is a function called a `rule`.
Rules can be [parameterized](https://chevrotain.io/docs/features/parameterized_rules.html), although I have not found a scenario where that is usefully.
Personally I create a function that can be used to create multiple `ParserRule` objects.
The result of a rule should match the type provided in the `ParserRule` definition, and is the result of a call of `SUBRULE` with that rule.
##### Testing the correctness of your parser
By default, the parser builder will construct a parser that does not perform validation (to be more speedy).
When creating a parser, you best enable the validation by passing a context to the parser builder like:
```typescript
const context = {
tokenVocabulary: myLexerVoc,
lexerConfig: {
skipValidations: false,
ensureOptimizations: true,
},
parserConfig: {
skipValidations: false,
},
}
```
#### Patching rules
When a rule definition calls to a subrule using `SUBRULE(mySub)`, the implementation itself is not necessarily called.
That is because the SUBRULE function will call the function with the same name as `mySub` that is present in the current grammarBuilder.
A builder is thus free to override definitions as it pleases. Doing so does however **break the types** and should thus only be done with care.
An example patch is:
```typescript
const myBuilder = Builder
.createBuilder(<const> [selectOrDescribe, selectRule, describeRule])
.patchRule(selectRuleAlternative);
```
When `selectOrDescribe` calls what it thinks to be `selectRule`,
it will instead call `selectRuleAlternative` since it overwrote the function `selectRule` with the same name.
When you are creating a new parser,
it might be good to test your parser by setting `skipValidations: false` in the context of the `.build` function.
### Generator Builder
The generator builder function in much the same as the [parser builder](#parser-builder).
Your builder expects objects of type [GeneratorRule](lib/generator-builder/generatorTypes.ts),
containing the implementation of the generator in the `gImpl` member.
The `gImpl` function gets essential functions to create a generator rule (capitalized members),
returning a function that will get the AST and context, returning a string.
For generator rules, you can unpack the context since no recording phase is present in this case.
The idea is that GeneratorRules and ParserRules can be tied together in the same object, as such, similar behaviour is grouped together.
```typescript
/**
* Parses a named node, either as an IRI or as a prefixed name.
* [[136]](https://www.w3.org/TR/sparql11-query/#riri)
*/
export const iri: GeneratorRule<'iri', IriTerm> = <const> {
name: 'iri',
gImpl: ({ PRINT }) => ast => { PRINT(ast.value) },
};
```
While implementing a generator, you can easily support pretty print indentation manipulating `traqulaIndentation` context item.
The key for this context item can be accessed like:
```typescript
import { traqulaIndentation } from '@traqula/core';
C[traqulaIndentation] += 2;
```
### A word on round tripping:
The generator builder can significantly help you with creating a round tripping parser.
Basically what that allows you to do is to keep information that the AST finds 'unimportant' within the generated string.
Take for example capitalization and spaces in the sparql spec.
Both are ignored in the AST, but if you want to generate the same string out of your AST, yuo need to store them somewhere.
Traqula helps you store this information using it's `Node` `Localization`.
Localization basically allows you to remember what _portion of the original string_ a node represents.
Take for example the `SENTENCE`: `I Love Traqula`, If we ignore spaces and caps in the ast, a valid representation would be:
```
SENTENCE-node{ words: [ WORD-node{ value: "i" }, WORD-node{ value: "love" }, WORD-node{ value: "traqula" } ] }
```
If we generated we would loe the capitalisation and get: `i love traqula` for example.
Round tripping will add a `source localization` for each node,
we therefore register that our SENTENCE starts at 0 and ends at 19, while our words have ranges 0-1, 2-6, 12-19.
Using this information our generator can reconstruct the original string (given the original string).
The magic happens when we start manipulating the words, so imagine we want to lowercase the word 'Love',
we would simply annotate in the `localization` that the node should be generated (and not reconstructed),
and we can generate the sentence: `I love Traqula`.
To support this feature, the generator requires that your AST follows a tree structure with respect to the ranges.
That means that a node cannot start later, or end earlier than its children.
In our example: A sentence cannot start after the first word start, nor can it end before the last word ends.