@doist/filterist-definitions

filterist-definitions

filterist-definitions contains three components:

• Filter definition files. The files define the regular expression codes to be used to parse filters in different languages.
• Filter client tests.
• Definition generator. The generator will pull the latest filter translations and generate the definitions file and filter client tests.

filterist-definitions by itself did not run any tests, but it generates files that will be added to your own client so you can base your tests on it.

It's up to the client to decide the method on how to add files to their repositories (copy & paste, git submodule, etc.)

Todoist filter queries are defined in this Google Spreadsheet

File anatomy

• dist/: filter definition files
• test_json/: filter client tests
• src/: definition generator

Definition generator

Setup

Prerequisite: the generator requires Node.js > 10.0. For convenience, a .nvmrc file is provided so node version managers like fnm can switch to an appropriate version.

A credentials.json file containing OAuth 2.0 client secrets will need to be placed in the root of the project, which identifies the generator when asking users to authenticate. To download this file, either visit this url, or go to the Google Cloud Console, look for the "Filterist definitions" project under the Doist.io organization, and go to APIs & Services -> Credentials. From there, find the "Filterist definitions" OAuth 2.0 Client ID, and use the "Download JSON" button.

Lastly, to pull dependencies hosted on the GitHub package registry from the @doist scope, npm will need to be authenticated with GitHub. Instructions are available in the handbook here.

Then in the repo root directory, run:

npm install

Generate definitions files and test files

To run the generator, run:

npm run generate

The first time this is run, an "Authorize this app by visiting this url" message will appear. Open the URL to go through the authentication flow and you'll land on a http://localhost/?code url. Copy the value of the code parameter, and paste it into your terminal. This will then write a token for your account into token.json, allowing spreadsheets from the Doist org to be read.

Support a new filter query

To introduce support a new filter query

• Define the new query in the filter spreadsheet. The detail of how the sheet works could be found here.

• Update the generator config. The generator config defines a set of queries it expects to generate, and also the expected query variable for each query. Add the new query to the generator config.

After the completion above steps, the generator should be able to generate the updated the filter definition files and tests with the new query.

Instructions for Client Teams

Frontend

In order to use the generated files from this repository it's necessary to release this package. Follow these instructions:

• Generate the files using npm run generate and make sure the dist folder is updated accordingly
• Update the version number following Semantic Versioning with npm version <major|minor|patch>
  • If existing translations or tokens are changed, a major release should be created as our users' filters can break, and we should ensure that the Support team is aware and the appropriate changes are made in the Help Center
  • If tokens are added, then a minor release can be used
• Push the new commit and tag upstream with git push --follow-tags
• Ensure you're logged into npm using the GitHuib Package Registry, using a Personal Access Token that grants the write:packages scope. See Code Sharing in the handbook for more details
• Run npm publish to publish a new version to the GitHub Package Registry
• Update the dependency in todoist-web

Android

To release new filterist version follow these steps:

• Generate the files using npm run generate. This step may be omitted if the latest build artifacts have already been commited into main. You can check the tags page to see if the latest changes have already been released.
• Move generated files from /dist/kt/ to FilteristKt
• Move generated test files (including those in lexer_i18n_tests) from /test_json/ to FilteristKt
• Run ./gradlew build to ensure things are correct
• Open PR in FilteristKt with updated definitions and, once merged, follow release instructions

iOS

Follow these manual steps to use the generated files from this repository until we integrate FilteristKt into the iOS app:

• Generate the files using npm run generate. This step may be omitted if the latest build artifacts have already been commited into main. You can check the tags page to see if the latest changes have already been released.
• Move the generated swift files from /dist/swift/ to TDFilter
• Move the json files from /test_json/ to TDFiltersTests
• Run tests in TDFilter