@gholk/tsjson

Template S-expression to JSON

A Template literal converts s-expression to json which support variable embedding, include a sxml like mini template engine.

you can play tsjson and sxml in browser in playground.html.

usage

import tsj from '@gholk/tsjson' // or 'path/to/tsjson/index.esm.js'
const j = tsj.j

const jo = j `1 2 a b ("str1" "str2") (:ok false :val null)`
// [1, 2, 'a', 'b', ['str1', 'str2'], {ok: false, val: null}]

const jv = j `type-${jo[2]} ${x => x*2} "1 + 2 = ${1+2}\\n"`
// ['type-a', x=>x*2, "1 + 2 = 3\n"]

with plain string:

j('1 2 (x 0) ${1+2}\nl2 \t "escape sequence \\n \\\\ \\t end"')
// [1, 2, ['x', 0], '${1+2}', 'l2', 'escape sequence \n \\ \t end']

array

The whole s-expression is wrap in brackets automatically.

j `1` // [1]
j `1 2 3` // [1, 2, 3]
j `` // []
j `1 (2 3) () 4` // [1, [2, 3], [], 4]

literal

number, string, boolean, null and undefined literal are support.

j `0 -1 0.1 999 -0` // [0, -1, 0.1, 999, -0]
j `"foo" 'bar'` // ['foo', 'bar']
j `true false null (NaN undefined)` // [true, false, null, [NaN, undefined]]

tsjson does not support lisp-style nil and t.

symbol

other symbol (lisp) are treated encode to string directly. symbol are strings which contain no space and special character.

j `symbol1 s2 a-symbol *star* under_score`
// ['symbol1', 's2', 'a-symbol', '*star*', 'under_score']

(most cases, in js, symbol is just string. we write addEventListener('click', ...), where the click is a symbol.)

string

string are treated as string. string can use double quote and single quote.

j `"abc" 'def ghi'` // ['abc', 'def ghi']

tsj will use the raw string, so the result will be intuitive when using template literal string. (this mean that you don't need escape special chars twice, in backquote string and in tsj string.)

j `"a \\ \n \\n \\\n \r \t  \" \' \` \${1} ${1}"`
// ['a \\ \n \\n \\\n \r \t " \' ` ${1} 1']

/* in *raw*:
   a \ <LF> \n \<LF> <CR> <TAB> " ' ` ${1} 1
*/

A newline prefixed with backslash would be ignored:

j `"a\
b"`
// ['ab']

variable interpolation

template string allow variable interpolation, tsj will handle this in a intuitive way.

variable standalone

A standalone variable will keep as it was.

j `a ${'a b'} c` // ['a', 'a b', 'c']
j `a (${ {n: 3} } 2) ${[null]}` // ['a', [{n:3}, 2], [null]]
j `${x=>x*2} ${/[a-z]/g}` // [x=>x*2, /[a-z]/g]

variable concat symbol

when a variable adjoins a symbol or another variable, they are concated and treat as a single symbol.

let x = 'string'
j `sym-${x}-end sym${x} ${x}sym` 
// ['sym-string-end', 'symstring', 'stringsym']

let y = 'string with space'
j `${x}${y} ${y}s${y}`
// ['stringstring with space', 'sstring with space']

variable inside string

when a variable is inside a string, its value is direct concat in the string. (the variable in string will not get its content unescape.)

let s = 'a\\nb'
j `"1 ${s} 2"` // ["1 a\\nb 2"]

object

if a array's item are symbols and prefix with colon, then it is treated as an object.

this is an object:

j `(:key k :value v)` // [{key: 'k', value: 'v'}]

this contains string but not symbol, so it is not an object:

j `(":key" k ":value" v)`// [[':key', 'k', ':value', 'v']]

after a symbol concat string, it is still a symbol, so key can be a symbol concat variable:

j `(:${'foo'} foo)` // [{foo: 'foo'}]
j `(:key-${'foo'} foo)` // [{'key-foo': 'foo'}]
j `(${':'}key 3)` // [{key: 3}]

colon omission

in fact, only the first item's prefix colon is neccessory, so you can skip the colon after that.

j `(:key 1 k2 2)` // [{key: 1, k2: 2}]

but if the key is prefix with colon, the colon will get remove. to add key prefix with colon, write 2 colon:

j `(::key 1 ::k2 2)` // [{':key': 1, ':k2': 2}]

colon only key

if the first item is just a colon, it will get skipped but not cause a empty string key, and the list will be treat as object.

this feature can produce a empty object.

j `(: key 1 k2 2)` // [{key: 1, k2: 2}]
j `(:)` // [{}]
j `(':')` // [[':']]

since the top level is automatically wrap, it can be a object too.

j `:key 1` // {key: 1}
j `:` // {}
j `":"` // [":"]

string key

only the first item need to be a symbol. you can use string as key in the following keys.

j `:k1 1 "k2" 2 "k e y 3" 3` // {k1: 1, k2: 2, 'k e y 3': 3}
j `: "k 1" 1 "k 2" 2` // {'k 1': 1, 'k 2': 2}

A string key's prefix colon will not get removed. only the symbol key's prefix colon is removed.

j `: :k1 1 ":k2" 2` // {k1: 1, ':k2': 2}

nest

both object and array can nest.

j `:k1 (:k2 2 :a (1 2 3)) :k3 null`
// {k1: {k2: 2, a: [1,2,3]}, k3: null}

splice

the @ can splice the following array, object, or variable, similar to at-sign ,@ in lisp quasiquote ` macro.

the space between @ and variable is optional.

array

array can be anything iteratable except Map.

j `1 2 @ (3 4) 5` // [1, 2, 3, 4, 5]
j `1 2 @(3 (4 5) 6) 7` // [1, 2, 3, [4, 5], 6, 7]
j `1 2 @ ${[3, 4]} 5` // [1,2,3,4,5]
j `1 2 @${[3, 4]}  5` // [1,2,3,4,5]

object

object are maps or any other things.

j `:k 1 @(: k2 2 k3 3) k4 4` // {k:1, k2:2, k3:3, k4:4}
j `: k 1 @${{k2:2, k3:3}} k4 4` // {k:1, k2:2, k3:3, k4:4}

undefined behavior

do not splice object out of order, the key will become value.

j `:k 1 :k2 @(:k3 3 :k4 4) 2 k5 5`

type conversion

values are convert to string if:

1. it is not string or symbol (js symbol, the Symbol constructor), and appear as a object key.
2. it is a variable which concat to a symbol.
3. it is a variable which inside a string.

example:

j `${1} x${1} "${1}"` // [1, 'x1', '1']

j `:${1} v1 ${2} v2 ${Symbol.for('v3')} v3`
// {'1': 'v1', '2': 'v2', [Symbol.for('v3')]: 'v3'}

change bracket

you can use square bracket instead of round bracket (if you do not want to press shift-9 and shift-0 all the time)

tsj.bracket = '[ ]'.split(' ')
j `a [b c] [:]` // ['a', ['b', 'c'], {}]

cache

you can enable cache for lexing by tsj.enableCache(). this will save 30% time while parsing a same template string. (test on the npm run bench.) the template strings are same if they have same string parts. for example, these strings are same:

tsj.j `a ${someVar} (1 2 @${someList}) ${e => 'even function'}`
tsj.j `a ${var2} (1 2 @${l2.concat(l3)}) ${null}`

but not this:

tsj.j `a  ${someVar} (1 2 @${someList}) ${e => 'even function'}`
tsj.j `a ${someVar}${var2} (1 2 @${someList}) ${e => 'even function'}`

the toJson method has no cached now.

sxml

tsjson can produce html element with a sxml like syntax in browser, but the s-expression do not use (@ (key value) ...) syntax to define attributes. we use the colon prefix attribute name (the object syntax): (:key value :k2 v2)

see the example if tldr.

tsj.html will return a document fragment from the s-expression, or return the element if there is only one element in sxml.

syntax

mostly like sxml, but string and symbol is mostly identical, and dom node can show up as variable in s-expression.

(element (:attribute-name attribute-value ...)
 child)

element

element can be a symbol, string or variable. if it is symbol, string or a variable contain string, tsj will create corresponding element. if it is a node, it will be used directly.

following examples are identical:

(ol), ("ol"), (${'ol'}), (${'o'}l), or (${document.createElement('ol')})

this work too:

(${document.createElement('a')} (:href '..' :target _blank) parent)

attribute

then, attributes in attribute object will assign to the element. (in browser) if the dom object contain that property, property value will be assigned directly, ot it is stringify and assign.

the whole attribute object can be a variable:

(a ${{href: '..', target: '_blank'}} parent)

following examples are identical:

• (script (:type application/javascript :src index.js))
• (script (:type "application/javascript" :src "index.js"))
• (script ${{type: 'application/javascript', src: 'index.js'}})
• (script (:type ${'application/javascript'} src index.js))

children

children can be text, another sxml or a node variable.

these are identical: (div "text"), (div text), or (div ${document.createTextNode('text')})

multiple children are append sequently, without space join.

macro

user can define macro to transform the dom tree. if node names or attribute names match a macro name, the macro will be execute. for nodes, the macro is apply on the list before convert them to nodes. for attributes, if a macro return anything except undefined, the attribute is set to that value.

define a macro

node macro:

domTool['macro:my-div'] = (list) => {
  list[0] = 'div'
  if (!domTool.isDict(list[1]) || domTool.isNode(list[1])) {
    list.splice(1, 0, {})
  }
  if (!list[1]['class']) list[1]['class'] = []
  list[1]['class'].push('my-div')
}
tsj.html `(my-div (:class another-class) "some text")`
// <div class="another-class my-div">some text</div>

attribute macro is prefix with colon, so there will be 2 colons.

// only work in browser
domTool['macro::range'] = (node, k, v, dict) => {
  const [min, max, step] = v
  const dict = {min, max, step, type: 'number'}
  for (const k in dict) domTool.setAttribute(node, k, dict[k])
}
tsj.html `(input (:range (0 1 0.1) :name num :value 0))`
// <input type="number" min="0" max="1" step="0.1" name="num" value="0">

following are built-in macro:

::id

::id is a attribute macro with additional colon prefix. we use double colon here to distinct from html id attribute. and it will produce nothing in output html.

if a element has ::id attribute, it will be store to the context object's corresponding key. the default context object is tsjson.domTool.context.

const menu = tsj.html `(menu
  (li (button (::id b1 :onclick ${handleClick}) b1))
  (li (button (::id b2 :onclick ${handleClick}) b2))
  (li (button (::id b3 :onclick ${handleClick}) b3)))`

const {b1, b2, b3} = tsj.domTool.context
addFancyAnimation(b1)
addDebounce(b3)

if you want to store in specified object but not use the global object:

const ctx = {}
const menu = tsj.html(ctx) `(menu
  (li (button (::id b1 :onclick ${handleClick}) b1))
  (li (button (::id b2 :onclick ${handleClick}) b2))
  (li (button (::id b3 :onclick ${handleClick}) b3)))`

const {b1, b2, b3} = ctx
addFancyAnimation(b1)
addDebounce(b3)

this could be useful if there are multiple rendering call mix in async context.

::call

this attribute will pass the element to the callback function.

const detail = tsj.html `
  (details
   (summary (::call ${addFancyAnimation}) 'open me')
   "i am open")`

note that the parent and children are not connected when the oncreate is called, and only the preceding attributes are set.

:class

:class is a simple macro convert array to a space joined class string. (div (:class (header outfit))) become <div class="header outfit"></div>.

style

style macro can convert s-expression in style element to css syntax. a list will convert to a style block.

tsj.html `(style
  (body
   background black
   color white)
  ((header , footer)
   margin 1em
   border (inset black 1px))
  ((header p, footer p)
   font-size smaller))`

if selector is a list, it will be join with space. following item are paired as key-value. if the value is a list, it is join with space too.

the output:

<style>
body {
  background: black;
  color: white;
}
header , footer {
  margin: 1em;
  border: inset black 1px;
}
header p, footer p {
  font-size: smaller;
}
</style>

if you need comma, square-bracket or other special character, quote them as string. like: (style ("input[name=number]" display block))

:style

this is a attribute macro for style. convert:

(div (:style
 (background black
  font-size larger
  border (yellow 1px solid))
 foo))

to:

<div style="background: black; font-size: larger; border: yellow 1px solid;">foo</div>

sxml example

with event handler:

tsj.html `(button (:onclick ${() => alert('hello world')})
           "hello world ${n}!")`
// <button>hello world ${n}!</button> with onclick set to the function

a larger document:

tsj.html `
(html
 (head
  (style "body { background: black; color: white; }")
  (script (:src index.js :type application/javascript))
  (title index))
 (body
  (h1 "a index html")
  (p "hey user ${user}")
  (p "go "
     (a (:href "..") "back"))
  (script "alert('welcome to index!')")))`

the playground.html is generated from playground.sxml.js

more macro

you can find more macro in lib/macro-more.js. to use this:

import mmacro from '@gholk/tsjson/lib/macro-more.esm.js'
Object.assign(tsj.domTool, mmacro.domTool)

or (without es module in browser):

<script src="tsjson/lib/macro-more.browser.js"></script>
<script>
  Object.assign(tsj.domTool, macroMore.domTool)
</script>

script

if a children is a function, use its body as code in script.

tsj.html `(script ${() => {
  var user = 'username'
  alert(user)
})`

will output:

<script>
  var user = 'username'
  alert(user)
</script>

macro

define a macro in inline.

tsj.html `
(macro checkbox ${l => {
    const t = tsj.domTool
    l[0] = 'label'
    let d = {}
    if (t.isAttributeDict(l[1])) [d] = l.splice(1, 1)
    d.type = 'checkbox'
    const input = ['input', d]
    const [name] = l.splice(1, 1, input)
    d.name = name
    if (!l[2]) l[2] = name.replace(/[-_]/g, ' ')
}})
(checkbox a-chk-box)`

output:

<label><input type="checkbox" name="a-chk-box">a chk box</label>

note that this will set tsj.domTool['macro:checkbox'] to the function after the sxml is evaluated, so use this macro carefully.

do

the do macro just execute the passed function.

let n = 'before'
tsj.html `(do ${() => n = 'after'})`
console.assert(n == 'after')

the do macro become a document fragment after evaluation. if the functions return a non-undefined value, it will append to the fragment. so the above code will produce a text node after

non-browser environment

To use this feature outside browser or without document object, you need to overwrite the method tsjson.domTool.* .

cheerio

A domTool for cheerio are included in lib/cheerio-dom-tool.js. example:

import tsjson from '@gholk/tsjson'
import {domTool as chdt} from '@gholk/tsjson/lib/cheerio-dom-tool.js'
import cheerio from 'cheerio'

const domTool = Object.assign(tsjson.domTool, chdt)

const $ = cheerio.load(`<!DOCTYPE html>
<html><body></body></html>`)
domTool.setCheerio($)

// without doctype cause quirk mode. or just
// domTool.setCheerioModule(cheerio)

const $div = tsjson.html `(div (:id a-div) "i am a div")`
console.log(domTool.toHtml($div))

$('body').append($div)
console.log($.html())

note that the event handler and non-string attributes will not preserve in cheerio. all attributes are converted to string in cheerio.

tabular

tsj.jtable `
(name key summary)
book1    1 'the book 1'
'book 2' 2 'the book 2'`

// [{name: 'book1', key: 1, summary: 'the book 1'},
//  {name: 'book 2', key: 2, summary: 'the book 2'}]

install

npm install @gholk/tsjson or npm install the tarball, or just unzip the tarball and require the index.esm.js

to import as es-module, import the *.esm.js file if possible. to run in browser without es-module, load the *.browser.js (if exists), and you will have a tsjson global variable. the *.js is es-module, or common-js module if *.esm.js exist.

static js file on gitlab:

• https://gholk.gitlab.io/tsjson/index.browser.js
• https://gholk.gitlab.io/tsjson/index.esm.js
• https://gholk.gitlab.io/tsjson/index.js
• https://gholk.gitlab.io/tsjson/lib/macro-more.browser.js
• https://gholk.gitlab.io/tsjson/lib/macro-more.esm.js
• https://gholk.gitlab.io/tsjson/lib/macro-more.js

cli tool

cli tools are in bin. cli tools accept arguments or read from stdin if no argument. to enable print non-json value (like function or regexp), you can install the optional dependency stringify-object.

(to enable stringify-object in [playground.html] , install and npm run build-stro, or just wget it from online playground.)

if the inputs first non-space char is backquote `, then it will chop the first and the last non-space chars. (so a sxml can be a legal js file which contains only a backquote string.)

~/tsjson $ bin/tsj.js '1 2 a b ("str1" "str2") (:ok false :val null)'
[
  1,
  2,
  'a',
  'b',
  ['str1', 'str2'],
  {ok: false, val: null}
]

if you need the json output, add the --json parameter

~/tsjson $ bin/tsj.js --json '1 2 a b ("str1" "str2") (:ok false :val null)'
[
  1,
  2,
  "a",
  "b",
  [
    "str1",
    "str2"
  ],
  {
    "ok": false,
    "val": null
  }
]

the tsj-html.js requires cheerio installed, and the lib/macro-more.js is import default, so you can use macro and others macro. cheerio is a optional dependency for tsjson.

~/tsjson $ echo '(div (:id div1)) (div (:id div2))' | bin/tsj-html.js
<div id="div1"></div><div id="div2"></div>

~/tsjson $ bin/tsj-html.js '(div (:id div1)) (div (:id div2))'
<div id="div1"></div><div id="div2"></div>

~/tsjson $ bin/tsj-html.js < playground.sxml.js > playground.html

auto detection

backquote

These commands will ignore the first char and the last char if they were backquote.

bracket

It also detect if the input was in square bracket and change the bracket.

css

if tsj-html.js output has only a css element, it will output the raw css without html entity encoding.

~ $ cat style.css.lsp
[css
  [blockquote::before
    content "'> '"]]
~ $ tsj-html.js < style.css.lsp

blockquote::before {
  content: '> ';
}

why

Write large json is tedious. You need comma, colon and quotation marks.

For a string array: ['a', 'b', 'c'] contains 15 characters. For every string in array, it need about 3 addition character, 2 quotes and 1 camma. why not just qw `a b c` ?

with s-expression, you don't need comma, the things matter are only spaces and brackets.

license

this project is fork from the sexp-tokenizer, and re-license as AGPL3+.

todo

• find a better name and publish in global scope.
• unzip table to dict, reuse code
• use and extend painless error to sum type (styp)
• cache template string parse result
• fix macro collide with old attribute problem
• refactor future code
• move toHtml to instance method, and maybe context and macro define in prototype/instance, so user can patch them
• explicit cache
• hex number 0xff 0o12 0b111 7E10
• long unicode
• underscore in number?
• use weakref* in state.js to make derive gc friendly
  • how to distinct side-effect derive and pure node derive
• doc raw css usage in test/html.js