🎩 You're Invited:Meet the Socket team at Black Hat in Las Vegas, August 3-6.RSVP →
Sign In

@toon-format/toon

Package Overview
Dependencies
Maintainers
1
Versions
20
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

@toon-format/toon - npm Package Compare versions

Comparing version
4.0.0
 to
4.1.0
+59
-17
dist/index.d.mts

@@ -52,2 +52,6 @@ //#region src/constants.d.ts

*/
indentSize?: number;
/**
* @deprecated Use `indentSize` instead.
*/
indent?: number;

@@ -67,3 +71,3 @@ /**

}
type ResolvedEncodeOptions = Readonly<Required<Omit<EncodeOptions, "replacer">>> & Pick<EncodeOptions, "replacer">;
type ResolvedEncodeOptions = Readonly<Required<Omit<EncodeOptions, "replacer" | "indent">>> & Pick<EncodeOptions, "replacer">;
interface DecodeOptions {

@@ -74,2 +78,6 @@ /**

*/
indentSize?: number;
/**
* @deprecated Use `indentSize` instead.
*/
indent?: number;

@@ -82,6 +90,4 @@ /**

}
type ResolvedDecodeOptions = Readonly<Required<DecodeOptions>>;
/**
* Options for streaming decode operations.
*/
type ResolvedDecodeOptions = Readonly<Required<Omit<DecodeOptions, "indent">>>;
/** Options for streaming decode operations. */
type DecodeStreamOptions = DecodeOptions;

@@ -124,2 +130,41 @@ type JsonStreamEvent = {

//#endregion
//#region src/encode/raw-string.d.ts
/**
* Pre-formatted string that the encoder emits verbatim at a primitive value
* position, bypassing quoting, escaping, and number/keyword detection.
*
* Returned from a replacer for an object or array value, it is ignored and
* the container is encoded normally.
*/
declare class RawString {
readonly value: string;
constructor(value: string);
}
/**
* Wraps a pre-formatted string for verbatim emission, typically returned from
* an encode `replacer`. Compose with `escapeString` to control quoting yourself.
*
* @param value - The exact text to emit at the value position
* @returns A `RawString` marker honored at primitive value positions
*
* @example
* ```ts
* encode({ name: 'Ada', age: 30 }, {
* replacer: (key, value) => rawString(`"${escapeString(String(value))}"`)
* })
* // name: "Ada"
* // age: "30"
* ```
*/
declare function rawString(value: string): RawString;
//#endregion
//#region src/shared/string-utils.d.ts
/**
* Escapes special characters in a string for encoding.
*
* @remarks
* Control characters outside `\n`, `\r`, `\t`, `\\`, and `"` are emitted as `\uXXXX`.
*/
declare function escapeString(value: string): string;
//#endregion
//#region src/index.d.ts

@@ -147,3 +192,3 @@ /**

*
* encode(data, { indent: 4 })
* encode(data, { indentSize: 4 })
* ```

@@ -202,5 +247,4 @@ */

*
* This is a convenience wrapper around the streaming decoder that builds
* the full value in memory. Useful when you already have lines as an array
* or iterable and want the standard decode behavior.
* Convenience wrapper around the streaming decoder that builds the full
* value in memory.
*

@@ -222,5 +266,4 @@ * @param lines - Iterable of TOON lines (without newlines)

*
* This function yields structured events (startObject, endObject, startArray, endArray,
* key, primitive) that represent the JSON data model without building the full value tree.
* Useful for streaming processing, custom transformations, or memory-efficient parsing.
* Yields structured events (startObject, endObject, startArray, endArray, key,
* primitive) that represent the JSON data model without building the full value tree.
*

@@ -247,6 +290,5 @@ * @param lines - Iterable of TOON lines (without newlines)

*
* This function yields structured events (startObject, endObject, startArray, endArray,
* key, primitive) that represent the JSON data model without building the full value tree.
* Supports both sync and async iterables for maximum flexibility with file streams,
* network responses, or other async sources.
* Yields structured events (startObject, endObject, startArray, endArray, key,
* primitive) that represent the JSON data model without building the full value tree.
* Supports both sync and async iterables.
*

@@ -273,2 +315,2 @@ * @param source - Async or sync iterable of TOON lines (without newlines)

//#endregion
export { DEFAULT_DELIMITER, DELIMITERS, type DecodeOptions, type DecodeStreamOptions, type Delimiter, type DelimiterKey, type EncodeOptions, type EncodeReplacer, type JsonArray, type JsonObject, type JsonPrimitive, type JsonStreamEvent, type JsonValue, type ResolvedDecodeOptions, type ResolvedEncodeOptions, ToonDecodeError, decode, decodeFromLines, decodeStream, decodeStreamSync, encode, encodeLines };
export { DEFAULT_DELIMITER, DELIMITERS, type DecodeOptions, type DecodeStreamOptions, type Delimiter, type DelimiterKey, type EncodeOptions, type EncodeReplacer, type JsonArray, type JsonObject, type JsonPrimitive, type JsonStreamEvent, type JsonValue, type RawString, type ResolvedDecodeOptions, type ResolvedEncodeOptions, ToonDecodeError, decode, decodeFromLines, decodeStream, decodeStreamSync, encode, encodeLines, escapeString, rawString };
+2
-2
package.json
{
"name": "@toon-format/toon",
"type": "module",
"version": "4.0.0",
"version": "4.1.0",
"description": "Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware encoding of JSON for LLM prompts",

@@ -36,3 +36,3 @@ "author": "Johann Schopplich <hello@johannschopplich.com>",

"devDependencies": {
"@toon-format/spec": "^4.0.0"
"@toon-format/spec": "^4.1.0"
},

@@ -39,0 +39,0 @@ "scripts": {

+384
-426
README.md

@@ -1,2 +0,2 @@

![TOON logo with step‑by‑step guide](./.github/og.png)
![Summary card: JSON encodes to TOON for LLM prompts, with token and accuracy benchmarks](./.github/og.png)

@@ -7,3 +7,3 @@ # Token-Oriented Object Notation (TOON)

[![npm version](https://img.shields.io/npm/v/@toon-format/toon.svg?labelColor=1b1b1f&color=fef3c0)](https://www.npmjs.com/package/@toon-format/toon)
[![SPEC v4.0](https://img.shields.io/badge/spec-v4.0-fef3c0?labelColor=1b1b1f)](https://github.com/toon-format/spec)
[![SPEC v4.1](https://img.shields.io/badge/spec-v4.1-fef3c0?labelColor=1b1b1f)](https://github.com/toon-format/spec)
[![npm downloads (total)](https://img.shields.io/npm/dt/@toon-format/toon.svg?labelColor=1b1b1f&color=fef3c0)](https://www.npmjs.com/package/@toon-format/toon)

@@ -14,6 +14,4 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-fef3c0?labelColor=1b1b1f)](./LICENSE)

TOON combines YAML's indentation-based structure for nested objects with a CSV-style tabular layout for uniform arrays. TOON's sweet spot is uniform arrays of objects (multiple fields per row, same structure across items), achieving CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably. For deeply nested or non-uniform data, JSON may be more efficient.
TOON combines YAML's indentation-based structure for nested objects with a CSV-style tabular form for uniform arrays. Its sweet spot is uniform arrays of objects – multiple fields per row, same structure across items – reaching CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably. For deeply nested or non-uniform data, JSON may be more efficient.
The similarity to CSV is intentional: CSV is simple and ubiquitous, and TOON aims to keep that familiarity while remaining a lossless, drop-in representation of JSON for Large Language Models.
Think of it as a translation layer: use JSON programmatically, and encode it as TOON for LLM input.

@@ -31,8 +29,7 @@

- [Installation & Quick Start](#installation--quick-start)
- [Playgrounds](#playgrounds)
- [Editor Support](#editor-support)
- [CLI](#cli)
- [Format Overview](#format-overview)
- [Using TOON with LLMs](#using-toon-with-llms)
- [Ecosystem](#ecosystem)
- [Documentation](#documentation)
- [Media Type & File Extension](#media-type--file-extension)
- [Other Implementations](#other-implementations)

@@ -47,32 +44,38 @@ - [📋 Full Specification](https://github.com/toon-format/spec/blob/main/SPEC.md)

{
"context": {
"task": "Our favorite hikes together",
"location": "Boulder",
"season": "spring_2025"
"location": {
"city": "Berlin",
"country": "DE",
"units": "metric"
},
"friends": ["ana", "luis", "sam"],
"hikes": [
"alerts": [
"frost",
"wind"
],
"forecast": [
{
"id": 1,
"name": "Blue Lake Trail",
"distanceKm": 7.5,
"elevationGain": 320,
"companion": "ana",
"wasSunny": true
"day": "Mon",
"temp": {
"min": -2,
"max": 4
},
"condition": "snow",
"rainChance": 80
},
{
"id": 2,
"name": "Ridge Overlook",
"distanceKm": 9.2,
"elevationGain": 540,
"companion": "luis",
"wasSunny": false
"day": "Tue",
"temp": {
"min": 1,
"max": 7
},
"condition": "cloudy",
"rainChance": 20
},
{
"id": 3,
"name": "Wildflower Loop",
"distanceKm": 5.1,
"elevationGain": 180,
"companion": "sam",
"wasSunny": true
"day": "Wed",
"temp": {
"min": 3,
"max": 11
},
"condition": "sunny",
"rainChance": 5
}

@@ -87,29 +90,28 @@ ]

```yaml
context:
task: Our favorite hikes together
location: Boulder
season: spring_2025
friends:
- ana
- luis
- sam
hikes:
- id: 1
name: Blue Lake Trail
distanceKm: 7.5
elevationGain: 320
companion: ana
wasSunny: true
- id: 2
name: Ridge Overlook
distanceKm: 9.2
elevationGain: 540
companion: luis
wasSunny: false
- id: 3
name: Wildflower Loop
distanceKm: 5.1
elevationGain: 180
companion: sam
wasSunny: true
location:
city: Berlin
country: DE
units: metric
alerts:
- frost
- wind
forecast:
- day: Mon
temp:
min: -2
max: 4
condition: snow
rainChance: 80
- day: Tue
temp:
min: 1
max: 7
condition: cloudy
rainChance: 20
- day: Wed
temp:
min: 3
max: 11
condition: sunny
rainChance: 5
```

@@ -122,43 +124,87 @@

```yaml
context:
task: Our favorite hikes together
location: Boulder
season: spring_2025
friends[3]: ana,luis,sam
hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
1,Blue Lake Trail,7.5,320,ana,true
2,Ridge Overlook,9.2,540,luis,false
3,Wildflower Loop,5.1,180,sam,true
location:
city: Berlin
country: DE
units: metric
alerts[2]: frost,wind
forecast[3]{day,temp{min,max},condition,rainChance}:
Mon,-2,4,snow,80
Tue,1,7,cloudy,20
Wed,3,11,sunny,5
```
Three things are happening at once. Two are **forms** – one rendering of a value, picked automatically from the data's shape – and the third is a header feature:
- `alerts[2]: frost,wind` is **inline form**: a primitive array on its header line.
- `forecast[3]{day,…}:` is **tabular form**: the field list is declared once in the header, then one row per element.
- `temp{min,max}` inside that header is a **nested field group**: the uniform nested `temp` objects fold into the header while rows stay flat.
The third form is **keyed tabular**, for objects whose values are uniform objects – config maps, feature flags, records by ID. The colon after the length (`[2:]`) marks it, and each row carries its own key:
<table>
<tr><th>JSON</th><th>TOON</th></tr>
<tr><td>
```json
{
"environments": {
"production": { "region": "eu-central-1", "replicas": 6, "debug": false },
"staging": { "region": "eu-central-1", "replicas": 2, "debug": true }
}
}
```
</td><td>
```toon
environments[2:]{region,replicas,debug}:
production: eu-central-1,6,false
staging: eu-central-1,2,true
```
</td></tr>
</table>
Anything that fits none of these – mixed types, non-uniform objects – falls back to the fourth form, **list form**: one `- ` item per element, or a bare `-` for an empty object. Those four cover the shapes; the [Format Overview](https://toonformat.dev/guide/format-overview) covers the rest.
> [!TIP]
> Try it on your own data – no install required:
>
> ```bash
> cat data.json | npx @toon-format/cli --stats
> ```
>
> It prints the TOON alongside what the conversion saved:
>
> ```
> ℹ Token estimates: ~117 (JSON) → ~66 (TOON)
> ✔ Saved ~51 tokens (-43.6%)
> ```
## Key Features
- 📊 **Token-Efficient & Accurate:** TOON reaches 76.4% accuracy (vs JSON's 75.0%) while using ~40% fewer tokens in mixed-structure benchmarks across 4 models.
- 📊 **Token-Efficient & Accurate:** Matches JSON's retrieval accuracy while using 42.6% fewer tokens – see [Benchmarks](#benchmarks).
- 🔁 **JSON Data Model:** Encodes the same objects, arrays, and primitives as JSON with deterministic, lossless round-trips.
- 🛤️ **LLM-Friendly Guardrails:** Explicit [N] lengths and {fields} headers give models a clear schema to follow, improving parsing reliability.
- 🛤️ **LLM-Friendly Guardrails:** Explicit `[N]` lengths and `{fields}` field lists give models a clear schema to follow, improving parsing reliability.
- 📐 **Minimal Syntax:** Uses indentation instead of braces and minimizes quoting, giving YAML-like readability with CSV-style compactness.
- 🧺 **Tabular Arrays:** Uniform arrays of objects collapse into tables that declare fields once and stream row values line by line.
- 🧺 **Tabular Forms:** Uniform arrays of objects – and objects of uniform objects – collapse into tables that declare the field list once and stream row values line by line.
- 🌐 **Multi-Language Ecosystem:** Spec-driven implementations in TypeScript, Python, Go, Rust, .NET, and other languages.
## Media Type & File Extension
By convention, TOON files use the `.toon` extension and the provisional media type `text/toon` for HTTP and content-type–aware contexts. TOON documents are always UTF-8 encoded; the `charset=utf-8` parameter may be specified but defaults to UTF-8 when omitted. See [SPEC.md §17](https://github.com/toon-format/spec/blob/main/SPEC.md#17-iana-considerations) for normative details.
## When Not to Use TOON
TOON excels with uniform arrays of objects, but there are cases where other formats are better:
TOON excels with uniform arrays of objects. Reach for something else when:
- **Deeply nested or non-uniform structures** (tabular eligibility ≈ 0%): JSON-compact often uses fewer tokens. Example: complex configuration objects with many nested levels.
- **Semi-uniform arrays** (~40–60% tabular eligibility): Token savings diminish. Prefer JSON if your pipelines already rely on it.
- **Pure tabular data**: CSV is smaller than TOON for flat tables. TOON adds minimal overhead (~5–10%) to provide structure (array length declarations, field headers, delimiter scoping) that improves LLM reliability.
- **Latency-critical applications**: If end-to-end response time is your top priority, benchmark on your exact setup. Some deployments (especially local/quantized models like Ollama) may process compact JSON faster despite TOON's lower token count. Measure TTFT, tokens/sec, and total time for both formats and use whichever is faster.
- **Structures are deeply nested or non-uniform** (tabular eligibility ≈ 0%) – compact JSON often wins outright.
- **Arrays are semi-uniform** (~40–60% eligibility) – savings shrink; stay on JSON if your pipeline already speaks it.
- **Data is purely tabular** – CSV is smaller. TOON's ~5–10% overhead buys declared lengths, field lists, and delimiter scoping, which is a reliability trade, not a size one.
- **Latency dominates** – some deployments (notably local or quantized models) process compact JSON faster despite the higher token count. Measure TTFT and total time on your own setup.
See [benchmarks](#benchmarks) for concrete comparisons across different data structures.
[Benchmarks](#benchmarks) below quantify the token and accuracy trade-offs; latency is the one you have to measure yourself.
## Benchmarks
Benchmarks are organized into two tracks to ensure fair comparisons:
Two tracks, so every comparison is like-for-like:
- **Mixed-Structure Track**: Datasets with nested or semi-uniform structures (TOON vs JSON, YAML, XML). CSV excluded as it cannot properly represent these structures.
- **Flat-Only Track**: Datasets with flat tabular structures where CSV is applicable (CSV vs TOON vs JSON, YAML, XML).
- **Mixed-Structure Track**: Nested and semi-uniform datasets (TOON vs JSON, YAML, XML). CSV is excluded – it cannot represent these structures without lossy flattening.
- **Flat-Only Track**: Flat, fully tabular-eligible datasets, where CSV is a fair competitor.

@@ -169,3 +215,3 @@ ### Retrieval Accuracy

Benchmarks test LLM comprehension across different input formats using 209 data retrieval questions on 4 models.
Benchmarks test LLM comprehension across different input formats using 244 data retrieval questions on 4 models.

@@ -184,8 +230,10 @@ <details>

| Semi-uniform event logs | 75 | semi-uniform | ✗ | 50% |
| Deeply nested configuration | 11 | deep | ✗ | 0% |
| Deeply nested configuration | 1 | deep | ✗ | 0% |
| Valid complete dataset (control) | 20 | uniform | ✓ | 100% |
| Array truncated: 3 rows removed from end | 17 | uniform | ✓ | 100% |
| Extra rows added beyond declared length | 23 | uniform | ✓ | 100% |
| Array truncated: 3 rows removed from end | 20 | uniform | ✓ | 100% |
| Extra rows added beyond declared length | 20 | uniform | ✓ | 100% |
| Inconsistent field count (missing salary in row 10) | 20 | uniform | ✓ | 100% |
| Missing required fields (no email in multiple rows) | 20 | uniform | ✓ | 100% |
| Feature flags keyed by name | 40 | uniform | ✗ | 100% |
| Contacts with nested address and plan groups | 50 | nested | ✗ | 100% |

@@ -200,3 +248,3 @@ **Structure classes:**

**Eligibility:** Percentage of arrays that qualify for TOON's tabular format (uniform objects with primitive values)
**Eligibility:** Percentage of arrays that qualify for TOON's tabular form (uniform objects with primitive values)

@@ -210,7 +258,7 @@ </details>

```
TOON ████████████████████ 27.7 acc%/1K tok │ 76.4% acc │ 2,759 tokens
JSON compact █████████████████░░░ 23.7 acc%/1K tok │ 73.7% acc │ 3,104 tokens
YAML ██████████████░░░░░░ 19.9 acc%/1K tok │ 74.5% acc │ 3,749 tokens
JSON ████████████░░░░░░░░ 16.4 acc%/1K tok │ 75.0% acc │ 4,587 tokens
XML ██████████░░░░░░░░░░ 13.8 acc%/1K tok │ 72.1% acc │ 5,221 tokens
TOON ████████████████████ 29.2 acc%/1K tok │ 72.2% ±2.8 acc │ 2,474 tokens
JSON compact ████████████████░░░░ 23.8 acc%/1K tok │ 69.0% ±2.9 acc │ 2,892 tokens
YAML ██████████████░░░░░░ 20.1 acc%/1K tok │ 70.1% ±2.9 acc │ 3,487 tokens
JSON ███████████░░░░░░░░░ 16.6 acc%/1K tok │ 71.4% ±2.8 acc │ 4,308 tokens
XML ██████████░░░░░░░░░░ 14.4 acc%/1K tok │ 70.7% ±2.9 acc │ 4,909 tokens
```

@@ -221,59 +269,73 @@

> [!TIP]
> TOON achieves **76.4%** accuracy (vs JSON's 75.0%) while using **39.9% fewer tokens**.
> TOON achieves **72.2%** accuracy (vs JSON's 71.4%) while using **42.6% fewer tokens**.
**Note on CSV:** Excluded from ranking as it only supports 109 of 209 questions (flat tabular data only). While CSV is highly token-efficient for simple tabular data, it cannot represent nested structures that other formats handle.
> [!NOTE]
> CSV is excluded from the ranking as it only supports 109 of 244 questions (flat tabular data only). While CSV is highly token-efficient for simple tabular data, it cannot represent nested structures that other formats handle.
#### Accuracy on Flat Datasets
Every format answers the same 109 flat-dataset questions per model, so CSV can be compared on equal footing here.
| Format | Accuracy | Correct/Total | Avg Tokens |
| ------ | -------- | ------------- | ---------- |
| `toon` | 63.1% ±4.5 | 275/436 | 1,994 |
| `csv` | 62.2% ±4.5 | 271/436 | 1,851 |
| `json-pretty` | 60.3% ±4.6 | 263/436 | 3,950 |
| `xml` | 60.1% ±4.6 | 262/436 | 4,516 |
| `yaml` | 59.9% ±4.6 | 261/436 | 3,270 |
| `json-compact` | 58.0% ±4.6 | 253/436 | 2,718 |
#### Per-Model Accuracy
Accuracy across 4 LLMs on 209 data retrieval questions:
Accuracy across 4 LLMs on 244 data retrieval questions:
```
claude-haiku-4-5-20251001
→ TOON ████████████░░░░░░░░ 59.8% (125/209)
JSON ███████████░░░░░░░░░ 57.4% (120/209)
YAML ███████████░░░░░░░░░ 56.0% (117/209)
XML ███████████░░░░░░░░░ 55.5% (116/209)
JSON compact ███████████░░░░░░░░░ 55.0% (115/209)
CSV ██████████░░░░░░░░░░ 50.5% (55/109)
→ TOON █████████████░░░░░░░ 65.6% ±5.9 (160/244)
JSON █████████████░░░░░░░ 63.5% ±6.0 (155/244)
XML ████████████░░░░░░░░ 62.3% ±6.0 (152/244)
YAML ████████████░░░░░░░░ 62.3% ±6.0 (152/244)
JSON compact ████████████░░░░░░░░ 61.9% ±6.0 (151/244)
CSV ██████████░░░░░░░░░░ 49.5% ±9.2 (54/109)
gemini-3-flash-preview
XML ████████████████████ 98.1% (205/209)
JSON ███████████████████░ 97.1% (203/209)
YAML ███████████████████░ 97.1% (203/209)
→ TOON ███████████████████░ 96.7% (202/209)
JSON compact ███████████████████░ 96.7% (202/209)
CSV ███████████████████░ 96.3% (105/109)
gemini-3.6-flash
→ TOON ██████████████░░░░░░ 69.3% ±5.8 (169/244)
JSON ██████████████░░░░░░ 68.4% ±5.8 (167/244)
YAML ██████████████░░░░░░ 67.6% ±5.8 (165/244)
XML █████████████░░░░░░░ 65.2% ±5.9 (159/244)
JSON compact █████████████░░░░░░░ 63.5% ±6.0 (155/244)
CSV ████████████░░░░░░░░ 57.8% ±9.1 (63/109)
gpt-5-nano
→ TOON ██████████████████░░ 90.9% (190/209)
JSON compact ██████████████████░░ 90.9% (190/209)
JSON ██████████████████░░ 89.0% (186/209)
CSV ██████████████████░░ 89.0% (97/109)
YAML █████████████████░░░ 87.1% (182/209)
XML ████████████████░░░░ 80.9% (169/209)
gpt-5.4-nano
XML ████████████░░░░░░░░ 59.4% ±6.1 (145/244)
JSON ███████████░░░░░░░░░ 57.4% ±6.2 (140/244)
→ TOON ███████████░░░░░░░░░ 57.0% ±6.2 (139/244)
JSON compact ███████████░░░░░░░░░ 54.9% ±6.2 (134/244)
YAML ███████████░░░░░░░░░ 54.5% ±6.2 (133/244)
CSV █████████░░░░░░░░░░░ 46.8% ±9.2 (51/109)
grok-4-1-fast-non-reasoning
→ TOON ████████████░░░░░░░░ 58.4% (122/209)
YAML ████████████░░░░░░░░ 57.9% (121/209)
JSON ███████████░░░░░░░░░ 56.5% (118/209)
XML ███████████░░░░░░░░░ 54.1% (113/209)
JSON compact ██████████░░░░░░░░░░ 52.2% (109/209)
CSV ██████████░░░░░░░░░░ 51.4% (56/109)
grok-4.5
→ TOON ███████████████████░ 97.1% ±2.2 (237/244)
JSON ███████████████████░ 96.3% ±2.5 (235/244)
XML ███████████████████░ 95.9% ±2.6 (234/244)
YAML ███████████████████░ 95.9% ±2.6 (234/244)
JSON compact ███████████████████░ 95.5% ±2.7 (233/244)
CSV ███████████████████░ 94.5% ±4.5 (103/109)
```
> [!TIP]
> TOON achieves **76.4% accuracy** (vs JSON's 75.0%) while using **39.9% fewer tokens** on these datasets.
> [!NOTE]
> Accuracy figures include Wilson 95% confidence intervals (±); when two formats' intervals overlap, the difference between them is not statistically meaningful. CSV answers only the 109 flat-dataset questions, so its per-model cells cover a smaller, easier population than the other formats.
<details>
<summary><strong>Performance by dataset, model, and question type</strong></summary>
<summary><strong>Performance by dataset and question type</strong></summary>
#### Performance by Question Type
| Question Type | TOON | JSON | YAML | JSON compact | XML | CSV |
| Question Type | TOON | JSON | XML | YAML | JSON compact | CSV |
| ------------- | ---- | ---- | ---- | ---- | ---- | ---- |
| Field Retrieval | 99.6% | 99.3% | 98.5% | 98.5% | 98.9% | 100.0% |
| Aggregation | 61.9% | 61.9% | 59.9% | 58.3% | 54.4% | 50.9% |
| Filtering | 56.8% | 53.1% | 56.3% | 55.2% | 51.6% | 50.9% |
| Structure Awareness | 89.0% | 87.0% | 84.0% | 84.0% | 81.0% | 85.9% |
| Structural Validation | 70.0% | 60.0% | 60.0% | 55.0% | 85.0% | 80.0% |
| Field Retrieval | 97.8% | 99.2% | 99.2% | 99.7% | 98.9% | 100.0% |
| Aggregation | 48.4% | 48.4% | 46.0% | 46.0% | 45.2% | 32.8% |
| Filtering | 38.0% | 41.1% | 37.5% | 40.1% | 38.0% | 33.3% |
| Structure Awareness | 90.3% | 84.0% | 84.0% | 79.2% | 78.5% | 82.8% |
| Structural Validation | 100.0% | 50.0% | 80.0% | 50.0% | 45.0% | 80.0% |

@@ -286,8 +348,8 @@ #### Performance by Dataset

| ------ | -------- | ------ | ------------- |
| `csv` | 73.2% | 2,334 | 120/164 |
| `toon` | 73.2% | 2,498 | 120/164 |
| `json-compact` | 73.8% | 3,924 | 121/164 |
| `yaml` | 73.8% | 4,959 | 121/164 |
| `json-pretty` | 73.8% | 6,331 | 121/164 |
| `xml` | 74.4% | 7,296 | 122/164 |
| `csv` | 64.6% | 2,336 | 106/164 |
| `toon` | 62.8% | 2,537 | 103/164 |
| `json-compact` | 62.2% | 3,919 | 102/164 |
| `yaml` | 64.0% | 4,982 | 105/164 |
| `json-pretty` | 62.2% | 6,326 | 102/164 |
| `xml` | 61.0% | 7,286 | 100/164 |

@@ -298,7 +360,7 @@ ##### E-commerce orders with nested structures

| ------ | -------- | ------ | ------------- |
| `toon` | 82.3% | 7,458 | 135/164 |
| `json-compact` | 78.7% | 7,110 | 129/164 |
| `yaml` | 79.9% | 8,755 | 131/164 |
| `json-pretty` | 79.3% | 11,234 | 130/164 |
| `xml` | 77.4% | 12,649 | 127/164 |
| `json-compact` | 70.7% | 6,875 | 116/164 |
| `toon` | 71.3% | 7,344 | 117/164 |
| `yaml` | 72.0% | 8,456 | 118/164 |
| `json-pretty` | 71.3% | 10,842 | 117/164 |
| `xml` | 74.4% | 12,180 | 122/164 |

@@ -309,8 +371,8 @@ ##### Time-series analytics data

| ------ | -------- | ------ | ------------- |
| `csv` | 75.0% | 1,411 | 90/120 |
| `toon` | 78.3% | 1,553 | 94/120 |
| `json-compact` | 74.2% | 2,354 | 89/120 |
| `yaml` | 75.8% | 2,954 | 91/120 |
| `json-pretty` | 75.0% | 3,681 | 90/120 |
| `xml` | 72.5% | 4,389 | 87/120 |
| `csv` | 64.2% | 1,408 | 77/120 |
| `toon` | 63.3% | 1,595 | 76/120 |
| `json-compact` | 59.2% | 2,351 | 71/120 |
| `yaml` | 62.5% | 2,951 | 75/120 |
| `json-pretty` | 65.0% | 3,678 | 78/120 |
| `xml` | 62.5% | 4,386 | 75/120 |

@@ -321,8 +383,8 @@ ##### Top 100 GitHub repositories

| ------ | -------- | ------ | ------------- |
| `csv` | 65.9% | 8,527 | 87/132 |
| `toon` | 66.7% | 8,779 | 88/132 |
| `yaml` | 65.2% | 13,141 | 86/132 |
| `json-compact` | 59.8% | 11,464 | 79/132 |
| `json-pretty` | 63.6% | 15,157 | 84/132 |
| `xml` | 56.1% | 17,105 | 74/132 |
| `toon` | 57.6% | 9,017 | 76/132 |
| `csv` | 54.5% | 8,726 | 72/132 |
| `json-compact` | 53.8% | 11,650 | 71/132 |
| `yaml` | 53.8% | 13,350 | 71/132 |
| `json-pretty` | 55.3% | 15,350 | 73/132 |
| `xml` | 53.8% | 17,304 | 71/132 |

@@ -333,7 +395,7 @@ ##### Semi-uniform event logs

| ------ | -------- | ------ | ------------- |
| `json-compact` | 68.3% | 4,839 | 82/120 |
| `toon` | 65.0% | 5,819 | 78/120 |
| `json-pretty` | 69.2% | 6,817 | 83/120 |
| `yaml` | 61.7% | 5,847 | 74/120 |
| `xml` | 58.3% | 7,729 | 70/120 |
| `json-compact` | 56.7% | 4,793 | 68/120 |
| `toon` | 60.8% | 5,814 | 73/120 |
| `json-pretty` | 60.0% | 6,759 | 72/120 |
| `yaml` | 55.0% | 5,798 | 66/120 |
| `xml` | 50.8% | 7,668 | 61/120 |

@@ -344,7 +406,7 @@ ##### Deeply nested configuration

| ------ | -------- | ------ | ------------- |
| `json-compact` | 90.5% | 568 | 105/116 |
| `toon` | 94.8% | 655 | 110/116 |
| `json-compact` | 91.4% | 562 | 106/116 |
| `yaml` | 93.1% | 675 | 108/116 |
| `json-pretty` | 92.2% | 924 | 107/116 |
| `xml` | 91.4% | 1,013 | 106/116 |
| `toon` | 91.4% | 669 | 106/116 |
| `json-pretty` | 94.8% | 918 | 110/116 |
| `xml` | 94.0% | 1,007 | 109/116 |

@@ -355,8 +417,8 @@ ##### Valid complete dataset (control)

| ------ | -------- | ------ | ------------- |
| `toon` | 100.0% | 535 | 4/4 |
| `json-compact` | 100.0% | 787 | 4/4 |
| `yaml` | 100.0% | 992 | 4/4 |
| `json-pretty` | 100.0% | 1,274 | 4/4 |
| `xml` | 25.0% | 1,462 | 1/4 |
| `csv` | 0.0% | 483 | 0/4 |
| `toon` | 100.0% | 566 | 4/4 |
| `json-compact` | 100.0% | 772 | 4/4 |
| `yaml` | 100.0% | 984 | 4/4 |
| `json-pretty` | 100.0% | 1,259 | 4/4 |
| `xml` | 0.0% | 1,441 | 0/4 |
| `csv` | 0.0% | 473 | 0/4 |

@@ -367,8 +429,8 @@ ##### Array truncated: 3 rows removed from end

| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 413 | 4/4 |
| `xml` | 100.0% | 1,243 | 4/4 |
| `toon` | 0.0% | 462 | 0/4 |
| `json-pretty` | 0.0% | 1,085 | 0/4 |
| `yaml` | 0.0% | 843 | 0/4 |
| `json-compact` | 0.0% | 670 | 0/4 |
| `csv` | 100.0% | 408 | 4/4 |
| `toon` | 100.0% | 498 | 4/4 |
| `xml` | 100.0% | 1,229 | 4/4 |
| `json-pretty` | 0.0% | 1,075 | 0/4 |
| `yaml` | 0.0% | 841 | 0/4 |
| `json-compact` | 0.0% | 660 | 0/4 |

@@ -379,8 +441,8 @@ ##### Extra rows added beyond declared length

| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 550 | 4/4 |
| `toon` | 75.0% | 605 | 3/4 |
| `json-compact` | 75.0% | 901 | 3/4 |
| `xml` | 100.0% | 1,678 | 4/4 |
| `yaml` | 75.0% | 1,138 | 3/4 |
| `json-pretty` | 50.0% | 1,460 | 2/4 |
| `csv` | 100.0% | 547 | 4/4 |
| `toon` | 100.0% | 644 | 4/4 |
| `xml` | 100.0% | 1,663 | 4/4 |
| `json-pretty` | 0.0% | 1,452 | 0/4 |
| `yaml` | 0.0% | 1,135 | 0/4 |
| `json-compact` | 0.0% | 893 | 0/4 |

@@ -391,8 +453,8 @@ ##### Inconsistent field count (missing salary in row 10)

| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 480 | 4/4 |
| `json-compact` | 100.0% | 782 | 4/4 |
| `yaml` | 100.0% | 985 | 4/4 |
| `toon` | 100.0% | 1,008 | 4/4 |
| `json-pretty` | 100.0% | 1,266 | 4/4 |
| `xml` | 100.0% | 1,453 | 4/4 |
| `csv` | 100.0% | 470 | 4/4 |
| `toon` | 100.0% | 563 | 4/4 |
| `json-compact` | 75.0% | 767 | 3/4 |
| `xml` | 100.0% | 1,432 | 4/4 |
| `yaml` | 75.0% | 977 | 3/4 |
| `json-pretty` | 75.0% | 1,251 | 3/4 |

@@ -403,123 +465,42 @@ ##### Missing required fields (no email in multiple rows)

| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 340 | 4/4 |
| `xml` | 100.0% | 1,409 | 4/4 |
| `toon` | 75.0% | 974 | 3/4 |
| `json-pretty` | 50.0% | 1,225 | 2/4 |
| `yaml` | 25.0% | 951 | 1/4 |
| `json-compact` | 0.0% | 750 | 0/4 |
| `csv` | 100.0% | 442 | 4/4 |
| `toon` | 100.0% | 535 | 4/4 |
| `xml` | 100.0% | 1,386 | 4/4 |
| `yaml` | 75.0% | 941 | 3/4 |
| `json-pretty` | 75.0% | 1,207 | 3/4 |
| `json-compact` | 50.0% | 732 | 2/4 |
#### Performance by Model
##### Feature flags keyed by name
##### claude-haiku-4-5-20251001
| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `toon` | 97.1% | 931 | 66/68 |
| `json-compact` | 94.1% | 1,264 | 64/68 |
| `yaml` | 92.6% | 1,443 | 63/68 |
| `json-pretty` | 95.6% | 1,873 | 65/68 |
| `xml` | 95.6% | 2,306 | 65/68 |
| Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- |
| `toon` | 59.8% | 125/209 |
| `json-pretty` | 57.4% | 120/209 |
| `yaml` | 56.0% | 117/209 |
| `xml` | 55.5% | 116/209 |
| `json-compact` | 55.0% | 115/209 |
| `csv` | 50.5% | 55/109 |
##### Contacts with nested address and plan groups
##### gemini-3-flash-preview
| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `toon` | 94.4% | 1,444 | 68/72 |
| `json-compact` | 91.7% | 2,357 | 66/72 |
| `yaml` | 94.4% | 2,797 | 68/72 |
| `json-pretty` | 97.2% | 4,014 | 70/72 |
| `xml` | 98.6% | 4,534 | 71/72 |
| Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- |
| `xml` | 98.1% | 205/209 |
| `json-pretty` | 97.1% | 203/209 |
| `yaml` | 97.1% | 203/209 |
| `toon` | 96.7% | 202/209 |
| `json-compact` | 96.7% | 202/209 |
| `csv` | 96.3% | 105/109 |
##### gpt-5-nano
| Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- |
| `toon` | 90.9% | 190/209 |
| `json-compact` | 90.9% | 190/209 |
| `json-pretty` | 89.0% | 186/209 |
| `csv` | 89.0% | 97/109 |
| `yaml` | 87.1% | 182/209 |
| `xml` | 80.9% | 169/209 |
##### grok-4-1-fast-non-reasoning
| Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- |
| `toon` | 58.4% | 122/209 |
| `yaml` | 57.9% | 121/209 |
| `json-pretty` | 56.5% | 118/209 |
| `xml` | 54.1% | 113/209 |
| `json-compact` | 52.2% | 109/209 |
| `csv` | 51.4% | 56/109 |
</details>
#### What's Being Measured
#### Run Configuration
This benchmark tests **LLM comprehension and data retrieval accuracy** across different input formats. Each LLM receives formatted data and must answer questions about it. This does **not** test the model's ability to generate TOON output – only to read and understand it.
- **Models tested**: `claude-haiku-4-5-20251001`, `gemini-3.6-flash`, `gpt-5.4-nano`, `grok-4.5`
- **Formats compared**: TOON, JSON, XML, YAML, JSON compact, CSV
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding (GPT-5 tokenizer). Other providers tokenize differently, so absolute counts are tokenizer-specific; relative differences between formats hold directionally.
- **Reasoning**: Disabled via the AI SDK's universal `reasoning: 'none'` (Gemini 3 floors at minimal thinking, `grok-4.5` at `low`)
- **Temperature**: Not set (models use their defaults)
- **Total evaluations**: 244 questions × 6 formats × 4 models = 5,856 LLM calls
#### Datasets Tested
What the datasets contain, how the questions are generated, and how answers are validated is documented in [the benchmark README](https://github.com/toon-format/toon/tree/main/benchmarks#retrieval-accuracy-benchmark).
Eleven datasets designed to test different structural patterns and validation capabilities:
**Primary datasets:**
1. **Tabular** (100 employee records): Uniform objects with identical fields – optimal for TOON's tabular format.
2. **Nested** (50 e-commerce orders): Complex structures with nested customer objects and item arrays.
3. **Analytics** (60 days of metrics): Time-series data with dates and numeric values.
4. **GitHub** (100 repositories): Real-world data from top GitHub repos by stars.
5. **Event Logs** (75 logs): Semi-uniform data with ~50% flat logs and ~50% with nested error objects.
6. **Nested Config** (1 configuration): Deeply nested configuration with minimal tabular eligibility.
**Structural validation datasets:**
7. **Control**: Valid complete dataset (baseline for validation)
8. **Truncated**: Array with 3 rows removed from end (tests `[N]` length detection)
9. **Extra rows**: Array with 3 additional rows beyond declared length
10. **Width mismatch**: Inconsistent field count (missing salary in row 10)
11. **Missing fields**: Systematic field omissions (no email in multiple rows)
#### Question Types
209 questions are generated dynamically across five categories:
- **Field retrieval (33%)**: Direct value lookups or values that can be read straight off a record (including booleans and simple counts such as array lengths)
- Example: "What is Alice's salary?" → `75000`
- Example: "How many items are in order ORD-0042?" → `3`
- Example: "What is the customer name for order ORD-0042?" → `John Doe`
- **Aggregation (30%)**: Dataset-level totals and averages plus single-condition filters (counts, sums, min/max comparisons)
- Example: "How many employees work in Engineering?" → `17`
- Example: "What is the total revenue across all orders?" → `45123.50`
- Example: "How many employees have salary > 80000?" → `23`
- **Filtering (23%)**: Multi-condition queries requiring compound logic (AND constraints across fields)
- Example: "How many employees in Sales have salary > 80000?" → `5`
- Example: "How many active employees have more than 10 years of experience?" → `8`
- **Structure awareness (12%)**: Tests format-native structural affordances (TOON's `[N]` count and `{fields}`, CSV's header row)
- Example: "How many employees are in the dataset?" → `100`
- Example: "List the field names for employees" → `id, name, email, department, salary, yearsExperience, active`
- Example: "What is the department of the last employee?" → `Sales`
- **Structural validation (2%)**: Tests ability to detect incomplete, truncated, or corrupted data using structural metadata
- Example: "Is this data complete and valid?" → `YES` (control dataset) or `NO` (corrupted datasets)
- Tests TOON's `[N]` length validation and `{fields}` consistency checking
- Demonstrates CSV's lack of structural validation capabilities
#### Evaluation Process
1. **Format conversion**: Each dataset is converted to all 6 formats (TOON, JSON, YAML, JSON compact, XML, CSV).
2. **Query LLM**: Each model receives formatted data + question in a prompt and extracts the answer.
3. **Validate deterministically**: Answers are validated using type-aware comparison (e.g., `50000` = `$50,000`, `Engineering` = `engineering`, `2025-01-01` = `January 1, 2025`) without requiring an LLM judge.
#### Models & Configuration
- **Models tested**: `claude-haiku-4-5-20251001`, `gemini-3-flash-preview`, `gpt-5-nano`, `grok-4-1-fast-non-reasoning`
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding (GPT-5 tokenizer)
- **Temperature**: Not set (models use their defaults)
- **Total evaluations**: 209 questions × 6 formats × 4 models = 5,016 LLM calls
<!-- /automd -->

@@ -542,7 +523,7 @@

│
TOON █████████████░░░░░░░ 73,126 tokens
├─ vs JSON (−33.3%) 109,599 tokens
├─ vs JSON compact (+5.3%) 69,459 tokens
├─ vs YAML (−14.4%) 85,415 tokens
└─ vs XML (−40.7%) 123,344 tokens
TOON █████████████░░░░░░░ 72,832 tokens
├─ vs JSON (−32.9%) 108,611 tokens
├─ vs JSON compact (+5.6%) 68,944 tokens
├─ vs YAML (−14.0%) 84,701 tokens
└─ vs XML (−40.4%) 122,119 tokens

@@ -559,14 +540,30 @@ 🧾 Semi-uniform event logs ┊ Tabular: 50%

│
TOON ██████████████░░░░░░ 620 tokens
├─ vs JSON (−31.9%) 911 tokens
├─ vs JSON compact (+11.1%) 558 tokens
├─ vs YAML (−6.3%) 662 tokens
└─ vs XML (−38.2%) 1,003 tokens
TOON █████████████░░░░░░░ 589 tokens
├─ vs JSON (−34.9%) 905 tokens
├─ vs JSON compact (+6.7%) 552 tokens
├─ vs YAML (−11.0%) 662 tokens
└─ vs XML (−40.9%) 997 tokens
📊 Feature flags keyed by name ┊ Tabular: 100%
│
TOON █████████░░░░░░░░░░░ 10,503 tokens
├─ vs JSON (−54.6%) 23,141 tokens
├─ vs JSON compact (−32.8%) 15,635 tokens
├─ vs YAML (−41.3%) 17,905 tokens
└─ vs XML (−63.3%) 28,655 tokens
📊 Contacts with nested address and plan groups ┊ Tabular: 100%
│
TOON ███████░░░░░░░░░░░░░ 26,726 tokens
├─ vs JSON (−66.5%) 79,779 tokens
├─ vs JSON compact (−42.9%) 46,791 tokens
├─ vs YAML (−51.8%) 55,475 tokens
└─ vs XML (−70.4%) 90,306 tokens
──────────────────────────────────── Total ────────────────────────────────────
TOON ████████████████░░░░ 227,830 tokens
├─ vs JSON (−21.9%) 291,711 tokens
├─ vs JSON compact (+14.7%) 198,546 tokens
├─ vs YAML (−5.7%) 241,474 tokens
└─ vs XML (−31.0%) 330,206 tokens
TOON █████████████░░░░░░░ 264,734 tokens
├─ vs JSON (−32.7%) 393,637 tokens
├─ vs JSON compact (+1.6%) 260,451 tokens
├─ vs YAML (−15.7%) 314,140 tokens
└─ vs XML (−40.9%) 447,936 tokens
```

@@ -576,3 +573,3 @@

Datasets with flat tabular structures where CSV is applicable.
Datasets with flat, fully tabular-eligible data where CSV is applicable.

@@ -582,8 +579,8 @@ ```

│
CSV ███████████████████░ 47,102 tokens
TOON ████████████████████ 49,919 tokens (+6.0% vs CSV)
├─ vs JSON (−60.7%) 127,063 tokens
├─ vs JSON compact (−36.9%) 79,059 tokens
├─ vs YAML (−50.1%) 100,011 tokens
└─ vs XML (−65.9%) 146,579 tokens
CSV ███████████████████░ 47,153 tokens
TOON ████████████████████ 49,978 tokens (+6.0% vs CSV)
├─ vs JSON (−60.7%) 127,061 tokens
├─ vs JSON compact (−36.8%) 79,057 tokens
├─ vs YAML (−50.0%) 100,054 tokens
└─ vs XML (−65.9%) 146,605 tokens

@@ -601,18 +598,20 @@ 📈 Time-series analytics data ┊ Tabular: 100%

│
CSV ███████████████████░ 8,512 tokens
TOON ████████████████████ 8,744 tokens (+2.7% vs CSV)
├─ vs JSON (−42.3%) 15,144 tokens
├─ vs JSON compact (−23.7%) 11,454 tokens
├─ vs YAML (−33.4%) 13,128 tokens
└─ vs XML (−48.9%) 17,095 tokens
CSV ███████████████████░ 8,711 tokens
TOON ████████████████████ 8,937 tokens (+2.6% vs CSV)
├─ vs JSON (−41.7%) 15,337 tokens
├─ vs JSON compact (−23.2%) 11,640 tokens
├─ vs YAML (−33.0%) 13,337 tokens
└─ vs XML (−48.3%) 17,294 tokens
──────────────────────────────────── Total ────────────────────────────────────
CSV ███████████████████░ 63,997 tokens
TOON ████████████████████ 67,778 tokens (+5.9% vs CSV)
├─ vs JSON (−58.8%) 164,452 tokens
├─ vs JSON compact (−35.3%) 104,724 tokens
├─ vs YAML (−48.3%) 130,997 tokens
└─ vs XML (−64.4%) 190,290 tokens
CSV ███████████████████░ 64,247 tokens
TOON ████████████████████ 68,030 tokens (+5.9% vs CSV)
├─ vs JSON (−58.7%) 164,643 tokens
├─ vs JSON compact (−35.2%) 104,908 tokens
├─ vs YAML (−48.2%) 131,249 tokens
└─ vs XML (−64.3%) 190,515 tokens
```
Token counts use `gpt-tokenizer` with `o200k_base` encoding (GPT-5 tokenizer). Other providers tokenize differently, so absolute counts are tokenizer-specific; relative differences between formats hold directionally.
<details>

@@ -689,5 +688,5 @@ <summary><strong>Show detailed examples</strong></summary>

**Savings:** 6,400 tokens (42.3% reduction vs JSON)
**Savings:** 6,400 tokens (41.7% reduction vs JSON)
**JSON** (15,144 tokens):
**JSON** (15,337 tokens):

@@ -698,15 +697,2 @@ ```json

{
"id": 28457823,
"name": "freeCodeCamp",
"repo": "freeCodeCamp/freeCodeCamp",
"description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
"createdAt": "2014-12-24T17:49:19Z",
"updatedAt": "2025-10-28T11:58:08Z",
"pushedAt": "2025-10-28T10:17:16Z",
"stars": 430886,
"watchers": 8583,
"forks": 42146,
"defaultBranch": "main"
},
{
"id": 132750724,

@@ -717,7 +703,7 @@ "name": "build-your-own-x",

"createdAt": "2018-05-09T12:03:18Z",
"updatedAt": "2025-10-28T12:37:11Z",
"pushedAt": "2025-10-10T18:45:01Z",
"stars": 430877,
"watchers": 6332,
"forks": 40453,
"updatedAt": "2026-07-23T18:57:15Z",
"pushedAt": "2026-07-14T19:25:58Z",
"stars": 530712,
"watchers": 6778,
"forks": 50205,
"defaultBranch": "master"

@@ -731,8 +717,21 @@ },

"createdAt": "2014-07-11T13:42:37Z",
"updatedAt": "2025-10-28T12:40:21Z",
"pushedAt": "2025-10-27T17:57:31Z",
"stars": 410052,
"watchers": 8017,
"forks": 32029,
"updatedAt": "2026-07-23T18:57:24Z",
"pushedAt": "2026-06-30T18:21:16Z",
"stars": 488074,
"watchers": 8292,
"forks": 36010,
"defaultBranch": "main"
},
{
"id": 28457823,
"name": "freeCodeCamp",
"repo": "freeCodeCamp/freeCodeCamp",
"description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
"createdAt": "2014-12-24T17:49:19Z",
"updatedAt": "2026-07-22T07:01:33Z",
"pushedAt": "2026-07-21T18:00:51Z",
"stars": 452380,
"watchers": 8590,
"forks": 45624,
"defaultBranch": "main"
}

@@ -743,9 +742,9 @@ ]

**TOON** (8,744 tokens):
**TOON** (8,937 tokens):
```
repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2025-10-28T11:58:08Z","2025-10-28T10:17:16Z",430886,8583,42146,main
132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-28T12:37:11Z","2025-10-10T18:45:01Z",430877,6332,40453,master
21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-28T12:40:21Z","2025-10-27T17:57:31Z",410052,8017,32029,main
132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2026-07-23T18:57:15Z","2026-07-14T19:25:58Z",530712,6778,50205,master
21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2026-07-23T18:57:24Z","2026-06-30T18:21:16Z",488074,8292,36010,main
28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2026-07-22T07:01:33Z","2026-07-21T18:00:51Z",452380,8590,45624,main
```

@@ -759,19 +758,3 @@

### CLI (No Installation Required)
Try TOON instantly with npx:
```bash
# Convert JSON to TOON
npx @toon-format/cli input.json -o output.toon
# Pipe from stdin
echo '{"name": "Ada", "role": "dev"}' | npx @toon-format/cli
```
See the [CLI section](#cli) for all options and examples.
### TypeScript Library
```bash
# npm

@@ -787,2 +770,8 @@ npm install @toon-format/toon

To keep the [CLI](#cli) around instead of invoking it through `npx`, install it globally:
```bash
npm install -g @toon-format/cli
```
**Example usage:**

@@ -833,3 +822,3 @@

// name: Ada
// email: alice@example.com
// email: ada@example.com

@@ -847,42 +836,7 @@ // Transform values

> [!TIP]
> The `replacer` function provides fine-grained control over encoding, similar to `JSON.stringify`'s replacer but with path tracking. See the [API Reference](https://toonformat.dev/reference/api#replacer-function) for more examples.
> The `replacer` function provides fine-grained control over encoding, similar to `JSON.stringify`'s replacer but with path tracking. See the [API Reference](https://toonformat.dev/reference/api#replacer-function) for more examples, including verbatim output with [`rawString`](https://toonformat.dev/reference/api#raw-string-output).
## Playgrounds
Experiment with TOON format interactively using these tools for token comparison, format conversion, and validation.
### Official Playground
The [TOON Playground](https://toonformat.dev/playground) lets you convert JSON or YAML to TOON in real time, compare token counts, and share your experiments via URL.
### Community Playgrounds
- [Format Tokenization Playground](https://www.curiouslychase.com/playground/format-tokenization-exploration)
- [TOON Tools](https://toontools.vercel.app/)
## Editor Support
### VS Code
[TOON Language Support](https://marketplace.visualstudio.com/items?itemName=vishalraut.vscode-toon) – Syntax highlighting, validation, conversion, and token analysis.
```bash
code --install-extension vishalraut.vscode-toon
```
### Tree-sitter Grammar
[tree-sitter-toon](https://github.com/3swordman/tree-sitter-toon) – Grammar for Tree-sitter-compatible editors (Neovim, Helix, Emacs, Zed).
### Neovim
[toon.nvim](https://github.com/thalesgelinger/toon.nvim) – Lua-based plugin.
### Other Editors
Use YAML syntax highlighting as a close approximation.
## CLI
Command-line tool for quick JSON↔TOON conversions, token analysis, and pipeline integration. Auto-detects format from file extension, supports stdin/stdout workflows, and offers delimiter options for maximum efficiency.
Command-line tool for quick JSON↔TOON conversions, token analysis, and pipeline integration. Auto-detects format from file extension, supports stdin/stdout workflows, and offers delimiter options (comma, tab, pipe) that trade readability for fewer tokens.

@@ -910,16 +864,16 @@ ```bash

## Format Overview
## Using TOON with LLMs
Detailed syntax references, implementation guides, and quick lookups for understanding and using the TOON format.
TOON works best when you show the format instead of describing it. Once a model sees one tabular example, the header – `[N]` length plus `{fields}` field list – tells it how to read the rest. Wrap data in ` ```toon` code blocks for input, and show the expected header template when asking models to generate TOON. Tab delimiters buy further token savings. Full-line `#` comments are stripped on decode, so hand-annotated prompt data – and model output with explainer lines – still decodes cleanly.
- [Format Overview](https://toonformat.dev/guide/format-overview) – Complete syntax documentation
- [Syntax Cheatsheet](https://toonformat.dev/reference/syntax-cheatsheet) – Quick reference
- [API Reference](https://toonformat.dev/reference/api) – Encode/decode usage (TypeScript)
Follow the detailed [LLM integration guide](https://toonformat.dev/guide/llm-prompts) for strategies, examples, and validation techniques.
## Using TOON with LLMs
## Ecosystem
TOON works best when you show the format instead of describing it. The structure is self-documenting – models parse it naturally once they see the pattern. Wrap data in ` ```toon` code blocks for input, and show the expected header template when asking models to generate TOON. Use tab delimiters for even better token efficiency.
**Playgrounds** – the [official playground](https://toonformat.dev/playground) converts JSON or YAML to TOON in real time, compares token counts, and shares experiments by URL. Community alternatives: [Format Tokenization Playground](https://www.curiouslychase.com/playground/format-tokenization-exploration), [TOON Tools](https://toontools.vercel.app/).
Follow the detailed [LLM integration guide](https://toonformat.dev/guide/llm-prompts) for strategies, examples, and validation techniques.
**Editors** – [TOON Language Support](https://marketplace.visualstudio.com/items?itemName=vishalraut.vscode-toon) for VS Code (`code --install-extension vishalraut.vscode-toon`) adds highlighting, validation, and token analysis. [tree-sitter-toon](https://github.com/3swordman/tree-sitter-toon) covers Neovim, Helix, Emacs, and Zed; [toon.nvim](https://github.com/thalesgelinger/toon.nvim) is a Lua-native alternative. Elsewhere, YAML highlighting is a close approximation.
**Tooling** – [Tooner](https://github.com/chaindead/tooner) is an MCP proxy that converts JSON tool responses to TOON.
## Documentation

@@ -939,3 +893,2 @@

- [Playgrounds](https://toonformat.dev/ecosystem/tools-and-playgrounds) – Interactive tools
- [Tooner](https://github.com/chaindead/tooner) – MCP proxy that converts JSON tool responses to TOON
- [Using TOON with LLMs](https://toonformat.dev/guide/llm-prompts) – Prompting strategies & validation

@@ -948,3 +901,8 @@

- [Specification](https://github.com/toon-format/spec/blob/main/SPEC.md) – Normative rules for implementers
- [Glossary](https://github.com/toon-format/spec/blob/main/CONTEXT.md) – One name per concept, for contributors and tooling
## Media Type & File Extension
TOON files use the `.toon` extension and the provisional media type `text/toon`. Documents are always UTF-8; the `charset=utf-8` parameter may be given but is assumed when absent. See [SPEC.md §17](https://github.com/toon-format/spec/blob/main/SPEC.md#17-iana-considerations) for normative details.
## Other Implementations

@@ -951,0 +909,0 @@ 

dist/index.mjs

Sorry, the diff of this file is too big to display