lean-reader - npm Package Compare versions

Comparing version

0.1.0

0.1.1

+9

-2

package.json

		{
		"name": "lean-reader",
		"version": "0.1.0",
		"version": "0.1.1",
		"private": false,
		@@ -23,4 +23,11 @@ "type": "module",
		"llm",
		"llm-context",
		"url-to-markdown",
		"html-to-markdown",
		"clean-text",
		"content-extraction",
		"web-scraping",
		"fetch-url",
		"token-optimization",
		"tokens",
		"scraper",
		"readability",
		@@ -27,0 +34,0 @@ "markdown"

+4

-4

README.md

		@@ -39,5 +39,5 @@ # Lean Reader

		- ~32% fewer tokens than Mozilla Readability (the standard extractor) at the median, while keeping ~99% of the body text. Be honest about where that edge comes from: it's the `minimize` post-pass (link/image/footnote/whitespace strip), not smarter extraction — run both through `minimize` and they're roughly par. Lean actually runs Readability as one of its two extractors (see Honest limits), so it doesn't lose to it.
		- Versus raw page HTML the multiple is much larger (median ~15×, 100×+ on script-heavy docs) — but that's HTML nobody feeds an LLM, so read it as "don't dump raw pages," not as a competitive claim.
		- Versus Jina Reader (measured, anonymous tier): ~1.6× fewer tokens on a like-for-like body, ~4.8× if you count the nav and reference dumps Jina also returns. Firecrawl is not yet measured (needs an API key).
		- ~29% fewer tokens than Mozilla Readability (the standard extractor) at the median, while keeping ~99% of the body text. Be honest about where that edge comes from: it's the `minimize` post-pass (link/image/footnote/whitespace strip), not smarter extraction — run both through `minimize` and they're roughly par. Lean actually runs Readability as one of its two extractors (see Honest limits), so it doesn't lose to it.
		- Versus raw page HTML the multiple is much larger (median ~8.7×, down to ~3.1× on already-clean blog prose, 100×+ on script-heavy docs) — but that's HTML nobody feeds an LLM, so read it as "don't dump raw pages," not as a competitive claim.
		- Versus Jina Reader (measured, anonymous tier): ~1.6× fewer tokens on a like-for-like body, ~4.3× if you count the nav and reference dumps Jina also returns. Firecrawl is not yet measured (needs an API key).

		@@ -49,3 +49,3 @@ The receipt uses the `o200k_base` tokenizer (GPT-4o/4.1 class); the model and tokenizer are always shown, and counts are vs the raw page HTML so you can check the math.
		- Static HTML only (v1). Pages whose body is client-rendered (some SPAs, GitHub repo landing pages) return little — Lean Reader flags `partial` instead of emitting empty text. Jina/Firecrawl render JS and will beat us there.
		- Two extractors, body-max selection. Defuddle and Mozilla Readability each silently drop the body on different pages (Defuddle on some large Wikipedia articles, Readability on some docs/SPAs). Lean runs both and keeps whichever recovers more body, so neither's blind spot becomes a silent content drop. A real ROUGE-L ground-truth fidelity pass is still the next step (see the bench repo).
		- Two extractors, body-max selection. Defuddle and Mozilla Readability each silently drop the body on different pages (Defuddle on some large Wikipedia articles, Readability on some docs/SPAs). Lean runs both and keeps whichever recovers more body, so neither's blind spot becomes a silent content drop. A ROUGE-L ground-truth pass on a 14-page hand-labeled sample is done: reference-body recall 0.99, equal to Readability on the same ground truth, so the word-count gap is noise removal, not body loss (see the bench repo).
		- Token counts are `o200k_base`; Claude/Gemini tokenize differently.
		@@ -52,0 +52,0 @@

		@@ -39,5 +39,5 @@ # Lean Reader

		- ~32% fewer tokens than Mozilla Readability (the standard extractor) at the median, while keeping ~99% of the body text. Be honest about where that edge comes from: it's the `minimize` post-pass (link/image/footnote/whitespace strip), not smarter extraction — run both through `minimize` and they're roughly par. Lean actually runs Readability as one of its two extractors (see Honest limits), so it doesn't lose to it.
		- Versus raw page HTML the multiple is much larger (median ~15×, 100×+ on script-heavy docs) — but that's HTML nobody feeds an LLM, so read it as "don't dump raw pages," not as a competitive claim.
		- Versus Jina Reader (measured, anonymous tier): ~1.6× fewer tokens on a like-for-like body, ~4.8× if you count the nav and reference dumps Jina also returns. Firecrawl is not yet measured (needs an API key).
		- ~29% fewer tokens than Mozilla Readability (the standard extractor) at the median, while keeping ~99% of the body text. Be honest about where that edge comes from: it's the `minimize` post-pass (link/image/footnote/whitespace strip), not smarter extraction — run both through `minimize` and they're roughly par. Lean actually runs Readability as one of its two extractors (see Honest limits), so it doesn't lose to it.
		- Versus raw page HTML the multiple is much larger (median ~8.7×, down to ~3.1× on already-clean blog prose, 100×+ on script-heavy docs) — but that's HTML nobody feeds an LLM, so read it as "don't dump raw pages," not as a competitive claim.
		- Versus Jina Reader (measured, anonymous tier): ~1.6× fewer tokens on a like-for-like body, ~4.3× if you count the nav and reference dumps Jina also returns. Firecrawl is not yet measured (needs an API key).

		@@ -49,3 +49,3 @@ The receipt uses the `o200k_base` tokenizer (GPT-4o/4.1 class); the model and tokenizer are always shown, and counts are vs the raw page HTML so you can check the math.
		- Static HTML only (v1). Pages whose body is client-rendered (some SPAs, GitHub repo landing pages) return little — Lean Reader flags `partial` instead of emitting empty text. Jina/Firecrawl render JS and will beat us there.
		- Two extractors, body-max selection. Defuddle and Mozilla Readability each silently drop the body on different pages (Defuddle on some large Wikipedia articles, Readability on some docs/SPAs). Lean runs both and keeps whichever recovers more body, so neither's blind spot becomes a silent content drop. A real ROUGE-L ground-truth fidelity pass is still the next step (see the bench repo).
		- Two extractors, body-max selection. Defuddle and Mozilla Readability each silently drop the body on different pages (Defuddle on some large Wikipedia articles, Readability on some docs/SPAs). Lean runs both and keeps whichever recovers more body, so neither's blind spot becomes a silent content drop. A ROUGE-L ground-truth pass on a 14-page hand-labeled sample is done: reference-body recall 0.99, equal to Readability on the same ground truth, so the word-count gap is noise removal, not body loss (see the bench repo).
		- Token counts are `o200k_base`; Claude/Gemini tokenize differently.
		@@ -52,0 +52,0 @@

lean-reader - npm Package Compare versions

Improved metrics