fasttext.wasm.js

fasttext.wasm.js

Node and Browser env supported WebAssembly version of fastText: Library for efficient text classification and representation learning.

WebAssembly version of fastText with compressed lid.176.ftz model (~900KB) and a typescript wrapper. This project focuses on cross-platform, zero-dependency and out-of-the-box.

Features

• Written in TypeScript
• Node, Browser and Browser extension support
• Language identification integrated

Usage

// Node
import {
  LanguageIdentificationModel,
  initializeFastTextModule,
} from 'fasttext.wasm.js'

await initializeFastTextModule()
// Use lid.176.ftz as default model
const model = new LanguageIdentificationModel()
await model.load()
const result = await model.identify('Hello, world!')
console.log(result) // 'en'

// Others
import {
  LanguageIdentificationModel,
  initializeFastTextModule,
} from 'fasttext.wasm.js/common'

// It will load fasttext.common.wasm from public root directly by default,
// You can download it from https://github.com/yunsii/fasttext.wasm.js/blob/master/src/core/fastText.common.wasm
// You can also use `locateFile` callback to custom fasttext.common.wasm full path.
await initializeFastTextModule()
const model = new LanguageIdentificationModel({
  // Specific model path under public directory,
  // You can download it from https://fasttext.cc/docs/en/language-identification.html
  modelHref: '/models/lid.176.ftz',
})
await model.load()
const result = await model.identify('Hello, world!')
console.log(result) // 'en'

Benchmark

codesandbox/language-detect-benchmark

• 10x Faster and very more accurate than languagedetect
• ~3x Faster and more accurate than franc

Related

• awesome-exhibition/fasttext.wasm.js - Next.js example.

Credits

• DreamOfIce/fastText.wasm
• facebookresearch/fastText/webassembly

References

• Language identification

Build & Publish

Requirements

• emsdk
• xmake

Pay attention, add source ./emsdk_env.sh to shell profile to auto load emsdk env, and export EMSDK_QUIET=1 can be used to suppress these messages.

• npm run build
• npx changeset
• npx changeset version
• git commit
• npx changeset publish
• git push --follow-tags

changeset prerelease doc

License

MIT License © 2023 Yuns

Changelog

0.0.1-next.5

Patch Changes

• ✨ feat: support common usage