intl-segmenter-polyfill

Intl Segmenter Polyfill

Provides .wasm module built with icu4c for breaking text into words, so that we can polyfill Intl Segmenter Proposal with full compatibility, even on browsers that do not expose v8BreakIterator api.

By default it bundles only Thai language dictionary. Modify filters.json if you need to support other exotic languages.

Usage

npm install --save intl-segmenter-polyfill

Web – fetch

This is the most efficient way as you can lazily load the wasm module only when you need it and use instantiateStreaming for the best performance. Serve break_iterator.wasm as a static asset with application/wasm content-type and you are good to go.

index.js

import { createIntlSegmenterPolyfill } from 'intl-segmenter-polyfill'
;(async function () {
  const Segmenter = await createIntlSegmenterPolyfill(
    fetch('/path/to/break_iterator.wasm'),
  )

  const segmenter = new Segmenter('en', { granularity: 'word' })
  const segments = segmenter.segment('foo bar baz')
})()

Web – bundle with base64 encoded module

This is the simplest way to use the polyfill, at the cost of base64 encoded module – it's ~33% bigger and cannot be loaded on demand.

index.js

import { createIntlSegmenterPolyfill } from 'intl-segmenter-polyfill/bundled'
;(async function () {
  const Segmenter = await createIntlSegmenterPolyfill()
  const segmenter = new Segmenter('en', { granularity: 'word' })
  const segments = segmenter.segment('foo bar baz')
  console.log(segments)
})()

OR using plain old

<script src="bundled.js"></script>
<script>
  IntlSegmenterPolyfillBundled.createIntlSegmenterPolyfill().then(function (
    Segmenter,
  ) {
    const segmenter = new Segmenter('en', { granularity: 'word' })
    const segments = segmenter.segment('foo bar baz')
    console.log(segments)
  })
</script>

Web – Rollup / Webpack wasm loader

@rollup/plugin-wasm and webpack wasm-loader can be used with createIntlSegmenterPolyfillFromFactory

rollup.config.js

import commonjs from '@rollup/plugin-commonjs'
import { wasm } from '@rollup/plugin-wasm'

export default {
  input: 'index.js',
  output: {
    file: 'out.js',
    format: 'iife',
  },
  plugins: [commonjs(), wasm()],
}

index.js

import { createIntlSegmenterPolyfillFromFactory } from 'intl-segmenter-polyfill'
import break_iterator from 'intl-segmenter-polyfill/break_iterator.wasm'
;(async function () {
  const Segmenter = await createIntlSegmenterPolyfillFromFactory(break_iterator)

  const segmenter = new Segmenter('en', { granularity: 'word' })
  const segments = segmenter.segment('foo bar baz')
})()

Node

const {createIntlSegmenterPolyfill} = require('intl-segmenter-polyfill')
const fs = require('fs')

const wasmBuffer = fs.readFileSync('node_modules/intl-segmenter-polyfill/break_iterator.wasm')
let wasmBinary = new Uint8Array(wasmBuffer)

;(async () => {
  const Segmenter = await createIntlSegmenterPolyfill(wasmBinary);
  const segmenter = new Segmenter("en", { granularity: 'word' });
  const segments = segmenter.segment("foo bar baz");
)()

Supported browsers

Besides Chrome, Firefox and Safari with reasonable versions, it polyfills TextEncoder/TextDecoder to support Edge 18 (non-chromium).

Building

Running ./build.sh while having docker installed should output break_iterator.wasm ready to be used in Node, browsers or Wasmer without a lot of special treatment (see examples above or examples/).