Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

sentence-splitter

Package Overview
Dependencies
Maintainers
1
Versions
38
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

sentence-splitter

split {japanese, english} text into sentences.

  • 4.0.1
  • Source
  • npm
  • Socket score

Version published
Weekly downloads
19K
decreased by-69.33%
Maintainers
1
Weekly downloads
 
Created
Source

sentence-splitter

Split {Japanese, English} text into sentences.

Installation

npm install sentence-splitter

Usage

export interface SeparatorParserOptions {
    /**
     * Recognize each characters as separator
     * Example [".", "!", "?"]
     */
    separatorCharacters?: string[]
}

export interface splitOptions {
    /**
     * Separator options
     */
    SeparatorParser?: SeparatorParserOptions;
}

/**
 * split `text` into Sentence nodes
 */
export declare function split(text: string, options?: splitOptions): SentenceSplitterTxtNode[];

/**
 * Convert Paragraph Node to Sentence node.
 * Paragraph Node is defined in textlint's TxtAST.
 * See https://github.com/textlint/textlint/blob/master/docs/txtnode.md
 */
export declare function splitAST(paragraphNode: TxtParentNode, options?: splitOptions): SentenceSplitterTxtNode;

See also TxtAST.

Example

Node

This node is based on TxtAST.

Node's type

  • Str: Str node has value. It is same as TxtAST's Str node.
  • Sentence: Sentence Node has Str, WhiteSpace, or Punctuation nodes as children
  • WhiteSpace: WhiteSpace Node has \n.
  • Punctuation: Punctuation Node has .,

Get these SentenceSplitterSyntax constants value from the module:

import { SentenceSplitterSyntax } from "sentence-splitter";

console.log(SentenceSplitterSyntax.Sentence);// "Sentence"

Node's interface

export type TxtSentenceNode = Omit<TxtParentNode, "type"> & {
    readonly type: "Sentence";
};
export type TxtWhiteSpaceNode = Omit<TxtTextNode, "type"> & {
    readonly type: "WhiteSpace";
};
export type TxtPunctuationNode = Omit<TxtTextNode, "type"> & {
    readonly type: "Punctuation";
};

Fore more details, Please see TxtAST.

Node layout

Node layout image.

This is 1st sentence. This is 2nd sentence.

<Sentence>
    <Str />                      |This is 1st sentence| 
    <Punctuation />              |.|
</Sentence>  
<WhiteSpace />                   | |
<Sentence>
    <Str />                      |This is 2nd sentence|
    <Punctuation />              |.|
</Sentence>

Note: This library will not split Str into Str and WhiteSpace(tokenize) Because, Tokenize need to implement language specific context.

For textlint rule

You can use splitAST for textlint rule. splitAST function can preserve original AST's position unlike split function.

import { splitAST, SentenceSplitterSyntax } from "sentence-splitter";

export default function(context, options = {}) {
    const { Syntax, RuleError, report, getSource } = context;
    return {
        [Syntax.Paragraph](node) {
            const parsedNode = splitAST(node);
            const sentenceNodes = parsedNode.children.filter(childNode => childNode.type === SentenceSplitterSyntax.Sentence);
            console.log(sentenceNodes); // => Sentence nodes
        }
    }
}

Examples

Reference

This library use "Golden Rule" test of pragmatic_segmenter for testing.

Tests

Run tests:

npm test

Create input.json from _input.md

npm run createInputJson    

Update snapshots(output.json):

npm run updateSnapshot

Adding snapshot testcase

  1. Create test/fixtures/<test-case-name>/ directory
  2. Put test/fixtures/<test-case-name>/_input.md with testing content
  3. Run npm run updateSnapshot
  4. Check the test/fixtures/<test-case-name>/output.json
  5. If it is ok, commit it

Contributing

  1. Fork it!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request :D

License

MIT

Keywords

FAQs

Package last updated on 10 Feb 2023

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc