transformers

What is transformers?

The 'transformers' npm package provides a suite of tools for working with transformer models, particularly those developed by Hugging Face. It allows users to easily integrate state-of-the-art natural language processing (NLP) models into their applications for tasks such as text generation, translation, summarization, and more.

What are transformers's main functionalities?

Text Generation

This feature allows you to generate text based on a given prompt using a pre-trained model like GPT-2.

const { pipeline } = require('transformers');
const generator = pipeline('text-generation', 'gpt2');
generator('Once upon a time,');

Text Classification

This feature enables sentiment analysis, allowing you to classify the sentiment of a given text.

const { pipeline } = require('transformers');
const classifier = pipeline('sentiment-analysis');
classifier('I love using transformers!');

Translation

This feature allows you to translate text from one language to another using pre-trained translation models.

const { pipeline } = require('transformers');
const translator = pipeline('translation_en_to_fr');
translator('Hello, how are you?');

Summarization

This feature enables text summarization, allowing you to generate a concise summary of a longer text.

const { pipeline } = require('transformers');
const summarizer = pipeline('summarization');
summarizer('The quick brown fox jumps over the lazy dog.');

Other packages similar to transformers

spacy

spaCy is an open-source software library for advanced NLP in Python. It is designed specifically for production use and provides a wide range of functionalities including tokenization, part-of-speech tagging, named entity recognition, and more. Compared to transformers, spaCy is more focused on traditional NLP tasks and less on transformer-based models.

nltk

The Natural Language Toolkit (NLTK) is a suite of libraries and programs for symbolic and statistical natural language processing for English written in the Python programming language. NLTK provides easy-to-use interfaces to over 50 corpora and lexical resources. While NLTK is comprehensive for traditional NLP tasks, it does not focus on transformer models like the transformers package.

README

transformers

String/Data transformations for use in templating libraries, static site generators and web frameworks. This gathers the most useful transformations you can apply to text or data into one library with a consistent API. Transformations can be pretty much anything but most are either compilers or templating engines.

Supported transforms

To use each of these transforms you will also need to install the associated npm module for that transformer.

Template engines

• atpl - Compatible with twig templates
• coffeecup - pure coffee-script templates (fork of coffeekup)
• dot (website) - focused on speed
• dust (website) - asyncronous templates
• eco - Embedded CoffeeScript templates
• ect (website) - Embedded CoffeeScript templates
• ejs - Embedded JavaScript templates
• haml (website) - dry indented markup
• haml-coffee (website) - haml with embedded CoffeeScript
• handlebars (website) - extension of mustache templates
• hogan (website) - Mustache templates
• jade (website) - robust, elegant, feature rich template engine
• jazz
• jqtpl (website) - extensible logic-less templates
• JUST - EJS style template with some special syntax for layouts/partials etc.
• liquor - extended EJS with significant white space
• mustache - logic less templates
• QEJS - Promises + EJS for async templating
• swig (website) - Django-like templating engine
• templayed (website) - Mustache focused on performance
• toffee - templating language based on coffeescript
• underscore (website)
• walrus - A bolder kind of mustache
• whiskers - logic-less focused on readability

Stylesheet Languages

• less (website) - LESS extends CSS with dynamic behavior such as variables, mixins, operations and functions.
• stylus (website) - revolutionary CSS generator making braces optional
• sass (website) - Sassy CSS

Minifiers

• uglify-js - No need to install anything, just minifies/beautifies JavaScript
• uglify-css - No need to install anything, just minifies/beautifies CSS
• ugilify-json - No need to install anything, just minifies/beautifies JSON

Other

• cdata - No need to install anything, just wraps input as <![CDATA[${INPUT_STRING]]> with the standard escape for ]]> (]]]]><![CDATA[>).
• cdata-js - as cdata, but with surrounding comments suitable for inclusion into a HTML/JavaScript <script> block: //<![CDATA[\n${INPUT_STRING\n//]]>.
• cdata-css - as cdata, but with surrounding comments suitable for inclusion into a HTML/CSS <style> block: /*<![CDATA[*/\n${INPUT_STRING\n/*]]>*/.
• verbatim - No need to install anything, acts as a verbatim passthrough ${INPUT_STRING}
• escape-html - No need to install anything, just replaces special characters to sanitize input for html/xml
• coffee-script - npm install coffee-script
• cson - coffee-script based JSON format
• markdown - You can use marked, supermarked, markdown-js or markdown

Pull requests to add more transforms will always be accepted providing they are open-source, come with unit tests, and don't cause any of the tests to fail.

API

The exported object transformers is a collection of named transformers. To access an individual transformer just do:

var transformer = require('transformers')['transformer-name']

Transformer

The following options are given special meaning by transformers:

• filename is set by transformers automatically if using the renderFile APIs. It is used if cache is enabled.
• cache if true, the template function will be cached where possible (templates are still updated if you provide new options, so this can be used in most live applications).
• sudoSync used internally to put some asyncronous transformers into "sudo syncronous" mode. Don't touch this.
• minify if set to true on a transformer that isn't a minifier, it will cause the output to be minified. e.g. coffeeScript.renderSync(str, {minify: true}) will result in minified JavaScript.

Transformer.engines

Returns an array of engines that can be used to power this transformer. The first of these that's installed will be used for the transformation.

To enable a transformation just take [engine] = Transformer.engines[0] and then do npm install [engine]. If [engine] is . there is no need to install an engine from npm to use the transformer.

Transformer.render(str, options, cb)

Tranform the string str using the Transformer with the provided options and call the callback cb(err, res).

If no cb is provided, this method returns a promises/A+ promise.

Transformer.renderSync(str, options)

Synchronous version of Transformer.render

Transformer.renderFile(filename, options, cb)

Reads the file at filename into str and sets options.filename = filename then calls Transform.render(str, options, cb).

If no cb is provided, this method returns a promises/A+ promise.

Tranformer.renderFileSync(filename, options)

Synchronous version of Tranformer.renderFile

Transformer.outputFormat

A string, one of:

• 'xml'
• 'css'
• 'js'
• 'json'
• 'text'

Adding to this list will not result in a major version change, so you should handle unexpected types gracefully (I'd suggest default to assuming 'text').

Transformer.sync

true if the transformer can be used syncronously, false otherwise.

Libraries that don't work synchronously

The following transformations will always throw an exception if you attempt to run them synchronously:

1. dust
2. qejs
3. html2jade

The following transformations sometimes throw an exception if run syncronously, typically they only throw an exception if you are doing something like including another file. If you are not doing the things that cause them to fail then they are consistently safe to use syncronously.

• jade (only when using then-jade instead of jade)
• less (when @import is used with a url instead of a filename)
• jazz (When one of the functions passed as locals is asyncronous)

The following libraries look like they might sometimes throw exceptions when used syncronously (if you read the source) but they never actually do so:

• just
• ect
• stylus