transformers

What is transformers?

The 'transformers' npm package provides a suite of tools for working with transformer models, particularly those developed by Hugging Face. It allows users to easily integrate state-of-the-art natural language processing (NLP) models into their applications for tasks such as text generation, translation, summarization, and more.

What are transformers's main functionalities?

Text Generation

This feature allows you to generate text based on a given prompt using a pre-trained model like GPT-2.

const { pipeline } = require('transformers');
const generator = pipeline('text-generation', 'gpt2');
generator('Once upon a time,');

Text Classification

This feature enables sentiment analysis, allowing you to classify the sentiment of a given text.

const { pipeline } = require('transformers');
const classifier = pipeline('sentiment-analysis');
classifier('I love using transformers!');

Translation

This feature allows you to translate text from one language to another using pre-trained translation models.

const { pipeline } = require('transformers');
const translator = pipeline('translation_en_to_fr');
translator('Hello, how are you?');

Summarization

This feature enables text summarization, allowing you to generate a concise summary of a longer text.

const { pipeline } = require('transformers');
const summarizer = pipeline('summarization');
summarizer('The quick brown fox jumps over the lazy dog.');

Other packages similar to transformers

spacy

spaCy is an open-source software library for advanced NLP in Python. It is designed specifically for production use and provides a wide range of functionalities including tokenization, part-of-speech tagging, named entity recognition, and more. Compared to transformers, spaCy is more focused on traditional NLP tasks and less on transformer-based models.

nltk

The Natural Language Toolkit (NLTK) is a suite of libraries and programs for symbolic and statistical natural language processing for English written in the Python programming language. NLTK provides easy-to-use interfaces to over 50 corpora and lexical resources. While NLTK is comprehensive for traditional NLP tasks, it does not focus on transformer models like the transformers package.

README

transformers

String/Data transformations for use in templating libraries, static site generators and web frameworks. This gathers the most useful transformations you can apply to text or data into one library with a consistent API. Transformations can be pretty much anything but most are either compilers or templating engines.

API

To apply a transformation called foo-bar to a string str and a file at path pth with options opts:

N.B. note that you will need to also have the foo-bar library installed

var transfomers = require('tramsformers');

var transformer = transformers['foo-bar'];

var rendered = transformer.renderSync(str, opts);

transformer.render(str, opts, function (err, rendered) {
  //Some transformations can only be used asyncronously
});

var rendered = transformer.renderFileSync(pth, opts);

transformer.renderFile(pth, opts, function (err, res) {
  //Some transformations can only be used asyncronously
});

You can also make use of:

transformers['foo-bar'].outputFormat;// => `xml`, `css`, `js`, `json` or `text`
transformers['foo-bar'].sync;// => `true` if it can be used syncronously
transformers['foo-bar'].sudoSync;// => `true` if not always syncronous (see `Libraries that don't work synchronously`)
transformers['foo-bar'].engines;// => an array of possible npm packages to use as the implementation

Libraries that don't work synchronously

The following transformations will always throw an exception if you attempt to run them synchronously:

1. dust
2. qejs

The following transformations sometimes throw an exception if run syncronously, typically they only throw an exception if you are doing something like including another file. If you are not doing the things that cause them to fail then they are consistently safe to use syncronously.

• jade (only when using then-jade instead of jade)
• less (when @import is used)
• stylus (when @import is used)
• jazz (I don't have an example of where this fails, but I also don't know whether it does or not)

The following libraries look like they might sometimes throw exceptions when used syncronously (if you read the source) but they never actually do so:

• just
• ect

Supported transforms

Template engines

• atpl - Compatible with twig templates
• coffeecup - pure coffee-script templates (fork of coffeekup)
• dust (website) - asyncronous templates
• eco - Embedded CoffeeScript templates
• ect (website) - Embedded CoffeeScript templates
• ejs - Embedded JavaScript templates
• haml (website) - dry indented markup
• haml-coffee (website) - haml with embedded CoffeeScript
• handlebars (website) - extension of mustache templates
• hogan (website) - Mustache templates
• jade (website) - robust, elegant, feature rich template engine
• jazz
• jqtpl (website) - extensible logic-less templates
• JUST
• liquor - extended EJS with significant white space
• mustache - logic less templates
• QEJS - Promises + EJS for async templating
• swig (website) - Django-like templating engine
• toffee - templating language based on coffeescript
• underscore (website)
• walrus - A bolder kind of mustache
• whiskers - logic-less focused on readability

Other

• cdata - No need to install anything, just wrapps text in <![CDATA[\nYOUR TEXT\n]]>
• coffee - npm install coffee-script
• cson - coffee-script based JSON format
• markdown - You can use marked, supermarked, markdown-js or `markdown
• uglify - npm install uglify-js minifies javascript

Pull requests to add more transforms will always be accepted providing they are open-source, come with unit tests, and don't cause any of the tests to fail.