leetfilter

leetfilter - Extensible Swear Filter

This is a library to help you filter out swear words and people who attempt to bypass filters using leets!

What are 'leets'?

Using leets is when you type in a symbol that looks like a letter.

If the original word is testswear, a version with leets would be:

+35+5vv34r
testsw.ear
 ...or...
|e$7?uu&@12
testsw.ear.

These are a bit extreme, but you get the idea. This library can detect these.

Leetfilter also supports being used in discord bots, since it filters out regional indicators too.

How to Use

Install via npm -i leetfilter or yarn add leetfilter.

Leetfilter works with both CJS and ESM! Here are examples of how to initialize it:

CJS

const { LeetFilter } = require("leetfilter");

const swear = new LeetFilter({
  debug: false,
  filler: "*",
  dict: 'small',
  precompile: true,
  customwords: ["testswear"]
});

ESM

import { LeetFilter } from "leetfilter";

const swear = new LeetFilter({
  debug: false,
  filler: "*",
  dict: 'small',
  precompile: true,
  customwords: ["testswear"]
});

Leetfilter was created in TypeScript, so TypeScript is also supported!

Documentation

Initialization

Leetfilter takes in 5 optional parameters on initialization.

Parameter	Description	Type	Default
debug	If on, logs out debug information about compilation and filtering.	boolean	false
filler	Letter used to censor swear words.	string	'#'
dict	Default dictionary used. More info in the following section.	Dictionary | 'tiny' | 'small' | 'large'	Dictionary.Small
precompile	Whether to compile the leets during initialization. This value is suggested to be kept at true. More info in the following section.	boolean	true
customwords	Custom words that you want to add to the filter list.	string[]	[]

Dictionaries and Precompiling

There are currently 3 different dictionaries: tiny, small, and large.

Enum

There is an enum for dictionary type, which can be imported from leetfilter/other. There are three values, Tiny, Small, and Large. However, this enum isn't required to be used in the dict parameter.

Tiny Dictionary

The tiny dictionary only contains one word: testswear. This is solely used for testing.

Value for dict: Dictionary.Tiny or 'tiny'

Small Dictionary

The small dictionary contains about 200 swear words. This is the reccomended dictionary to use since the large one takes a long time to use.

Timings for compiling: 0.1s
Timings for filtering: 0.1-0.3s

Timings performed on an unboosted hacker repl on replit.com

Value for dict: Dictionary.Small or 'small'

Large Dictionary

The large dictionary contains about 2000 swear words. This dictionary is not reccomended because of it's large size. However, if you have a powerful host, it may be possible to use.

Timings for compiling: 1.2s
Timings for filtering: 2-3s

Timings performed on an unboosted hacker repl on replit.com

Value for dict: Dictionary.Large or 'large'

Precompiling

Precompiling should usually be kept on so that if the dictionary ever gets changed or if custom words are added.

If custom words are added, leetfilter will give a warning if precompiling is turned off.

If you ever have issues with words being filtered when they are not meant to or vice versa, you should turn on precompiling.

Precompiling ONLY occurs when the LeetFilter class is initialized. It will not affect the speed of your running program.

async LeetFilter.compile(): Promise<LeetFormat>

An asynchronous function that returns a Promise of type LeetFormat ([index: string]: string). It compiles from the dictionary and the leet file.

LeetFilter.compileSync(): LeetFormat

A synchronous function that does the same as LeetFilter.compile(), used for precompiling.

async LeetFilter.compileToFile(): Promise<LeetFormat>

Same as asynchronous LeetFilter.compile() but it saves it to the buffer file to be used later.

LeetFilter.compileToFileSync(): LeetFormat

Same as before, but synchronous. Used for precompiling.

LeetFilter.setConfig(config: Config)

Same parameters as constructor. Calls constructor again with these parameters.

LeetFilter.censorText(word: string, amnt: number = 3): string

Takes in a word and a number, and uses this.config.filler to change the last amnt characters of word and replace them with this.config.filler.

get LeetFilter.getDict(): string

Returns file path to dictionary file.

async LeetFilter.filter(input: string): Promise<string | false>

Checks against compiled leet regex file to see whether input contains swears, and if so returns a string with all the letters in the swear replaced with this.config.filler. If there are no swears, it returns false.

Example code

This section is WIP.

Using an asynchronous function

// CJS
const { LeetFilter } = require("leetfilter");
// ESM
import { LeetFilter } from "leetfilter";

const swear = new LeetFilter({
  debug: false,
  filler: "*",
  dict: 'small',
  precompile: true,
  customwords: ["testswear"] // added 'testswear' as a custom word
});

const main = async () => {
  // Testing 'not a swear'
  let res = await swear.filter("not a swear");
  console.log(res); // result: false

  // Testing 'testswear'
  res = await swear.filter("testswear");
  console.log(res); // result: '*********'

  // Testing 'testsweara'
  res = await swear.filter("testsweara");
  console.log(res); // result: '*********a'

  // Testing 'a testswear a'
  res = await swear.filter("a testswear a");
  console.log(res); // result: 'a ******** a'

  // Testing '†3575vv34r'
  res = await swear.filter("†3575vv34r");
  console.log(res); // result: '*********'
}

main();

Contribution Information

./src

This directory contains all the source files. They should be .ts files in camel case. All variable/methods should be camel case, but classes should be upper camel case.

./src/index.ts

The main file with the LeetFilter class in it.

./src/other.ts

Where all the enums and types are kept.

./src/data

Directory that contains all the JSON data needed.

./src/data/compiled.leets.json

Where compiling to file puts the compiled data into. This shouldn't be touched.

./src/data/dictionary.*.json

Dictionaries files. Contains all the swear words. There is a tiny, small, and large dictionary.

./src/data/leets.json

A JSON file containing a regex for every possible leet for each letter. More leets can be added.

./dist

Where the compiled code sits. CJS files are .js and ESM files are .mjs. Compiled using tsup.

./example

Directory containing example code.

./example/index.js

Example code, runnable using yarn example.

./package.json

Scripts

yarn build

Builds CJS and ESM files in ./dist and runs yarn data. Uses tsup.

yarn example

Runs example code. This doesn't use ESM and also doesn't import the library. It just imports the ./dist/index.js file. (File located in ./example/index.js)

yarn data

Copies the data directory inside ./src to ./dist.

Credits

By Anvay Mathur ([e-mail]) and Alex Shen ([e-mail])

Licensed under the MIT license, view more at LICENSE.txt.