Regular expression detection implemented as a Transform steam; and regex-based buffer replacement stream to replace incoming data on-the-fly.
yarn add -E restream
restream(regex: RegExp): TransformReplaceable Class
The package contains the default restream and Replaceable functions, as well as functions to create markers and their cut and paste rules.
import restream, {
Replaceable,
makeMarkers, makeCutRule, makePasteRule,
} from 'restream'
restream(regex: RegExp,): TransformCreate a Transform stream which will buffer incoming data and push regex results when matches can be made, i.e. when regex.exec returns non-null value. When the g flag is added to the regex, multiple matches will be detected.
/** yarn example/restream.js */
import restream from 'restream'
import { createReadable, createWritable } from './lib'
(async () => {
try {
const rs = createReadable('test-string-{12345}-{67890}')
const stream = restream(/{(\d+)}/g) // create a transform stream
rs.pipe(stream)
const { data, ws } = createWritable()
stream.pipe(ws)
ws.once('finish', () => {
console.log(data)
})
} catch (err) {
console.error(err)
}
})()
[ [ '{12345}',
'12345',
index: 12,
input: 'test-string-{12345}-{67890}' ],
[ '{67890}',
'67890',
index: 20,
input: 'test-string-{12345}-{67890}' ] ]
Replaceable ClassA Replaceable transform stream can be used to transform data according to a single or multiple rules.
Rule TypeReplaceable uses rules to determine how to transform data. Below is the description of the Rule type.
| Property | Type | Description | Example |
|---|---|---|---|
re* | RegExp | A regular expression. | Detect inline code blocks in markdown: /`(.+?)`/. |
replacement* | string | function | async function | A replacer either as a string, function, or async function. It will be passed to the string.replace(re, replacement) native JavaScript method. | As a string: INLINE_CODE. |
String ReplacementReplacement as a string. Given a simple string, it will replace a match detected by the rule's regular expression, without consideration for the capturing groups.
Function ReplacerReplacement as a function. See MDN for more documentation on how the replacer function should be implemented.
The example below allows to replace strings like %NPM: documentary% and %NPM: @rqt/aqt% into a markdown badge (used in documentary).
|
Async Function ReplacerAn asynchronous function to get replacements. The stream won't push any data until the replacer's promise is resolved. Due to implementation details, the regex will have to be run against incoming chunks twice, therefore it might be not ideal for heavy-load applications with many matches.
This example will replace strings like %FORK-js: example example/Replaceable.js% into the output of a forked JavaScript program (used in documentary).
|
constructor(rule: Rule|Rules[],): ReplaceableCreate a Transform stream which will make data available when an incoming chunk has been updated according to the specified rule or rules.
Matches can be replaced using a string, function or async function. When multiple rules are passed as an array, the string will be replaced multiple times if the latter rules also modify the data.
/** yarn example/Replaceable.js */
import { Replaceable } from 'restream'
import { createReadable } from './lib'
const dateRule = {
re: /%DATE%/g,
replacement: new Date().toLocaleString(),
}
const emRule = {
re: /__(.+?)__/g,
replacement(match, p1) {
return `<em>${p1}</em>`
},
}
const authorRule = {
re: /^%AUTHOR_ID: (.+?)%$/mg,
async replacement(match, id) {
const name = await new Promise(resolve => {
// pretend to lookup author name from the database
const authors = { 5: 'John' }
resolve(authors[id])
})
return `Author: <strong>${name}</strong>`
},
}
const STRING = `
Hello __Fred__, your username is __fred__.
You have __5__ stars.
%AUTHOR_ID: 5%
on __%DATE%__
`
;(async () => {
const replaceable = new Replaceable([
dateRule,
emRule,
authorRule,
])
const rs = createReadable(STRING)
rs
.pipe(replaceable)
.pipe(process.stdout)
})()
Output:
Hello <em>Fred</em>, your username is <em>fred</em>.
You have <em>5</em> stars.
Author: <strong>John</strong>
on <em>2018-9-5 05:28:04</em>
Replacer ContextReplacer functions will be executed with their context set to a Replaceable instance to which they belong. Both sync and async replacers can use the this keyword to access their Replaceable instance and modify its properties and/or emit events. This is done so that there's a mechanism by which replacers can share data between themselves.
For example, we might want to read and parse an external file first, but remember its data for use in following replacers.
Given an external file example/types.json:
{
"TypeA": "A new type with certain properties.",
"TypeB": "A type to represent the state of the world."
}
Replaceable can read it in the first typesRule rule, and reference its data in the second paramRule rule:
/** yarn example/context.js */
import Catchment from 'catchment'
import { createReadStream } from 'fs'
import { Replaceable } from 'restream'
import { createReadable } from './lib'
const typesRule = {
re: /^%types: (.+?)%$/mg,
async replacement(match, location) {
const rs = createReadStream(location)
const { promise } = new Catchment({ rs })
const d = await promise
const j = JSON.parse(d)
this.types = j // remember types for access in following rules
return match
},
}
const paramRule = {
re: /^ \* @typedef {(.+?)} (.+)(?: .*)?/mg,
replacement(match, type, typeName) {
const description = this.types[typeName]
if (!description) return match
return ` * @typedef {${type}} ${typeName} ${description}`
},
}
const STRING = `
%types: example/types.json%
/**
* @typedef {Object} TypeA
*/
`
;(async () => {
const replaceable = new Replaceable([
typesRule,
paramRule,
])
const rs = createReadable(STRING)
rs
.pipe(replaceable)
.pipe(process.stdout)
})()
%types: example/types.json%
/**
* @typedef {Object} TypeA A new type with certain properties.
*/
As can be seen above, the description of the type was automatically updated based on the data read from the file.
brake(): voidThe brake method allows to stop further rules from processing incoming chunks. If a a replacer function is run with a global regex, the succeeding replacements will also have no effect.
import { Replaceable } from 'restream'
(async () => {
const replaceable = new Replaceable([
{
re: /AAA/g,
replacement() {
this.brake() // prevent further replacements
return 'BBB'
},
},
{
re: /AAA/g,
replacement() {
return 'RRR'
},
},
])
replaceable.pipe(process.stdout)
replaceable.end('AAA AAA AAA AAA')
})()
BBB AAA AAA AAA
Replacer ErrorsIf an error happens in a sync or async replacer function, the Replaceable will emit it and close.
/** yarn example/errors.js */
import { Replaceable } from 'restream'
import { createReadable } from './lib'
const replace = () => {
throw new Error('An error occurred during a replacement.')
}
(async () => {
const rs = createReadable('example-string')
const replaceable = new Replaceable([
{
re: /.*/,
replacement(match) {
return replace(match)
},
},
])
rs
.pipe(replaceable)
.on('error', (error) => {
console.log(error)
})
})()
Error: An error occurred during a replacement.
at replace (/Users/zavr/adc/restream/example/errors.js:6:9)
at Replaceable.replacement (/Users/zavr/adc/restream/example/errors.js:16:16)
To be able to collect stream data into memory, the catchment package can be used. It will create a promise resolved when the stream finishes.
/** yarn example/catchment.js */
import { Replaceable } from 'restream'
import { createReadable } from './lib'
import Catchment, { collect } from 'catchment'
import { equal } from 'assert'
(async () => {
try {
//0. SETUP: create a replaceable and readable input streams,
// and pipe the input stream into the replaceable.
const replaceable = new Replaceable([
{
re: /hello/i,
replacement() {
return 'WORLD'
},
},
{
re: /world/,
replacement() {
return 'hello'
},
},
])
const rs = createReadable('HELLO world')
rs
.pipe(replaceable)
// 1. Create a writable catchment using constructor.
const catchment = new Catchment()
replaceable.pipe(catchment)
// OR 1. Create a writable catchment and automatically
// pipe into it.
const { promise } = new Catchment({
rs: replaceable,
})
// OR 1+2. Use the collect method which uses a catchment
// internally.
const data = await collect(replaceable)
// 2. WAIT for the catchment streams to finish.
const data2 = await catchment.promise
const data3 = await promise
// Validate that results are the same.
equal(data, data2); equal(data2, data3)
console.log(data)
} catch ({ stack }) {
console.log(stack)
}
})()
WORLD hello
Markers can be used to cut some portion of input text according to a regular expression, run necessary replacement rules on the remaining parts, and then restore the cut chunks. In this way, those chunks do not take part in transformations produced by rules, and can be re-inserted into the stream in their original form.
An example use case would be a situation when markdown code blocks need to be transformed into html, however those code blocks don't need to be processed when inside of a comment, such as:
<!--
The following line should be preseved:
**Integrity is the ability to stand by an idea.**
-->
But the next lines should be transformed into HTML:
**Civilization is the process of setting man free from men.**
**Every building is like a person. Single and unrepeatable.**
When using a naïve transformation with a replacement rule for changing ** into <strong>, both lines will be transformed.
import { Replaceable } from 'restream'
import { createReadStream } from 'fs'
const FILE = 'example/markers/example.md'
const strongRule = {
re: /\*\*(.+?)\*\*/g,
replacement(match, p1) {
return `<strong>${p1}</strong>`
},
}
;(async () => {
const rs = createReadStream(FILE)
const replaceable = new Replaceable(strongRule)
rs
.pipe(replaceable)
.pipe(process.stdout)
})()
<!--
The following line should be preseved:
<strong>Integrity is the ability to stand by an idea.</strong>
-->
But the next lines should be transformed into HTML:
<strong>Civilization is the process of setting man free from men.</strong>
<strong>Every building is like a person. Single and unrepeatable.</strong>
In the output above, the ** in the comment is also transformed using the rule. To prevent this, the strategy is to cut comments out first using markers, then perform the transformation using the strong rule, and finally place the comments back into the text.
const { comments } = makeMarkers({
comments: /<!--([\s\S]+?)-->/g,
})
const cutComments = makeCutRule(comments)
const pasteComments = makePasteRule(comments)
const replaceable = new Replaceable([
cutComments,
strongRule,
pasteComments,
])
<!--
The following line should be preseved:
**Integrity is the ability to stand by an idea.**
-->
But the next lines should be transformed into HTML:
<strong>Civilization is the process of setting man free from men.</strong>
<strong>Every building is like a person. Single and unrepeatable.</strong>
makeMarkers(matchers: { [name]: RegExp },config?: MakeMarkersConfig,): { [name]: Marker }This function will create markers from the hash of passed matchers object. The markers are then used to create cut and paste rules.
When a RegExp specified for a marker is matched, the chunk will be replaced with a string. By default, the string has the %%_RESTREAM_MARKER_NAME_REPLACEMENT_INDEX_%% format.
| Rules (source) | Text after cut |
|---|---|
|
|
This format can be modified with the additional configuration passed as the second argument by providing a function to generate replacement strings, and their respecitve regular expressions to replace them back with their original values.
MakeMarkersConfig: Additional configuration.
| Name | Type | Description | Default |
|---|---|---|---|
| getReplacement | (name: string, index: number) => string | A function used to create a replacement when some text needs to be cut. | - |
| getRegex | (name: string) => RegExp | A function used to create a RegExp to detect replaced chunks. | - |
By default, %%_RESTREAM_${name.toUpperCase()}_REPLACEMENT_${index}_%% replacement is used with new RegExp(`%%_RESTREAM_${name.toUpperCase()}_REPLACEMENT_(\d+)_%%`, 'g') regex to detect it and restore the original value.
makeCutRule(marker: Marker,): RuleMake a rule for the Repleceable to cut out marked chunks so that they don't participate in further transformations.
makePasteRule(marker: Marker,): RuleMake a rule for the Repleceable to paste back chunks replaced earlier.
Sometimes, it might be necessary to access the value replaced by a marker's regular expression. In the example below, all inner code blocks are cut at first to preserve them as they are, then the LINKS rule is applied to generate anchors in a text. However, it is also possible that an inner code block will form part of a link, but because it has been replaced with a marker, the link rule will not work properly.
| Rules (source) | Input |
|---|---|
|
|
| Output | |
|
To prevent this from happening, a check must be performed in the LINKS rule replacement function to see if matched text has any inner code blocks in it. If it does, the value can be accessed and placed back for the correct generation of the link name. This is achieved with the replace function.
const getName = (title) => {
const name = title.toLowerCase()
.replace(/\s+/g, '-')
.replace(/[^\w-]/g, '')
return name
}
const { code } = makeMarkers({
code: /`(.+?)`/g,
})
const cutCode = makeCutRule(code)
const pasteCode = makePasteRule(code)
const linkRule = {
re: /\[(.+?)\]\(#LINK\)/g,
replacement(match, title) {
const realTitle = title.replace(code.regExp, (m, i) => {
const val = code.map[i]
return val
})
const name = getName(realTitle)
return `<a name="${name}">${title}</a>`
},
}
const replaceable = new Replaceable([
cutCode,
linkRule,
pasteCode,
])
`a code block`
`[link in a code block](#LINK)`
<a name="just-link">just link</a>
<a name="a-code-block-in-a-link">`A code block` in a link</a>
Now, the link is generated correctly using the title with the text inside of the code block, and not its replaced marker. Also, because the code marker's regex is used with .replace, its lastIndex property won't change so there's no side effects (compared to using .exec method of a regular expression). This simple example shows how some markers can gain access to replacements made by other markers, which can have more compres applications.
zoroaster masks.The following relevant packages might be of interest.
| Name | Description |
|---|---|
catchment | Collect all data flowing in from the stream into memory, and provide a promise resolved when the stream finishes. |
pedantry | Read a directory as a stream. |
which-stream | Create or choose source and destination (including stdout) streams easily. |
spawncommand | Spawn or fork a process and return a promise resolved with stdout and stderr data when it exits. |
(c) Art Deco 2018
FAQs
Regular Expression Detection & Replacement streams.
The npm package restream receives a total of 3,215 weekly downloads. As such, restream popularity was classified as popular.
We found that restream demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware payment rates hit an all-time low in 2024 as law enforcement crackdowns, stronger defenses, and shifting policies make attacks riskier and less profitable.
Security News
require(esm) backported to Node.js 20, easing the transition to ESM-only packages and reducing complexity for developers as Node 18 nears end-of-life.
Security News
PyPI now supports iOS and Android wheels, making it easier for Python developers to distribute mobile packages.