@transformation/process

@transformation/process

A package for using child processes.

• exec
• spawn
• startProcess/childProcess

exec

Spawns a shell and executes the given commands. See Node child_process.spawn for more information.

import { exec } from "@transform/process";
import { concat, lines } from "@transform/stream";
import { emitItems, interleave, skipLast } from "@transform/core";

await expect(
  pipeline(
    emitItems("Hello", "beautiful", "world!"),
    interleave("\n"),
    exec(`sed 's/^/> /g' | grep -v beautiful`),
    concat()
  ),
  "to yield items",
  ["> Hello\n> world!\n"]
);

You can also pipe data into a sub-process.

await expect(
  pipeline(
    emitItems("Hello\nfantastic\nworld"),
    exec("grep -v fantastic"),
    lines()
  ),
  "to yield items",
  ["Hello", "world", ""]
);

spawn

Spawns a new sub-process. See Node child_process.spawn for more information.

import { spawn } from "@transform/process";
import { lines } from "@transform/stream";
import { pipeline, skipLast } from "@transform/core";

You can use it to emit items into the pipeline.

await expect(
  pipeline(spawn("ls", [testDir]), lines(), skipLast()),
  "to yield items",
  ["0.txt", "1.txt", "2.txt"]
);

You can also pipe data into a sub-process.

await expect(
  pipeline(
    emitItems("Hello\nfantastic\nworld"),
    spawn("grep", ["-v", "fantastic"]),
    lines(),
    skipLast()
  ),
  "to yield items",
  ["Hello", "world"]
);

Multiple sub-processes can be combined.

await expect(
  pipeline(spawn("ls", [testDir]), spawn("grep", ["0"]), lines(), skipLast()),
  "to yield items",
  ["0.txt"]
);

startProcess/childProcess

Starts a child process pipeline in a new Node instance.

const { startProcess, childProcess } = require("@transformation/process");
import { pipeline } from "@transform/core";

Notice this is only useful for cases where your pipeline is more CPU intensive than the overhead of communicating with the child process.

As an example let's try to square numbers in a child process.

We start by defining the child process pipeline (square.js).

module.exports = childProcess(map(n => n * n));

Now we can load start the process as part of our pipeline.

await expect(
  pipeline(
    emitItems(0, 1, 2, 3, 4, 5, 6, 7, 8, 9),
    startProcess("square.js"))
  ),
  "to yield items",
  [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
);

What is happening here, is that every item is serialized and send into the child process for processing. When the child process emits new items, they are serialized and passed back to the main pipeline.

Changelog

v3.5.1 (2020-11-06)

Pull requests

• #54 Make sure we reset the state for uniqBy (Sune Simonsen)

Commits to master

• v3.5.1 (Sune Simonsen)
• Update changelog (Sune Simonsen)