args-tokenizer

args-tokenizer

args-tokenizer is a lightweight JavaScript library for parsing shell commands with arguments into an argv array. This makes it easy to work with command-line tools and libraries that expect an array format for arguments, such as tinyexec.

Features

• Simple and intuitive API.
• Handles quoted strings and escapes correctly.
• Supports multiline input.
• Ideal for parsing human-readable shell commands, especially curl-style commands.

---

  Made at Webstudio, open source website builder.

---

Installation

Install args-tokenizer:

npm install args-tokenizer

Usage

Here's how you can use args-tokenizer to parse shell commands:

import { tokenizeArgs } from "args-tokenizer";

const args = tokenizeArgs(`ls -la "./src"`);
console.log(args); // ["ls", "-la", "./src"]

Multiline Input Support

args-tokenizer also supports multiline commands, such as:

const args = tokenizeArgs(`
  curl \\
    -X POST \\
    "https://my-url.com"
`);
console.log(args); // ["curl", "-X", "POST", "https://my-url.com"]

Example with tinyexec

One common use case is passing more human-readable commands into the tinyexec library:

import { tokenizeArgs } from "args-tokenizer";
import { x } from "tinyexec";

const [command, ...args] = tokenizeArgs("ls -la");
const result = await x(command, args);
console.log(result.stdout);

API

tokenizeArgs(command: string): string[]

Parses a shell command string into an array of arguments. Properly handles:

• Quoted strings (e.g., '"./path/to/file"').
• Escaped characters (e.g., \").
• Multiline commands (e.g., lines ending with \\).

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests to improve the library.