lingest (Local Ingest) is a command-line tool that generates a single Markdown file named lingest_output.md. It recursively finds all text files in the current directory and its subdirectories and concatenates their content. Each file's content is preceded by a header indicating its relative path.

This is particularly useful for creating a single context file for Large Language Models (LLMs) from your local projects, enabling them to understand the structure and content of your codebase or documentation.

Features

High Performance: Core logic implemented in Rust for blazing fast directory traversal and file processing
Smart Filtering: Automatically excludes binary files, images, videos, and other non-text content
Directory Tree Generation: Outputs a visual tree of the directory structure
Recursive Scanning: Traverses directories deep to find all relevant files
Content Concatenation: Combines the content of all found text files into one output
Clear File Headers: Each file's content is demarcated with FILE: path/to/file.ext for easy identification
Customizable Output: Specify the name of the generated Markdown file
Flexible Filtering:
- Ignore Patterns: Use glob patterns to exclude specific files or directories (e.g., build artifacts, temporary files)
- Include Patterns: Focus the tool on specific file types or directories using glob patterns
Overwrite Protection: Prevents accidental overwriting of existing output files unless explicitly forced
Quiet Mode: Suppresses informational logs, ideal for scripting or CI/CD pipelines
Dry Run Mode: Preview which files would be included and where the output would be saved, without writing anything
Standard CLI Interface: Includes --help and --version flags

Installation

The easiest way to use lingest is with npx:

npx lingest [options]

Alternatively, install it globally using npm, Yarn, or pnpm:

npm install -g lingest

# or

yarn global add lingest

# or

pnpm add -g lingest

Once installed globally, run it directly:

lingest [options]

Note: The first time you install lingest, it will build the native Rust module automatically. This requires Rust to be installed on your system. If you don't have Rust, you can install it from rustup.rs.

Usage

Navigate to the root directory of the project you want to process, then run:

lingest [options]

Options

Option	Alias	Description	Default
`--output`	`-o`	Output file name for the generated Markdown.	`lingest_output.md`
`--ignore`	`-i`	Comma-separated list of glob patterns to ignore (files or directories). These are added to default ignores.	`""`
`--include`	`-n`	Comma-separated list of glob patterns to include. If specified, only files matching these patterns will be processed (still respects ignore patterns).	`""` (include all)
`--force`	`-f`	Force overwrite of the output file if it already exists.	`false`
`--quiet`	`-q`	Suppress informational messages; only errors will be shown.	`false`
`--no-tree`		Do not include the directory tree structure in the output file.	`false`
`--dry-run`		List files that would be processed and the final output path, but don't actually write the file.	`false`
`--help`	`-h`	Show this help message and exit.
`--version`	`-v`	Show the program's version number and exit.

Default Ignored Items

By default, lingest ignores:

node_modules/** and .git/** directories
.DS_Store and thumbs.db system files
Binary executables: *.exe, *.dll, *.so, *.dylib, etc.
Image files: *.jpg, *.png, *.gif, *.svg, etc.
Video files: *.mp4, *.avi, *.mov, etc.
Audio files: *.mp3, *.wav, *.flac, etc.
Archive files: *.zip, *.tar, *.gz, *.rar, etc.
Document files: *.pdf, *.doc, *.docx, etc.
Font files: *.ttf, *.otf, *.woff, etc.
Database files: *.db, *.sqlite, etc.
Compiled files: *.pyc, *.class, *.jar, etc.
Minified files: *.min.js, *.min.css
Common build/output directories: dist/, build/, coverage/, .cache/, .next/, .nuxt/, out/
The output file itself (e.g., lingest_output.md)

This ensures that only UTF-8 text files (mainly source code and documentation) are included in the output.

Glob Patterns

lingest uses micromatch for glob pattern matching. Common examples:

**/*.js: Matches all .js files in any directory.
src/**: Matches all files and folders within the src directory.
*.log: Matches all .log files in the current directory.
!src/important.js: (Negation not supported directly but illustrates how to exclude files conceptually.)

Examples

Basic usage (generates lingest_output.md in the current directory):
```
lingest
```
Or using npx:
```
npx lingest
```

Specify a custom output file name:

lingest --output project_snapshot.md

Or:

lingest -o project_snapshot.md

Add custom ignore patterns (e.g., all dist folders and .log files):
```
lingest --ignore "**/dist/**,**/*.log"
```
Only include JavaScript and TypeScript files:
```
lingest --include "**/*.js,**/*.ts"
```

Include only .md files from the docs folder, excluding drafts:

lingest --include "docs/**/*.md" --ignore "docs/drafts/**"

Force overwrite an existing output file:
```
lingest --force
```
Or:
```
lingest -f
```
Dry run for Python files:
```
lingest --dry-run --include "**/*.py"
```
Quiet mode for scripting:
```
lingest -q -o context.md
```

Output Format

Each file's content in the generated Markdown file will be structured like this:

# FILE: path/to/your/file.ext

(Content of file.ext)

================================================

# FILE: another/path/file.js

================================================

(Content of file.js)

Handling Non-Text Files

If lingest encounters a non-UTF-8 or unreadable file, it will:

Log a warning (unless in quiet mode).
Include the file's header.
Add [Content not included: Could not be read as UTF-8 text...].

This preserves file structure and alerts you about skipped files.

Contributing

Contributions are welcome!

Fork the repository.
Create a branch.
Make your changes.
Submit a pull request with details.

Issues and suggestions? Open one on GitHub (replace with actual link).

License

MIT License. See LICENSE file for details.

Technical Details

lingest uses a hybrid architecture:

The CLI interface is implemented in JavaScript for npm compatibility
The core directory traversal and file processing logic is implemented in Rust for maximum performance
The Rust code is compiled to a native Node.js module using napi-rs

This design provides the best of both worlds: easy installation via npm/npx and high-performance file processing.