lingest (Local Ingest) is a command-line tool that generates a single Markdown file named lingest_output.md. It recursively finds all text files in the current directory and its subdirectories and concatenates their content. Each file's content is preceded by a header indicating its relative path.

This is particularly useful for creating a single context file for Large Language Models (LLMs) from your local projects, enabling them to understand the structure and content of your codebase or documentation.

Features

Directory Tree Visualization: Automatically generates a visual directory tree structure at the beginning of the output, showing the project layout with standard tree characters.
Recursive Scanning: Traverses directories deep to find all relevant files.
Content Concatenation: Combines the content of all found text files into one output.
Clear File Headers: Each file's content is demarcated with FILE: path/to/file.ext for easy identification.
Customizable Output: Specify the name of the generated Markdown file.
Flexible Filtering:
- Ignore Patterns: Use glob patterns to exclude specific files or directories (e.g., build artifacts, temporary files).
- Include Patterns: Focus the tool on specific file types or directories using glob patterns.
Overwrite Protection: Prevents accidental overwriting of existing output files unless explicitly forced.
Quiet Mode: Suppresses informational logs, ideal for scripting or CI/CD pipelines.
Dry Run Mode: Preview which files would be included and where the output would be saved, without writing anything.
Standard CLI Interface: Includes --help and --version flags.

Installation

The easiest way to use lingest is with npx:

npx lingest [options]

Alternatively, install it globally using npm, Yarn, or pnpm:

npm install -g lingest

# or

yarn global add lingest

# or

pnpm add -g lingest

Once installed globally, run it directly:

lingest [options]

Usage

Navigate to the root directory of the project you want to process, then run:

lingest [options]

Options

Option	Alias	Description	Default
`--output`	`-o`	Output file name for the generated Markdown.	`lingest_output.md`
`--ignore`	`-i`	Comma-separated list of glob patterns to ignore (files or directories). These are added to default ignores.	`""`
`--include`	`-n`	Comma-separated list of glob patterns to include. If specified, only files matching these patterns will be processed (still respects ignore patterns).	`""` (include all)
`--force`	`-f`	Force overwrite of the output file if it already exists.	`false`
`--quiet`	`-q`	Suppress informational messages; only errors will be shown.	`false`
`--dry-run`		List files that would be processed and the final output path, but don't actually write the file.	`false`
`--no-tree`		Skip generating the directory tree structure at the beginning of the output.	`false`
`--help`	`-h`	Show this help message and exit.
`--version`	`-v`	Show the program's version number and exit.

Default Ignored Items

By default, lingest ignores common files and directories that are typically not needed in code analysis:

Directories:

node_modules, .git, __pycache__, .pytest_cache, .tox, .nox, .mypy_cache, .ruff_cache, .hypothesis
bower_components, .npm, .yarn, .pnpm-store, .gradle, build, .settings, .build, .bundle
vendor/bundle, target, pkg, obj, packages, bin, .svn, .hg
venv, .venv, env, .env, virtualenv, .idea, .vscode, .vs
.cache, .sass-cache, dist, out, site-packages, .docusaurus, .next, .nuxt
.terraform, vendor, xcuserdata

Files:

Python: *.pyc, *.pyo, *.pyd, .coverage, poetry.lock, Pipfile.lock
JavaScript/Node: package-lock.json, yarn.lock, bun.lock, bun.lockb
Java: *.class, *.jar, *.war, *.ear, *.nar, .classpath, *.gradle, .project
C/C++: *.o, *.obj, *.dll, *.dylib, *.exe, *.lib, *.out, *.a, *.pdb
Swift/Xcode: *.xcodeproj, *.xcworkspace, *.xcuserstate, .swiftpm
Ruby: *.gem, Gemfile.lock, .ruby-version, .rvmrc
Rust: Cargo.lock, **/*.rs.bk
.NET/C#: *.suo, *.user, *.nupkg
Version control: .gitignore, .gitattributes, .gitmodules
Media files: *.svg, *.png, *.jpg, *.jpeg, *.gif, *.ico, *.pdf, *.mov, *.mp4, *.mp3, *.wav
Temporary/cache: *.log, *.bak, *.swp, *.tmp, *.temp, .DS_Store, Thumbs.db
Build artifacts: *.egg-info, *.egg, *.whl, *.so
Minified files: *.min.js, *.min.css, *.map
Terraform: *.tfstate*
The output file itself (e.g., lingest_output.md, or whatever is specified by --output)

Glob Patterns

lingest uses micromatch for glob pattern matching. Common examples:

**/*.js: Matches all .js files in any directory.
src/**: Matches all files and folders within the src directory.
*.log: Matches all .log files in the current directory.
!src/important.js: (Negation not supported directly but illustrates how to exclude files conceptually.)

Examples

Basic usage (generates lingest_output.md in the current directory):
```
lingest
```
Or using npx:
```
npx lingest
```

Specify a custom output file name:

lingest --output project_snapshot.md

Or:

lingest -o project_snapshot.md

Add custom ignore patterns (e.g., all dist folders and .log files):
```
lingest --ignore "**/dist/**,**/*.log"
```
Only include JavaScript and TypeScript files:
```
lingest --include "**/*.js,**/*.ts"
```

Include only .md files from the docs folder, excluding drafts:

lingest --include "docs/**/*.md" --ignore "docs/drafts/**"

Force overwrite an existing output file:
```
lingest --force
```
Or:
```
lingest -f
```
Dry run for Python files:
```
lingest --dry-run --include "**/*.py"
```
Quiet mode for scripting:
```
lingest -q -o context.md
```
Generate output without directory tree:
```
lingest --no-tree
```

Output Format

Each file’s content in the generated Markdown file will be structured like this:

# FILE: path/to/your/file.ext

(Content of file.ext)

================================================

# FILE: another/path/file.js

================================================

(Content of file.js)

Handling Non-Text Files

If lingest encounters a non-UTF-8 or unreadable file, it will:

Log a warning (unless in quiet mode).
Include the file’s header.
Add [Content not included: Could not be read as UTF-8 text...].

This preserves file structure and alerts you about skipped files.

Contributing

Contributions are welcome!

Fork the repository.
Create a branch.
Make your changes.
Submit a pull request with details.

Issues and suggestions? Open one on GitHub (replace with actual link).

License

MIT License. See LICENSE file for details.

Keywords

markdown

concatenate

files

lingest