@node-llm/core

@node-llm/core

The production-grade LLM engine for Node.js. Provider-agnostic by design.

@node-llm/core provides a single, unified API for interacting with over 540+ models across all major providers. It is built for developers who need stable infrastructure, standard streaming, and automated tool execution without vendor lock-in.

🚀 Key Features

• Unified API: One interface for OpenAI, Anthropic, Gemini, DeepSeek, OpenRouter, and Ollama.
• Automated Tool Loops: Recursive tool execution handled automatically—no manual loops required.
• Streaming + Tools: Seamlessly execute tools and continue the stream with the final response.
• Structured Output: Native Zod support for rigorous schema validation (.withSchema()).
• Multimodal engine: Built-in handling for Vision, Audio (Whisper), and Video (Gemini).
• Security-First: Integrated circuit breakers for timeouts, max tokens, and infinite tool loops.

📋 Supported Providers

Provider	Supported Features
OpenAI	Chat, Streaming, Tools, Vision, Audio, Images, Transcription, Reasoning
Anthropic	Chat, Streaming, Tools, Vision, PDF, Structured Output, Extended Thinking (Claude 3.7)
Gemini	Chat, Streaming, Tools, Vision, Audio, Video, Embeddings
DeepSeek	Chat (V3), Extended Thinking (R1), Streaming, Tools
Bedrock	Chat, Streaming, Tools, Image Gen (Titan/SD), Embeddings, Prompt Caching
OpenRouter	540+ models, Chat, Streaming, Tools, Vision, Embeddings, Reasoning
Ollama	Local Inference, Chat, Streaming, Tools, Vision, Embeddings
Mistral	Chat, Streaming, Tools, Vision, Embeddings, Transcription, Moderation, Reasoning (Magistral)

⚡ Quick Start

Installation

npm install @node-llm/core

Basic Chat & Streaming

NodeLLM automatically reads your API keys from environment variables (e.g., OPENAI_API_KEY).

import { createLLM } from "@node-llm/core";

const llm = createLLM({ provider: "openai" });

// 1. Standard Request
const res = await llm.chat("gpt-4o").ask("What is the speed of light?");
console.log(res.content);

// 2. Real-time Streaming
for await (const chunk of llm.chat().stream("Tell me a long story")) {
  process.stdout.write(chunk.content);
}

Structured Output (Zod)

Stop parsing markdown. Get typed objects directly.

import { z } from "@node-llm/core";

const PlayerSchema = z.object({
  name: z.string(),
  powerLevel: z.number(),
  abilities: z.array(z.string())
});

const chat = llm.chat("gpt-4o-mini").withSchema(PlayerSchema);
const response = await chat.ask("Generate a random RPG character");

console.log(response.parsed.name); // Fully typed!

🛡️ Security Circuit Breakers

NodeLLM protects your production environment with four built-in safety pillars:

const llm = createLLM({
  requestTimeout: 15000, // 15s DoS Protection
  maxTokens: 4096, // Cost Protection
  maxRetries: 3, // Retry Storm Protection
  maxToolCalls: 5 // Infinite Loop Protection
});

🔌 Middleware System

NodeLLM 1.9.0 introduces a powerful lifecycle hook system for audit, security, and observability.

import { createLLM, PIIMaskMiddleware, UsageLoggerMiddleware } from "@node-llm/core";

const llm = createLLM({
  provider: "openai",
  middlewares: [
    new PIIMaskMiddleware(), // Redact emails/phone numbers automatically
    new UsageLoggerMiddleware() // Log structured token usage & costs
  ]
});

// All chats created from this instance inherit these middlewares
const chat = llm.chat("gpt-4o");

Decisive Tool Safety

Middlewares can control the engine's recovery strategy during tool failures.

const safetyMiddleware = {
  name: "Audit",
  onToolCallError: async (ctx, tool, error) => {
    if (tool.function.name === "delete_user") return "STOP"; // Kill the loop
    return "RETRY"; // Attempt recovery
  }
};

🤖 Agent Class

Define reusable, class-configured agents with a declarative DSL:

import { Agent, Tool, z } from "@node-llm/core";

class LookupOrderTool extends Tool<{ orderId: string }> {
  name = "lookup_order";
  description = "Look up an order by ID";
  schema = z.object({ orderId: z.string() });

  async execute({ orderId }: { orderId: string }) {
    return { status: "shipped", eta: "Tomorrow" };
  }
}

class SupportAgent extends Agent {
  static model = "gpt-4.1";
  static instructions = "You are a helpful support agent.";
  static tools = [LookupOrderTool];
  static temperature = 0.2;
}

// Use anywhere in your app
const agent = new SupportAgent();
const response = await agent.ask("Where is order #123?");
console.log(response.content);

ToolHalt - Early Loop Termination

Stop the agentic loop early when a definitive answer is found:

class FinalAnswerTool extends Tool<{ answer: string }> {
  name = "final_answer";
  description = "Return the final answer to the user";
  schema = z.object({ answer: z.string() });

  async execute({ answer }: { answer: string }) {
    return this.halt(answer); // Stops the loop, returns this result
  }
}

💾 Ecosystem

Looking for persistence? use @node-llm/orm.

• Automatically saves chat history to PostgreSQL/MySQL/SQLite via Prisma.
• Tracks tool execution results and API metrics (latency, cost, tokens).

📚 Full Documentation

Visit nodellm.dev for:

• Deep Dive into Tool Calling
• Multi-modal Vision & Audio Guide
• Custom Provider Plugin System

License

MIT © [NodeLLM Contributors]