logs-interceptor-node14

📊 Logs Interceptor

Enterprise-grade, production-ready log interceptor for Node.js applications with Grafana Loki integration. Zero-impact performance, automatic log collection from all sources, and built-in security features.

✨ Key Features

• 🚀 Zero Performance Impact - Circuit breaker, memory management, and adaptive sampling
• 🔒 Security First - Automatic sanitization of sensitive data (passwords, tokens, credit cards)
• 📝 Universal Logging - Intercepts console, Winston, Pino, Morgan, and Bunyan
• 🌐 Distributed Tracing - OpenTelemetry integration with trace/span correlation
• 💾 Smart Buffering - Automatic batching with memory-aware flushing
• 🔄 Resilient - Retry logic, circuit breaker, and emergency file fallback
• 📊 Observability - Built-in metrics, health checks, and performance monitoring
• 🎯 Context Propagation - AsyncLocalStorage for request-scoped logging
• 🗜️ Compression - Automatic gzip compression for network efficiency

📦 Installation

npm install logs-interceptor
# or
yarn add logs-interceptor
# or
pnpm add logs-interceptor

🚀 Quick Start

Basic Usage

import { init } from 'logs-interceptor';

const logger = init({
  transport: {
    url: 'https://loki.example.com/loki/api/v1/push',
    tenantId: 'my-tenant',
    authToken: process.env.LOKI_AUTH_TOKEN,
  },
  appName: 'my-app',
  environment: 'production',
  interceptConsole: true, // Automatically capture console.log
});

// Now all console.log, console.error, etc. are captured
console.log('This goes to Loki!');
logger.info('So does this!');

Environment Variables Setup

Create a .env file:

LOGS_INTERCEPTOR_URL=https://loki.example.com/loki/api/v1/push
LOGS_INTERCEPTOR_TENANT_ID=my-tenant
LOGS_INTERCEPTOR_AUTH_TOKEN=your-auth-token
LOGS_INTERCEPTOR_APP_NAME=my-app
LOGS_INTERCEPTOR_ENVIRONMENT=production
LOGS_INTERCEPTOR_ENABLED=true

Then simply:

import { init } from 'logs-interceptor';
const logger = init(); // Auto-configures from environment

🔧 Advanced Configuration

const logger = init({
  transport: {
    url: 'https://loki.example.com/loki/api/v1/push',
    tenantId: 'my-tenant',
    authToken: process.env.LOKI_AUTH_TOKEN,
    timeout: 10000,
    maxRetries: 5,
    compression: true,
  },
  
  appName: 'production-api',
  version: '2.0.0',
  environment: 'production',
  
  labels: {
    region: 'us-east-1',
    cluster: 'prod-cluster',
    service: 'api-gateway',
  },
  
  dynamicLabels: {
    hostname: () => require('os').hostname(),
    pid: () => process.pid,
  },
  
  buffer: {
    maxSize: 500,           // Max logs before auto-flush
    flushInterval: 10000,   // Auto-flush every 10s
    maxMemoryMB: 100,      // Max memory usage
  },
  
  filter: {
    levels: ['info', 'warn', 'error', 'fatal'],
    samplingRate: 0.8,     // Sample 80% of logs
    sanitize: true,        // Auto-sanitize sensitive data
    maxMessageLength: 8192,
    sensitivePatterns: [
      /password/i,
      /token/i,
      /api[_-]?key/i,
      /credit[_-]?card/i,
      /ssn/i,
      /cpf/i,
    ],
  },
  
  circuitBreaker: {
    enabled: true,
    failureThreshold: 10,
    resetTimeout: 120000,
  },
  
  integrations: {
    winston: true,
    pino: true,
    morgan: true,
  },
  
  performance: {
    useWorkers: true,
    compressionLevel: 6,
  },
});

🔌 Framework Integrations

Express.js

import express from 'express';
import { init } from 'logs-interceptor';
import { expressMiddleware } from 'logs-interceptor/middleware';

const app = express();
const logger = init({ /* config */ });

// Add request/response logging
app.use(expressMiddleware(logger));

// Morgan integration
import morgan from 'morgan';
app.use(morgan('combined', {
  stream: logger.getMorganStream(),
}));

Koa.js

import Koa from 'koa';
import { init } from 'logs-interceptor';
import { koaMiddleware } from 'logs-interceptor/middleware';

const app = new Koa();
const logger = init({ /* config */ });

app.use(koaMiddleware(logger));

Fastify

import fastify from 'fastify';
import { init } from 'logs-interceptor';
import { fastifyPlugin } from 'logs-interceptor/middleware';

const app = fastify();
const logger = init({ /* config */ });

app.register(fastifyPlugin(logger));

🔗 Logger Integrations

Winston

import winston from 'winston';

const logger = init({
  integrations: { winston: true },
});

const winstonLogger = winston.createLogger({
  transports: [
    logger.getWinstonTransport(),
    new winston.transports.Console(),
  ],
});

winstonLogger.info('This goes to both console and Loki!');

Pino

import pino from 'pino';

const logger = init({
  integrations: { pino: true },
});

const pinoLogger = pino({
  destination: logger.getPinoStream(),
});

pinoLogger.info('High-performance logging to Loki!');

🎯 Context Propagation

// Add context to all logs within a request
app.use((req, res, next) => {
  logger.runWithContext({
    userId: req.user?.id,
    sessionId: req.session?.id,
    requestId: req.headers['x-request-id'],
  }, next);
});

// All logs within this context will include the metadata
app.get('/api/users/:id', async (req, res) => {
  logger.info('Fetching user'); // Includes userId, sessionId, requestId
  
  try {
    const user = await getUserById(req.params.id);
    logger.info('User fetched successfully');
    res.json(user);
  } catch (error) {
    logger.error('Failed to fetch user', { error: error.message });
    res.status(500).json({ error: 'Internal server error' });
  }
});

🔒 Security Features

Automatic Sensitive Data Sanitization

// These will be automatically sanitized
logger.info('User login', {
  username: '[e-mail]',
  password: 'secret123',        // → [REDACTED]
  creditCard: '4111111111111111', // → [REDACTED]
  apiKey: 'sk_live_abc123',     // → [REDACTED]
});

Custom Sanitization Rules

const logger = init({
  filter: {
    sanitize: true,
    sensitivePatterns: [
      /password/i,
      /secret/i,
      /\b\d{3}-\d{2}-\d{4}\b/, // SSN
      /custom-pattern/,
    ],
  },
});

📊 Monitoring & Health Checks

// Health endpoint
app.get('/health', (req, res) => {
  const health = logger.getHealth();
  const metrics = logger.getMetrics();
  
  res.json({
    status: health.healthy ? 'healthy' : 'unhealthy',
    health: {
      ...health,
      circuitBreaker: health.circuitBreakerState,
      memoryUsageMB: health.memoryUsageMB,
      bufferUtilization: health.bufferUtilization,
    },
    metrics: {
      logsProcessed: metrics.logsProcessed,
      logsDropped: metrics.logsDropped,
      logsSanitized: metrics.logsSanitized,
      avgFlushTime: metrics.avgFlushTime,
      errorCount: metrics.errorCount,
    },
  });
});

🛡️ Error Handling & Graceful Shutdown

// Capture unhandled errors
process.on('unhandledRejection', (reason, promise) => {
  logger.fatal('Unhandled Promise Rejection', { reason });
});

process.on('uncaughtException', (error) => {
  logger.fatal('Uncaught Exception', {
    error: error.message,
    stack: error.stack,
  });
  
  logger.flush().then(() => process.exit(1));
});

// Graceful shutdown
async function gracefulShutdown(signal) {
  console.log(`Received ${signal}, shutting down gracefully...`);
  
  server.close();           // Stop accepting new connections
  await logger.flush();      // Flush remaining logs
  await logger.destroy();    // Cleanup resources
  
  process.exit(0);
}

process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));

🚀 Performance Optimization

Adaptive Sampling

const logger = init({
  filter: {
    samplingRate: 0.8, // Sample 80% of logs in production
    levels: ['info', 'warn', 'error', 'fatal'], // Skip debug in production
  },
});

Circuit Breaker

Automatically stops sending logs when Loki is down to prevent application impact:

const logger = init({
  circuitBreaker: {
    enabled: true,
    failureThreshold: 5,    // Open after 5 failures
    resetTimeout: 60000,     // Try again after 1 minute
    halfOpenRequests: 3,     // Test with 3 requests
  },
});

Memory Management

Automatic memory pressure handling:

const logger = init({
  buffer: {
    maxMemoryMB: 50, // Auto-flush when memory usage exceeds 50MB
    maxSize: 1000,   // Max 1000 logs in buffer
  },
});

📝 API Reference

Logger Methods

logger.debug(message: string, context?: object): void
logger.info(message: string, context?: object): void
logger.warn(message: string, context?: object): void
logger.error(message: string, context?: object): void
logger.fatal(message: string, context?: object): void
logger.trackEvent(eventName: string, properties?: object): void
logger.flush(): Promise<void>
logger.getMetrics(): LoggerMetrics
logger.getHealth(): HealthStatus
logger.runWithContext<T>(context: object, fn: () => T): T
logger.destroy(): Promise<void>

Environment Variables

Variable	Description	Default
LOGS_INTERCEPTOR_URL	Loki push endpoint	Required
LOGS_INTERCEPTOR_TENANT_ID	Loki tenant ID	Required
LOGS_INTERCEPTOR_AUTH_TOKEN	Authentication token	Optional
LOGS_INTERCEPTOR_APP_NAME	Application name	Required
LOGS_INTERCEPTOR_ENVIRONMENT	Environment name	production
LOGS_INTERCEPTOR_VERSION	App version	1.0.0
LOGS_INTERCEPTOR_BUFFER_SIZE	Buffer size	100
LOGS_INTERCEPTOR_FLUSH_INTERVAL	Flush interval (ms)	5000
LOGS_INTERCEPTOR_LOG_LEVEL	Log levels (comma-separated)	debug,info,warn,error,fatal
LOGS_INTERCEPTOR_SAMPLING_RATE	Sampling rate (0-1)	1.0
LOGS_INTERCEPTOR_CIRCUIT_BREAKER	Enable circuit breaker	true
LOGS_INTERCEPTOR_SANITIZE	Sanitize sensitive data	true
LOGS_INTERCEPTOR_MAX_MEMORY_MB	Max memory usage	50
LOGS_INTERCEPTOR_DEBUG	Debug mode	false
LOGS_INTERCEPTOR_ENABLED	Enable/disable logging	true

🧪 Testing

// Test mode - logs to memory instead of Loki
const logger = init({
  transport: {
    url: 'memory://test',
    tenantId: 'test',
  },
  debug: true,
});

// Access logs in tests
logger.on('log', (entry) => {
  console.log('Log captured:', entry);
});

📊 Grafana Dashboard

Import our pre-configured Grafana dashboard for monitoring:

• Import dashboard JSON from dashboards/logs-interceptor.json
• Configure Loki data source
• Set variables for app_name and environment

🐛 Troubleshooting

Logs not appearing in Loki

• Check circuit breaker status: logger.getHealth()
• Verify network connectivity to Loki
• Check authentication token
• Enable debug mode: LOGS_INTERCEPTOR_DEBUG=true

High memory usage

• Reduce buffer size
• Increase flush frequency
• Enable sampling
• Check for memory leaks in context data

Performance impact

• Enable circuit breaker
• Reduce sampling rate
• Use worker threads
• Increase compression level

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details.

📄 License

MIT © Leonardo Zwirtes

🔗 Links

• GitHub Repository
• NPM Package
• Documentation
• Examples

💬 Support

• GitHub Issues
• Discussions