nexus-dsm

🌐 Nexus DSM – Dataset Management Toolkit

Centralized tooling for parsing, validating, converting, and indexing structured datasets (CSV, JSON, and planned TSV). Built for modularity, clarity, and service-ready integration.

✨ Motivation: Building Trust in Your Data

Most data tools promise seamless automation—but often operate as opaque black boxes. This obscures subtle quality issues and forces blind trust in processes that should be transparent. When working with critical datasets, that lack of visibility can lead to costly errors or endless manual review.

Nexus DSM takes a different approach:

Your data is yours. You deserve control, clarity, and confidence.

Instead of a “set-it-and-forget-it” pipeline, Nexus DSM offers a developer-assisted validation engine that flags syntax issues, highlights inconsistencies (like malformed rows or ambiguous headers), and guides you through every step. Whether you're ingesting raw CSVs or preparing JSON for an API, Nexus DSM ensures every byte meets your standards—with full visibility and explainable feedback.

📦 Repo Scope

This repository contains the core logic for:

• 📥 Parsing and syntax validation
• ✅ Schema validation (via Zod or JSON Schema)
• 🔁 Format conversion (CSV ↔ JSON)
• 🧾 Metadata-driven eligibility checks
• 🗂 (Planned) Indexing for advanced dataset workflows

⚠️ Note: UI components, drag-and-drop tools, and frontend visualizations are maintained in a separate repository: nexus-dsm-ui

📚 Project Documentation

For more detailed information about the project's direction, contribution guidelines, and version history, please see the following documents:

• Project Roadmap - Our vision and development phases.
• Changelog - A detailed log of all version changes.
• Contribution Guide - How to get involved and contribute to the project.
• Security Policy - Our policy for reporting security vulnerabilities.

🚀 Usage

Installation

npm install nexus-dsm # or pnpm add nexus-dsm

Parsing a CSV

The parseCSV function can process a CSV from a file path or a raw string. It returns a detailed response object with the parsed data and rich metadata.

import { parseCSV } from "nexus-dsm";

// From a file path
const response = await parseCSV(undefined, "./data/sample.csv");

if (response.success) {
    console.log("Parsed Data:", response.data);
    console.log(
        "Is eligible for conversion?",
        response.meta.eligibleForConversion
    );
} else {
    console.error("Parsing Failed:", response.message);
    console.log("Error Details:", response.meta.diagnostics);
}

Parsing a JSON

The parseJSON function handles JSON files with a root-level array of objects or a single root object.

import { parseJSON } from "nexus-dsm";

const jsonObject = [{ id: 1, name: "Alice" }];
const response = await parseJSON(jsonObject);

if (response.success) {
    console.log("Parsed Data:", response.data);
    console.log("Structure Type:", response.meta.structureType);
} else {
    console.error("Parsing Failed:", response.message);
}

🔁 Converting Data

After parsing, you can convert between CSV and JSON formats using the conversion functions. The conversion functions take the entire successful response object from the parser.

JSON to CSV Conversion

The convertToCsv function can perform a "shallow" conversion (default) or a "deep" conversion.

import { parseJSON, convertToCsv } from "nexus-dsm";

const nestedJson = `[{"name":"Alice","profile":{"age":30},"tags":["dev"]}]`;
const jsonResponse = await parseJSON(nestedJson);

// Shallow conversion (default)
const shallow = convertToCsv(jsonResponse);
// shallow.content is: name,profile,tags\r\nAlice,"{""age"":30}","[""dev""]"

// Deep conversion
const deep = convertToCsv(jsonResponse, { flattening: 'deep' });
// deep.content is: name,profile.age,tags[0]\r\nAlice,30,dev

CSV to JSON Conversion

The convertToJson function automatically detects flattened headers (e.g., user.name) and reconstructs the nested JSON structure by default.

import { parseCSV, convertToJson } from "nexus-dsm";

const flatCsv = 'user.name,user.id\nAlice,1';
const csvResponse = await parseCSV(flatCsv);

// The function detects flattened headers and un-flattens by default.
const nestedJson = convertToJson(csvResponse);
// nestedJson.content is: '[{"user":{"name":"Alice","id":1}}]'

// To prevent this and get a flat JSON, pass `unflatten: false`.
const flatJson = convertToJson(csvResponse, { unflatten: false });
// flatJson.content is: '[{"user.name":"Alice","user.id":1}]'

🧠 API Reference

Primary API

These are the main functions intended for everyday use.

Function	Description
parseCSV(data?, filePath?)	Parses a CSV or TSV string/file, validates its structure, and returns a detailed response object.
parseJSON(data, filePath?)	Parses a JSON string or a pre-parsed object/array, analyzes its structure, and returns a detailed response object.
convertToCsv(response, options)	Converts a parsed JSON response into a CSV string, with "shallow" (default) or "deep" flattening.
convertToJson(response, options)	Converts a parsed CSV response into a JSON string, with automatic "un-flattening" of deep CSVs.

Advanced & Utility API

These functions and classes provide lower-level access for custom workflows and analysis.

Function/Class	Description
convertCsvStructure(response)	Normalizes a CsvResponse into a standard tabular payload, ready for transformation.
convertJsonStructure(data, meta)	Normalizes a JsonResponse into a structure-aware payload with detailed metadata.
ParsedFileMetaBuilder	A fluent builder class to programmatically construct metadata objects for testing or custom parsing.
calculateNestingDepth(obj)	Computes the maximum nesting depth of a JSON object or array (e.g., a flat object is depth 1).
checkKeyConsistency(array)	Verifies if all objects in an array share the same keys and returns a list of any inconsistent keys.

📊 Feature Comparison Table

Library	Best For	DX Simplicity	Streaming	Nested Support	Metadata	Validation	UI Pairing Potential
nexus-dsm	DX-first conversion, contributor tooling	⭐⭐⭐⭐⭐	✅ (planned)	✅ Deep + Shallow	✅ Rich	✅ Detailed	✅ High
PapaParse	Browser-based parsing, quick CSV import	⭐⭐⭐⭐	✅ Step-wise	❌ (flat only)	⚠️ Minimal	❌	⚠️ Limited
csvtojson	Quick CLI conversions, Node pipelines	⭐⭐⭐	✅ Stream	⚠️ Partial flattening	⚠️ Basic	⚠️ Basic	❌
fast-csv	High-performance streaming in Node	⭐⭐	✅ Stream	❌	❌	❌	❌
csv-parse	Configurable parsing, large datasets	⭐⭐⭐	✅ Stream	⚠️ Manual flattening	⚠️ Basic	⚠️ Manual	❌

⚙️ Workflow Overview

📥 Input file (.csv/.json)
    ↓
🔍 Parsing + Syntax Validation
    ↓
✅ Schema Validation
    ↓
🧾 Metadata Creation (syntax tree + eligibility flags)
    ↓
🔁 Conversion (CSV ↔ JSON, if eligible)
    ↓
📚 Optional Indexing & downstream usage

📚 Folder Structure

nexus-dsm/
├── src/
│   ├── parsers/          # CSV and JSON parsing logic
│   ├── converters/       # Format transformation modules
│   ├── validators/       # Syntax, quote balance, header, and schema checks
│   ├── schemas/          # Schema definitions
│   ├── indexers/         # (Planned) indexing logic
│   ├── constants/        # Shared constants
│   ├── adapters/         # Environment or format adapters
│   └── utils/            # Shared helpers
│   └── index.ts          # Main export file
├── __tests__/            # Unit and integration tests
├── lib/                  # External libraries (e.g. papaparse.min.js)
├── api/                  # Optional HTTP service layer (Phase 3)
├── cli/                  # Optional CLI wrapper (Phase 4)
├── docs/                 # Specs, architecture diagrams, usage notes
├── vitest.config.ts      # Vitest test runner config
├── tsconfig.json         # TypeScript configuration
├── CONTRIBUTING.md       # Contribution guidelines
├── LICENSE               # Project license
└── README.md             # This file

🧪 Testing & Validation

Use __tests__ with fixtures to simulate:

• CSV files with quote imbalances
• JSON inputs with nested structures
• Schema-conforming and non-conforming data
• Metadata eligibility checks
• Conversion edge cases

Built to support mock-driven unit tests and validation suites for CLI, API, or internal tooling.

🤝 Contribution

We welcome PRs, issues, and architectural suggestions. Whether you're extending validation stages, improving conversion logic, or building new adapters—your input helps make Nexus DSM more robust and accessible.

🙏 Acknowledgements

This toolkit's powerful and reliable CSV parsing capabilities are made possible by Papa Parse, the fastest in-browser CSV parser for JavaScript.

Modular, testable, and orchestration-ready. Built by Nexicore Digitals to empower developers with clarity and control.