@memberjunction/storage

@memberjunction/storage

The @memberjunction/storage library provides a unified interface for interacting with various cloud storage providers. It abstracts the complexities of different storage services behind a consistent API, making it easy to work with files stored across different cloud platforms.

Part of the MemberJunction framework.

Overview

This library is a key component of the MemberJunction platform, providing seamless file storage operations across multiple cloud providers. It offers a provider-agnostic approach to file management, allowing applications to switch between storage providers without code changes. The package supports both simple single-tenant deployments using environment variables and enterprise multi-tenant deployments using the MemberJunction Credential Engine for secure credential management.

Architecture

The library is organized around an abstract base class (FileStorageBase) with concrete driver implementations for each supported cloud storage provider. A set of high-level utility functions in util.ts bridge the gap between MemberJunction entities and the underlying drivers. Configuration is handled through Zod-validated schemas loaded from mj.config.cjs or environment variables.

graph TB
    subgraph Application["Application Layer"]
        style Application fill:#2d6a9f,stroke:#1a4971,color:#fff
        UtilFunctions["Utility Functions<br/>createUploadUrl, createDownloadUrl,<br/>moveObject, deleteObject, listObjects,<br/>copyObjectBetweenProviders,<br/>searchAcrossProviders"]
    end

    subgraph Core["Core Abstractions"]
        style Core fill:#7c5295,stroke:#563a6b,color:#fff
        FSBase["FileStorageBase<br/>(Abstract Base Class)"]
        Config["StorageConfig<br/>(Zod Schema)"]
    end

    subgraph Drivers["Storage Provider Drivers"]
        style Drivers fill:#2d8659,stroke:#1a5c3a,color:#fff
        AWS["AWSFileStorage<br/>'AWS S3'"]
        Azure["AzureFileStorage<br/>'Azure Blob Storage'"]
        GCS["GoogleFileStorage<br/>'Google Cloud Storage'"]
        GDrive["GoogleDriveFileStorage<br/>'Google Drive Storage'"]
        SP["SharePointFileStorage<br/>'SharePoint'"]
        Dropbox["DropboxFileStorage<br/>'Dropbox'"]
        Box["BoxFileStorage<br/>'Box'"]
    end

    subgraph MJ["MemberJunction Integration"]
        style MJ fill:#b8762f,stroke:#8a5722,color:#fff
        ClassFactory["MJGlobal ClassFactory<br/>@RegisterClass"]
        Entities["FileStorageProviderEntity<br/>FileStorageAccountEntity"]
        CredEngine["CredentialEngine<br/>Secure credential decryption"]
    end

    UtilFunctions --> FSBase
    UtilFunctions --> ClassFactory
    UtilFunctions --> Entities
    UtilFunctions --> CredEngine
    FSBase --> Config
    AWS --> FSBase
    Azure --> FSBase
    GCS --> FSBase
    GDrive --> FSBase
    SP --> FSBase
    Dropbox --> FSBase
    Box --> FSBase
    ClassFactory --> AWS
    ClassFactory --> Azure
    ClassFactory --> GCS
    ClassFactory --> GDrive
    ClassFactory --> SP
    ClassFactory --> Dropbox
    ClassFactory --> Box

Features

• Unified API: Consistent methods across all storage providers via the FileStorageBase abstract class
• Type-Safe: Full TypeScript support with comprehensive type definitions
• Flexible Provider Selection: Use any number of storage providers simultaneously based on your application needs
• Pre-authenticated URLs: Secure upload and download operations using time-limited URLs
• Metadata Support: Store and retrieve custom metadata with your files
• Error Handling: Provider-specific errors are normalized with clear error messages
• Zod-Validated Configuration: All configuration schemas are validated at load time via Zod
• Enterprise Credential Management: Integrates with @memberjunction/credentials for secure, multi-tenant credential storage and decryption
• Cross-Provider Copy: Copy files between different storage providers server-side
• Multi-Provider Search: Search for files across multiple providers or accounts in parallel
• Token Refresh Persistence: Automatic callback system to persist new OAuth tokens (critical for providers like Box that rotate refresh tokens)
• Extensible: Easy to add new storage providers by extending FileStorageBase

Supported Storage Providers

Provider	Driver Key	Search	Pre-auth URLs	Native Directories
AWS S3	AWS S3	No	Yes	Simulated
Azure Blob Storage	Azure Blob Storage	No	Yes	Simulated
Google Cloud Storage	Google Cloud Storage	No	Yes	Simulated
Google Drive	Google Drive Storage	Yes (content)	Yes	Native
Microsoft SharePoint	SharePoint	Yes (content)	Yes	Native
Dropbox	Dropbox	Yes (content)	Yes	Native
Box	Box	Yes (metadata)	Yes	Native

File Operations

• Upload files (via pre-authenticated URLs or direct Buffer upload)
• Download files (via pre-authenticated URLs or direct Buffer download)
• Copy and move files (within and across providers)
• Delete files and directories (with optional recursive deletion)
• List files and directories with full metadata
• Create and manage directories
• Get detailed file metadata without downloading content
• Check file/directory existence
• Search files using native provider search APIs

Installation

npm install @memberjunction/storage

Dependencies

This package depends on:

Package	Purpose
@memberjunction/core	Core MemberJunction functionality (logging, UserInfo)
@memberjunction/core-entities	Entity definitions (FileStorageProviderEntity, FileStorageAccountEntity)
@memberjunction/global	Global class factory and @RegisterClass decorator
@memberjunction/credentials	Credential Engine for secure credential decryption
@aws-sdk/client-s3, @aws-sdk/s3-request-presigner	AWS S3 SDK
@azure/storage-blob, @azure/identity	Azure Blob Storage SDK
@google-cloud/storage	Google Cloud Storage SDK
googleapis	Google Drive API
@microsoft/microsoft-graph-client	SharePoint via Microsoft Graph
dropbox	Dropbox SDK
box-node-sdk	Box SDK
cosmiconfig	Configuration file loading
zod	Configuration schema validation
env-var	Environment variable parsing
mime-types	MIME type detection

Usage

Initialization Flow

Every storage driver follows a two-step initialization pattern. The initialize() method is smart enough to handle both simple deployments (environment variables) and multi-tenant deployments (database credentials).

flowchart TD
    Start(["Create Driver Instance"]) --> Constructor["Constructor<br/>Loads env vars / config defaults"]
    Constructor --> Init{"Call initialize()"}

    Init -->|"No config"| EnvPath["Use environment variables<br/>already loaded by constructor"]
    Init -->|"With config"| ConfigPath["Override credentials<br/>with provided config"]

    EnvPath --> SetAccount["Set accountId / accountName<br/>if provided"]
    ConfigPath --> SetAccount

    SetAccount --> Reinit["Reinitialize SDK client<br/>with final credentials"]
    Reinit --> Ready(["Driver Ready"])

    style Start fill:#2d6a9f,stroke:#1a4971,color:#fff
    style Ready fill:#2d8659,stroke:#1a5c3a,color:#fff
    style Init fill:#b8762f,stroke:#8a5722,color:#fff
    style Constructor fill:#7c5295,stroke:#563a6b,color:#fff
    style EnvPath fill:#7c5295,stroke:#563a6b,color:#fff
    style ConfigPath fill:#7c5295,stroke:#563a6b,color:#fff
    style SetAccount fill:#7c5295,stroke:#563a6b,color:#fff
    style Reinit fill:#7c5295,stroke:#563a6b,color:#fff

Simple Deployment (Environment Variables)

For single-tenant applications, development, testing, or simple production deployments:

# Example: Azure Blob Storage
export STORAGE_AZURE_CONTAINER=your-container-name
export STORAGE_AZURE_ACCOUNT_NAME=your-account-name
export STORAGE_AZURE_ACCOUNT_KEY=your-account-key

import { AzureFileStorage } from '@memberjunction/storage';

// Constructor loads environment variables
const storage = new AzureFileStorage();

// ALWAYS call initialize() - no config needed for env var deployments
await storage.initialize();

// Provider is now ready to use
await storage.ListObjects('/');

Multi-Tenant Enterprise (Database Credentials)

For enterprise applications managing multiple storage accounts via the MemberJunction entity system and Credential Engine:

import { FileStorageEngine } from '@memberjunction/core-entities';
import { initializeDriverWithAccountCredentials } from '@memberjunction/storage';

// Load account from database
const engine = FileStorageEngine.Instance;
await engine.Config(false, contextUser);

const accountWithProvider = engine.GetAccountWithProvider(accountId);

// Infrastructure utility handles credential decryption and initialization
const storage = await initializeDriverWithAccountCredentials({
  accountEntity: accountWithProvider.account,
  providerEntity: accountWithProvider.provider,
  contextUser
});

// Provider is ready - credentials were automatically decrypted and initialized
await storage.ListObjects('/');

The initializeDriverWithAccountCredentials() utility:

• Creates the driver instance via the MJGlobal ClassFactory
• Retrieves the credential by ID from the Credential Engine
• Decrypts the credential values
• Configures a token refresh callback to persist rotated tokens back to the database
• Calls initialize() with decrypted values and account information

Enterprise Credential Flow

sequenceDiagram
    participant App as Application
    participant Util as initializeDriverWithAccountCredentials
    participant CF as ClassFactory
    participant CE as CredentialEngine
    participant Driver as Storage Driver

    App->>Util: { accountEntity, providerEntity, contextUser }
    Util->>CF: CreateInstance(FileStorageBase, driverKey)
    CF-->>Util: driver instance

    alt Account has CredentialID
        Util->>CE: Config(false, contextUser)
        Util->>CE: getCredentialById(credentialID)
        CE-->>Util: credentialEntity
        Util->>CE: getCredential(name, options)
        CE-->>Util: { values: decrypted credentials }
        Util->>Util: Create onTokenRefresh callback
        Util->>Driver: initialize({ accountId, ...decryptedValues, onTokenRefresh })
    else No CredentialID
        alt Provider has Configuration JSON
            Util->>Driver: initialize({ accountId, ...providerConfig })
        else No configuration
            Util->>Driver: initialize({ accountId })
        end
    end

    Driver-->>App: initialized driver

Using Utility Functions

The library provides high-level utility functions that work with MemberJunction's entity system:

import {
  createUploadUrl,
  createDownloadUrl,
  moveObject,
  deleteObject,
  listObjects,
  copyObject
} from '@memberjunction/storage';
import { FileStorageProviderEntity } from '@memberjunction/core-entities';
import { Metadata } from '@memberjunction/core';

async function fileOperationsExample() {
  const md = new Metadata();
  const provider = await md.GetEntityObject<FileStorageProviderEntity>('File Storage Providers');
  await provider.Load('your-provider-id');

  // Create pre-authenticated upload URL
  const { updatedInput, UploadUrl } = await createUploadUrl(
    provider,
    {
      ID: '123',
      Name: 'documents/report.pdf',
      ProviderID: provider.ID
    }
  );

  // The client can use the UploadUrl directly to upload the file
  console.log(`Upload URL: ${UploadUrl}`);
  console.log(`File status: ${updatedInput.Status}`); // 'Uploading'
  console.log(`Content type: ${updatedInput.ContentType}`); // 'application/pdf'

  // If a ProviderKey was returned, use it for future operations
  const fileIdentifier = updatedInput.ProviderKey || updatedInput.Name;

  // Create pre-authenticated download URL
  const downloadUrl = await createDownloadUrl(provider, fileIdentifier);

  // List directory contents
  const contents = await listObjects(provider, 'documents/');
  for (const file of contents.objects) {
    console.log(`${file.name} (${file.size} bytes)`);
  }

  // Copy a file
  await copyObject(provider, fileIdentifier, 'documents/report-backup.pdf');

  // Move a file
  await moveObject(provider, fileIdentifier, 'archive/report.pdf');

  // Delete a file
  await deleteObject(provider, 'archive/report.pdf');
}

Direct Provider Usage

You can work directly with a storage provider by instantiating it:

import { AzureFileStorage, FileStorageBase } from '@memberjunction/storage';
import { MJGlobal } from '@memberjunction/global';

async function directProviderExample() {
  // Method 1: Direct instantiation (simple deployment with env vars)
  const storage = new AzureFileStorage();
  await storage.initialize();

  // Method 2: Using class factory (dynamic provider selection)
  const storage2 = MJGlobal.Instance.ClassFactory.CreateInstance<FileStorageBase>(
    FileStorageBase,
    'Azure Blob Storage'
  );
  await storage2.initialize();

  // List all files in a directory
  const result = await storage.ListObjects('documents/');
  console.log('Files:', result.objects);
  console.log('Directories:', result.prefixes);

  // Upload a file directly with metadata
  const content = Buffer.from('Hello, World!');
  await storage.PutObject(
    'documents/reports/hello.txt',
    content,
    'text/plain',
    { author: 'John Doe', department: 'Engineering' }
  );

  // Get file metadata without downloading content
  const metadata = await storage.GetObjectMetadata({
    fullPath: 'documents/reports/hello.txt'
  });
  console.log('File metadata:', metadata);

  // Download file content
  const fileContent = await storage.GetObject({
    fullPath: 'documents/reports/hello.txt'
  });
  console.log('File content:', fileContent.toString('utf8'));

  // Copy a file
  await storage.CopyObject(
    'documents/reports/hello.txt',
    'documents/archive/hello-backup.txt'
  );

  // Check if a file exists
  const exists = await storage.ObjectExists('documents/reports/hello.txt');

  // Check if a directory exists
  const dirExists = await storage.DirectoryExists('documents/reports/');

  // Delete a directory and all its contents
  await storage.DeleteDirectory('documents/reports/', true);
}

Cross-Provider File Copy

Transfer files between different storage providers server-side:

import { copyObjectBetweenProviders } from '@memberjunction/storage';

const result = await copyObjectBetweenProviders(
  sourceProviderEntity,
  destProviderEntity,
  'documents/report.pdf',
  'imported/report.pdf',
  {
    sourceUserContext: { userID: currentUser.ID, contextUser },
    destinationUserContext: { userID: currentUser.ID, contextUser }
  }
);

if (result.success) {
  console.log(`Transferred ${result.bytesTransferred} bytes`);
}

flowchart LR
    Source["Source Provider<br/>(e.g. Dropbox)"] -->|"GetObject()"| Server["MJ Server<br/>(in-memory Buffer)"]
    Server -->|"PutObject()"| Dest["Destination Provider<br/>(e.g. Google Drive)"]

    style Source fill:#2d6a9f,stroke:#1a4971,color:#fff
    style Server fill:#b8762f,stroke:#8a5722,color:#fff
    style Dest fill:#2d8659,stroke:#1a5c3a,color:#fff

Searching Files

Providers with native search capabilities support the SearchFiles method:

import { FileStorageBase, UnsupportedOperationError } from '@memberjunction/storage';

try {
  // Simple search
  const results = await storage.SearchFiles('quarterly report');

  for (const file of results.results) {
    console.log(`  ${file.path} (${file.size} bytes)`);
    if (file.excerpt) {
      console.log(`    Excerpt: ${file.excerpt}`);
    }
  }

  // Advanced search with filters
  const filtered = await storage.SearchFiles('budget 2024', {
    fileTypes: ['pdf', 'docx'],
    modifiedAfter: new Date('2024-01-01'),
    pathPrefix: 'documents/finance/',
    maxResults: 50,
    searchContent: true
  });

  if (filtered.hasMore) {
    console.log(`Total matches: ${filtered.totalMatches}`);
  }
} catch (error) {
  if (error instanceof UnsupportedOperationError) {
    console.log('This provider does not support file search');
  }
}

Multi-Provider and Multi-Account Search

Search across multiple providers or accounts in parallel:

import { searchAcrossProviders, searchAcrossAccounts } from '@memberjunction/storage';

// Search across multiple providers
const providerResults = await searchAcrossProviders(
  [googleDriveProvider, dropboxProvider, boxProvider],
  'quarterly report',
  {
    maxResultsPerProvider: 25,
    fileTypes: ['pdf', 'docx'],
    providerUserContexts: new Map([
      [googleDriveProvider.ID, { userID: currentUser.ID, contextUser }],
      [dropboxProvider.ID, { userID: currentUser.ID, contextUser }]
    ])
  }
);

for (const pr of providerResults.providerResults) {
  if (pr.success) {
    console.log(`${pr.providerName}: ${pr.results.length} results`);
  } else {
    console.log(`${pr.providerName}: ${pr.errorMessage}`);
  }
}

// Enterprise: Search across multiple accounts (including multiple accounts of same type)
const accountResults = await searchAcrossAccounts(
  [
    { accountEntity: researchDropbox, providerEntity: dropboxProvider },
    { accountEntity: marketingDropbox, providerEntity: dropboxProvider },
    { accountEntity: engineeringGDrive, providerEntity: gdriveProvider }
  ],
  'quarterly report',
  {
    maxResultsPerAccount: 25,
    fileTypes: ['pdf', 'docx'],
    contextUser: currentUser
  }
);

console.log(`Total results: ${accountResults.totalResultsReturned}`);
console.log(`Successful: ${accountResults.successfulAccounts}`);
console.log(`Failed: ${accountResults.failedAccounts}`);

Configuration

Configuration File (mj.config.cjs)

Storage providers can be configured via the mj.config.cjs file at the repository root. The configuration is validated using Zod schemas at load time.

module.exports = {
  storageProviders: {
    aws: {
      accessKeyID: 'your-key',
      secretAccessKey: 'your-secret',
      region: 'us-east-1',
      defaultBucket: 'my-bucket',
      keyPrefix: '/'
    },
    azure: {
      accountName: 'your-account',
      accountKey: 'your-key',
      connectionString: 'optional-conn-string',
      defaultContainer: 'my-container'
    },
    googleCloud: {
      projectID: 'your-project',
      keyFilename: '/path/to/keyfile.json',
      keyJSON: '{"type":"service_account",...}',
      defaultBucket: 'my-bucket'
    },
    googleDrive: {
      clientID: 'your-client-id',
      clientSecret: 'your-client-secret',
      refreshToken: 'your-refresh-token',
      rootFolderID: 'optional-root-folder'
    },
    dropbox: {
      accessToken: 'your-access-token',
      refreshToken: 'your-refresh-token',
      clientID: 'your-app-key',
      clientSecret: 'your-app-secret',
      rootPath: '/optional/root'
    },
    box: {
      clientID: 'your-client-id',
      clientSecret: 'your-client-secret',
      accessToken: 'your-access-token',
      refreshToken: 'your-refresh-token',
      enterpriseID: 'your-enterprise-id',
      rootFolderID: '0'
    },
    sharePoint: {
      clientID: 'your-client-id',
      clientSecret: 'your-client-secret',
      tenantID: 'your-tenant-id',
      siteID: 'your-site-id',
      driveID: 'your-drive-id',
      rootFolderID: 'optional-root-folder'
    }
  }
};

Environment Variables

Each provider also supports environment variable configuration. Environment variables take lower priority than config file values (config file wins if both are present).

AWS S3

Variable	Description
STORAGE_AWS_ACCESS_KEY_ID	AWS access key ID
STORAGE_AWS_SECRET_ACCESS_KEY	AWS secret access key
STORAGE_AWS_REGION	AWS region (e.g., us-east-1)
STORAGE_AWS_BUCKET_NAME	S3 bucket name
STORAGE_AWS_KEY_PREFIX	Key prefix (defaults to /)

Azure Blob Storage

Variable	Description
STORAGE_AZURE_ACCOUNT_NAME	Storage account name
STORAGE_AZURE_ACCOUNT_KEY	Storage account key
STORAGE_AZURE_CONNECTION_STRING	Connection string (alternative to name/key)
STORAGE_AZURE_CONTAINER or STORAGE_AZURE_DEFAULT_CONTAINER	Container name

Google Cloud Storage

Variable	Description
STORAGE_GOOGLE_KEY_JSON	JSON string of service account credentials
STORAGE_GOOGLE_CLOUD_KEY_FILENAME	Path to service account key file
STORAGE_GOOGLE_BUCKET_NAME or STORAGE_GOOGLE_CLOUD_DEFAULT_BUCKET	GCS bucket name
STORAGE_GOOGLE_CLOUD_PROJECT_ID	Google Cloud project ID

Google Drive

Variable	Description
STORAGE_GOOGLE_DRIVE_CLIENT_ID	OAuth client ID
STORAGE_GOOGLE_DRIVE_CLIENT_SECRET	OAuth client secret
STORAGE_GOOGLE_DRIVE_REFRESH_TOKEN	OAuth refresh token
STORAGE_GOOGLE_DRIVE_REDIRECT_URI	OAuth redirect URI
STORAGE_GDRIVE_ROOT_FOLDER_ID	Root folder ID (optional)
STORAGE_GDRIVE_KEY_FILE	Service account key file (legacy)
STORAGE_GDRIVE_CREDENTIALS_JSON	Service account credentials JSON (legacy)

SharePoint

Variable	Description
STORAGE_SHAREPOINT_CLIENT_ID	Azure AD client ID
STORAGE_SHAREPOINT_CLIENT_SECRET	Azure AD client secret
STORAGE_SHAREPOINT_TENANT_ID	Azure AD tenant ID
STORAGE_SHAREPOINT_SITE_ID	SharePoint site ID
STORAGE_SHAREPOINT_DRIVE_ID	Document library drive ID
STORAGE_SHAREPOINT_ROOT_FOLDER_ID	Root folder ID (optional)

Dropbox

Variable	Description
STORAGE_DROPBOX_ACCESS_TOKEN	Dropbox access token
STORAGE_DROPBOX_REFRESH_TOKEN	Dropbox refresh token
STORAGE_DROPBOX_CLIENT_ID or STORAGE_DROPBOX_APP_KEY	App key
STORAGE_DROPBOX_CLIENT_SECRET or STORAGE_DROPBOX_APP_SECRET	App secret
STORAGE_DROPBOX_ROOT_PATH	Root path (optional)

Box

Variable	Description
STORAGE_BOX_CLIENT_ID	Box client ID
STORAGE_BOX_CLIENT_SECRET	Box client secret
STORAGE_BOX_ACCESS_TOKEN	Box access token
STORAGE_BOX_REFRESH_TOKEN	Box refresh token
STORAGE_BOX_ENTERPRISE_ID	Box enterprise ID (for JWT auth)
STORAGE_BOX_ROOT_FOLDER_ID	Root folder ID (optional)

API Reference

Core Types

CreatePreAuthUploadUrlPayload

type CreatePreAuthUploadUrlPayload = {
  UploadUrl: string;       // Pre-authenticated URL for upload
  ProviderKey?: string;    // Optional provider-specific key for future reference
};

GetObjectParams / GetObjectMetadataParams

type GetObjectParams = {
  objectId?: string;   // Provider-specific ID (preferred for performance)
  fullPath?: string;   // Full path to the object (fallback)
};

StorageObjectMetadata

type StorageObjectMetadata = {
  name: string;           // Object name (filename)
  path: string;           // Directory path
  fullPath: string;       // Complete path including name
  size: number;           // Size in bytes
  contentType: string;    // MIME type
  lastModified: Date;     // Last modification date
  isDirectory: boolean;   // Whether this is a directory
  etag?: string;          // Entity tag for caching
  cacheControl?: string;  // Cache control directives
  customMetadata?: Record<string, string>;
};

StorageListResult

type StorageListResult = {
  objects: StorageObjectMetadata[];  // Files found
  prefixes: string[];                // Directories found
};

FileSearchOptions

type FileSearchOptions = {
  maxResults?: number;                       // Maximum results (default: 100)
  fileTypes?: string[];                      // Filter by MIME types or extensions
  modifiedAfter?: Date;                      // Only files modified after this date
  modifiedBefore?: Date;                     // Only files modified before this date
  pathPrefix?: string;                       // Search within specific directory
  searchContent?: boolean;                   // Search file contents (default: false)
  providerSpecific?: Record<string, unknown>;// Provider-specific options
};

FileSearchResult

type FileSearchResult = {
  path: string;                              // Full path to file
  name: string;                              // Filename only
  size: number;                              // Size in bytes
  contentType: string;                       // MIME type
  lastModified: Date;                        // Last modification date
  relevance?: number;                        // Relevance score (0.0-1.0)
  excerpt?: string;                          // Text excerpt with match context
  matchInFilename?: boolean;                 // Whether match is in filename
  objectId?: string;                         // Provider-specific ID for direct access
  customMetadata?: Record<string, string>;
  providerData?: Record<string, unknown>;
};

FileSearchResultSet

type FileSearchResultSet = {
  results: FileSearchResult[];    // Array of matching files
  totalMatches?: number;          // Total matches (if available)
  hasMore: boolean;               // More results available?
  nextPageToken?: string;         // Token for next page
};

StorageProviderConfig

interface StorageProviderConfig {
  accountId?: string;        // FileStorageAccount ID (multi-tenant tracking)
  accountName?: string;      // Account display name (logging)
  [key: string]: unknown;   // Provider-specific configuration values
}

UnsupportedOperationError

Custom error class thrown when a provider does not support a specific operation (e.g., SearchFiles on AWS S3).

FileStorageBase Methods

All storage providers implement these methods:

Method	Returns	Description
initialize(config?)	Promise<void>	Initialize the driver. Always call after construction.
CreatePreAuthUploadUrl(objectName)	Promise<CreatePreAuthUploadUrlPayload>	Generate pre-authenticated upload URL
CreatePreAuthDownloadUrl(objectName)	Promise<string>	Generate pre-authenticated download URL
MoveObject(oldName, newName)	Promise<boolean>	Move/rename a file
DeleteObject(objectName)	Promise<boolean>	Delete a file
ListObjects(prefix, delimiter?)	Promise<StorageListResult>	List files and directories
CreateDirectory(directoryPath)	Promise<boolean>	Create a directory
DeleteDirectory(path, recursive?)	Promise<boolean>	Delete a directory
GetObjectMetadata(params)	Promise<StorageObjectMetadata>	Get file metadata without downloading
GetObject(params)	Promise<Buffer>	Download file content
PutObject(name, data, contentType?, metadata?)	Promise<boolean>	Upload file content directly
CopyObject(source, destination)	Promise<boolean>	Copy a file
ObjectExists(objectName)	Promise<boolean>	Check if a file exists
DirectoryExists(directoryPath)	Promise<boolean>	Check if a directory exists
SearchFiles(query, options?)	Promise<FileSearchResultSet>	Search files (throws UnsupportedOperationError if not supported)
get IsConfigured	boolean	Check if driver is properly configured
get AccountId	string | undefined	Get the associated account ID
get AccountName	string | undefined	Get the associated account name

Utility Functions

High-level functions that integrate with MemberJunction's entity system:

Function	Description
createUploadUrl(provider, input, userContext?)	Create pre-authenticated upload URL with automatic MIME type detection
createDownloadUrl(provider, keyOrName, userContext?)	Create pre-authenticated download URL
moveObject(provider, oldKey, newKey, userContext?)	Move a file within a provider
copyObject(provider, source, destination, userContext?)	Copy a file within a provider
deleteObject(provider, keyOrName, userContext?)	Delete a file
listObjects(provider, prefix, delimiter?, userContext?)	List files and directories
copyObjectBetweenProviders(source, dest, sourcePath, destPath, options?)	Transfer file between providers
searchAcrossProviders(providers, query, options?)	Search multiple providers in parallel
searchAcrossAccounts(accounts, query, options)	Search multiple accounts in parallel (enterprise)
initializeDriverWithAccountCredentials(options)	Initialize driver using enterprise credential model
initializeDriverWithUserCredentials(options)	Initialize driver with user context (deprecated)

Configuration Functions

Function	Description
getStorageConfig()	Get full storage configuration (loads from mj.config.cjs on first call)
getStorageProvidersConfig()	Get just the storage providers configuration
getProviderConfig(provider)	Get configuration for a specific provider by key
clearStorageConfig()	Clear cached configuration (useful for testing)

Implementing Additional Providers

The library is designed to be extensible. To add a new storage provider:

1. Create a New Provider Class

import {
  FileStorageBase,
  StorageProviderConfig,
  StorageObjectMetadata,
  StorageListResult,
  CreatePreAuthUploadUrlPayload,
  GetObjectParams,
  GetObjectMetadataParams,
  FileSearchOptions,
  FileSearchResultSet,
} from '@memberjunction/storage';
import { RegisterClass } from '@memberjunction/global';

@RegisterClass(FileStorageBase, 'My Custom Storage')
export class MyCustomStorage extends FileStorageBase {
  protected readonly providerName = 'My Custom Storage';

  private _apiKey: string | undefined;
  private _isConfigured = false;

  constructor() {
    super();
    // Load from environment variables
    this._apiKey = process.env.STORAGE_MYCUSTOM_API_KEY;
  }

  public async initialize(config?: StorageProviderConfig): Promise<void> {
    await super.initialize(config); // Sets accountId and accountName
    if (config) {
      // Override env var defaults with config values if present
      if (config.apiKey) this._apiKey = config.apiKey as string;
    }
    this._isConfigured = !!this._apiKey;
  }

  public get IsConfigured(): boolean {
    return this._isConfigured;
  }

  // Implement all abstract methods...

  public async SearchFiles(query: string, options?: FileSearchOptions): Promise<FileSearchResultSet> {
    // If not supported:
    this.throwUnsupportedOperationError('SearchFiles');
  }
}

2. Export from Index

Add to src/index.ts:

export * from './drivers/MyCustomStorage';

Class Hierarchy

classDiagram
    class FileStorageBase {
        <<abstract>>
        #providerName: string
        #_accountId: string
        #_accountName: string
        +initialize(config?) Promise~void~
        +get IsConfigured() boolean
        +get AccountId() string
        +get AccountName() string
        +CreatePreAuthUploadUrl(name) Promise
        +CreatePreAuthDownloadUrl(name) Promise
        +MoveObject(old, new) Promise
        +DeleteObject(name) Promise
        +ListObjects(prefix, delimiter?) Promise
        +CreateDirectory(path) Promise
        +DeleteDirectory(path, recursive?) Promise
        +GetObjectMetadata(params) Promise
        +GetObject(params) Promise
        +PutObject(name, data, type?, meta?) Promise
        +CopyObject(source, dest) Promise
        +ObjectExists(name) Promise
        +DirectoryExists(path) Promise
        +SearchFiles(query, options?) Promise
        #throwUnsupportedOperationError(method) never
    }

    class AWSFileStorage {
        @RegisterClass 'AWS S3'
    }
    class AzureFileStorage {
        @RegisterClass 'Azure Blob Storage'
    }
    class GoogleFileStorage {
        @RegisterClass 'Google Cloud Storage'
    }
    class GoogleDriveFileStorage {
        @RegisterClass 'Google Drive Storage'
    }
    class SharePointFileStorage {
        @RegisterClass 'SharePoint'
    }
    class DropboxFileStorage {
        @RegisterClass 'Dropbox'
    }
    class BoxFileStorage {
        @RegisterClass 'Box'
    }

    FileStorageBase <|-- AWSFileStorage
    FileStorageBase <|-- AzureFileStorage
    FileStorageBase <|-- GoogleFileStorage
    FileStorageBase <|-- GoogleDriveFileStorage
    FileStorageBase <|-- SharePointFileStorage
    FileStorageBase <|-- DropboxFileStorage
    FileStorageBase <|-- BoxFileStorage

Error Handling

UnsupportedOperationError

Thrown when a provider does not support a specific operation:

try {
  await storage.SearchFiles('quarterly report');
} catch (error) {
  if (error instanceof UnsupportedOperationError) {
    console.log(`Provider doesn't support search: ${error.message}`);
    // Fall back to ListObjects with client-side filtering
  }
}

Provider-Specific Errors

Each provider may throw errors specific to its underlying SDK. These are not wrapped, allowing you to handle provider-specific error conditions:

try {
  await storage.GetObject({ fullPath: 'non-existent-file.txt' });
} catch (error) {
  // Handle provider-specific errors
  if (error.code === 'NoSuchKey') {        // AWS S3
    console.log('File not found');
  } else if (error.code === 'BlobNotFound') { // Azure
    console.log('Blob not found');
  }
}

Testing

The package includes Jest-based unit tests for the base class behavior and the enterprise credential initialization flow.

# Run tests
npm test

# Run tests in watch mode
npm run test:watch

# Run tests with coverage
npm run test:coverage

Test files are located at:

• src/__tests__/FileStorageBase.test.ts -- Tests for the FileStorageBase abstract class, initialize() method, and account information handling
• src/__tests__/util.test.ts -- Tests for initializeDriverWithAccountCredentials and the enterprise credential model

Build

# Build the package
npm run build

# Watch mode for development
npm run watch

The package uses TypeScript with tsc and tsc-alias for path alias resolution. Output is emitted to the dist/ directory.

Best Practices

• Always call initialize(): After creating a provider instance, always call initialize() before using the driver, even if no configuration is needed.
• Use ProviderKey: Always check for and use ProviderKey if returned by CreatePreAuthUploadUrl for subsequent operations.
• Use objectId when available: The GetObject and GetObjectMetadata methods accept either objectId or fullPath. Using objectId bypasses path resolution and is significantly faster.
• Use enterprise credential model: Prefer initializeDriverWithAccountCredentials over direct instantiation for multi-tenant applications.
• Handle unsupported operations: Wrap SearchFiles calls in try/catch for UnsupportedOperationError since not all providers support search.
• Directory paths: Always use trailing slashes for directory paths (e.g., documents/ not documents).
• Content types: Always specify content types for better browser handling and security.
• Buffer operations: GetObject and PutObject load entire files into memory; consider pre-authenticated URLs for large files.
• Batch searches: Use searchAcrossProviders or searchAcrossAccounts for parallel multi-source search instead of sequential calls.

License

ISC