REST-based file store -- StoreClient provides CRUD operations (has, get, put, remove, list) via proxy endpoints
Flat-queue download orchestration -- DownloadManager with per-file and global concurrency limits
Intelligent chunking -- files > 100MB split into 50MB chunks with prioritized headers and trailers for fast video start
Download resume -- partial downloads resume automatically; expired URLs defer to next collection cycle
Stale media detection -- CacheAnalyzer identifies orphaned files and evicts when storage exceeds threshold
Widget HTML preprocessing -- base tag injection, dependency rewriting for iframe sandboxing

No Cache API is used. All content is stored on the filesystem via the proxy's ContentStore.

Architecture

CMS (RequiredFiles: size, URL, MD5)
        |
        v
+-------------------------------------+
|      DownloadManager (Facade)       |
|  +- enqueue(fileInfo)               |
|  +- prioritizeLayoutFiles(mediaIds) |
|  +- urgentChunk(type, id, idx)      |
+-------------------------------------+
             |
             v
      DownloadQueue (Flat)
    +-----------------------------+
    | [Task, Task, BARRIER,       |
    |  Task, Task, Task]          |
    |                             |
    | Global concurrency: 6       |
    | Per-file chunks: 2-3        |
    +-----------------------------+
       |
       +-> DownloadTask (chunk-0, high priority)
       +-> DownloadTask (chunk-last, high priority)
       +-> BARRIER (gate: waits for above to finish)
       +-> DownloadTask (bulk chunks, normal priority)
               |
               v
         HTTP Range GET
               |
               v
       ContentStore (proxy)
         /store/media/{id}
         /store/layout/{id}
         /store/widget/{L}/{R}/{M}

Installation

npm install @xiboplayer/cache

Usage

StoreClient -- direct REST access to ContentStore

import { StoreClient } from '@xiboplayer/cache';

const store = new StoreClient();

// Check existence
const exists = await store.has('media', '123');

// Retrieve file
const blob = await store.get('media', '456');

// Store widget HTML
const html = new Blob(['<h1>Widget</h1>'], { type: 'text/html' });
await store.put('widget', 'layout/1/region/2/media/3', html, 'text/html');

// List all cached files
const files = await store.list();

// Delete orphaned files
await store.remove([
  { type: 'media', id: '999' },
  { type: 'media', id: '1000' },
]);

DownloadManager -- orchestrated downloads

import { DownloadManager } from '@xiboplayer/cache';

const dm = new DownloadManager({
  concurrency: 6,
  chunkSize: 50 * 1024 * 1024,
  chunksPerFile: 2,
});

// Enqueue a file
const media = dm.enqueue({
  id: '123',
  type: 'media',
  path: 'https://cdn.example.com/video.mp4',
  size: 250 * 1024 * 1024,
  md5: 'abc123def456',
});

// Wait for completion
const blob = await media.wait();

// Get progress
const progress = dm.getProgress();
// { 'media/123': { downloaded: 50M, total: 250M, percent: '20.0', state: 'downloading' } }

Prioritization and emergency chunks

// Boost files needed by current layout
dm.prioritizeLayoutFiles(['123', '456']);

// Emergency: chunk needed for video playback is stalled
// Moves to front of queue, reduces global concurrency to 2
dm.urgentChunk('media', '789', 3);

CacheAnalyzer -- storage health

import { CacheAnalyzer } from '@xiboplayer/cache';

const analyzer = new CacheAnalyzer(store, { threshold: 80 });

const report = await analyzer.analyze(requiredFiles);
console.log(`Storage: ${report.storage.percent}%`);
console.log(`Orphaned: ${report.files.orphaned} files`);
console.log(`Evicted: ${report.evicted.length} files`);

Download Pipeline

Files flow through stages:

Enqueueing -- deduplication via type/id key, URL refresh if new URL has longer expiry
Preparation -- HEAD request determines chunking (> 100MB = chunked)
Task creation -- chunk-0 and chunk-last get high priority (video headers/trailers); BARRIER separates from bulk chunks
Processing -- concurrency-aware: 6 global connections, 2-3 per file
Execution -- HTTP Range GET with retries and exponential backoff
Storage -- proxy ContentStore saves chunks to filesystem
Assembly -- chunks concatenated; progressive callback enables playback before full download

Chunk strategy

Default chunk size: 50MB
Threshold: files > 100MB get chunked
Header+trailer first: MP4 moov atom in chunk-0 and chunk-last allows instant playback start
Barriers: critical chunks download before bulk chunks begin
Resume: cached chunks tracked in skipChunks Set; only missing chunks downloaded
Expired URLs: deferred (not failed) -- next collection provides fresh URL

Retry strategy by type

Type	Max retries	Backoff	Notes
media	3	500ms	Large, cacheable files
layout	3	500ms	Layout XML, stable URL
dataset	4	15s, 30s, 60s, 120s	Re-enqueues 5x for "cache not ready"
static	3	500ms	Widget dependencies (CSS, fonts)

API Reference

StoreClient

Method	Signature	Returns	Description
`has()`	`has(type, id)`	`Promise<boolean>`	Check if file exists
`get()`	`get(type, id)`	`Promise<Blob \| null>`	Retrieve file as Blob
`put()`	`put(type, id, body, contentType?)`	`Promise<boolean>`	Store file
`remove()`	`remove(files)`	`Promise<{deleted, total}>`	Delete files
`list()`	`list()`	`Promise<Array>`	List all cached files

DownloadManager

new DownloadManager({ concurrency?, chunkSize?, chunksPerFile? })

Method	Signature	Returns	Description
`enqueue()`	`enqueue(fileInfo)`	`FileDownload`	Add file to download queue
`getTask()`	`getTask(key)`	`FileDownload \| null`	Get task by "type/id" key
`getProgress()`	`getProgress()`	`Record<string, Progress>`	All in-progress downloads
`prioritizeLayoutFiles()`	`prioritizeLayoutFiles(mediaIds)`	`void`	Boost priority for layout files
`urgentChunk()`	`urgentChunk(type, id, chunkIndex)`	`boolean`	Move chunk to front, reduce concurrency
`clear()`	`clear()`	`void`	Cancel all, clear queue

CacheAnalyzer

new CacheAnalyzer(storeClient, { threshold: 80 })

Method	Signature	Returns	Description
`analyze()`	`analyze(requiredFiles)`	`Promise<Report>`	Compare cache vs required, evict if needed

Report:

{
  storage: { usage, quota, percent },
  files: { required, orphaned, total },
  orphaned: [{ id, type, size, cachedAt }],
  evicted: [{ id, type, size }],
  threshold: 80
}

CacheManager

Method	Signature	Returns	Description
`getCacheKey()`	`getCacheKey(type, id)`	`string`	Get cache key path
`addDependant()`	`addDependant(mediaId, layoutId)`	`void`	Track layout -> media reference
`removeLayoutDependants()`	`removeLayoutDependants(layoutId)`	`string[]`	Remove layout, return orphaned media IDs
`isMediaReferenced()`	`isMediaReferenced(mediaId)`	`boolean`	Check if media is still used