What is file-entry-cache?
The file-entry-cache npm package provides a simple way to cache the metadata of files in a directory. It is primarily used to speed up file processing tasks by caching the results of expensive file operations. This can be particularly useful in tasks like linting, where files that have not changed since the last run can be skipped. The package allows for creating, reading, and updating caches, and it can work with both flat file caches and more complex, structured data.
What are file-entry-cache's main functionalities?
Create and manage a file cache
This feature allows you to create a new cache or load an existing one from disk. You can add or remove files from the cache and save the updated cache back to disk. This is useful for tracking which files have been processed and which have not.
const fileEntryCache = require('file-entry-cache');
const cache = fileEntryCache.create('.myCache');
// To add a file to the cache
cache.addEntry('path/to/file.js');
// To remove a file from the cache
cache.removeEntry('path/to/file.js');
// To save the cache to disk
cache.reconcile();
Check if a file has changed since last cached
This feature allows you to check if a file has been modified since it was last added to the cache. This is particularly useful for tools that perform operations on files and want to skip processing for files that have not changed.
const fileEntryCache = require('file-entry-cache');
const cache = fileEntryCache.create('.myCache');
// Assuming 'path/to/file.js' was previously added to the cache
const fileHasChanged = cache.hasFileChanged('path/to/file.js');
if (fileHasChanged) {
console.log('File has changed since last cache.');
} else {
console.log('File has not changed.');
}
Other packages similar to file-entry-cache
chokidar
Chokidar is a more comprehensive file watching library that provides a high-level API to watch files and directories for changes. It is similar to file-entry-cache in that it can be used to optimize file processing tasks by reacting to changes in real-time, but it operates on a different principle, focusing on event-based file watching rather than caching metadata.
node-persist
node-persist is a local storage library for Node.js, allowing data to be stored and retrieved across sessions. While it is not specifically designed for caching file metadata, it can be used for similar purposes as file-entry-cache by manually managing file metadata as part of the stored data. The main difference is that node-persist offers a broader set of storage capabilities beyond just file metadata.
file-entry-cache
Super simple cache for file metadata, useful for process that work on a given series of files and that only need to repeat the job on the changed ones since the previous run of the process
install
npm i --save file-entry-cache
Usage
The module exposes two functions create
and createFromFile
.
create(cacheName, [directory, useCheckSum, currentWorkingDir])
- cacheName: the name of the cache to be created
- directory: Optional the directory to load the cache from
- usecheckSum: Whether to use md5 checksum to verify if file changed. If false the default will be to use the mtime and size of the file.
- currentWorkingDir: Optional the current working directory to use when resolving relative paths
createFromFile(pathToCache, [useCheckSum, currentWorkingDir])
- pathToCache: the path to the cache file (this combines the cache name and directory)
- useCheckSum: Whether to use md5 checksum to verify if file changed. If false the default will be to use the mtime and size of the file.
- currentWorkingDir: Optional the current working directory to use when resolving relative paths
var fileEntryCache = require('file-entry-cache');
var cache = fileEntryCache.create('testCache');
var files = expand('../fixtures/*.txt');
var oFiles = cache.getUpdatedFiles(files);
cache.reconcile();
var cache2 = fileEntryCache.create('testCache');
var oFiles = cache.getUpdatedFiles(files);
cache.removeEntry('path/to/file');
var oFiles = cache.normalizeEntries(files);
entry = {
key: 'some/name/file', the path to the file
changed: true,
meta: {
size: 3242,
mtime: 231231231,
data: {}
}
}
Motivation for this module
I needed a super simple and dumb in-memory cache with optional disk persistence (write-back cache) in order to make
a script that will beautify files with esformatter
to execute only on the files that were changed since the last run.
In doing so the process of beautifying files was reduced from several seconds to a small fraction of a second.
This module uses flat-cache a super simple key/value
cache storage with
optional file persistance.
The main idea is to read the files when the task begins, apply the transforms required, and if the process succeed,
then store the new state of the files. The next time this module request for getChangedFiles
will return only
the files that were modified. Making the process to end faster.
This module could also be used by processes that modify the files applying a transform, in that case the result of the
transform could be stored in the meta
field, of the entries. Anything added to the meta field will be persisted.
Those processes won't need to call getChangedFiles
they will instead call normalizeEntries
that will return the
entries with a changed
field that can be used to determine if the file was changed or not. If it was not changed
the transformed stored data could be used instead of actually applying the transformation, saving time in case of only
a few files changed.
In the worst case scenario all the files will be processed. In the best case scenario only a few of them will be processed.
Important notes
- The values set on the meta attribute of the entries should be
stringify-able
ones if possible, flat-cache uses circular-json
to try to persist circular structures, but this should be considered experimental. The best results are always obtained with non circular values - All the changes to the cache state are done to memory first and only persisted after reconcile.
License
MIT (c) Jared Wray