Super simple cache for file metadata, useful for process that work o a given series of files and that only need to repeat the job on the changed ones since the previous run of the process

jaredwray

file-entry-cache

royriojas

  createFromFile: function (filePath, useChecksum) {

    return this.create(fname, dir, useChecksum);

  create: function (cacheId, _path, useChecksum) {

    var flatCache = require('flat-cache');

    var cache = flatCache.load(cacheId, _path);

    var removeNotFoundFiles = function removeNotFoundFiles() {

      const cachedEntries = cache.keys();

      cachedEntries.forEach(function remover(fPath) {

       * the flat cache storage used to persist the metadata of the `files

       * Given a buffer, calculate md5 hash of its content.

       * @param  {Buffer} buffer   buffer to calculate hash on

       * @return {String}          content hash digest

        return crypto.createHash('md5').update(buffer).digest('hex');

       * Return whether or not a file has changed since last time reconcile was called.

       * @param  {String}  file  the filepath to check

       * @return {Boolean}       wheter or not the file has changed

        return this.getFileDescriptor(file).changed;

       * given an array of file paths it return and object with three arrays:

       *  - changedFiles: Files that changed since previous run

       *  - notChangedFiles: Files that haven't change

       *  - notFoundFiles: Files that were not found, probably deleted

       * @param  {Array} files the files to analyze and compare to the previous seen files

       * @return {[type]}       [description]

        me.normalizeEntries(files).forEach(function (entry) {

            res.changedFiles.push(entry.key);

            res.notFoundFiles.push(entry.key);

          res.notChangedFiles.push(entry.key);

      getFileDescriptor: function (file) {

          return { key: file, notFound: true, err: ex };

        var meta = Object.assign(cacheEntry.meta, { hash: hash });

      _getMetaForFileUsingMtimeAndSize: function (cacheEntry) {

        var stat = fs.statSync(cacheEntry.key);

        var meta = Object.assign(cacheEntry.meta, {

       * Sync the files and persist them to the cache

        noPrune = typeof noPrune === 'undefined' ? true : noPrune;

        keys.forEach(function (entryName) {

          var cacheEntry = entries[entryName];

              ? me._getMetaForFileUsingCheckSum(cacheEntry)

              : me._getMetaForFileUsingMtimeAndSize(cacheEntry);

            cache.setKey(entryName, meta);

            // if the file does not exists we don't save it

            // other errors are just thrown

  - Delete previous mtime when checksum is used and vice versa - [abcf0f9]( https://github.com/royriojas/file-entry-cache/commit/abcf0f9 ), [Milos Djermanovic](https://github.com/Milos Djermanovic), 19/02/2021 18:19:43

  - Adds travis jobs on ppc64le - [92e4d4a]( https://github.com/royriojas/file-entry-cache/commit/92e4d4a ), [dineshks1](https://github.com/dineshks1), 25/11/2020 04:52:11

  - Align file-entry-cache with latest eslint - [4c6f1fb]( https://github.com/royriojas/file-entry-cache/commit/4c6f1fb ), [Roy Riojas](https://github.com/Roy Riojas), 08/11/2020 02:43:09

  - Upgrade deps - [8ab3257]( https://github.com/royriojas/file-entry-cache/commit/8ab3257 ), [Roy Riojas](https://github.com/Roy Riojas), 08/11/2020 02:41:53

  - updated packages - [3dd4231]( https://github.com/royriojas/file-entry-cache/commit/3dd4231 ), [Roy Riojas](https://github.com/Roy Riojas), 08/11/2020 02:29:37

  - Upgrade flat-cache to version 3 - [d7c60ef]( https://github.com/royriojas/file-entry-cache/commit/d7c60ef ), [Roy Riojas](https://github.com/Roy Riojas), 08/11/2020 01:18:04

  - Fix missing checksum comparison from reconcile since now we use mtime and size by default. - [e858aa9]( https://github.com/royriojas/file-entry-cache/commit/e858aa9 ), [Roy Riojas](https://github.com/Roy Riojas), 04/02/2019 09:30:22

    Old mode using checkSum can still be used by passing the `useCheckSum` parameter to the `create` or `createFromFile` methods.

  - Make checksum comparison optional - [b0f9ae0]( https://github.com/royriojas/file-entry-cache/commit/b0f9ae0 ), [Roy Riojas](https://github.com/Roy Riojas), 03/02/2019 18:17:39

    To determine if a file has changed we were using the checksum in the newer versions, but eslint was relying on the old behavior where we use the mtime and file size to determine if a file changed. That's why we decided to make the checksum check optional.

    // to make the cache use the checkSum check do the following:

    var fCache = fileEntryCache.create(cacheName, dir, useCheckSum); // pass the third parameter as true

    var otherCache = fileEntryCache.createFromFile(cacheName, useCheckSum); // pass the second parameter as true

  - use the same node versions eslint use - [563cfee]( https://github.com/royriojas/file-entry-cache/commit/563cfee ), [Roy Riojas](https://github.com/Roy Riojas), 08/01/2019 20:29:34

  - Remove object-assign dependency. - [d0f598e]( https://github.com/royriojas/file-entry-cache/commit/d0f598e ), [Corey Farrell](https://github.com/Corey Farrell), 08/01/2019 20:09:51

    node.js >=4 is required so object-assign is no longer needed, the native

  - Upgrade flat-cache dep to latest - [078b0df]( https://github.com/royriojas/file-entry-cache/commit/078b0df ), [Roy Riojas](https://github.com/Roy Riojas), 08/01/2019 18:54:40

  - Commit new package-lock.json file - [245fe62]( https://github.com/royriojas/file-entry-cache/commit/245fe62 ), [Roy Riojas](https://github.com/Roy Riojas), 08/01/2019 17:56:21

  - add eslintrc file - [6dd32d8]( https://github.com/royriojas/file-entry-cache/commit/6dd32d8 ), [Roy Riojas](https://github.com/Roy Riojas), 22/08/2018 09:58:17

  - Move variable definition out of else block - [ea05441]( https://github.com/royriojas/file-entry-cache/commit/ea05441 ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 25/04/2017 11:19:00

  - Add script and cmd to test hash/checksum performance - [7f60e0a]( https://github.com/royriojas/file-entry-cache/commit/7f60e0a ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 24/04/2017 14:43:12

  - Calculate md5 hexdigest instead of Adler-32 checksum - [f9e5c69]( https://github.com/royriojas/file-entry-cache/commit/f9e5c69 ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 24/04/2017 14:43:12

  - How to reproduce - [4edc2dc]( https://github.com/royriojas/file-entry-cache/commit/4edc2dc ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 24/04/2017 13:49:32

  - Test handling of removed files - [09d9ec5]( https://github.com/royriojas/file-entry-cache/commit/09d9ec5 ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 19/04/2017 19:51:50

  - Use content checksum instead of mtime and fsize - [343b340]( https://github.com/royriojas/file-entry-cache/commit/343b340 ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 19/04/2017 19:51:47

  - Revert "How to reproduce" - [4b4e54a]( https://github.com/royriojas/file-entry-cache/commit/4b4e54a ), [Zakhar Shapurau](https://github.com/Zakhar Shapurau), 25/04/2017 11:15:36

    This reverts commit 4edc2dcec01574247bfc2e0a2fe26527332b7df3.

  - do not persist and prune removed files from cache. Relates to [#2](https://github.com/royriojas/file-entry-cache/issues/2) - [408374d]( https://github.com/royriojas/file-entry-cache/commit/408374d ), [Roy Riojas](https://github.com/Roy Riojas), 16/08/2016 13:47:58

  - remove older node version - [0a26ac4]( https://github.com/royriojas/file-entry-cache/commit/0a26ac4 ), [Roy Riojas](https://github.com/Roy Riojas), 01/08/2016 04:09:17

  - Add an option to not prune non visited keys. Closes [#2](https://github.com/royriojas/file-entry-cache/issues/2) - [b1a64db]( https://github.com/royriojas/file-entry-cache/commit/b1a64db ), [Roy Riojas](https://github.com/Roy Riojas), 01/08/2016 03:52:12

  - Expose the flat-cache instance - [f34c557]( https://github.com/royriojas/file-entry-cache/commit/f34c557 ), [royriojas](https://github.com/royriojas), 23/09/2015 18:26:33

  "description": "Super simple cache for file metadata, useful for process that work o a given series of files and that only need to repeat the job on the changed ones since the previous run of the process",

  "repository": "royriojas/file-entry-cache",

    "eslint": "eslint --cache --cache-location=node_modules/.cache/ 'cache.js' 'test/**/*.js' 'perf.js'",

    "autofix": "npm run eslint -- --fix",

    "install-hooks": "prepush install && changelogx install-hook && precommit install",

    "changelog": "changelogx -f markdown -o ./changelog.md",

    "do-changelog": "npm run changelog && git add ./changelog.md && git commit -m 'DOC: Generate changelog' --no-verify",

    "post-v": "npm run do-changelog && git push --no-verify && git push --tags --no-verify",

    "bump-major": "npm run pre-v && npm version major -m 'BLD: Release v%s' && npm run post-v",

    "bump-minor": "npm run pre-v && npm version minor -m 'BLD: Release v%s' && npm run post-v",

    "bump-patch": "npm run pre-v && npm version patch -m 'BLD: Release v%s' && npm run post-v",

    "test": "npm run eslint --silent && mocha -R spec test/specs",

    "cover": "istanbul cover test/runner.js html text-summary",

    "watch": "watch-run -i -p 'test/specs/**/*.js' istanbul cover test/runner.js html text-summary"

    "commitURL": "https://github.com/royriojas/file-entry-cache/commit/{0}",

    "authorURL": "https://github.com/{0}",

    "issueIDURL": "https://github.com/royriojas/file-entry-cache/issues/{0}",

		@@ -237,2 +237,4 @@ var path = require('path');
		var meta = Object.assign(cacheEntry.meta, { hash: hash });
		delete meta.size;
		delete meta.mtime;
		return meta;
		@@ -247,2 +249,3 @@ },
		});
		delete meta.hash;
		return meta;
		@@ -249,0 +252,0 @@ },


		# file-entry-cache - Changelog
		## v6.0.1
		- Other changes
		- Delete previous mtime when checksum is used and vice versa - [abcf0f9]( https://github.com/royriojas/file-entry-cache/commit/abcf0f9 ), [Milos Djermanovic](https://github.com/Milos Djermanovic), 19/02/2021 18:19:43


		- Adds travis jobs on ppc64le - [92e4d4a]( https://github.com/royriojas/file-entry-cache/commit/92e4d4a ), [dineshks1](https://github.com/dineshks1), 25/11/2020 04:52:11


		## v6.0.0
		@@ -4,0 +12,0 @@ - Refactoring

		{
		"name": "file-entry-cache",
		"version": "6.0.0",
		"version": "6.0.1",
		"description": "Super simple cache for file metadata, useful for process that work o a given series of files and that only need to repeat the job on the changed ones since the previous run of the process",
		@@ -5,0 +5,0 @@ "repository": "royriojas/file-entry-cache",

Improved metrics