enmap - npm Package Compare versions

Comparing version 6.0.0-alpha-0.0.6 to 6.0.0-alpha-0.0.7

package.json

 {
   "name": "enmap",
-  "version": "6.0.0-alpha-0.0.6",
+  "version": "6.0.0-alpha-0.0.7",
   "description": "A simple database wrapper to make sqlite database interactions much easier for beginners, with additional array helper methods.",
@@ -5,0 +5,0 @@ "main": "src/index.js",

src/index.js

@@ -787,15 +787,22 @@ import {
    */
-  map(pathOrFn) {
-    const stmt = this.#db.prepare(`SELECT key, value FROM ${this.#name}`);
-    const results = [];
-    for (const row of stmt.iterate()) {
-      const parsed = this.#parse(row.value);
-      if (isFunction(pathOrFn)) {
-        results.push(pathOrFn(parsed, row.key));
-      } else {
-        results.push(_get(parsed, pathOrFn));
-      }
+    map(pathOrFn) {
+      this.#db.aggregate('map', {
+        start: [],
+        step: (accumulator, value, key) => {
+          const parsed = this.#parse(value);
+          if (isFunction(pathOrFn)) {
+            accumulator.push(pathOrFn(parsed, key));
+          } else {
+            accumulator.push(_get(parsed, pathOrFn));
+          }
+          return accumulator;
+        },
+        result: (accumulator) => JSON.stringify(accumulator),
+      });
+      const results = this.#db
+        .prepare(`SELECT map(value, key) FROM ${this.#name}`)
+        .pluck()
+        .get();
+      return JSON.parse(results);
     }
-    return results;
-  }
 
@@ -886,6 +893,6 @@ /**
       start: [],
-      step: (accumulator, currentValue) => {
+      step: (accumulator, currentValue, key) => {
         const parsed = this.#parse(currentValue);
         if (isFunction(pathOrFn)) {
-          if (pathOrFn(parsed)) {
+          if (pathOrFn(parsed, key)) {
             accumulator.push(parsed);
@@ -909,3 +916,3 @@ }
     const results = this.#db
-      .prepare(`SELECT filter(value) FROM ${this.#name}`)
+      .prepare(`SELECT filter(value, key) FROM ${this.#name}`)
       .pluck()
@@ -912,0 +919,0 @@ .get();

UpgradeV6.md

@@ -1,6 +0,120 @@
-# Updating from v5 to v6
+# Why update to v6???
 
-Enmap version 6 is a major rewrite and contains multiple changes that must be taken into consideration when updating.
-Though a lot of the API has been tweaked, there should not be a loss of overall capacity, but you'll have some code changes to do.
+So, you might be wondering what the main driver of Enmap Version 6 is. Let me give you a little bit of historical contet here.
 
+Before Enmap, there was [djs-collection-persistent](https://www.npmjs.com/package/djs-collection-persistent). This module was born
+from using discord.js' Collection, and the idea was, "What if I could use that, but save it in a database?" and thus, this whole
+adventure started. It saved data on `leveldb`, and it was alright. But who the hell wants to remember that name?
+
+And thus, Enmap 1.0 was born. The *Enhanced* Map, which detached the name from djs itself. Enmap 1.0 already established Providers,
+including leveldb, postgresql, sqlite, rethinkdb, and mongo.
+
+Enmap 2 and 3 added more features, moved things around, but generally stayed the same. Lot of bug fixes, added math, that sort of thing.
+
+In Enmap 4, the main major change was that I removed all providers. I discovered (better-sqlite3)[https://www.npmjs.com/package/better-sqlite3],
+the first (and only?) synchronous database module. This started changing everything, but I didn't fully trust its performance yet. So Enmap 4
+is sqlite only, persistent, but it still has its cache... that is to say, it's still an enhanced Map structure with persistence. Enmap 5 is
+more of the same, updates, small breaking changes, new features, etc. 
+
+But there's a problem : Enmap... is stil cached. It still uses a lot of memory, and that makes it slower than it should be. better-sqlite3 is *fast*
+and now I'm updating both the cache (Map) and the database! But I left this sleeping for the last few years as I was doing other things with life.
+
+And here we are. Enmap 6.0 just removes caching, and updates all the map/array methods to directly interact with the database, no cache needed. This
+not only simplifies the code, and reduces RAM usage by a *wide* margin, it also makes Enmap **FAST**. I mean... really fast. 
+
+## The SPEED
+
+HOW FAST? Let's take a look!
+
+### Loading
+
+Loading of data remains approximately the same when empty, but can be much faster in Enmap 6 the larger your database is, if `autoFetch` is `true`.
+With the 1 million rows, Enmap 6 loads in 6ms (milliseconds) but Enmap 5 loads in 20s (seconds). That's a massive difference, because of caching.
+
+### Adding Data
+
+This test inserts 1 million rows in a simple for loop. Each entry is an object with multiple randomly generated numbers.
+
+Here's the actual test!
+```js
+const rnum = size => Math.floor(Math.random() * (size + 1));
+
+for (let i = 1; i <= 1_000_000; i++) {
+  enmap.set(`obj${i}`, {
+    key: `obj${i}`,
+    a: rnum(100),
+    b: rnum(100),
+    c: rnum(100),
+    d: rnum(100),
+    e: rnum(100),
+  });
+}
+```
+
+```
+1 million enmap5: 2:57.741 (m:ss.mmm)
+1 million enmap6: 2:44.252 (m:ss.mmm)
+```
+As you can see, the insert time is almost the same. I tried a few times, the time are around 2 minutes 50 seconds, +- 10 seconds.
+The speed does not change if the data already exists since it's all new data anyway (this means "key creation" doesn't cost anything).
+
+### Looping over data
+
+So here's where we have the meat of the magic. The Unicorn steak, if you will. Enmap, looping over data, is up to 10 times faster. 
+
+You've read that right. ***__10 times faster__***. 
+
+Here's the tests and results. I tried more than once, and it's all the same ballpark: 
+
+```js
+console.time('partition enmap');
+const [one, two] = enmap.partition((value) => value.a % 2 === 0);
+console.timeEnd('partition enmap');
+
+console.time('partition enmap6');
+const [one6, two6] = enmap6.partition((value) => value.a % 2 === 0);
+console.timeEnd('partition enmap6');
+```
+
+```diff
+-partition enmap5: 51.221s
++partition enmap6:  6.048s
+```
+
+```js
+console.time('filter enmap');
+const filtered = enmap.filter((value) => value.a % 2 === 0);
+console.timeEnd('filter enmap');
+
+console.time('filter enmap6');
+const filtered6 = enmap6.filter((value) => value.a % 2 === 0);
+console.timeEnd('filter enmap6');
+```
+
+```diff
+- filter enmap5: 28.315s
++ filter enmap6:  5.560s
+```
+
+I **almost** missed the difference in magnitude here: enmap.map() is slower by a lot. 
+I'm not sure why and won't expend more time on this, and I don't feel guilty, because 
+loading the 1M values took 17s for enmap5 versus the 6ms uncached enmap6. Still a clear
+value winner either way.
+
+```js
+console.time('map enmap');
+const mapped = enmap.map((value) => value.a * 2);
+console.timeEnd('map enmap');
+
+console.time('map enmap6');
+const mapped6 = enmap6.map((value) => value.a * 2);
+console.timeEnd('map enmap6');
+```
+
+```diff
+-map enmap5: 47.295ms
++map enmap6:  6.271s
+```
+
+
 ## Move to ESM
@@ -7,0 +121,0 @@

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

183096

increased by2.63%

Lines of code

2225

increased by0.32%

No dependency changes