efrt

compression of key-value data

npm install efrt
(or alternatively:)
npm install efrt-unpack

efrt turns a javascript object into a very-compressed prefix trie format, so that any redundancies in key-value paris are compressed, and nothing is repeated.

it is based on lookups by Mike Koss, tamper by the nyTimes, and bits.js by Steve Hanov

• squeeze a key-value object into a very compact form
• reduce filesize/bandwidth a bunch
• ensure the unpacking overhead is negligible
• word-lookups are critical-path

By doing the fancy stuff ahead-of-time, efrt lets you ship much bigger key-value data to the client-side, without much hassle. The whole library is 8kb, the unpack-half is only 2.5kb.

var efrt = require('efrt')
var data = {
  bedfordshire   : 'England',
  aberdeenshire  : 'Scotland',
  berkshire      : 'England',
  buckinghamshire: 'England',
  argyllshire    : 'Scotland',
  bambridgeshire : 'England',
  angus          : 'Scotland',
  bristol        : 'England',
  cheshire       : 'England',
  ayrshire       : 'Scotland',
  banffshire     : 'Scotland',
  berwickshire   : 'Scotland'
}

//pack these words as tightly as possible
var compressed = efrt.pack(data);
//{"England":"b0che2;ambridge1e0ristol,uckingham1;dford0rk0;shire","Scotland":"a1b0;anff1erwick1;berdeen0ngus,rgyll0yr0;shire"}

//create a lookup-trie
var objAgain = efrt.unpack(compressed);

//hit it!
console.log(objAgain['bedfordshire']);//'England'
console.log(objAgain.hasOwnProperty('miles davis'));//false

Demo!

the keys you input are pretty normalized. Spaces and unicode are good, but numbers, case-sensitivity, and some punctuation (semicolon, comma, exclamation-mark) are not (yet) supported.

an element may have more than one category. It will accept an array of strings, and pack them into multiple tries like this:

var foods = {
  strawberry: 'fruit',
  blueberry: 'fruit',
  blackberry: 'fruit',
  tomato: ['fruit', 'vegetable'],
  cucumber: 'vegetable',
  pepper: 'vegetable'
};
var str = efrt.pack(foods);
//'{"fruit":"bl0straw1tomato;ack0ue0;berry","vegetable":"cucumb0pepp0tomato;er"}'
var obj=efrt.unpack(str)
console.log(obj.tomato)
//['fruit', 'vegetable']

efrt is used in compromise, to greatly expand the amount of word-data it can fit onto the client-side. If you find another use for efrt, please drop us a line🎈

Performance

efrt is tuned to be very quick to unzip. It is O(1) to lookup. Packing-up the data is the slowest part, which is usually cool.

var compressed = efrt.pack(skateboarders);//1k words (on a macbook)
var trie = efrt.unpack(compressed)
// unpacking-step: 5.1ms

trie.hasOwnProperty('tony hawk')
// cached-lookup: 0.02ms

Size

efrt will pack filesize down as much as possible, depending upon the redundancy of the prefixes/suffixes in the words, and the size of the list.

• list of countries - 1.5k -> 0.8k (46% compressed)
• all adverbs in wordnet - 58k -> 24k (58% compressed)
• all adjectives in wordnet - 265k -> 99k (62% compressed)
• all nouns in wordnet - 1,775k -> 692k (61% compressed)

but there are some things to consider:

• bigger files compress further (see 🎈 birthday problem)
• using efrt will reduce gains from gzip compression, which most webservers quietly use
• english is more suffix-redundant than prefix-redundant, so non-english words may benefit from other styles

Assuming your data has a low category-to-data ratio, you will hit-breakeven with at about 250 keys. If your data is in the thousands, you can very be confident about saving your users some considerable bandwidth.

Use

IE9+

<script src="https://unpkg.com/efrt@latest/builds/efrt.min.js"></script>
<script>
  var smaller=efrt.pack(['larry','curly','moe'])
  var trie=efrt.unpack(smaller)
  console.log(trie['moe'])
</script>

if you're doing the second step in the client, you can load just the unpack-half of the library(~3k):

npm install efrt-unpack

<script src="https://unpkg.com/efrt@latest/builds/efrt-unpack.min.js"></script>
<script>
  var trie=unpack(compressedStuff);
  trie.hasOwnProperty('miles davis');
</script>

MIT

Changelog

1.1.0

• adds support for object inputs, instead of just arrays