Security News
Cloudflare Adds Security.txt Setup Wizard
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
compress-json
Advanced tools
Store JSON data in space efficient manner.
Inspired by compressed-json and jsonpack.
This library is optimized to compress json object in compact format, which can save network bandwidth and disk space. It is not optimized for writing nor querying throughput. Although the reduced IO may speed up usage of lmdb on frequently redundant data, that is not the design goal.
npm i -S compress-json
import { compress, decompress } from 'compress-json'
let data = {
user: 'Alice',
// more fields of any json values (string, number, array, object, e.t.c.)
}
let compressed = compress(data) // the result is an array
fetch('/submit', {
method: 'post',
body: JSON.stringify(compressed) // convert into string if needed
})
let reversed = decompress(compressed)
data === reversed // will be false
JSON.stringify(data) === JSON.stringify(reversed) // will be true
Sample data:
let longStr = 'A very very long string, that is repeated'
let data = {
int: 42,
float: 12.34,
str: 'Alice',
longStr,
longNum: 9876543210.123455,
bool: true,
bool2: false,
arr: [42, longStr],
arr2: [42, longStr], // identical values will be deduplidated, including array and object
obj: { // nested values are supported
id: 123,
name: 'Alice',
role: [ 'Admin', 'User', 'Guest' ],
longStr: 'A very very long string, that is repeated',
longNum: 9876543210.123455
},
escape: [ 's|str', 'n|123', 'o|1', 'a|1', 'b|T', 'b|F' ]
}
Compressed data:
// [ encoded value array, root value index ]
let compressed = [
[ // encoded value array
'int', // string
'float',
'str',
'longStr',
'longNum',
'bool',
'bool2',
'arr',
'arr2',
'obj',
'escape',
'a|0|1|2|3|4|5|6|7|8|9|A',
'n|g', // number (integer) (base62-encoded)
'n|C.h', // number (float) (integer part and decimals are base62-encoded separately)
'Alice',
'A very very long string, that is repeated',
'n|AmOy42.2KCf',
'b|T', // boolean (true)
'b|F', // boolean (false)
'a|C|F', // array
'id',
'name',
'role',
'a|K|L|M|3|4',
'n|1z',
'Admin',
'User',
'Guest',
'a|P|Q|R',
'o|N|O|E|S|F|G', // object
's|s|str', // escaped string
's|n|123', // escaped number
's|o|1',
's|a|1',
's|b|T', // escaped boolean
's|b|F',
'a|U|V|W|X|Y|Z',
'o|B|C|D|E|F|G|H|I|J|J|T|a'
],
'b' // root value index
]
import { compress } from 'compress-json'
import { trimUndefined, trimUndefinedRecursively } from 'compress-json'
let user = { name: 'Alice', role: undefined }
compress(user) // will throw an error since undefined field is not supported
trimUndefined(user) // explicitly remove undefined fields
compress(user) // now it will not throw error since user.role is deleted
let a = { name: 'a', extra: undefined }
let b = { name: 'b', a }
trimUndefinedRecursively(b)
compress(b) // now it will not throw error since b.a.extra is deleted
Test file: compress-test.ts
Sample data in use: json data of 109,164 threads and 724,905 post crawled from a discuz forum. Truncated in varies of size for testing.
Algorithms in comparison:
JSON.stringify
without indentation)(Binary compression algorithm is not considered)
sample | JSON | compressed-json | jsonpack | compress-json |
---|---|---|---|---|
all | 263M | 199M | - | 176M |
100,000 | 235M | 178M | - | 158M |
50,000 | 70M | 55M | - | 50M |
10,000 | 34M | 26M | - | 23M |
2,000 | 6.6M | 5.0M | 5.3M | 4.4M |
1,000 | 4.8M | 3.7M | 3.8M | 3.3M |
100 | 335K | 265K | 271K | 243K |
10 | 4.0K | 3.3K | 3.0K | 3.2K |
sample | JSON | compressed-json | jsonpack | compress-json |
---|---|---|---|---|
all | 1,654ms | 12,674ms | timeout* | 15,788ms |
100,000 | 1,500ms | 10,921ms | timeout* | 12,715ms |
50,000 | 462ms | 3,047ms | timeout* | 3,935ms |
10,000 | 146ms | 1,278ms | timeout* | 1,733ms |
2,000 | 35ms | 328ms | 21,018ms | 456ms |
1,000 | 20ms | 270ms | 12,960ms | 390ms |
100 | 1ms | 18ms | 47ms | 37ms |
10 | 0.3ms | 1.8ms | 1.6ms | 1.9ms |
*timeout: excess 1 minute
sample | JSON | compressed-json | jsonpack | compress-json |
---|---|---|---|---|
all | 1,908ms | 4,611ms | - | 9,225ms |
100,000 | 1,744ms | 3,740ms | - | 7,576ms |
50,000 | 558ms | 1,066ms | - | 2,452ms |
10,000 | 173ms | 460ms | - | 1,055ms |
2,000 | 47ms | 108ms | 189ms | 317ms |
1,000 | 34ms | 90ms | 160ms | 263ms |
100 | 2ms | 11ms | 16ms | 19ms |
10 | 2.9ms | 2.1ms | 1.1ms | 1.3ms |
BSD 2-Clause License (Free Open Sourced Software)
FAQs
convert JSON data to space efficient format
The npm package compress-json receives a total of 22,794 weekly downloads. As such, compress-json popularity was classified as popular.
We found that compress-json demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
Security News
The Socket Research team breaks down a malicious npm package targeting the legitimate DOMPurify library. It uses obfuscated code to hide that it is exfiltrating browser and crypto wallet data.
Security News
ENISA’s 2024 report highlights the EU’s top cybersecurity threats, including rising DDoS attacks, ransomware, supply chain vulnerabilities, and weaponized AI.