xprezzo-buffer

xprezzo-buffer

Use the new Node.js Buffer APIs (Buffer.from, Buffer.alloc, Buffer.allocUnsafe, Buffer.allocUnsafeSlow) in all versions of Node.js.

Uses the built-in implementation when available. (After Node.js v12)

Modified from safe-buffer

It also expose xprezzo-mixin

Philosophy of Xprezzo

Problems faced:

• Too many requires which creates problem when project grow
• The dependencies update are slow
• Test cases of difficult to design

How Xprezzo try to tackle those problems:

• Useful internal libraries/packages are exposed
• Merge small libraries into a larger one.
• Provide easy to use test framework

install

npm install xprezzo-buffer

usage

The goal of this package is to provide a safe replacement for the node.js Buffer.

It's a drop-in replacement for Buffer. You can use it by adding one require line to the top of your node.js modules:

var Buffer = require('xprezzo-buffer').Buffer

// Existing buffer code will continue to work without issues:

new Buffer('hey', 'utf8')
new Buffer([1, 2, 3], 'utf8')
new Buffer(obj)
new Buffer(16) // create an uninitialized buffer (potentially unsafe)

// But you can use these new explicit APIs to make clear what you want:

Buffer.from('hey', 'utf8') // convert from many types to a Buffer
Buffer.alloc(16) // create a zero-filled buffer (safe)
Buffer.allocUnsafe(16) // create an uninitialized buffer (potentially unsafe)

api

Class Method: Buffer.from(array)

• array {Array}

Allocates a new Buffer using an array of octets.

const buf = Buffer.from([0x62,0x75,0x66,0x66,0x65,0x72]);
  // creates a new Buffer containing ASCII bytes
  // ['b','u','f','f','e','r']

A TypeError will be thrown if array is not an Array.

Class Method: Buffer.from(arrayBuffer[, byteOffset[, length]])

• arrayBuffer {ArrayBuffer} The .buffer property of a TypedArray or a new ArrayBuffer()
• byteOffset {Number} Default: 0
• length {Number} Default: arrayBuffer.length - byteOffset

When passed a reference to the .buffer property of a TypedArray instance, the newly created Buffer will share the same allocated memory as the TypedArray.

const arr = new Uint16Array(2);
arr[0] = 5000;
arr[1] = 4000;

const buf = Buffer.from(arr.buffer); // shares the memory with arr;

console.log(buf);
  // Prints: <Buffer 88 13 a0 0f>

// changing the TypedArray changes the Buffer also
arr[1] = 6000;

console.log(buf);
  // Prints: <Buffer 88 13 70 17>

The optional byteOffset and length arguments specify a memory range within the arrayBuffer that will be shared by the Buffer.

const ab = new ArrayBuffer(10);
const buf = Buffer.from(ab, 0, 2);
console.log(buf.length);
  // Prints: 2

A TypeError will be thrown if arrayBuffer is not an ArrayBuffer.

Class Method: Buffer.from(buffer)

• buffer {Buffer}

Copies the passed buffer data onto a new Buffer instance.

const buf1 = Buffer.from('buffer');
const buf2 = Buffer.from(buf1);

buf1[0] = 0x61;
console.log(buf1.toString());
  // 'auffer'
console.log(buf2.toString());
  // 'buffer' (copy is not changed)

A TypeError will be thrown if buffer is not a Buffer.

Class Method: Buffer.from(str[, encoding])

• str {String} String to encode.
• encoding {String} Encoding to use, Default: 'utf8'

Creates a new Buffer containing the given JavaScript string str. If provided, the encoding parameter identifies the character encoding. If not provided, encoding defaults to 'utf8'.

const buf1 = Buffer.from('this is a tést');
console.log(buf1.toString());
  // prints: this is a tést
console.log(buf1.toString('ascii'));
  // prints: this is a tC)st

const buf2 = Buffer.from('7468697320697320612074c3a97374', 'hex');
console.log(buf2.toString());
  // prints: this is a tést

A TypeError will be thrown if str is not a string.

Class Method: Buffer.alloc(size[, fill[, encoding]])

• size {Number}
• fill {Value} Default: undefined
• encoding {String} Default: utf8

Allocates a new Buffer of size bytes. If fill is undefined, the Buffer will be zero-filled.

const buf = Buffer.alloc(5);
console.log(buf);
  // <Buffer 00 00 00 00 00>

The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a [RangeError][] is thrown. A zero-length Buffer will be created if a size less than or equal to 0 is specified.

If fill is specified, the allocated Buffer will be initialized by calling buf.fill(fill). See [buf.fill()][] for more information.

const buf = Buffer.alloc(5, 'a');
console.log(buf);
  // <Buffer 61 61 61 61 61>

If both fill and encoding are specified, the allocated Buffer will be initialized by calling buf.fill(fill, encoding). For example:

const buf = Buffer.alloc(11, 'aGVsbG8gd29ybGQ=', 'base64');
console.log(buf);
  // <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>

Calling Buffer.alloc(size) can be significantly slower than the alternative Buffer.allocUnsafe(size) but ensures that the newly created Buffer instance contents will never contain sensitive data.

A TypeError will be thrown if size is not a number.

Class Method: Buffer.allocUnsafe(size)

• size {Number}

Allocates a new non-zero-filled Buffer of size bytes. The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a [RangeError][] is thrown. A zero-length Buffer will be created if a size less than or equal to 0 is specified.

The underlying memory for Buffer instances created in this way is not initialized. The contents of the newly created Buffer are unknown and may contain sensitive data. Use [buf.fill(0)][] to initialize such Buffer instances to zeroes.

const buf = Buffer.allocUnsafe(5);
console.log(buf);
  // <Buffer 78 e0 82 02 01>
  // (octets will be different, every time)
buf.fill(0);
console.log(buf);
  // <Buffer 00 00 00 00 00>

A TypeError will be thrown if size is not a number.

Note that the Buffer module pre-allocates an internal Buffer instance of size Buffer.poolSize that is used as a pool for the fast allocation of new Buffer instances created using Buffer.allocUnsafe(size) (and the deprecated new Buffer(size) constructor) only when size is less than or equal to Buffer.poolSize >> 1 (floor of Buffer.poolSize divided by two). The default value of Buffer.poolSize is 8192 but can be modified.

Use of this pre-allocated internal memory pool is a key difference between calling Buffer.alloc(size, fill) vs. Buffer.allocUnsafe(size).fill(fill). Specifically, Buffer.alloc(size, fill) will never use the internal Buffer pool, while Buffer.allocUnsafe(size).fill(fill) will use the internal Buffer pool if size is less than or equal to half Buffer.poolSize. The difference is subtle but can be important when an application requires the additional performance that Buffer.allocUnsafe(size) provides.

Class Method: Buffer.allocUnsafeSlow(size)

• size {Number}

Allocates a new non-zero-filled and non-pooled Buffer of size bytes. The size must be less than or equal to the value of require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is (2^31)-1). Otherwise, a [RangeError][] is thrown. A zero-length Buffer will be created if a size less than or equal to 0 is specified.

The underlying memory for Buffer instances created in this way is not initialized. The contents of the newly created Buffer are unknown and may contain sensitive data. Use [buf.fill(0)][] to initialize such Buffer instances to zeroes.

When using Buffer.allocUnsafe() to allocate new Buffer instances, allocations under 4KB are, by default, sliced from a single pre-allocated Buffer. This allows applications to avoid the garbage collection overhead of creating many individually allocated Buffers. This approach improves both performance and memory usage by eliminating the need to track and cleanup as many Persistent objects.

However, in the case where a developer may need to retain a small chunk of memory from a pool for an indeterminate amount of time, it may be appropriate to create an un-pooled Buffer instance using Buffer.allocUnsafeSlow() then copy out the relevant bits.

// need to keep around a few small chunks of memory
const store = [];

socket.on('readable', () => {
  const data = socket.read();
  // allocate for retained data
  const sb = Buffer.allocUnsafeSlow(10);
  // copy the data into the new allocation
  data.copy(sb, 0, 0, 10);
  store.push(sb);
});

Use of Buffer.allocUnsafeSlow() should be used only as a last resort after a developer has observed undue memory retention in their applications.

A TypeError will be thrown if size is not a number.

People

Xprezzo and related projects are maintained by Cloudgen Wong.

License

MIT