s3-readstream

s3-readstream

AWS S3 Read Stream made easy

Simple wrapper around AWS S3 getObject's grab-by-range call allowing intuitive and stable smart streaming.

• Simple interface for streaming any size file from AWS S3
• Easily speed-up, and slow down, the streaming at any point
• All of the functionaly you love with NodeJS Readable streams
• Drop in replacement for AWS.S3.getObject().createReadStream()

Installing the package

To install the package:

npm install s3-readstream

Using the package

You can integrate the S3ReadStream class with the aws-sdk package easily:

import * as AWS from 'aws-sdk';
import {S3ReadStream} from 's3-readstream';
// Pass in your AWS S3 credentials
const s3 = new AWS.S3({
  accessKeyId: s3Env.accessKey,
  secretAccessKey: s3Env.secret
});

const bucketParams = {
  Bucket: s3Env.bucket, // S3 Bucket Path
  Key: s3Env.key // S3 file
};

// Check the headobject like normal to get the length of the file
s3.headObject(bucketParams, (error, data) => {
    const options = {
        parameters: bucketParams,
        s3,
        maxLength: data.ContentLength,
        byteRange: 1024 * 1024 // 1 MiB (optional - defaults to 64kb)
    };
    // Instantiate the S3ReadStream in place of s3.getObject().createReadStream()
    const stream = new S3ReadStream(options);
});

Adjusting the read stream

To adjust the range of bytes grabbed from S3:

// You can adjust the range at any point during the stream (adjusting the speed)
stream.adjustByteRange(1024 * 1024 * 5); // 5 MiB

To adjust cursor position:

// You can move the cursor forwards to skip ahead (or back) in the file
// By default, the stream will skip ahead by the current Range
stream.moveCursorForward();
stream.moveCursorBack();

// Both of these methods also take in a `bytes` parameter for finer control
stream.moveCursorForward(10 * 1024); // Move cursor forward 10 KiB in file
stream.moveCursorBack(5 * 1024); // Move cursor back 5 KiB in file

Inherited features from NodeJS Readable class

You can alse use this S3ReadStream like any other NodeJS Readable stream, setting an event listener is exactly the same:

stream.on('data', (chunk) => {
  console.log(`read: ${chunk.toString()}`);
});
stream.on('end', () => {
  console.log('end');
});

To work with zipped files:

import {createGunzip} from 'zlib';

const gzip = createGunzip();
// pipe into gzip to unzip files as you stream!
stream.pipe(gzip);

API

S3ReadStream(options: S3ReadStreamOptions)

Instantiates a new S3ReadStream object.

Parameters:

• options (S3ReadStreamOptions) - Container object to hold options
  • options.parameters (S3.GetObjectRequest) - Parameters to pass into S3 getObject method call
  • options.s3 (S3) - Resolved S3 object
  • options.maxLength (number) - Total length of file in S3 bucket
  • options.byteRange (number) - (optional) Range of bytes to grab in S3 getObject call (defaults to 64 KiB)
• nodeReadableStreamOptions (ReadableOptions) - (optional) NodeJs Readable options to pass to super call

adjustByteRange(bytes: number)

Adjusts the S3ReadStream._s3DataRange property. Can be used to slow down or speed up the stream by grabbing a smaller or larger range of bytes.

Parameter:

• bytes (number) - New range of bytes to set

moveCursorForward(bytes: number)

Drains the internal buffer and adjusts the S3ReadStream._currentCusorPosition property moving the cursor forward bytes amount. If current cursor position + number of bytes to move forward is > the length of the file, set cursor at end of file

Parameter:

• bytes (number) - (optional) Number of bytes to move forward (defaults to current range)

moveCursorBack(bytes: number)

Drains the internal buffer and adjusts the S3ReadStream._currentCusorPosition property moving the cursor back bytes amount. If current cursor position - number of bytes to move back is <= 0, set cursor at begining of file

Parameter:

• bytes (number) - (optional) Number of bytes to move forward (defaults to current range)

Example

See s3-readstream in action in an HD video streaming app example and read a blog on its origins.