@openfin/multer-s3

Multer S3

A tool for uploading files to S3 using the express multer module.

Why create this over the other?

We provide a way to alter the data while it is being uploaded. We also have better hooks

Keys

Optional are keys with a ? next to the name. We suggest adding a key, but hey, do you. Otherwise, multer will create a random name for you but will put it in the root of your bucket(as long as you have full access to that bucket).

Some of the properties are promises so you can do things asynchronously. IE, if you want to have a call to get a key name from somewhere else, that is fine. If you accidentally have a promise that never resolves, the file will never be uploaded.

For more info on these properties, check the aws documentation: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html

s3?: AWS.S3; bucket: string | PromisedFn; acl?: string | PromisedFn; key?: PromisedFn; serverSideEncryptionKMSKeyId?: string | PromisedFn; serverSideEncryption?: string | PromisedFn; storageClass?: string | PromisedFn; cacheControl?: string | PromisedFn; contentDisposition?: string | PromisedFn; metadata?: { [key: string]: any } | PromisedFn<{ [key: string]: any }>; contentType?: PromisedFn; serverSideEncryptionCustomerKey?: | string | Buffer | PromisedFn<string | Buffer>; serverSideEncryptionCustomerKeyMD5?: string | PromisedFn; serverSideEncryptionCustomerAlgorithm?: string | PromisedFn; onUpload?: (req: express.Request, storage: S3Storage, file: S3File) => any; streamTransformers?: TransformFuncs[]; detectContentType?: boolean;

Usage

An example of how to use the library.

// Create class to get checksum as it is uploaded
class Chunker extends Stream.Transform {
  public checksum: crypto.Hash;
  public file: S3File;
  constructor(file: S3File) {
    super();
    this.file = file;
    this.checksum = crypto.createHash("sha256");
  }

  _transform(chunk: any, encoding: string, callback: any) {
    this.checksum.update(chunk);
    this.push(chunk);
    callback();
  }

  _flush(callback: any) {
    this.file.checksum = this.checksum.digest("hex");
    callback();
  }
}
const s = new S3Storage({
    s3,
    bucket: "bucketName",
    key: function() {
      return Promise.resolve("andre");
    },
    streamTransformers: [
      async (file: S3File, S3Storage, req) => {
        return new Chunker(file);
      }
    ]
  });
multer({
  storage: s
}).single("file")

Configuring environment variables

MULTER_S3_TIMEOUT: The number of milliseconds to wait for the stream to come in before giving up. The reason we had to add this is because we are trying to hook to the data event when data is coming in. However, when a empty file comes in, no data comes through which results in the stream never starting or resolving which will result in the server timing out. We cannot use the open event since either multer or express is using the open event.