multer-s3

What is multer-s3?

The multer-s3 package is a middleware for handling multipart/form-data, which is primarily used for uploading files. It integrates with AWS S3 to store files directly in an S3 bucket.

What are multer-s3's main functionalities?

Basic File Upload

This code demonstrates a basic file upload to an S3 bucket using multer-s3. It configures AWS SDK with credentials, sets up multer to use S3 as the storage engine, and defines an endpoint to handle file uploads.

const multer = require('multer');
const multerS3 = require('multer-s3');
const AWS = require('aws-sdk');

AWS.config.update({
  accessKeyId: 'your-access-key-id',
  secretAccessKey: 'your-secret-access-key',
  region: 'your-region'
});

const s3 = new AWS.S3();

const upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'your-bucket-name',
    key: function (req, file, cb) {
      cb(null, Date.now().toString() + '-' + file.originalname);
    }
  })
});

app.post('/upload', upload.single('file'), (req, res) => {
  res.send('File uploaded successfully!');
});

Custom File Naming

This code sample shows how to customize the file naming convention when uploading to S3. The file name is prefixed with the user's ID and a timestamp.

const upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'your-bucket-name',
    key: function (req, file, cb) {
      const fileName = `${req.user.id}/${Date.now().toString()}-${file.originalname}`;
      cb(null, fileName);
    }
  })
});

Setting Metadata

This code demonstrates how to set metadata for the files being uploaded to S3. The metadata function allows you to add custom metadata to the uploaded files.

const upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'your-bucket-name',
    metadata: function (req, file, cb) {
      cb(null, { fieldName: file.fieldname });
    },
    key: function (req, file, cb) {
      cb(null, Date.now().toString() + '-' + file.originalname);
    }
  })
});

Other packages similar to multer-s3

aws-sdk

The aws-sdk package is the official AWS SDK for JavaScript. It provides a comprehensive set of tools for interacting with AWS services, including S3. While it does not provide middleware for handling file uploads like multer-s3, it can be used in conjunction with other middleware to achieve similar functionality.

s3-upload-stream

The s3-upload-stream package allows you to stream data directly to an S3 bucket. It is useful for handling large files or streaming data in real-time. Unlike multer-s3, it does not handle multipart/form-data parsing, so it needs to be used with other middleware for file uploads.

s3fs

The s3fs package provides a file system-like interface for interacting with S3. It allows you to perform file operations such as read, write, and delete on S3 objects. While it does not handle file uploads directly, it can be used to manage files stored in S3.

README

Multer S3

Streaming multer storage engine for AWS S3.

This project is mostly an integration piece for existing code samples from Multer's storage engine documentation with a call to S3 as the substitution piece for file system. Existing solutions I found required buffering the multipart uploads into the actual filesystem which is difficult to scale.

AWS SDK Versions

3.x.x releases of multer-s3 use AWS JavaScript SDK v3. Specifically, it uses the Upload class from @aws-sdk/lib-storage which in turn calls the modular S3Client.

2.x.x releases for multer-s3 use AWS JavaScript SDK v2 via a call to s3.upload.

Installation

npm install --save multer-s3

Usage

const { S3Client } = require('@aws-sdk/client-s3')
const express = require('express')
const multer = require('multer')
const multerS3 = require('multer-s3')

const app = express()

const s3 = new S3Client()

const upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    metadata: function (req, file, cb) {
      cb(null, {fieldName: file.fieldname});
    },
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

app.post('/upload', upload.array('photos', 3), function(req, res, next) {
  res.send('Successfully uploaded ' + req.files.length + ' files!')
})

File information

Each file contains the following information exposed by multer-s3:

Key	Description	Note
size	Size of the file in bytes
bucket	The bucket used to store the file	S3Storage
key	The name of the file	S3Storage
acl	Access control for the file	S3Storage
contentType	The mimetype used to upload the file	S3Storage
metadata	The metadata object to be sent to S3	S3Storage
location	The S3 url to access the file	S3Storage
etag	The etagof the uploaded file in S3	S3Storage
contentDisposition	The contentDisposition used to upload the file	S3Storage
storageClass	The storageClass to be used for the uploaded file in S3	S3Storage
versionId	The versionId is an optional param returned by S3 for versioned buckets.	S3Storage
contentEncoding	The contentEncoding used to upload the file	S3Storage

Setting ACL

ACL values can be set by passing an optional acl parameter into the multerS3 object.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    acl: 'public-read',
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

Available options for canned ACL.

ACL Option	Permissions added to ACL
private	Owner gets FULL_CONTROL. No one else has access rights (default).
public-read	Owner gets FULL_CONTROL. The AllUsers group gets READ access.
public-read-write	Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. Granting this on a bucket is generally not recommended.
aws-exec-read	Owner gets FULL_CONTROL. Amazon EC2 gets READ access to GET an Amazon Machine Image (AMI) bundle from Amazon S3.
authenticated-read	Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
bucket-owner-read	Object owner gets FULL_CONTROL. Bucket owner gets READ access. If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
bucket-owner-full-control	Both the object owner and the bucket owner get FULL_CONTROL over the object. If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
log-delivery-write	The LogDelivery group gets WRITE and READ_ACP permissions on the bucket. For more information on logs.

Setting Metadata

The metadata option is a callback that accepts the request and file, and returns a metadata object to be saved to S3.

Here is an example that stores all fields in the request body as metadata, and uses an id param as the key:

var opts = {
    s3: s3,
    bucket: config.originalsBucket,
    metadata: function (req, file, cb) {
      cb(null, Object.assign({}, req.body));
    },
    key: function (req, file, cb) {
      cb(null, req.params.id + ".jpg");
    }
  };

Setting Cache-Control header

The optional cacheControl option sets the Cache-Control HTTP header that will be sent if you're serving the files directly from S3. You can pass either a string or a function that returns a string.

Here is an example that will tell browsers and CDNs to cache the file for one year:

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    cacheControl: 'max-age=31536000',
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

Setting Custom Content-Type

The optional contentType option can be used to set Content/mime type of the file. By default the content type is set to application/octet-stream. If you want multer-s3 to automatically find the content-type of the file, use the multerS3.AUTO_CONTENT_TYPE constant. Here is an example that will detect the content type of the file being uploaded.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    contentType: multerS3.AUTO_CONTENT_TYPE,
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

You may also use a function as the contentType, which should be of the form function(req, file, cb).

Setting StorageClass

storageClass values can be set by passing an optional storageClass parameter into the multerS3 object.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    acl: 'public-read',
    storageClass: 'REDUCED_REDUNDANCY',
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

Setting Content-Disposition

The optional contentDisposition option can be used to set the Content-Disposition header for the uploaded file. By default, the contentDisposition isn't forwarded. As an example below, using the value attachment forces the browser to download the uploaded file instead of trying to open it.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    acl: 'public-read',
    contentDisposition: 'attachment',
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

Using Server-Side Encryption

An overview of S3's server-side encryption can be found in the [S3 Docs] (http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html); be advised that customer-managed keys (SSE-C) is not implemented at this time.

You may use the S3 server-side encryption functionality via the optional serverSideEncryption and sseKmsKeyId parameters. Full documentation of these parameters in relation to the S3 API can be found [here] (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property) and [here] (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html).

serverSideEncryption has two valid values: 'AES256' and 'aws:kms'. 'AES256' utilizes the S3-managed key system, while 'aws:kms' utilizes the AWS KMS system and accepts the optional sseKmsKeyId parameter to specify the key ID of the key you wish to use. Leaving sseKmsKeyId blank when 'aws:kms' is specified will use the default KMS key. Note: You must instantiate the S3 instance with signatureVersion: 'v4' in order to use KMS-managed keys [[Docs]] (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version), and the specified key must be in the same AWS region as the S3 bucket used.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    acl: 'authenticated-read',
    contentDisposition: 'attachment',
    serverSideEncryption: 'AES256',
    key: function(req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

Setting Content-Encoding

The optional contentEncoding option can be used to set the Content-Encoding header for the uploaded file. By default, the contentEncoding isn't forwarded. As an example below, using the value gzip, a file can be uploaded as a gzip file - and when it is downloaded, the browser will uncompress it automatically.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: 'some-bucket',
    acl: 'public-read',
    contentEncoding: 'gzip',
    key: function (req, file, cb) {
      cb(null, Date.now().toString())
    }
  })
})

You may also use a function as the contentEncoding, which should be of the form function(req, file, cb).

Testing

The tests mock all access to S3 and can be run completely offline.

npm test