@aws-sdk/lib-storage

What is @aws-sdk/lib-storage?

The @aws-sdk/lib-storage package is part of the AWS SDK for JavaScript (v3) and provides high-level abstractions for uploading and downloading large objects to and from Amazon S3. It simplifies the process of handling multipart uploads and downloads, making it easier to work with large files or streams in a scalable and efficient manner.

What are @aws-sdk/lib-storage's main functionalities?

Multipart Upload

This feature allows for the uploading of large files to Amazon S3 in parts. The code sample demonstrates how to create a multipart upload using the Upload class. It also shows how to listen for upload progress events.

const { S3Client } = require('@aws-sdk/client-s3');
const { Upload } = require('@aws-sdk/lib-storage');
const client = new S3Client({});
const upload = new Upload({
  client,
  params: { Bucket: 'bucket-name', Key: 'file-key', Body: fileStream }
});
upload.on('httpUploadProgress', (progress) => {
  console.log(progress);
});
await upload.done();

Parallel Uploads

This feature enhances the multipart upload by allowing multiple parts of the file to be uploaded in parallel, which can significantly speed up the upload process for large files. The code sample demonstrates how to specify the number of parts to upload in parallel.

const { S3Client } = require('@aws-sdk/client-s3');
const { Upload } = require('@aws-sdk/lib-storage');
const client = new S3Client({});
const upload = new Upload({
  client,
  params: { Bucket: 'bucket-name', Key: 'file-key', Body: fileStream },
  queueSize: 4 // Number of parts to upload in parallel
});
await upload.done();

Multipart Download

While the primary focus of @aws-sdk/lib-storage is on uploading, it provides the foundation for handling large objects that could be extended for multipart downloads, though as of the last update, explicit multipart download functionality is not directly exposed.

N/A

Other packages similar to @aws-sdk/lib-storage

aws-sdk

The older version of the AWS SDK for JavaScript. It also supports uploading and downloading files to/from S3, but @aws-sdk/lib-storage offers a more modular and efficient approach, especially for multipart uploads.

s3-upload-stream

A package that was designed to support streaming uploads to S3. While it provides similar functionality for multipart uploads, @aws-sdk/lib-storage benefits from being part of the official AWS SDK, ensuring better integration and support.

multer-s3

A package that integrates with Multer to upload files to Amazon S3. It's more focused on handling file uploads in web applications, particularly with Express.js. Unlike @aws-sdk/lib-storage, it's not designed for general-purpose S3 interactions and lacks the comprehensive support for multipart uploads.

README

@aws-sdk/lib-storage

Upload

Upload allows for easy and efficient uploading of buffers, blobs, or streams, using a configurable amount of concurrency to perform multipart uploads where possible. This abstraction enables uploading large files or streams of unknown size due to the use of multipart uploads under the hood.

import { Upload } from "@aws-sdk/lib-storage";
import { S3Client, S3 } from "@aws-sdk/client-s3";

try {
  const parallelUploads3 = new Upload({
    client: new S3({}) || new S3Client({}),
    params: { Bucket, Key, Body },

    // optional tags
    tags: [
      /*...*/
    ],

    // additional optional fields show default values below:

    // (optional) concurrency configuration
    queueSize: 4,

    // (optional) size of each part, in bytes, at least 5MB
    partSize: 1024 * 1024 * 5,

    // (optional) when true, do not automatically call AbortMultipartUpload when
    // a multipart upload fails to complete. You should then manually handle
    // the leftover parts.
    leavePartsOnError: false,
  });

  parallelUploads3.on("httpUploadProgress", (progress) => {
    console.log(progress);
  });

  await parallelUploads3.done();
} catch (e) {
  console.log(e);
}

Changelog

3.717.0 (2024-12-20)

Features

• client-bedrock-agent-runtime: bedrock agents now supports long term memory and performance configs. Invokeflow supports performance configs. RetrieveAndGenerate performance configs (8cc6104)
• client-bedrock-agent: Support for custom user agent and max web pages crawled for web connector. Support app only credentials for SharePoint connector. Increase agents memory duration limit to 365 days. Support to specify max number of session summaries to include in agent invocation context. (808e71b)
• client-billing: Added new API's for defining and fetching Billing Views. (3398a2a)
• client-connect: This release supports adding NotAttributeCondition and Range to the RoutingCriteria object. (329a39d)
• client-cost-explorer: Support for retrieving cost, usage, and forecast for billing view. (e4e1078)
• client-docdb: Support AWS Secret Manager managed password for AWS DocumentDB instance-based cluster. (ba90110)
• client-eks: This release expands the catalog of upgrade insight checks (a4d9117)
• client-macie2: This release adds support for identifying S3 general purpose buckets that exceed the Amazon Macie quota for preventative control monitoring. (5f8c6de)
• client-outposts: Add CS8365C as a supported power connector for Outpost sites. (fb7afd4)
• client-sagemaker: This release adds support for c6i, m6i and r6i instance on SageMaker Hyperpod and trn1 instances in batch (47d77a9)
• clients: update client endpoints as of 2024-12-20 (2330ab1)