s3-sync-client - npm Package Compare versions

Comparing version 3.0.3 to 4.0.0

package.json

 {
   "name": "s3-sync-client",
-  "version": "3.0.3",
+  "version": "4.0.0",
   "description": "AWS CLI s3 sync for Node.js provides a modern client to perform S3 sync operations between file systems and S3 buckets in the spirit of the official AWS CLI command",
-  "main": "lib/index.js",
+  "keywords": [
+    "aws",
+    "aws-cli",
+    "aws-cli-sync",
+    "aws-s3",
+    "aws-s3-sync",
+    "aws-sdk",
+    "aws-sync",
+    "s3-bucket",
+    "s3-sync",
+    "s3-syncing",
+    "sync"
+  ],
+  "main": "dist/index.js",
+  "engines": {
+    "node": ">=16.0.0"
+  },
   "files": [
-    "/lib"
+    "/dist/**/*"
   ],
   "scripts": {
-    "test": "jest --runInBand"
+    "build": "tsc -p tsconfig.build.json",
+    "watch": "tsc -w",
+    "test": "ts-node -T ./test/S3SyncClient.test.ts",
+    "docgen": "typedoc --plugin typedoc-plugin-markdown --out docs src/index.ts"
   },
@@ -23,14 +42,23 @@ "repository": {
   "peerDependencies": {
-    "@aws-sdk/abort-controller": "^3.0.0",
-    "@aws-sdk/client-s3": "^3.0.0"
+    "@aws-sdk/abort-controller": "^3.x.x",
+    "@aws-sdk/client-s3": "^3.x.x"
   },
   "devDependencies": {
-    "@aws-sdk/abort-controller": "^3.110.0",
-    "@aws-sdk/client-s3": "^3.113.0",
-    "eslint": "^8.18.0",
+    "@aws-sdk/abort-controller": "^3.329.0",
+    "@aws-sdk/client-s3": "^3.332.0",
+    "@typescript-eslint/eslint-plugin": "^5.59.6",
+    "@typescript-eslint/parser": "^5.59.6",
+    "eslint": "^8.40.0",
     "eslint-config-airbnb-base": "^15.0.0",
-    "eslint-plugin-import": "^2.26.0",
-    "jest": "^28.1.1",
-    "tar": "^6.1.11"
+    "eslint-config-airbnb-typescript": "^17.0.0",
+    "eslint-config-prettier": "^8.8.0",
+    "eslint-plugin-import": "^2.27.5",
+    "eslint-plugin-prettier": "^4.2.1",
+    "prettier": "^2.8.8",
+    "tar": "^6.1.14",
+    "ts-node": "^10.9.1",
+    "typedoc": "^0.24.7",
+    "typedoc-plugin-markdown": "^3.15.3",
+    "typescript": "^5.0.4"
   }
 }

README.md

 # AWS CLI s3 sync for Node.js
 
-![test](https://github.com/jeanbmar/s3-sync-client/actions/workflows/test.yml/badge.svg)
+![npm](https://img.shields.io/npm/v/s3-sync-client.svg) ![downloads](https://img.shields.io/npm/dm/s3-sync-client.svg)
 
-**AWS CLI ``s3 sync`` for Node.js** provides a modern client to perform S3 sync operations between file systems and S3 buckets in the spirit of the official [AWS CLI command](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html).    
+**AWS CLI ``s3 sync`` for Node.js** is a modern TypeScript client to perform S3 sync operations between file systems and S3 buckets, in the spirit of the official [AWS CLI command](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html).    
 AWS CLI installation is **NOT** required by this module.
@@ -15,4 +15,4 @@
 - Support AWS CLI options `--delete`, `--dryrun`, `--size-only`, `--include` and `--exclude`
-- Support AWS SDK command input options
-- Monitor object sync progress
+- Support AWS SDK native command input options
+- Monitor sync progress
 - Sync **any** number of objects (no 1000 objects limit)
@@ -24,7 +24,7 @@ - Transfer objects concurrently
 
-1. There is no way to achieve S3 sync using the AWS SDK for JavaScript v3 alone.
-1. AWS CLI installation is **NOT** required.
-1. The package contains no external dependency.
-1. The AWS SDK peer dependency is up-to-date ([AWS SDK for JavaScript v3](https://github.com/aws/aws-sdk-js-v3)).
-1. The module overcomes a set of common limitations listed at the bottom of this README.
+1. There is [no way](https://github.com/aws/aws-sdk-js-v3/issues/2096#issuecomment-858098764) to achieve S3 sync using the AWS SDK for JavaScript v3 alone
+1. AWS CLI installation is **NOT** required
+1. The module contains no external dependency
+1. The AWS SDK peer dependency is up-to-date ([AWS SDK for JavaScript v3](https://github.com/aws/aws-sdk-js-v3))
+1. The module overcomes a set of common limitations listed at the bottom of this README
 
@@ -35,3 +35,3 @@ # Table of Contents
     1. [Install](#install)
-    2. [Code Examples](#code-examples)
+    2. [Quick Start](#quick-start)
         1. [Client initialization](#client-initialization)
@@ -41,14 +41,15 @@ 2. [Sync a remote S3 bucket with the local file system](#sync-a-remote-s3-bucket-with-the-local-file-system)
         4. [Sync two remote S3 buckets](#sync-two-remote-s3-buckets)
-        5. [Monitor transfer progress](#monitor-transfer-progress)
-        6. [Use AWS SDK command input options](#use-aws-sdk-command-input-options)
-        7. [Relocate objects during sync](#relocate-objects-during-sync)
-        8. [Filter source files](#filter-source-files)
+        5. [Monitor transfer progress](#monitor-sync-progress)
+        6. [Abort sync](#abort-sync)
+        7. [Use AWS SDK command input options](#use-aws-sdk-command-input-options)
+        8. [Relocate objects during sync](#relocate-objects-during-sync)
+        9. [Filter source files](#filter-source-files)
 1. [API Reference](#api-reference)
     - [Class: S3SyncClient](#class-s3-sync-client)
       - [new S3SyncClient(configuration)](#new-s3-sync-client)
-      - [client.sync(localDir, bucketUri[, options])](#sync-bucket-with-local)
-      - [client.sync(bucketUri, localDir[, options])](#sync-local-with-bucket)
-      - [client.sync(sourceBucketUri, targetBucketUri[, options])](#sync-bucket-with-bucket)
+      - [client.sync(localDir, bucketPrefix[, options])](#sync-bucket-with-local)
+      - [client.sync(bucketPrefix, localDir[, options])](#sync-local-with-bucket)
+      - [client.sync(sourceBucketPrefix, targetBucketPrefix[, options])](#sync-bucket-with-bucket)
 1. [Change Log](#change-log)
-1. [Comparison with other modules](#comparison-with-other-modules)
+1. [Benchmark](#benchmark)
 
@@ -61,3 +62,3 @@ ## Getting Started
 
-### Code Examples
+### Quick Start
 
@@ -69,4 +70,4 @@ #### Client initialization
 ```javascript
-const S3Client = require('@aws-sdk/client-s3');
-const S3SyncClient = require('s3-sync-client');
+import S3Client from '@aws-sdk/client-s3';
+import { S3SyncClient } from 's3-sync-client';
 
@@ -95,4 +96,4 @@ const s3Client = new S3Client({ /* ... */ });
 // aws s3 sync s3://mybucket2 /path/to/local/dir --dryrun
-const syncOps = await sync('s3://mybucket2', '/path/to/local/dir', { dryRun: true });
-console.log(syncOps); // log download and delete operations to perform
+const diff = await sync('s3://mybucket2', '/path/to/local/dir', { dryRun: true });
+console.log(diff); // log operations to perform
 ```
@@ -107,12 +108,9 @@
 
-#### Monitor transfer progress
+#### Monitor sync progress
 
 ```javascript
-const EventEmitter = require('events');
-const { TransferMonitor } = require('s3-sync-client');
+import { TransferMonitor } from 's3-sync-client';
 
 const monitor = new TransferMonitor();
 monitor.on('progress', (progress) => console.log(progress));
-setTimeout(() => monitor.abort(), 30000); // optional abort
-
 await sync('s3://mybucket', '/path/to/local/dir', { monitor });
@@ -127,3 +125,2 @@
 ...
-and aborts unfinished sync after 30s (promise rejected with an AbortError) 
 */
@@ -140,10 +137,22 @@
 
+#### Abort sync
+
+```javascript
+import { AbortController } from '@aws-sdk/abort-controller';
+import { TransferMonitor } from 's3-sync-client';
+
+const abortController = new AbortController();
+setTimeout(() => abortController.abort(), 30000);
+
+await sync('s3://mybucket', '/path/to/local/dir', { abortSignal: abortController.signal });
+```
+
 #### Use AWS SDK command input options
 
 ```javascript
-const mime = require('mime-types');
+import mime from 'mime-types';
 /*
 commandInput properties can either be:
-  - fixed values
-  - functions, in order to set dynamic values (e.g. using the object key)
+  - a plain object
+  - a function returning a plain object
 */
@@ -160,10 +169,21 @@
 await sync('s3://mybucket1', 's3://mybucket2', {
-    commandInput: {
-        ContentType: (syncCommandInput) => (
-            mime.lookup(syncCommandInput.Key) || 'text/html'
-        ),
-    },
+    commandInput: (input) => ({
+        ContentType: mime.lookup(input.Key) || 'text/html',
+    }),
 });
 ```
 
+#### Filter source files
+
+```javascript
+// aws s3 sync s3://my-source-bucket s3://my-target-bucket --exclude "*" --include "*.txt" --include "flowers/*"
+await sync('s3://my-source-bucket', 's3://my-target-bucket', {
+    filters: [
+        { exclude: () => true }, // exclude everything
+        { include: (key) => key.endsWith('.txt') }, // include .txt files
+        { include: (key) => key.startsWith('flowers/') }, // also include everything inside the flowers folder
+    ],
+});
+```
+
 #### Relocate objects during sync
@@ -189,18 +209,8 @@
 
-#### Filter source files
+Additional examples are available in the repo test directory.
 
-```javascript
-// aws s3 sync s3://my-source-bucket s3://my-target-bucket --exclude "*" --include "*.txt" --include "flowers/*"
-await sync('s3://my-source-bucket', 's3://my-target-bucket', {
-    filters: [
-        { exclude: () => true }, // exclude everything
-        { include: (key) => key.endsWith('.txt') }, // include .txt files
-        { include: (key) => key.startsWith('flowers/') }, // also include everything inside the flowers folder
-    ],
-});
-```
+## API Reference
 
-Additional code examples are available in the test folder.
+A complete API reference is available in the repo docs directory.
 
-## API Reference
 <a name="class-s3-sync-client"></a>
@@ -211,69 +221,72 @@ ### Class: S3SyncClient
 
-- `options` *<Object\>*
+- `options` *<S3SyncClientConfig\>*
   - `client` *<S3Client\>* instance of [AWS SDK S3Client](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html).
 
 <a name="sync-bucket-with-local"></a>
-#### ``client.sync(localDir, bucketUri[, options])``
+#### ``client.sync(localDir, bucketPrefix[, options])``
 
+Sync a remote S3 bucket with the local file system.  
+Similar to AWS CLI ``aws s3 sync localDir bucketPrefix [options]``.
+
 - `localDir` *<string\>* Local directory
-- `bucketUri` *<string\>* Remote bucket name which may contain a prefix appended with a `/` separator 
-- `options` *<Object\>*
-  - `commandInput` *<Object\>* Set any of the SDK [*<PutObjectCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/putobjectcommandinput.html) or [*<CreateMultipartUploadCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/createmultipartuploadcommandinput.html) options to uploads
+- `bucketPrefix` *<string\>* Remote bucket name which may contain a prefix appended with a `/` separator 
+- `options` *<SyncBucketWithLocalOptions\>*
+  - `dryRun` *<boolean\>* Equivalent to CLI ``--dryrun`` option
   - `del` *<boolean\>* Equivalent to CLI ``--delete`` option
-  - `dryRun` *<boolean\>* Equivalent to CLI ``--dryrun`` option
-  - `filters` *<Array\>* [Almost](https://github.com/jeanbmar/s3-sync-client/issues/30) equivalent to CLI ``--exclude`` and ``--include`` options. Filters can be specified using plain objects including either an `include` or `exclude` property. The `include` and `exclude` properties are functions that take an object key and return a boolean.
   - `sizeOnly` *<boolean\>* Equivalent to CLI ``--size-only`` option
-  - `monitor` *<S3SyncClient.TransferMonitor\>*
+  - `relocations` *<Relocation[]\>* Allows uploading objects to remote folders without mirroring the source directory structure. Each relocation should be specified as an *<Array\>* of `[sourcePrefix, targetPrefix]`.
+  - `filters` *<Filter[]\>* [Almost](https://github.com/jeanbmar/s3-sync-client/issues/30) equivalent to CLI ``--exclude`` and ``--include`` options. Filters can be specified using plain objects including either an `include` or `exclude` property. The `include` and `exclude` properties are functions that take an object key and return a boolean.
+  - `abortSignal` *<AbortSignal\>* Allows aborting the sync
+  - `commandInput` *<CommandInput<PutObjectCommandInput\>\> | <CommandInput<CreateMultipartUploadCommandInput\>\>* Set any of the SDK [*<PutObjectCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/putobjectcommandinput.html) or [*<CreateMultipartUploadCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/createmultipartuploadcommandinput.html) options to uploads
+  - `monitor` *<TransferMonitor\>*
     - Attach `progress` event to receive upload progress notifications
-    - Emit `abort` event to stop object uploads immediately
+    - Call `getStatus()` to retrieve progress info on demand
   - `maxConcurrentTransfers` *<number\>* Each upload generates a Promise which is resolved when a local object is written to the S3 bucket. This parameter sets the maximum number of upload promises that might be running concurrently.
   - `partSize` *<number\>* Set the part size in **bytes** for multipart uploads. Default to 5 MB.
-  - `relocations` *<Array\>* Allows uploading objects to remote folders without mirroring the source directory structure. Each relocation should be specified as an *<Array\>* of `[sourcePrefix, targetPrefix]`.
-- Returns: *<Promise\>* Fulfills with an *<Object\>* of sync operations upon success.
+- Returns: *<Promise<SyncBucketWithLocalCommandOutput\>\>* Fulfills with sync operations upon success.
 
-Sync a remote S3 bucket with the local file system.  
-Similar to AWS CLI ``aws s3 sync localDir bucketUri [options]``.
-
 <a name="sync-local-with-bucket"></a>
-#### ``client.sync(bucketUri, localDir[, options])``
+#### ``client.sync(bucketPrefix, localDir[, options])``
 
-- `bucketUri` *<string\>* Remote bucket name which may contain a prefix appended with a ``/`` separator
+Sync the local file system with a remote S3 bucket.  
+Similar to AWS CLI ``aws s3 sync bucketPrefix localDir [options]``.
+
+- `bucketPrefix` *<string\>* Remote bucket name which may contain a prefix appended with a ``/`` separator
 - `localDir` *<string\>* Local directory
-- `options` *<Object\>*
-  - `commandInput` *<Object\>* Set any of the SDK [*<GetObjectCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/getobjectcommandinput.html) options to downloads
+- `options` *<SyncLocalWithBucketOptions\>*
+  - `dryRun` *<boolean\>* Equivalent to CLI ``--dryrun`` option
   - `del` *<boolean\>* Equivalent to CLI ``--delete`` option
-  - `dryRun` *<boolean\>* Equivalent to CLI ``--dryrun`` option
-  - `filters` *<Array\>* [Almost](https://github.com/jeanbmar/s3-sync-client/issues/30) equivalent to CLI ``--exclude`` and ``--include`` options. Filters can be specified using plain objects including either an `include` or `exclude` property. The `include` and `exclude` properties are functions that take an object key and return a boolean.
   - `sizeOnly` *<boolean\>* Equivalent to CLI ``--size-only`` option
-  - `monitor` *<S3SyncClient.TransferMonitor\>*
+  - `relocations` *<Relocation[]\>* Allows downloading objects to local directories without mirroring the source folder structure. Each relocation should be specified as an *<Array\>* of `[sourcePrefix, targetPrefix]`.
+  - `filters` *<Filter[]\>* [Almost](https://github.com/jeanbmar/s3-sync-client/issues/30) equivalent to CLI ``--exclude`` and ``--include`` options. Filters can be specified using plain objects including either an `include` or `exclude` property. The `include` and `exclude` properties are functions that take an object key and return a boolean.
+  - `abortSignal` *<AbortSignal\>* Allows aborting the sync
+  - `commandInput` *<CommandInput<GetObjectCommandInput\>\>* Set any of the SDK [*<GetObjectCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/getobjectcommandinput.html) options to downloads
+  - `monitor` *<TransferMonitor\>*
     - Attach `progress` event to receive download progress notifications
-    - Emit `abort` event to stop object downloads immediately
+    - Call `getStatus()` to retrieve progress info on demand
   - `maxConcurrentTransfers` *<number\>* Each download generates a Promise which is resolved when a remote object is written to the local file system. This parameter sets the maximum number of download promises that might be running concurrently.
-  - `relocations` *<Array\>* Allows downloading objects to local directories without mirroring the source folder structure. Each relocation should be specified as an *<Array\>* of `[sourcePrefix, targetPrefix]`.
-- Returns: *<Promise\>* Fulfills with an *<Object\>* of sync operations upon success.
+- Returns: *<Promise<SyncLocalWithBucketCommandOutput\>\>* Fulfills with sync operations upon success.
 
-Sync the local file system with a remote S3 bucket.  
-Similar to AWS CLI ``aws s3 sync bucketUri localDir [options]``.
-
 <a name="sync-bucket-with-bucket"></a>
-#### ``client.sync(sourceBucketUri, targetBucketUri[, options])``
+#### ``client.sync(sourceBucketPrefix, targetBucketPrefix[, options])``
 
-- `sourceBucketUri` *<string\>* Remote reference bucket name which may contain a prefix appended with a ``/`` separator
-- `targetBucketUri` *<string\>* Remote bucket name to sync which may contain a prefix appended with a ``/`` separator
-- `options` *<Object\>*
-  - `commandInput` *<Object\>* Set any of the SDK [*<CopyObjectCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/copyobjectcommandinput.html) options to copy operations
+Sync two remote S3 buckets.  
+Similar to AWS CLI ``aws s3 sync sourceBucketPrefix targetBucketPrefix [options]``.
+
+- `sourceBucketPrefix` *<string\>* Remote reference bucket name which may contain a prefix appended with a ``/`` separator
+- `targetBucketPrefix` *<string\>* Remote bucket name to sync which may contain a prefix appended with a ``/`` separator
+- `options` *<SyncBucketWithBucketOptions\>*
+  - `dryRun` *<boolean\>* Equivalent to CLI ``--dryrun`` option
   - `del` *<boolean\>* Equivalent to CLI ``--delete`` option
-  - `dryRun` *<boolean\>* Equivalent to CLI ``--dryrun`` option
-  - `filters` *<Array\>* [Almost](https://github.com/jeanbmar/s3-sync-client/issues/30) equivalent to CLI ``--exclude`` and ``--include`` options. Filters can be specified using plain objects including either an `include` or `exclude` property. The `include` and `exclude` properties are functions that take an object key and return a boolean.
   - `sizeOnly` *<boolean\>* Equivalent to CLI ``--size-only`` option
-  - `monitor` *<S3SyncClient.TransferMonitor\>*
+  - `relocations` *<Relocation[]\>* Allows copying objects to remote folders without mirroring the source folder structure. Each relocation should be specified as an *<Array\>* of `[sourcePrefix, targetPrefix]`.
+  - `filters` *<Filter[]\>* [Almost](https://github.com/jeanbmar/s3-sync-client/issues/30) equivalent to CLI ``--exclude`` and ``--include`` options. Filters can be specified using plain objects including either an `include` or `exclude` property. The `include` and `exclude` properties are functions that take an object key and return a boolean.
+  - `abortSignal` *<AbortSignal\>* Allows aborting the sync
+  - `commandInput` *<CommandInput<CopyObjectCommandInput\>\>* Set any of the SDK [*<CopyObjectCommandInput\>*](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/copyobjectcommandinput.html) options to copy operations
+  - `monitor` *<TransferMonitor\>*
     - Attach `progress` event to receive copy progress notifications
-    - Emit `abort` event to stop object copy operations immediately
+    - Call `getStatus()` to retrieve progress info on demand
   - `maxConcurrentTransfers` *<number\>* Each copy generates a Promise which is resolved after the object has been copied. This parameter sets the maximum number of copy promises that might be running concurrently.
-  - `relocations` *<Array\>* Allows copying objects to remote folders without mirroring the source folder structure. Each relocation should be specified as an *<Array\>* of `[sourcePrefix, targetPrefix]`.
-- Returns: *<Promise\>* Fulfills with an *<Object\>* of sync operations upon success.
+- Returns: *<Promise<SyncBucketWithBucketCommandOutput\>\>* Fulfills with sync operations upon success.
 
-Sync two remote S3 buckets.  
-Similar to AWS CLI ``aws s3 sync sourceBucketUri targetBucketUri [options]``.
-
 # Change Log
@@ -283,5 +296,5 @@
 
-# Comparison with other modules
+# Benchmark
 
-**AWS CLI ``s3 sync`` for Node.js** has been developed to solve the S3 syncing limitations of the existing GitHub repo and NPM modules.
+**AWS CLI ``s3 sync`` for Node.js** has been developed to solve the S3 sync limitations of the existing GitHub repo and NPM modules.
 
@@ -288,0 +301,0 @@ Most of the existing repo and NPM modules suffer one or more of the following limitations:

New alerts

Major refactor

Supply chain risk

Package has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.

Found 1 instance in 1 package

New author

Supply chain risk

A new npm collaborator published a version of the package for the first time. New collaborators are usually benign additions to a project, but do indicate a change to the security surface area of a package.

Found 1 instance in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Fixed alerts

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

192281

increased by308.55%

Number of package files

138

increased by411.11%

Lines of code

2747

increased by227.41%

Number of lines in readme file

306

increased by4.44%

Number of low supply chain risk alerts

1

decreased by-83.33%

Worsened metrics

Dev dependency count

16

increased by128.57%

Number of medium supply chain risk alerts

2

increased byInfinity%

No dependency changes