lambda-s3-mv

lambda-s3-mv

npm install lambda-s3-mv --save

Introduction

This nodejs module will move S3 files to another S3 location.

Description

The AWS S3 sdk doesn't come out with a built-in move files operation. This module will handle that for you using the available built-in operations, such copyObject (and deleteObjects if "createCopy" set to false).

This module can move single S3 file, or can move all the files in the specified S3 sourcePath parameter.

/*
 * This nodejs module will move files to another S3 location. It has also an option to just copy the files.
 * @param {options} - parameters, as follows
    {
        sourceBucket: 's3-source-bucket-name',
        sourcePath: 's3-source-prefix-path',
        sourceFiles: ['source-file1.txt', 'source-file2.txt'],
        targetBucket: 's3-target-bucket-name',
        targetPath: 's3-target-prefix-path',
        targetFiles: ['source-file1-moved.txt', 'source-file2-moved.txt'],
        createCopy: false
    }
 * @return {object} - a JSON object containing the details of the files that has been moved or copied.
    {
        s3Bucket: 's3-target-bucket-name',
        s3Objects: [
            '/s3-target-prefix-path/source-file1.txt', 
            '/s3-target-prefix-path/source-file2.txt'
        ]
    }
 */

Options	Required	Description
sourceBucket	REQUIRED	the bucket name containing the source files
sourcePath	REQUIRED	the s3 prefix path containing the source files
sourceFiles	OPTIONAL	If not specified, by default it will include all files in the $sourcePath
targetBucket	OPTIONAL	If not specified, by default the $sourceBucket will be used as the $targetBucket
targetPath	REQUIRED	the s3 prefix path to store the target files
targetFiles	OPTIONAL	the modified name of the files after moved. This should match the number of files in $sourceFiles, else it will fallback to the name in $sourceFiles. If not specified, by default it will use the name of the files in $sourceFiles.
createCopy	OPTIONAL	If not specified, by default it is set to false

Usage

1. Move ALL files in the specified S3 sourcePath into the same S3 bucket.

   const s3mv = require('lambda-s3-mv');
   
   const result = await s3mv.move({
       sourceBucket: 's3-source-bucket-name',
       sourcePath: 's3-source-prefix-path',
       targetPath: 's3-target-prefix-path'
   });
   console.log(result);
   

2. Move ALL files in the specified S3 sourcePath into different S3 bucket.

   const s3mv = require('lambda-s3-mv');
   
   const result = await s3mv.move({
       sourceBucket: 's3-source-bucket-name',
       sourcePath: 's3-source-prefix-path',
       targetBucket: 's3-target-bucket-name',
       targetPath: 's3-target-prefix-path'
   });
   console.log(result);
   

3. Move specified list of files in the specified S3 sourcePath into the same S3 bucket.

   const s3mv = require('lambda-s3-mv');
   
   const result = await s3mv.move({
       sourceBucket: 's3-source-bucket-name',
       sourcePath: 's3-source-prefix-path',
       sourceFiles: ['source-file1.txt', 'source-file2.txt'],
       targetPath: 's3-target-prefix-path'
   });
   console.log(result);
   

4. Move specified list of files in the specified S3 sourcePath into the same S3 bucket, and rename files after moved.

   const s3mv = require('lambda-s3-mv');
   
   const result = await s3mv.move({
       sourceBucket: 's3-source-bucket-name',
       sourcePath: 's3-source-prefix-path',
       sourceFiles: ['source-file1.txt', 'source-file2.txt'],
       targetPath: 's3-target-prefix-path',
       targetFiles: ['source-file1-moved.txt', 'source-file2-moved.txt']
   });
   console.log(result);
   

5. Copy ALL files in the specified S3 sourcePath into the same S3 bucket. "createCopy" set to true.

   const s3mv = require('lambda-s3-mv');
   
   const result = await s3mv.move({
       sourceBucket: 's3-source-bucket-name',
       sourcePath: 's3-source-prefix-path',
       sourceFiles: ['source-file1.txt', 'source-file2.txt'],
       targetPath: 's3-target-prefix-path',
       createCopy: true
   });
   console.log(result);
   

Important

Please make sure the lambda has read and write access to specifed source S3 Bucket.

Sample Lambda using the lambda-s3-mv, with Cloudformation, can be found here.

Test

npm run test