What is gcs-resumable-upload?
The gcs-resumable-upload npm package is designed to facilitate resumable uploads to Google Cloud Storage. It allows users to upload large files in chunks, which can be particularly useful for handling network interruptions and ensuring that uploads can be resumed from the point of failure.
What are gcs-resumable-upload's main functionalities?
Initiate a Resumable Upload
This feature allows you to initiate a resumable upload session by creating a unique upload URI. This URI can be used to upload the file in chunks.
const { Upload } = require('gcs-resumable-upload');
const upload = new Upload({
bucket: 'my-bucket',
file: 'my-file.txt',
authConfig: 'path/to/keyfile.json'
});
upload.createURI((err, uri) => {
if (err) {
console.error('Error creating URI:', err);
} else {
console.log('Upload URI:', uri);
}
});
Upload a File Chunk
This feature allows you to upload a file chunk to the previously created upload URI. It handles the actual data transfer to Google Cloud Storage.
const fs = require('fs');
const { Upload } = require('gcs-resumable-upload');
const upload = new Upload({
bucket: 'my-bucket',
file: 'my-file.txt',
authConfig: 'path/to/keyfile.json'
});
const uri = 'your-upload-uri';
const fileStream = fs.createReadStream('path/to/local/file');
upload.startUploading(uri, fileStream, (err, res) => {
if (err) {
console.error('Error uploading file:', err);
} else {
console.log('File uploaded successfully:', res);
}
});
Resume an Interrupted Upload
This feature allows you to resume an interrupted upload by continuing from the last successfully uploaded chunk. It ensures that the upload process can be completed even after a network failure.
const fs = require('fs');
const { Upload } = require('gcs-resumable-upload');
const upload = new Upload({
bucket: 'my-bucket',
file: 'my-file.txt',
authConfig: 'path/to/keyfile.json'
});
const uri = 'your-upload-uri';
const fileStream = fs.createReadStream('path/to/local/file');
upload.continueUploading(uri, fileStream, (err, res) => {
if (err) {
console.error('Error resuming upload:', err);
} else {
console.log('Upload resumed successfully:', res);
}
});
Other packages similar to gcs-resumable-upload
gcs-upload
The gcs-upload package is another library for uploading files to Google Cloud Storage. It supports both simple and resumable uploads. While it offers similar functionality to gcs-resumable-upload, it may not be as widely used or maintained.
gcs-resumable-upload
Upload a file to Google Cloud Storage with built-in resumable behavior
$ npm install --save gcs-resumable-upload
var upload = require('gcs-resumable-upload');
var fs = require('fs');
fs.createReadStream('titanic.mov')
.pipe(upload({ bucket: 'legally-owned-movies', file: 'titanic.mov' }))
.on('finish', function () {
});
Or from the command line:
$ npm install -g gcs-resumable-upload
$ cat titanic.mov | gcs-upload legally-owned-movies titanic.mov
If somewhere during the operation, you lose your connection to the internet or your tough-guy brother slammed your laptop shut when he saw what you were uploading, the next time you try to upload to that file, it will resume automatically from where you left off.
How it works
This module stores a file using ConfigStore that is written to when you first start an upload. It is aliased by the file name you are uploading to and holds the first 16kb chunk of data* as well as the unique resumable upload URI. (Resumable uploads are complicated)
If your upload was interrupted, next time you run the code, we ask the API how much data it has already, then simply dump all of the data coming through the pipe that it already has.
After the upload completes, the entry in the config file is removed. Done!
* The first 16kb chunk is stored to validate if you are sending the same data when you resume the upload. If not, a new resumable upload is started with the new data.
Authentication
Oh, right. This module uses google-auto-auth and accepts all of the configuration that module does to strike up a connection as config.authConfig
. See authConfig
.
API
upload = require('gcs-resumable-upload')
upload(config)
config
Configuration object.
config.authClient
If you want to re-use an auth client from google-auto-auth, pass an instance here.
config.authConfig
See authConfig
.
config.bucket
The name of the destination bucket.
config.file
The name of the destination file.
config.generation
This will cause the upload to fail if the current generation of the remote object does not match the one provided here.
config.metadata
Any metadata you wish to set on the object.
config.metadata.contentType
Set the content type of the incoming data.
config.origin
Set an Origin header when creating the resumable upload URI.
config.uri
If you already have a resumable URI from a previously-created resumable upload, just pass it in here and we'll use that.
--
Events
.on('error', function (err) {})
err
Invoked if the authorization failed, the request failed, or the file wasn't successfully uploaded.
.on('response', function (resp, metadata) {})
resp
The HTTP response from request
.
metadata
The file's new metadata.
.on('finish', function () {})
The file was uploaded successfully.
upload.createURI(config, callback)
callback(err, resumableURI)
callback.err
Invoked if the authorization failed or the request to start a resumable session failed.
callback.resumableURI
The resumable upload session URI.