azure-storage-fs
A drop-in "fs" replacement for accessing Azure Storage with Node.js "fs" API.
This package is designed to support ftpd.
How to use
azure-storage-fs
is designed to replace Node.js "fs" package and integrate with ftpd
package.
Replace Node.js "fs" package
const fs = require('azure-storage-fs').blob(accountName, secret, container);
fs.readFile('helloworld.txt', (err, data) => {
console.log(err || data);
});
Go advanced mode
If you want to bring your own BlobService
and prefer a Promise interface like mz/fs
.
const { createBlobService } = require('azure-storage');
const fs = require('azure-storage-fs').blob(createBlobService(), container).promise;
const data = await fs.readFile('helloworld.txt');
console.log(data);
Integration with ftpd
ftpd
supports custom "fs" implementation and azure-storage-fs
is designed to be a "fs" provider for ftpd
.
To use a custom "fs" implementation, in your ftpd
authorization code (command:pass
event), add require('azure-storage-fs').blob(accountName, secret, container)
to the success
callback. For example,
connection.on('command:pass', (password, success, failure) => {
if (auth(username, password)) {
success(username, require('azure-storage-fs').blob(accountName, secret, container));
} else {
failure();
}
});
Some ideas for ftpd
- Use different container for each user
- Username/password can be stored as container metadata, always salt and hash the password
- Trigger webhook when a file is uploaded to FTP
Some caveats for ftpd
- Azure Storage is eventually consistent, changes may not happen right away
- After uploaded a file, it may not appear in the file list immediately
- When listing files,
ftpd
will call fs.readdir()
first, then fs.stat()
for every file
- Listing a folder with 10 files will result in 11 requests to Azure
- Calling
fs.stat()
on a directory will result in 2 calls to Azure
Supported APIs
Path normalization
Paths will be normalized with the following rules:
- Turn backslashes (Windows style) into slashes, then
- Remove leading slashes
For example, \Users\Documents\HelloWorld.txt
will become Users/Documents/HelloWorld.txt
.
Blob service
Since Blob service is flat natively, it does not support directory tree. It use delimiter /
to provides a hierarchical view of its flat structure. A hidden empty blob named $$$.$$$
is used to represent empty directory.
Only block blob is supported and is the default blob type when creating a new blob.
createReadStream
- Only default options are supported
encoding
is not supported- New
snapshot
options for specifying ID of the snapshot to read
createWriteStream
- Only default options are supported
- Append is not supported
encoding
is not supported- New
metadata
option to specify blob metadata
mkdir
- Throw
EEXIST
if the directory already exists - Will create a hidden blob under the new folder, named
$$$.$$$
open
- Supported mode:
r
, w
, and wx
- Only default options are supported
- New
snapshot
options for specifying ID of the snapshot to open
readdir
readFile
- Implemented using
createReadStream
- Only default options are supported
encoding
is not supported- New
snapshot
options for specifying ID of the snapshot to read
rename
- Implemented as copy-and-delete
- Because rename is not natively supported, snapshots will be lost after rename
- Metadata will be retained
rmdir
- Will delete hidden blob
$$$.$$$
if exists - Checks if the directory is emptied, throw
ENOTEMPTY
if not
sas(path, options)
(New)
- Will create a Shared Access Signature token for a blob synchronously
- Options can be passed
flag
(optional)
start
(optional)
expiry
(optional)
setMetadata(metadata, options)
(New)
- Will modify metadata on existing blob
- Options can be passed
snapshot
(optional)
- The snapshot ID to modify the metadata on
snapshot(path, options)
(New)
- Will create a new snapshot based on existing blob
- Will return the new snapshot ID
- Options can be passed
snapshot
(optional)
- The snapshot ID to base the new snapshot on
stat(path, options)
- Only report the following properties
isDirectory()
mode
always equals to R_OK | W_OK
mtime
size
- New
url
for the actual URL (not support Shared Access Signature yet)
- Options can be passed as the second argument
metadata
(default set to false
)
- When set to
true
, the call will also return metadata - When paired with
snapshot
options, the call will also return metadata for snapshots
snapshot
(default set to false
)
- When set to
true
, the call will also return an array named snapshots
, each with the following properties
id
is the snapshot IDmtime
size
url
- When set to a string, the call will target the specified snapshot ID
- Otherwise, will target the default (i.e. most recent) blob
unlink(pathname, options)
- Options can be passed as the second argument
snapshot
(default set to true
)
true
will delete all snapshots associated with the blob- Otherwise, it will be treated as a string to specify the ID of the snapshot to delete
writeFile
- Implemented using
createWriteStream
- Only default options are supported
- Append is not supported
encoding
is not supported- New
metadata
option to specify blob metadata
Snapshot
Snapshot is supported and snapshot ID can be specified for most read APIs. stat
can also set to return snapshot information. By default this is disabled to save transaction cost.
File service
In future, we plan to support Azure Storage File, which is another file storage service accessible thru HTTP interface and SMB on Azure.
Changelog
0.4.0 (2017-12-07)
- Bring your own
BlobService
0.3.1 (2017-12-06)
- Fix not working on Linux due to file name capitalization
0.3.0 (2017-01-16)
- setMetadata: New API to set metadata on existing blob or snapshot
0.2.0 (2017-01-13)
- createWriteStream: Add
contentSettings
support - stat: Add
contentSettings
support - writeFile: Add
contentSettings
support
0.1.0 (2017-01-12)
- createWriteStream: Add
metadata
support - snapshot: Add
metadata
support - stat: Add
metadata
support - writeFile: Add
metadata
support
0.0.6 (2016-12-05)
- mkdir: Throw
EEXIST
when the directory already exists - rmdir: Throw
ENOTEMPTY
when the directory still contains one or more blobs after deleting $$$.$$$
placeholder - sas: Added check on absence of required
expiry
option - sas: Will default
flag
to r
if not set - Added more tests
0.0.5 (2016-09-28)
- sas: Added
sas()
to create a blob SAS token synchronously - snapshot: Added
snapshot()
to create a blob snapshot
0.0.4 (2016-08-11)
rmdir
now only delete hidden blob $$$.$$$
, instead of delete blobs recursivelyunlink
now fail if the blob does not exist- Add examples
0.0.3 (2016-08-11)
- Support all APIs required by ftpd
0.0.1 (2016-08-11)
Contributions
Like us? Star us.
Doesn't work as expected? File us an issue with minimal code for bug repro.