What is walk?
The 'walk' npm package is used for traversing directories and handling files within those directories. It provides a simple and efficient way to walk through a directory tree, allowing you to perform operations on files and directories as you encounter them.
What are walk's main functionalities?
Walking a directory
This feature allows you to walk through a directory and handle files and directories as they are encountered. The 'file' event is triggered for each file, and the 'directory' event is triggered for each directory.
const walk = require('walk');
const walker = walk.walk('/path/to/directory', {});
walker.on('file', function (root, fileStats, next) {
console.log('File: ' + fileStats.name);
next();
});
walker.on('directory', function (root, dirStats, next) {
console.log('Directory: ' + dirStats.name);
next();
});
walker.on('end', function () {
console.log('All files traversed.');
});
Filtering files
This feature allows you to filter out certain directories from being walked. In this example, the 'node_modules' and '.git' directories are excluded from the traversal.
const walk = require('walk');
const walker = walk.walk('/path/to/directory', {
filters: ['node_modules', '.git']
});
walker.on('file', function (root, fileStats, next) {
console.log('File: ' + fileStats.name);
next();
});
walker.on('end', function () {
console.log('All files traversed.');
});
Handling errors
This feature allows you to handle errors that occur during the directory traversal. The 'errors' event is triggered when an error is encountered, and you can log or handle the error as needed.
const walk = require('walk');
const walker = walk.walk('/path/to/directory', {});
walker.on('file', function (root, fileStats, next) {
console.log('File: ' + fileStats.name);
next();
});
walker.on('errors', function (root, nodeStatsArray, next) {
nodeStatsArray.forEach(function (n) {
console.error('Error: ' + n.error.message);
});
next();
});
walker.on('end', function () {
console.log('All files traversed.');
});
Other packages similar to walk
fs-extra
The 'fs-extra' package extends the native Node.js 'fs' module with additional methods and functionality, including methods for walking directories. It provides a more comprehensive set of file system utilities compared to 'walk'.
klaw
The 'klaw' package is a Node.js file system walker that provides a stream-based API for traversing directories. It is similar to 'walk' but offers a more modern and flexible approach with support for streams.
readdirp
The 'readdirp' package is a recursive version of 'fs.readdir' with additional features such as filtering and streaming. It is similar to 'walk' but provides a more powerful and flexible API for reading directories recursively.
node-walk
nodejs walk implementation.
This is somewhat of a port python's os.walk
, but using Node.JS conventions.
- EventEmitter
- Asynchronous
- Chronological (optionally)
- Built-in flow-control
- includes Synchronous version (same API as Asynchronous)
As few file descriptors are opened at a time as possible.
This is particularly well suited for single hard disks which are not flash or solid state.
Installation
npm install --save walk
Getting Started
(function () {
"use strict";
var walk = require('walk')
, fs = require('fs')
, walker
;
walker = walk.walk("/tmp", options);
walker.on("file", function (root, fileStats, next) {
fs.readFile(fileStats.name, function () {
next();
});
});
walker.on("errors", function (root, nodeStatsArray, next) {
next();
});
walker.on("end", function () {
console.log("all done");
});
}());
Common Events
All single event callbacks are in the form of function (root, stat, next) {}
.
All multiple event callbacks callbacks are in the form of function (root, stats, next) {}
, except names which is an array of strings.
All error event callbacks are in the form function (root, stat/stats, next) {}
.
stat.error
contains the error.
names
directory
directories
file
files
end
nodeError
(stat
failed)directoryError
(stat
succedded, but readdir
failed)errors
(a collection of any errors encountered)
A typical stat
event looks like this:
{ dev: 16777223,
mode: 33188,
nlink: 1,
uid: 501,
gid: 20,
rdev: 0,
blksize: 4096,
ino: 49868100,
size: 5617,
blocks: 16,
atime: Mon Jan 05 2015 18:18:10 GMT-0700 (MST),
mtime: Thu Sep 25 2014 21:21:28 GMT-0600 (MDT),
ctime: Thu Sep 25 2014 21:21:28 GMT-0600 (MDT),
birthtime: Thu Sep 25 2014 21:21:28 GMT-0600 (MDT),
name: 'README.md',
type: 'file' }
Advanced Example
Both Asynchronous and Synchronous versions are provided.
(function () {
"use strict";
var walk = require('walk')
, fs = require('fs')
, options
, walker
;
options = {
followLinks: false
, filters: ["Temp", "_Temp"]
};
walker = walk.walk("/tmp", options);
walker.on("names", function (root, nodeNamesArray) {
nodeNamesArray.sort(function (a, b) {
if (a > b) return 1;
if (a < b) return -1;
return 0;
});
});
walker.on("directories", function (root, dirStatsArray, next) {
next();
});
walker.on("file", function (root, fileStats, next) {
fs.readFile(fileStats.name, function () {
next();
});
});
walker.on("errors", function (root, nodeStatsArray, next) {
next();
});
walker.on("end", function () {
console.log("all done");
});
}());
Sync
Note: You can't use EventEmitter if you want truly synchronous walker
(although it's synchronous under the hood, it appears not to be due to the use of process.nextTick()
).
Instead you must use options.listeners
for truly synchronous walker.
Although the sync version uses all of the fs.readSync
, fs.readdirSync
, and other sync methods,
I don't think I can prevent the process.nextTick()
that EventEmitter
calls.
(function () {
"use strict";
var walk = require('walk')
, fs = require('fs')
, options
, walker
;
options = {
listeners: {
names: function (root, nodeNamesArray) {
nodeNamesArray.sort(function (a, b) {
if (a > b) return 1;
if (a < b) return -1;
return 0;
});
}
, directories: function (root, dirStatsArray, next) {
next();
}
, file: function (root, fileStats, next) {
fs.readFile(fileStats.name, function () {
next();
});
}
, errors: function (root, nodeStatsArray, next) {
next();
}
}
};
walker = walk.walkSync("/tmp", options);
console.log("all done");
}());
API
Emitted Values
-
on('XYZ', function(root, stats, next) {})
-
root
- the containing the files to be inspected
-
stats[Array] - a single stats
object or an array with some added attributes
- type - 'file', 'directory', etc
- error
- name - the name of the file, dir, etc
-
next - no more files will be read until this is called
Single Events - fired immediately
-
end
- No files, dirs, etc left to inspect
-
directoryError
- Error when fstat
succeeded, but reading path failed (Probably due to permissions).
-
nodeError
- Error fstat
did not succeeded.
-
node
- a stats
object for a node of any type
-
file
- includes links when followLinks
is true
-
directory
- NOTE you could get a recursive loop if followLinks
and a directory links to its parent
-
symbolicLink
- always empty when followLinks
is true
-
blockDevice
-
characterDevice
-
FIFO
-
socket
Events with Array Arguments - fired after all files in the dir have been stat
ed
-
names
- before any stat
takes place. Useful for sorting and filtering.
- Note: the array is an array of
string
s, not stat
objects - Note: the
next
argument is a noop
-
errors
- errors encountered by fs.stat
when reading ndes in a directory
-
nodes
- an array of stats
of any type
-
files
-
directories
- modification of this array - sorting, removing, etc - affects traversal
-
symbolicLinks
-
blockDevices
-
characterDevices
-
FIFOs
-
sockets
Warning beware of infinite loops when followLinks
is true (using walk-recurse
varient).
Comparisons
Tested on my /System
containing 59,490 (+ self) directories (and lots of files).
The size of the text output was 6mb.
find
:
time bash -c "find /System -type d | wc"
59491 97935 6262916
real 2m27.114s
user 0m1.193s
sys 0m14.859s
find.js
:
Note that find.js
omits the start directory
time bash -c "node examples/find.js /System -type d | wc"
59490 97934 6262908
# Test 1
real 2m52.273s
user 0m20.374s
sys 0m27.800s
# Test 2
real 2m23.725s
user 0m18.019s
sys 0m23.202s
# Test 3
real 2m50.077s
user 0m17.661s
sys 0m24.008s
In conclusion node.js asynchronous walk is much slower than regular "find".
LICENSE
node-walk
is available under the following licenses:
Copyright 2011 - Present AJ ONeal