rotating-file-stream

What is rotating-file-stream?

The rotating-file-stream npm package is used to create a stream that writes to a file and rotates the file based on a specified condition, such as file size or time interval. This is particularly useful for logging purposes where you want to manage log file sizes and avoid having a single, ever-growing log file.

What are rotating-file-stream's main functionalities?

Time-based rotation

This feature allows you to rotate the log file based on a time interval. In this example, the log file 'access.log' will be rotated daily and stored in the '/var/log' directory.

const rfs = require('rotating-file-stream');
const stream = rfs.createStream('access.log', {
  interval: '1d', // rotate daily
  path: '/var/log'
});

Size-based rotation

This feature allows you to rotate the log file based on its size. In this example, the log file 'access.log' will be rotated every time it reaches 10MB and stored in the '/var/log' directory.

const rfs = require('rotating-file-stream');
const stream = rfs.createStream('access.log', {
  size: '10M', // rotate every 10 MegaBytes written
  path: '/var/log'
});

Compression

This feature allows you to compress the rotated log files. In this example, the log file 'access.log' will be rotated daily and compressed using gzip, then stored in the '/var/log' directory.

const rfs = require('rotating-file-stream');
const stream = rfs.createStream('access.log', {
  interval: '1d', // rotate daily
  compress: 'gzip', // compress rotated files
  path: '/var/log'
});

Custom rotation

This feature allows you to specify a custom rotation policy. In this example, the log file 'access.log' will be rotated and up to 7 rotated files will be kept in the '/var/log' directory.

const rfs = require('rotating-file-stream');
const stream = rfs.createStream('access.log', {
  rotate: 7, // keep 7 rotated files
  path: '/var/log'
});

Other packages similar to rotating-file-stream

winston-daily-rotate-file

The winston-daily-rotate-file package is a transport for the winston logging library that allows you to rotate log files based on a time interval. It is similar to rotating-file-stream in that it supports time-based rotation and compression, but it is specifically designed to work with the winston logging library.

logrotate-stream

The logrotate-stream package provides a writable stream that rotates log files based on size or time. It is similar to rotating-file-stream in terms of functionality, but it offers a simpler API and fewer configuration options.

file-stream-rotator

The file-stream-rotator package is another option for rotating log files based on size or time. It offers similar features to rotating-file-stream, including time-based and size-based rotation, as well as compression. However, it is less actively maintained compared to rotating-file-stream.

README

rotating-file-stream

Description

Creates a stream.Writable to a file which is rotated. Rotation behaviour can be deeply customized; optionally, classical UNIX logrotate behaviour can be used.

Usage

var rfs    = require('rotating-file-stream');
var stream = rfs('file.log', {
    size:     '10M', // rotate every 10 MegaBytes written
    interval: '1d',  // rotate daily
    compress: 'gzip' // compress rotated files
});

Installation

With npm:

$ npm install --save rotating-file-stream

API

require('rotating-file-stream');

Returns RotatingFileStream constructor.

Class: RotatingFileStream

Extends stream.Writable.

[new] RotatingFileStream(filename, options)

Returns a new RotatingFileStream to filename as fs.createWriteStream does. The file is rotated following options rules.

filename {String|Function}

The most complex problem about file name is: "how to call the rotated file name?"

The answer to this question may vary in many forms depending on application requirements and/or specifications. If there are no requirements, a String can be used and default rotated file name generator will be used; otherwise a Function which returns the rotated file name can be used.

function filename(time, index)

• time: {Date} If both rotation by interval is enabled and options.rotationTime (see below) is false, the start time of rotation period, otherwise the time when rotation job started. If null, the not-rotated file name must be returned.
• index {Number} The progressive index of rotation by size in the same rotation period.

An example of a complex rotated file name generator function could be:

function pad(num) {
    return (num > 9 ? "" : "0") + num;
}

function generator(time, index) {
    if(! time)
        return "file.log";

    var month  = time.getFullYear() + "" + pad(time.getMonth() + 1);
    var day    = pad(time.getDate());
    var hour   = pad(time.getHours());
    var minute = pad(time.getMinutes());

    return "/storage/" + month + "/" + month +
        day + "-" + hour + minute + "-" + index + "-file.log";
}

var rfs    = require('rotating-file-stream');
var stream = rfs(generator, {
    size:     '10M',
    interval: '30m'
});

Note: if both rotation by interval and rotation by time are used, returned rotated file name must be function of both parameters time and index. Alternatively, rotationTime option can be used (to see below).

If classical logrotate behaviour is enabled rotated file name is only a function of index.

function filename(index)

• index {Number} The progressive index of rotation. If null, the not-rotated file name must be returned.

Note: The not-rotated file name must be only the filename, to specify a path the appropriate option must be used.

rfs('path/to/file.log'); // wrong
rfs('file.log', { path: 'path/to' }); // OK

Note: if part of returned destination path does not exists, the rotation job will try to create it.

options {Object}

• compress: {String|Function|True} (default: null) Specifies compression method of rotated files.
• highWaterMark: {Number} (default: null) Proxied to new stream.Writable
• history: {String} (default: null) Specifies the history filename.
• initialRotation: {Boolean} (default: null) Eventually makes an initial rotation based on not-rotated file timestamp.
• interval: {String} (default: null) Specifies the time interval to rotate the file.
• maxFiles: {Integer} (default: null) Specifies the maximum number of rotated files to keep.
• maxSize: {String} (default: null) Specifies the maximum size of rotated files to keep.
• mode: {Integer} (default: null) Proxied to fs.createWriteStream
• path: {String} (default: null) Specifies the base path for files.
• rotate: {Integer} (default: null) Enables the classical UNIX logrotate behaviour.
• rotationTime: {Boolean} (default: null) Makes rotated file name with time of rotation instead of start time of period.
• size: {String} (default: null) Specifies the file size to rotate the file.

path

If present, it is prepended to generated file names as well as for history file.

size

Accepts a positive integer followed by one of these possible letters:

• B: Bites
• K: KiloBites
• M: MegaBytes
• G: GigaBytes

  size: '300B', // rotates the file when size exceeds 300 Bytes
                // useful for tests

  size: '300K', // rotates the file when size exceeds 300 KiloBytes

  size: '100M', // rotates the file when size exceeds 100 MegaBytes

  size: '1G', // rotates the file when size exceeds a GigaByte

interval

Accepts a positive integer followed by one of these possible letters:

• s: seconds. Accepts integer divider of 60.
• m: minutes. Accepts integer divider of 60.
• h: hours. Accepts integer divider of 24.
• d: days

  interval: '5s', // rotates at seconds 0, 5, 10, 15 and so on
                  // useful for tests

  interval: '5m', // rotates at minutes 0, 5, 10, 15 and so on

  interval: '2h', // rotates at midnight, 02:00, 04:00 and so on

  interval: '1d', // rotates at every midnight

compress

Due the nature of Node.js compression may be done with an external command (to use other CPUs than the one used by Node.js) or with internal code (to use the CPU used by Node.js). This decision is left to you.

Following fixed strings are allowed to compress the files with internal libraries:

• bzip2 (not implemented yet)
• gzip

To enable external compression, a function can be used or simply the boolean true value to use default external compression. The function should accept source and dest file names and must return the shell command to be executed to compress the file. The two following code snippets have exactly the same effect:

var rfs    = require('rotating-file-stream');
var stream = rfs('file.log', {
    size:     '10M',
    compress: true
});

var rfs    = require('rotating-file-stream');
var stream = rfs('file.log', {
    size:     '10M',
    compress: function(source, dest) {
        return "cat " + source + " | gzip -c9 > " + dest;
    }
});

Note: the shell command to compress the rotated file should not remove the source file, it will be removed by the package if rotation job complete with success.

rotationTime

As specified above, if rotation by interval is enabled, the parameter time passed to rotatle name generator is the start time of rotation period. Setting this option to true, parameter time passed is time when rotation job started.

initialRotation

When program stops in a rotation period then restarts in a new rotation period, logs of differente rotation period will go in the next rotated file; in a few words: a rotation job is lost. If this option is set to true an initial check is performed against the not-rotated file timestamp and, if it falls in a previous rotation period, an initial rotation job is done as well.

Note: this option is ignored if rotationTime is set to true.

rotate

If specified, classical UNIX logrotate behaviour is enabled and the value of this option has same effect in logrotate.conf file.

Note: following options are ignored if rotate option is specified.

history

Due to the complexity that rotated file names can have because of the filename generator function, if number or size of rotated files should not exceed a given limit, the package needs a file where to store this information. This option specifies the name of that file. This option takes effects only if at least one of maxFiles or maxSize is used. If null, the not rotated filename with the '.txt' suffix is used.

maxFiles

If specified, it's value is the maximum number of rotated files to be kept.

maxSize

If specified, it's value must respect same syntax of size option and is the maximum size of rotated files to be kept.

Events

Custom Events are emitted by the stream.

var rfs    = require('rotating-file-stream');
var stream = rfs(...);

stream.on('error', function(err) {
    // here are reported blocking errors
    // once this event is emitted, the stream will be closed as well
});

stream.on('open', function() {
    // no rotated file is open (emitted after each rotation as well)
});

stream.on('removed', function(filename, number) {
    // rotation job removed the specified old rotated file
    // number == true, the file was removed to not exceed maxFiles
    // number == false, the file was removed to not exceed maxSize
});

stream.on('rotation', function() {
    // rotation job started
});

stream.on('rotated', function(filename) {
    // rotation job completed with success producing given filename
});

stream.on('warning', function(err) {
    // here are reported non blocking errors
});

Rotation logic

Regardless of when and why rotation happens, the content of a single stream.write will never be split among two files.

by size

Once the not-rotated file is opened first time, its size is checked and if it is greater or equal to size limit, a first rotation happens. After each stream.write, the same check is performed.

by interval

The package sets a Timeout to start a rotation job at the right moment.

Under the hood

Logs should be handled so carefully, so this package tries to never overwrite files.

At stream creation, if the not-rotated log file already exists and its size exceeds the rotation size, an initial rotation attempt is done.

At each rotation attempt a check is done to verify that destination rotated file does not exists yet; if this is not the case a new destination rotated file name is generated and the same check is performed before going on. This is repeated until a not existing destination file name is found or the package is exhausted. For this reason the rotated file name generator function may be called several times for each rotation job.

If requested by maxFiles or maxSize options, at the end of a rotation job, a check is performed to ensure that given limits are respected. This means that while rotation job is running both the limits could be not respected, the same can happend (if maxFiles or maxSize are changed) till the end of first rotation job. The first check performed is the one against maxFiles, in case some files are removed, than the check against maxSize is performed, finally other files can be removed. When maxFiles or maxSize are enabled for first time, an history file can be created with one rotated filename (as returned by filename generator function) at each line.

Once an error event is emitted, nothing more can be done: the stream is closed as well.

Compatibility

This package is written following Node.js 4.0 specifications always taking care about backward compatibility. The package is tested under several Node.js versions.

Required: Node.js 0.11

Licence

MIT Licence

Bugs

Do not hesitate to report any bug or inconsistency @github.

ChangeLog

ChangeLog

Donating

If you find useful this package, please consider the opportunity to donate some satoshis to this bitcoin address: 12p1p5q7sK75tPyuesZmssiMYr4TKzpSCN