gulp-robots

gulp-robots

Robots.txt generator for Gulp. Simple wrapper around robots-generator. Requires Node 4+. Installed through NPM with:

npm install gulp-robots --save-dev

Check out robots-generator for example options. This module scans your HTML for <link rel="sitemap" />. Example usage:

var robots = require('gulp-robots');

gulp.task('default', function () {
    gulp.src('index.html')
        .pipe(robots({
            useragent: '*',
            allow: ['folder1/', 'folder2/'],
            disallow: ['cgi-bin/']
        }))
        .pipe(gulp.dest('robots.txt'));
});

If you need an ES5 build for legacy purposes, just require the ES5 file:

var robots = require('gulp-robots/es5');

To build the ES5 version, run the following and remember to require the ES5 version.

npm install -g babel-cli
babel --presets es2015 index.js --out-file es5.js