Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

express-robots-txt

Package Overview
Dependencies
Maintainers
1
Versions
6
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

express-robots-txt

Express middleware to serve and generate robots.txt

  • 0.4.0
  • Source
  • npm
  • Socket score

Version published
Weekly downloads
23K
increased by21.48%
Maintainers
1
Weekly downloads
 
Created
Source

express-robots-txt npm version CircleCI

Express middleware for generating a robots.txt or responding with an existing file. Forked from weo-edu/express-robots.

Using a file

app.use(robots(__dirname + '/robots.txt'));

Using an object

Basic object

app.use(robots({UserAgent: '*', Disallow: '/'}))
Will produce:
UserAgent: *
Disallow: /

Crawl Delay

You can optionally pass a CrawlDelay in just like passing in Disallow

app.use(robots({UserAgent: '*', Disallow: '/', CrawlDelay: '5'}))
Will produce:
UserAgent: *
Disallow: /
Crawl-delay: 5

Sitemap

You can optionally pass a Sitemap in just like passing in CrawlDelay:

app.use(robots({UserAgent: '*', Disallow: '/', CrawlDelay: '5', Sitemap: 'https://nowhere.com/sitemap.xml'}))
Will produce:
UserAgent: *
Disallow: /
Crawl-delay: 5
Sitemap: https://nowhere.com/sitemap.xml

You can also pass in an array of Sitemaps:

app.use(robots({UserAgent: '*', Disallow: '/', CrawlDelay: '5', Sitemap: ['https://nowhere.com/sitemap.xml', 'https://nowhere.com/sitemap2.xml'] }))
Will produce:
UserAgent: *
Disallow: /
Crawl-delay: 5
Sitemap: https://nowhere.com/sitemap.xml
Sitemap: https://nowhere.com/sitemap2.xml

Or an array of objects

app.use(robots([
  {
    UserAgent: 'Googlebot',
    Disallow: '/no-google'
  },
  {
    UserAgent: 'Bingbot',
    Disallow: '/no-bing'
  }
]));
Will produce:
UserAgent: Googlebot
Disallow: /no-google
UserAgent: Bingbot
Disallow: /no-bing

Or either property (UserAgent | Disallow) as an array

app.use(robots([
  {
    UserAgent: [ 'Googlebot', 'Slurp' ],
    Disallow: [ '/no-google', '/no-yahoo' ]
  },
  {
    UserAgent: '*',
    Disallow: [ '/no-bots', '/still-no-bots' ]
  }
]));
Will produce:
UserAgent: Googlebot
UserAgent: Slurp
Disallow: /no-google
Disallow: /no-yahoo
UserAgent: *
Disallow: /no-bots
Disallow: /still-no-bots

Keywords

FAQs

Package last updated on 08 Jul 2018

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc