New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More →

astro-robots-txt

Package Overview

Dependencies

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

astro-robots-txt

Generate a robots.txt for Astro

0.1.5
Source
npm

Version published: 3 years ago

Maintainers: 1

Created: 3 years ago

Source

astro-robots-txt

This Astro integration generates a robots.txt for your Astro project during build.

Build Status

The robots.txt file informs search engines which pages on your website should be crawled. See Google's own advice on robots.txt to learn more.

Why astro-robots-txt?

For Astro project you usually create the robots.txt in a text editor and place it to the public/ directory. In that case you must manually synchronize site option in astro.config.* with Sitemap: record in robots.txt.
It brakes DRY principle.

Sometimes, especially during development, it's needed to prevent your site from being indexed. To achieve this you need place meta tag <meta name="robots" content="noindex"> in the <head> section of pages or add X-Robots-Tag: noindex in HTTP header response, then add lines User-agent: * and Disallow: \ to robots.txt.
Again you do it manually in two separate places.

astro-robots-txt could help in both two cases.

Installation

If you run into any hiccups, feel free to log an issue on my GitHub.

Install dependencies

First, install the astro-robots-txt integration like so:

#npm
npm install --save-dev astro-robots-txt

#yarn
yarn add -D astro-robots-txt

#pnpm
pnpm add -D astro-robots-txt

Then, apply this integration to your astro.config.* file using the integrations property:

astro.config.mjs

import { defineConfig } from 'astro/config';
import robotsTxt from 'astro-robots-txt';

export default defineConfig({
  // ...
  integrations: [robotsTxt()],
});

Getting started

astro-robots-txt requires a deployment / site URL for generation. Add your site's URL under your astro.config.* using the site property:

astro.config.mjs

import { defineConfig } from 'astro/config';
import robotsTxt from 'astro-robots-txt';

export default defineConfig({
  // ...
  site: 'https://example.com',
  integrations: [robotsTxt()],
});

Now, build your site for production via the astro build --experimental-integrations command. You should find your robots.txt under dist/robots.txt!

The robots.txt's content will be

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

You can also check our Astro Integration Documentation for more on integrations.

Configuration

Options

Name	Type	Default	Description
`host`	`String`	``	Host of your site
`sitemap`	`Boolean / String` / `String[]`	`true`	Resulting output will be `Sitemap: your-site-url/sitemap.xml`
			If `sitemap: false` - no `Sitemap` line in the output
			You could use for `sitemap` valid url string or array of url strings
`policy`	`Policy[]`	[{ allow: '/', userAgent: '*' }]	List of `Policy` rules

Policy

Name	Type	Required	Description
`userAgent`	`String`	Yes	You must provide name of user agent or wildcard
`disallow`	`String` / `String[]`	No	disallowed paths
`allow`	`String` / `String[]`	No	allowed paths
`crawlDelay`	`Number`	No
`cleanParam`	`String` / `String[]`	No

Sample of astro.config.mjs

import { defineConfig } from 'astro/config';
import robotsTxt from 'astro-robots-txt';

export default defineConfig({
  site: 'https://example.com',
  integrations: [
    robotsTxt({
      host: 'example.com',
      sitemap: [
        'https://example.com/main-sitemap.xml', 
        'https://example.com/images-sitemap.xml'
      ],
      policy: [
        {
          userAgent: 'Googlebot',
          allow: '/',
          disallow: ['/search'],
          crawlDelay: 2,
        },
        {
          userAgent: 'OtherBot',
          allow: ['/allow-for-all-bots', '/allow-only-for-other-bot'],
          disallow: ['/admin', '/login'],
          crawlDelay: 2,
        },
        {
          userAgent: '*',
          allow: '/',
          disallow: '/search',
          crawlDelay: 10,
          cleanParam: 'ref /articles/',
        },
      ],
    }),
  ],
});

:exclamation: Important Notes

Only official @astrojs/* integrations are currently supported by Astro at present moment.

To make astro-robots-txt integrations working, use the --experimental-integrations flag during build.

Use the following command to build your Astro site:

astro build --experimental-integrations

Inspirations:

Keywords

FAQs

What is astro-robots-txt?

Is astro-robots-txt well maintained?

Package last updated on 28 Apr 2022

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install