New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More →

astro-robots-txt

Package Overview

Dependencies

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

astro-robots-txt

Generate a robots.txt for Astro

0.3.4
Source
npm

Version published: 3 years ago

Weekly downloads: 13K; increased by19.74%

Maintainers: 1

Weekly downloads

Created: 3 years ago

Source

astro-robots-txt

This Astro integration generates a robots.txt for your Astro project during build.

The robots.txt file informs search engines which pages on your website should be crawled. See Google's own advice on robots.txt to learn more.

Why astro-robots-txt?

For Astro project you usually create the robots.txt in a text editor and place it to the public/ directory. In that case you must manually synchronize site option in astro.config.* with Sitemap: record in robots.txt.
It brakes DRY principle.

Sometimes, especially during development, it's needed to prevent your site from being indexed. To achieve this you need place meta tag <meta name="robots" content="noindex"> in the <head> section of pages or add X-Robots-Tag: noindex in HTTP header response, then add lines User-agent: * and Disallow: \ to robots.txt.
Again you do it manually in two separate places.

astro-robots-txt could help in both two cases on the robots.txt side. See details in this demo repo.

Installation

There are two ways to add astro-robots-txt integration to your Astro project.

Astro CLI tool

You should run astro add command in your project directory. This command after prompt will install required dependencies and apply changes to astro.config.*.

# Using NPM
npx astro add astro-robots-txt

# Using Yarn
yarn astro add astro-robots-txt

# Using PNPM
pnpx astro add astro-robots-txt

If you run into any hiccups, feel free to log an issue on my GitHub.

Install dependencies manually

First, install the astro-robots-txt integration like so:

# Using NPM
npm install --save-dev astro-robots-txt

# Using Yarn
yarn add -D astro-robots-txt

# Using PNPM
pnpm add -D astro-robots-txt

Then apply this integration to your astro.config.*. All details below in Getting started.

Getting started

The astro-robots-txt integration requires a deployment / site URL for generation. Add your site's URL under your astro.config.* using the site property.

:exclamation: Provide the experimental property to your astro.config.*, because only official @astrojs/* integrations are currently supported by Astro. Set the experimental.integrations value to true.

Then, apply this integration to your astro.config.* file using the integrations property.

astro.config.mjs

// astro.config.mjs
import { defineConfig } from 'astro/config';
import robotsTxt from 'astro-robots-txt';

export default defineConfig({
  // ...
  site: 'https://example.com',
  // Important!
  // Only official '@astrojs/*' integrations are currently supported by Astro.
  // Add 'experimental.integrations: true' to make 'astro-robots-txt' working
  // with 'astro build' command.
  experimental: {
    integrations: true,
  },
  integrations: [robotsTxt()],
});

Now, build your site for production via the astro build command. You should find your robots.txt under dist/robots.txt!

The robots.txt's content will be

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

You can also check Astro Integration Documentation for more on integrations.

Configuration

Options

Name	Type	Required	Default
`host`	`String`	No		Host of your site
`sitemap`	`Boolean` / `String` / `String[]`	No	`true`	Resulting output in a robots.txt will be `Sitemap: your-site-url/sitemap.xml`. If `sitemap: false` - no `Sitemap` line in the output. When the `sitemap` is `String` or `String[]` its values have to be a valid http url. Empty strings or other protocols are not allowed.
`policy`	`Policy[]`	No	[{ allow: `/`, userAgent: `*` }]	List of `Policy` rules
`sitemapBaseFileName`	`String`	No	`sitemap`	The name of a sitemap file before the file extension (`.xml`). It's used if the `sitemap` option is `true`.
`transform`	(content: `String`): `String` \| `Promise<String>`	No		Called just before writing the text output to disk. Sync or async.

Policy

Name	Type	Required	Description
`userAgent`	`String`	Yes	You must provide name of user agent or wildcard
`disallow`	`String` / `String[]`	No	Disallowed paths to index
`allow`	`String` / `String[]`	No	Allowed paths to index
`crawlDelay`	`Number`	No	Minimum interval (in secs) for the search robot to wait after loading one page, before starting other
`cleanParam`	`String` / `String[]`	No	Indicates that the page URL contains parameters that should be ignored when indexing

Sample of astro.config.mjs

// astro.config.mjs
import { defineConfig } from 'astro/config';
import robotsTxt from 'astro-robots-txt';

export default defineConfig({
  site: 'https://example.com',
  experimental: {
    integrations: true,
  },
  integrations: [
    robotsTxt({
      host: 'example.com',
      sitemap: ['https://example.com/main-sitemap.xml', 'https://example.com/images-sitemap.xml'],
      policy: [
        {
          userAgent: 'Googlebot',
          allow: '/',
          disallow: ['/search'],
          crawlDelay: 2,
        },
        {
          userAgent: 'OtherBot',
          allow: ['/allow-for-all-bots', '/allow-only-for-other-bot'],
          disallow: ['/admin', '/login'],
          crawlDelay: 2,
        },
        {
          userAgent: '*',
          allow: '/',
          disallow: '/search',
          crawlDelay: 10,
          cleanParam: 'ref /articles/',
        },
      ],
      transform(content) {
        return `# some comments before main content\n#second line\n${content}`;
      },
    }),
  ],
});

if you want to get a robots.txt without Sitemap: ... record please set the sitemap option to false.

// astro.config.mjs
import { defineConfig } from 'astro/config';
import robotsTxt from 'astro-robots-txt';

export default defineConfig({
  // ...
  site: 'https://example.com',
  experimental: {
    integrations: true,
  },
  integrations: [
    robotsTxt({
      sitemap: false,
      // ...
    }),
  ],
});

Using Configuration Files

You could configure the integration with external file robots-txt.config.* (js, cjs, mjs). Put it to the application root folder (see about root in official docs).

The external config must contain a default export statement:

// ESM
export default {
  ...
};

// CommonJS
module.exports = {
  ...
};

:exclamation: The current version of integration doesn't support typescript configs.

How does the integration internally resolve a config?

Options parameter provided?	External config exists?	Result
No	No	Default config used
Yes	No	Options parameter used
No	Yes	External config used
Yes	Yes	External config is merged with options parameter

The external configuration usage example is in the demo repo.

:exclamation: Important Notes

Only official @astrojs/* integrations are currently supported by Astro.

There are two possibilities to make astro-robots-txt integration working with current version of Astro.

Set the experimental.integrations option to true in your astro.config.*.

// astro.config.mjs
export default defineConfig({
  // ...
  experimental: {
    integrations: true,
  },
});

Or use the --experimental-integrations flag for build command.

astro build --experimental-integrations

Inspirations:

Keywords

FAQs

What is astro-robots-txt?

Is astro-robots-txt popular?

Is astro-robots-txt well maintained?

Package last updated on 19 Jun 2022

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install