New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More →
astro-robots-txt
Advanced tools
astro-robots-txt - npm Package Compare versions
Comparing version 0.1.5 to 0.1.7
@@ -0,0 +0,0 @@ import type { AstroIntegration } from 'astro'; |
@@ -0,0 +0,0 @@ import fs from 'node:fs'; |
| { | ||
| "name": "astro-robots-txt", | ||
| "version": "0.1.5", | ||
| "version": "0.1.7", | ||
| "description": "Generate a robots.txt for Astro", | ||
@@ -68,3 +68,3 @@ "keywords": [ | ||
| }, | ||
| "readme": "# astro-robots-txt\n\nThis **[Astro integration][astro-integration]** generates a _robots.txt_ for your Astro project during build.\n\n[](https://opensource.org/licenses/MIT) \n\n------\nThe _robots.txt_ file informs search engines which pages on your website should be crawled. [See Google's own advice on robots.txt](https://developers.google.com/search/docs/advanced/robots/intro) to learn more.\n\n## Why astro-robots-txt?\n\nFor Astro project you usually create the _robots.txt_ in a text editor and place it to the `public/` directory.\nIn that case you must manually synchronize `site` option in _astro.config.*_ with `Sitemap:` record in _robots.txt_. \nIt brakes DRY principle. \n\nSometimes, especially during development, it's needed to prevent your site from being indexed. To achieve this you need place meta tag `<meta name=\"robots\" content=\"noindex\">` in the `<head>` section of pages or add `X-Robots-Tag: noindex` in HTTP header response, then add lines `User-agent: *` and `Disallow: \\` to _robots.txt_. \nAgain you do it manually in two separate places.\n\n**astro-robots-txt** could help in both two cases.\n\n------\n\n## Installation\n\nIf you run into any hiccups, [feel free to log an issue on my GitHub](https://github.com/alextim/astro-robots-txt/issues).\n\n### Install dependencies\n\nFirst, install the **astro-robots-txt** integration like so:\n\n```sh\n#npm\nnpm install --save-dev astro-robots-txt\n\n#yarn\nyarn add -D astro-robots-txt\n\n#pnpm\npnpm add -D astro-robots-txt\n```\n\nThen, apply this integration to your _astro.config.*_ file using the `integrations` property:\n\n**astro.config.mjs**\n\n```js\nimport { defineConfig } from 'astro/config';\nimport robotsTxt from 'astro-robots-txt';\n\nexport default defineConfig({\n // ...\n integrations: [robotsTxt()],\n});\n```\n\n## Getting started\n\n`astro-robots-txt` requires a deployment / site URL for generation. Add your site's URL under your _astro.config.*_ using the `site` property:\n\n**astro.config.mjs**\n\n```js\nimport { defineConfig } from 'astro/config';\nimport robotsTxt from 'astro-robots-txt';\n\nexport default defineConfig({\n // ...\n site: 'https://example.com',\n integrations: [robotsTxt()],\n});\n```\n\nNow, [build your site for production](https://docs.astro.build/en/reference/cli-reference/#astro-build) via the `astro build --experimental-integrations` command. You should find your _robots.txt_ under `dist/robots.txt`!\n\nThe _robots.txt_'s content will be\n\n```text\nUser-agent: *\nAllow: /\nSitemap: https://example.com/sitemap.xml\n```\n\nYou can also check our [Astro Integration Documentation][astro-integration] for more on integrations.\n\n## Configuration\n\n## Options\n\n| Name | Type | Default | Description |\n| :-------: | :-----------------------------: | :------------------------------: | :------------------------------------------------------------------: |\n| `host` | `String` | `` | Host of your site |\n| `sitemap` | `Boolean / String` / `String[]` | `true` | Resulting output will be `Sitemap: your-site-url/sitemap.xml` |\n| | | | If `sitemap: false` - no `Sitemap` line in the output |\n| | | | You could use for `sitemap` valid url string or array of url strings |\n| `policy` | `Policy[]` | [{ allow: '/', userAgent: '*' }] | List of `Policy` rules |\n\n### Policy\n\n| Name | Type | Required | Description |\n| :----------: | :-------------------: | :------: | :---------------------------------------------: |\n| `userAgent` | `String` | Yes | You must provide name of user agent or wildcard |\n| `disallow` | `String` / `String[]` | No | disallowed paths |\n| `allow` | `String` / `String[]` | No | allowed paths |\n| `crawlDelay` | `Number` | No | |\n| `cleanParam` | `String` / `String[]` | No | |\n\n**Sample of _astro.config.mjs_**\n\n```js\nimport { defineConfig } from 'astro/config';\nimport robotsTxt from 'astro-robots-txt';\n\nexport default defineConfig({\n site: 'https://example.com',\n integrations: [\n robotsTxt({\n host: 'example.com',\n sitemap: [\n 'https://example.com/main-sitemap.xml', \n 'https://example.com/images-sitemap.xml'\n ],\n policy: [\n {\n userAgent: 'Googlebot',\n allow: '/',\n disallow: ['/search'],\n crawlDelay: 2,\n },\n {\n userAgent: 'OtherBot',\n allow: ['/allow-for-all-bots', '/allow-only-for-other-bot'],\n disallow: ['/admin', '/login'],\n crawlDelay: 2,\n },\n {\n userAgent: '*',\n allow: '/',\n disallow: '/search',\n crawlDelay: 10,\n cleanParam: 'ref /articles/',\n },\n ],\n }),\n ],\n});\n```\n\n:exclamation: Important Notes\n\nOnly official **@astrojs/\\*** integrations are currently supported by Astro at present moment. \n\nTo make **astro-robots-txt** integrations working, use the `--experimental-integrations` flag during build. \n\nUse the following command to build your Astro site:\n\n```sh\nastro build --experimental-integrations\n```\n\n[astro-integration]: https://docs.astro.build/en/guides/integrations-guide/\n\n\n**Inspirations:**\n\n- [gatsby-plugin-robots-txt](https://github.com/mdreizin/gatsby-plugin-robots-txt)\n- [generate-robotstxt](https://github.com/itgalaxy/generate-robotstxt)\n- [is-valid-hostname](https://github.com/miguelmota/is-valid-hostname)\n" | ||
| "readme": "# astro-robots-txt\n\nThis **[Astro integration][astro-integration]** generates a _robots.txt_ for your Astro project during build.\n\n[](https://opensource.org/licenses/MIT)\n\n------\nThe _robots.txt_ file informs search engines which pages on your website should be crawled. [See Google's own advice on robots.txt](https://developers.google.com/search/docs/advanced/robots/intro) to learn more.\n\n## Why astro-robots-txt?\n\nFor Astro project you usually create the _robots.txt_ in a text editor and place it to the `public/` directory.\nIn that case you must manually synchronize `site` option in _astro.config.*_ with `Sitemap:` record in _robots.txt_. \nIt brakes DRY principle. \n\nSometimes, especially during development, it's needed to prevent your site from being indexed. To achieve this you need place meta tag `<meta name=\"robots\" content=\"noindex\">` in the `<head>` section of pages or add `X-Robots-Tag: noindex` in HTTP header response, then add lines `User-agent: *` and `Disallow: \\` to _robots.txt_. \nAgain you do it manually in two separate places.\n\n**astro-robots-txt** could help in both two cases on the _robots.txt_ side. See details in the demo [repo](https://github.com/alextim/astro-robots-txt/tree/main/demo).\n\n------\n\n## Installation\n\nIf you run into any hiccups, [feel free to log an issue on my GitHub](https://github.com/alextim/astro-robots-txt/issues).\n\n### Install dependencies\n\nFirst, install the **astro-robots-txt** integration like so:\n\n```sh\n#npm\nnpm install --save-dev astro-robots-txt\n\n#yarn\nyarn add -D astro-robots-txt\n\n#pnpm\npnpm add -D astro-robots-txt\n```\n\nThen, apply this integration to your _astro.config.*_ file using the `integrations` property:\n\n**astro.config.mjs**\n\n```js\nimport { defineConfig } from 'astro/config';\nimport robotsTxt from 'astro-robots-txt';\n\nexport default defineConfig({\n // ...\n integrations: [robotsTxt()],\n});\n```\n\n## Getting started\n\n`astro-robots-txt` requires a deployment / site URL for generation. Add your site's URL under your _astro.config.*_ using the `site` property:\n\n**astro.config.mjs**\n\n```js\nimport { defineConfig } from 'astro/config';\nimport robotsTxt from 'astro-robots-txt';\n\nexport default defineConfig({\n // ...\n site: 'https://example.com',\n integrations: [robotsTxt()],\n});\n```\n\nNow, [build your site for production](https://docs.astro.build/en/reference/cli-reference/#astro-build) via the `astro build --experimental-integrations` command. You should find your _robots.txt_ under `dist/robots.txt`!\n\nThe _robots.txt_'s content will be\n\n```text\nUser-agent: *\nAllow: /\nSitemap: https://example.com/sitemap.xml\n```\n\nYou can also check our [Astro Integration Documentation][astro-integration] for more on integrations.\n\n## Configuration\n\n## Options\n\n| Name | Type | Default | Description |\n| :-------: | :-----------------------------: | :------------------------------: | :------------------------------------------------------------------: |\n| `host` | `String` | `` | Host of your site |\n| `sitemap` | `Boolean / String` / `String[]` | `true` | Resulting output will be `Sitemap: your-site-url/sitemap.xml` |\n| | | | If `sitemap: false` - no `Sitemap` line in the output |\n| | | | You could use for `sitemap` valid url string or array of url strings |\n| `policy` | `Policy[]` | [{ allow: '/', userAgent: '*' }] | List of `Policy` rules |\n\n### Policy\n\n| Name | Type | Required | Description |\n| :----------: | :-------------------: | :------: | :---------------------------------------------: |\n| `userAgent` | `String` | Yes | You must provide name of user agent or wildcard |\n| `disallow` | `String` / `String[]` | No | disallowed paths |\n| `allow` | `String` / `String[]` | No | allowed paths |\n| `crawlDelay` | `Number` | No | |\n| `cleanParam` | `String` / `String[]` | No | |\n\n**Sample of _astro.config.mjs_**\n\n```js\nimport { defineConfig } from 'astro/config';\nimport robotsTxt from 'astro-robots-txt';\n\nexport default defineConfig({\n site: 'https://example.com',\n integrations: [\n robotsTxt({\n host: 'example.com',\n sitemap: [\n 'https://example.com/main-sitemap.xml', \n 'https://example.com/images-sitemap.xml'\n ],\n policy: [\n {\n userAgent: 'Googlebot',\n allow: '/',\n disallow: ['/search'],\n crawlDelay: 2,\n },\n {\n userAgent: 'OtherBot',\n allow: ['/allow-for-all-bots', '/allow-only-for-other-bot'],\n disallow: ['/admin', '/login'],\n crawlDelay: 2,\n },\n {\n userAgent: '*',\n allow: '/',\n disallow: '/search',\n crawlDelay: 10,\n cleanParam: 'ref /articles/',\n },\n ],\n }),\n ],\n});\n```\n\n:exclamation: Important Notes\n\nOnly official **@astrojs/\\*** integrations are currently supported by Astro at the present moment. \n\nTo make **astro-robots-txt** integrations working, use the `--experimental-integrations` flag during build. \n\nUse the following command to build your Astro site:\n\n```sh\nastro build --experimental-integrations\n```\n\n[astro-integration]: https://docs.astro.build/en/guides/integrations-guide/\n\n**Inspirations:**\n\n- [gatsby-plugin-robots-txt](https://github.com/mdreizin/gatsby-plugin-robots-txt)\n- [generate-robotstxt](https://github.com/itgalaxy/generate-robotstxt)\n- [is-valid-hostname](https://github.com/miguelmota/is-valid-hostname)\n" | ||
| } |
@@ -5,3 +5,3 @@ # astro-robots-txt | ||
| [](https://opensource.org/licenses/MIT)  | ||
| [](https://opensource.org/licenses/MIT) | ||
@@ -20,3 +20,3 @@ ------ | ||
| **astro-robots-txt** could help in both two cases. | ||
| **astro-robots-txt** could help in both two cases on the _robots.txt_ side. See details in the demo [repo](https://github.com/alextim/astro-robots-txt/tree/main/demo). | ||
@@ -152,3 +152,3 @@ ------ | ||
| Only official **@astrojs/\*** integrations are currently supported by Astro at present moment. | ||
| Only official **@astrojs/\*** integrations are currently supported by Astro at the present moment. | ||
@@ -165,3 +165,2 @@ To make **astro-robots-txt** integrations working, use the `--experimental-integrations` flag during build. | ||
| **Inspirations:** | ||
@@ -168,0 +167,0 @@ |
No alert changes
Worsened metrics
- Total package byte prevSize
- decreased by-0.64%
30216
- Number of lines in readme file
- decreased by-0.6%
167
No dependency changes