Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →

Sign in Demo Install

Sign in Demo Install

robotstxt

Package Overview

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

robotstxt

0.5.4
Rubygems

Version published: 15 years ago

Maintainers: 1

Created: 15 years ago

= Robotstxt

Robotstxt is an Ruby robots.txt file parser.

Robotstxt Parser allows you to the check the accessibility of URLs and get other data.

Full support for the robots.txt RFC, wildcards and Sitemap: rules.

== Features

Check if the URL is allowed to be crawled from your Robot
Analyze the robots.txt file to return an Array containing the list of XML Sitemaps URLs

== Requirements

Ruby >= 1.8.7

== Installation

This library is intended to be installed via the RubyGems[http://rubyforge.org/projects/rubygems/] system.

$ gem install robotstxt

You might need administrator privileges on your system to install it.

== Author

Author:: {Simone Rinzivillo}[http://www.simonerinzivillo.it/] srinzivillo@gmail.com

== Resources

{Homepage}[http://www.simonerinzivillo.it/]

== License

Copyright (c) 2009 Simone Rinzivillo, Robotstxt is released under the MIT license.

FAQs

What is robotstxt?

Is robotstxt well maintained?

Package last updated on 13 Feb 2010

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Related posts