Vore quickly crawls websites and spits out text sans tags. It's written in Ruby and powered by Rust.
Install the gem and add to the application's Gemfile by executing:
$ bundle add vore
If bundler is not being used to manage dependencies, install the gem by executing:
$ gem install vore
crawler = Vore::Crawler.new
crawler.scrape_each_page("https://choosealicense.com") do |page|
puts page
end
Each page is a simple class consisting of the following values:
content: the text of the HTML document, sans tagstitle: the title of the HTML document (if any)meta: the document's meta tags (if any)path: the document's pathThe scraping is managed by spider-rs, so you know it's fast.
| Name | Description | Default |
|---|---|---|
delay | A value (in milliseconds) which introduces an artifical delay when crawling. Useful for situations where there's rate limiting involved. | 0 |
output_dir | Where the resulting HTML files are stored. | "tmp/vore" |
delete_after_yield | Whether the downloaded HTML files are deleted after the yield block finishes. | true |
log_level | The logging level. | :warn |
Vore processes HTML using handlers. By default, there are two:
MetaExtractor, which extracts information from your title and meta tagsTagRemover, which removes unnecessary elements like header, footer, scriptIf you wish to process the HTML further, you can provide your own handler:
Vore::Crawler.new(handlers: [MySpecialHandler.new])
Handlers are defined using Selma. Note that the MetaExtractor is always included and defined first, but if you pass in anything to the handler array, it'll overwrite Vore's other default handlers. You can of course choose to include them manually:
# preserve Vore's default content handler while adding your own;
# `MetaExtractor` is prefixed to the front
Vore::Crawler.new(handlers: [Vore::Handlers::TagRemover.new, MySpecialHandler.new])
Since the actual HTTP calls occur in a separate process, Vore will not integrate with libraries like VCR or Webmock by default. You'll need to require "vore/minitest_helper" to get a function that emulates the HTTP GET requests in a way Ruby can interpret.
Based on your needs, you can overwrite any of the existing methods to suit your application's needs. For example, if you prefer HTML to be generated by Faker, you can create and require a file that looks like the following:
require "vore/minitest_helper"
module Vore
module TestHelperExtension
DOCUMENT_TITLES = [
"Hello, I need help",
"I need to update my payment information",
]
DOCUMENT_CONTENT = [
"Hey, I'm having trouble with my computer. Can you help me?",
# v--- always creates three page chunks
"I need to update my payment information. Like, now. Right now. Now. Can you help me? Please? Now?" + "Can you help me? Please? Now?" * 100,
]
def content
@counter = -1 unless defined?(@counter)
@counter += 1
html = "<!DOCTYPE html><html><head><title>#{DOCUMENT_TITLES[@counter]}</title>"
meta_tag_count.times do # arbitrarily set to 5
html += "<meta name=\"#{Faker::Lorem.word}\" content=\"#{Faker::Lorem.word}\" />"
end
html += "</head><body>"
html += "<p>#{DOCUMENT_CONTENT[@counter]}</p>"
html += "</body></html>"
html
end
end
Vore::TestHelper.prepend(Vore::TestHelperExtension)
end
After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.
Bug reports and pull requests are welcome on GitHub at https://github.com/gjtorikian/vore.
The gem is available as open source under the terms of the MIT License.
FAQs
Unknown package
We found that vore demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.