Crawler
govuk_seed_crawler
Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.
recipe_crawler
This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz
rsocial
A ruby social media stat crawler
XDDCrawler
a simple wrapper for nokogiri/rest-client, aim to make capybara like dsl
the_country_identity
CIA World Factbook crawler
creepy-crawler
web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site
middleman-crawler
it starts a crawler for Middleman sites
legendastv-crawler
Crawler for http://legendas.tv to see the most dowloaded subtitles
bankcrawlers-hapoalim
A crappy crawler for a crappy bank interface
iudex-brutefuzzy-service
Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-service provides a fuzzy simhash lookup index as a distributed service.
fde-file_crawler
This file crawler helps to decect if there are new files in a directory.
crawler-movie-imdb
This rubygem does not have a description or summary.
iudex-brutefuzzy-protobuf
Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.
pagemunch
A client for the PageMunch web crawler API
bank-crawlers-hapoalim
A crappy crawler for a crappy bank interface
rcrawler
The wrapper of capybara for crawler
skyscraper
Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, siblings etc. Furthermore, you are able to download images, web pages, and store all content in the database. Please visit my Github account for more details.
otacrawler
This is web comic crawler that Akihabara otaku.
chanCrawlerGem
This gem scowers 4chan (or any chan copy theoretically) searching for threads that contains key words specified by you on boards specified by you and downloads all the images, gifs and webms to a specified folder
arb-bs
A demo of Web Crawler using arb-crawler
discovery-mission
Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.
grucrawler
Simple crawler using Redis as backend
fde-mail_crawler
Crawls the mails in a IMAP Folder
feeds-crawler
This gem allows to crawl news articles from RSS feeds.
torrent_crawler
Crawl multiple torrent sites.
sesc
A simple crawler to list SESC events schedule on Terminal
newrank
A Crawler for NewRank
octocrawler
Octocrawler is an intuitive and simple Github API wrapper
gildia_comics_crawler
Crawler for downloading comics from komiks.gildia.pl
embulk-filter-crawler
Crawler4J filter plugin for Embulk
coolCrawler
Simple Web Crawler
polipus-cassandra
Add support for Cassandra in Polipus crawler
carrierwave-nightcrawler-swift
A store for carrierwave that uses swift. NightcrawlerSwift client is used for authenication, list and store operations.
sledgehammer
Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.
options-lib
A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.
scrawler
Scrawler is a project management framework optimized for programmer happiness and sustainable productivity.
pttcrawler
Ruby ptt crawling tool
nikeplus_webcrawler
A webcrawler, which grabs fine-grained data of your personal Nike+ runs and saves these as XML and JSON files.
socialcrawler
It read file containing list of urls and produces output file with domain, page title, twitter, facebook and google plus handles found on the page
baidu_crawler
The Baidu Crawler is to crawl data with your demmand
rails-hush
Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.
simple-news-crawler
A simple news crawler. You can specify the structure of your xml or rss feeds.
rider
Ruby Web crawler
super_crawler
SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.
app-reviews
Mobile App Review Crawler
crawler_sample
This gem is a web crawler sample code.So I don't reccmmend that you use.
recluse
Friendly, neighborhood web crawler for quality assurance.
stupid_crawler
Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.
yamd
This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.
tag_crawler
The application crawls a URL and extracts links, tags and sequences. These features are written to an output file