A web crawler using Ruby and Redis.
A crawler for such as mailing lists.
MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run them periodically. Thanks to consistent Twitter API and twitter gem (http://twitter.rubyforge.org/), it is quite easy to track various types of timelines (such as user_timeline, home_timeline, search...)
plugin-container totally sucks..
check how many links are available inside the website
This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future
A simple web crawler gem
Lightweight web crawler
Give it a User-Agent string, get back what it is: 150 AI crawlers, search engines, SEO bots, archivers and dataset builders from 74 operators, bundled as one 124.3 KB JSON table. ai_crawler?(ua), identify(ua) -> {name, operator, category, verification_method}, robots_txt(stance). Zero dependencies, no network at require time, pure functions of the string you pass in. Data CC0-1.0 from the independent AI Crawler Index; code MIT.
Multithreaded web crawler with transparent DSL and requests caching.
A web crawler dsl.
A simple web crawler gem
Crawler Guru provides all basic functionalities to extract data from web pages
A crawler and parser to the Currículo Lattes.
Web crawler to get courses from XML pages at mymav.uta
Agentry replaces the browser gem without changing a call site, with the same Browser class, methods and require paths. It is faster, updates its bot list every week, names the category of each bot and checks the IP addresses of known crawlers.
Rack middleware and a Railtie: a firewall with rate limits, shared blocks, attack patterns and a browser check; error monitoring (Rails.error, Sidekiq); server speed per route, with its database queries, outgoing calls, job queues and how close the server is to its limits; crawler counts; server-side events, leads and audit log; feature flags and maintenance mode; and the gems, settings and machine figures the Security check needs. Standard library and rack only.
Generates /<slug>.md companions for each post, declared via rel=alternate type=text/markdown, so LLM crawlers and AI agents can fetch clean Markdown instead of parsing HTML. Each document links back to the site's llms.txt index.
A Rails engine that replaces per-app SEO/GEO code with one configuration file and one layout line: meta tags, canonical folding, JSON-LD, crawler fallback bodies, purpose-based AI crawler policy, robots.txt, sitemap.xml, llms.txt, Markdown negotiation, IndexNow and OG images. GEO means Generative Engine Optimization, not geolocation.
One deploy. After that, approved fixes arrive as a manifest pushed to your own origin and no further release is needed. The corrected HTML is produced on the server, so crawlers that do not run JavaScript still see it. Zero runtime dependencies; it never makes an outbound request.
A simple web crawler gem
A Rails engine for the easyFAA apps: each sits behind Cloudflare and shows the same error pages, limits, crawlers and N-numbers.
Bot Detector analyzes user agent strings to identify bots, crawlers, and automated software. It matches against a database of known bots and adds heuristics that catch automated clients not in the database, returning the bot's category, reputation and a composite risk score. Useful for security, analytics, and access control.
A simple crawler that gets posts and pages from wordpress websites that have an exposed api
A simple web crawler gem
Rack middleware that intercepts crawler requests and serves rendered HTML from render.getindexboost.com.
A Jekyll generator that writes a .md file alongside each rendered HTML page, so AI agents and crawlers can fetch clean Markdown (with a small machine- friendly frontmatter block) instead of parsing HTML. Configurable per collection.
Adds rich JSON-LD structured data, llms.txt, semantic HTML helpers, entity identity management, and AI crawler policies to Jekyll sites.
Protect your content from AI crawlers and monetize every request with Senthor. Real-time detection, crawler control, and detailed analytics.
Protect your content from AI crawlers and monetize every request with Senthor. Real-time detection, crawler control, and detailed analytics.
s7o crawler optimized for programmer happiness and sustainable productivity.
Another Web crawler running with Amazon SQS and ElastiCache(Redis)
A simple web crawler gem
With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the process, providing a clean and efficient solution.