Headless-browser crawling backend for SimpleCrawler using Microsoft Playwright, for fully JavaScript-rendered sites. Requires browsers via `playwright install`.
Headless-browser crawling backend for SimpleCrawler using PuppeteerSharp, for fully JavaScript-rendered sites. Downloads Chromium on first run.
Static-HTML crawling backend for SimpleCrawler using HtmlAgilityPack — the default, fastest backend for server-rendered sites.
Static-HTML crawling backend for SimpleCrawler using AngleSharp's standards-compliant HTML parser.
Use this filter that prerenders a javascript-rendered page using an external service and returns the HTML to the search engine crawler for SEO.
Web scraper / crawler / spider. Supports robots protocol and user agent.
A dedicated crawler agent for accessing public data from MangaKatana. Built on KamiYomu.CrawlerAgents.Core, it enables efficient search, metadata extraction, and integration with the KamiYomu platform.
Enterprise bot detection and anonymous entity resolution for ASP.NET Core — the detection engine behind StyloBot. Probabilistic and behavioural, not just User-Agent matching: UA/header/IP analysis, JA3/JA4/HTTP2/QUIC/TCP-IP fingerprinting, Leiden cluster discovery, 129-dim Markov session vectors, metastable fingerprint identity, and optional LLM classification. Zero-PII (HMAC-SHA256), SQLite-backed, single-call UseStyloBot() setup.
Spider of DogsBreeds.Crawler project that extract data from FCI
Link It is an object-oriented data integration library that make it easy to load different kinds of objects and link them together. LinkIt is not an object-relational mapping framework. It can be used for orchestrating the loading of objects and for linking the loaded objects togheter, not for defining how the objects are loaded. LinkIt is intended to be used in a variety of context such as data APIs, ETL processes, CQRS event handlers, web crawlers, etc. Features - Minimize coding effort by leveraging reuse and composition - Data source independant - Avoid the Select N + 1 problem - Built-in support for references between complex types - Support polymorphism out of the box - Favor convention over configuration - Perform complex projections easily with LinkIt AutoMapper Extensions
Detects the browser, engine, operating system, CPU, device and bot from a User-Agent string and User-Agent Client Hints (Windows 11, full browser versions, CPU, Android model). Rules based on ua-parser-js 1.0.41 and about 1,500 bots from crawler-user-agents, kept up to date; immutable thread-safe results, an optional cache, and GPU detection from WebGL renderer strings.
Indexes content into Typesense and searches it, so results come from an index the site controls rather than from an external crawler.
Web crawling extension for WebSpark.HttpClientUtility. Includes SiteCrawler and SimpleSiteCrawler with robots.txt compliance, HTML link extraction (HtmlAgilityPack), sitemap generation (Markdig), CSV export (CsvHelper), and real-time SignalR progress updates. Perfect for web scraping, SEO audits, and site analysis. Supports .NET 8 LTS (until Nov 2026), .NET 9, and .NET 10. Requires WebSpark.HttpClientUtility base package [2.5.0]. Install both packages and call AddHttpClientUtility() + AddHttpClientCrawler() in your DI registration.
A dedicated crawler agent for accessing public data from mundoavatar.com.br. Built on KamiYomu.CrawlerAgents.Core, it enables efficient search, metadata extraction, and integration with the KamiYomu platform.
DogsBreeds.Crawler project that extract data of dog breeds from many source
crawler framework , distributed crawler extractor. try ruiji scraper --- chrome web crawler https://chrome.google.com/webstore/detail/ruiji-scraper/klhahkhllngppofpkjdlbmnglnmnbbol?hl=zh-CN&authuser=0
A package to make your life even more happy and simple when messing with XML files.
Automatically ignore errors generated by bots, webcrawlers, spiders and similar. Deprecated: enable the Crawlers filter on the Filters tab.
A dedicated crawler agent for accessing public data from asuracomic.net. Built on KamiYomu.CrawlerAgents.Core, it enables efficient search, metadata extraction, and integration with the KamiYomu platform.
Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.
LucasSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Constellation is a collection of utilities for .NET CMS implementers. The Feature.SitemapXml namespace contains Http Handlers which intercept the request before it hits Sitecore. The /robots.txt handler returns a basic document stating that all agents have access to the sitemap.xml file. (there is an option to explicitly deny all agents as well). For /sitemap.xml, the request is passed to a Sitemap Generator, which uses a Crawler to identify which Items to put in the resulting XML document. The Generator also uses a Node class to parse the Items to identify what information belongs in each of the Items' records in the XML output. Because this document can be expensive to generate, there is a built-in caching mechanism. This library supports "global" defaults in the Sitecore *.config files, as well as Site-specific settings. The system will function on a basic level "out of the box". Developers are encouraged to extend/replace the crawler and node classes for their solution and/or individual Helix Projects within the solution to better suit their needs. Warning, your web.config must have a system.webServer/handlers section, and that section must have at least one handler in it (an "add" element) in order to install this package successfully. If you're using the default Sitecore web.config this won't be an issue. The handlers provided in this package must be inserted at the top of the stack of handlers. Configured for use with Sitecore 9.0. Requires Sitecore 9.0 NuGet packages, available here: https://sitecore.myget.org/F/sc-packages/api/v3/index.json
简单、易用、高效 一个有态度的开源.Net Http请求框架!可以用制作爬虫,api请求等等。让你感受一个简易到极致的HTTP编程. 让编程更简易,代码更简洁。用法请查看:https://github.com/stulzq/HttpCode.Core