This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams). This fork of HtmlAgilityPack has a fix in place for the RemoveChild(keepGrandChildren) bug reported here: https://htmlagilitypack.codeplex.com/workitem/9113
A dedicated crawler agent for accessing public data from mangaworld.mx. Built on KamiYomu.CrawlerAgents.Core, it enables efficient search, metadata extraction, and integration with the KamiYomu platform.
LucasSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
LucasSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
LucasSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
LucasSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Library allows you to extract different data from text or web sources, analyze text content, extract different information and write your own data mining & extracting applications and services.
Parses robots.txt files and sitemaps. Supports the proposed RFC9309 Robots Exclusion Protocol standard.
Searchine is a fast and easy to use Site Search solution that integrates well with Umbraco. We deliver ready made widgets to add to your site and provide you with this Umbraco package to further integrate Umbraco with Searchine. Any change made to Umbraco content will notify the Searchine Crawlers to re-index your content. More information about Searchine can be found at https://www.searchine.net
This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).
A package that helps you to scrap web pages. It shows you a lot of information about the page.
Yibi.NetSharper使用.net standard2.0创建,旨在利用HttpClient以及System.Net.Http相关实现进行rest api、站点资源(类似爬虫)等请求访问。 Yibi.NetSharper实现代码参考与借鉴了RestSharp代码的实现。 Yibi. NetSharper was created using. net standard 2.0 to make use of HttpClient and system. Net. Http-related implementations for rest api, site resources (like crawlers) and other requests. Yibi. NetSharper Implementation Code Reference and Reference. Yibi. NetSharper implementations refer to and learn from RestSharp implementations.
An API wrapper for the TF2 Outpost platform. A platform to find great deals for your Team Fortress 2, Counter-Strike: Global Offensive and Dota 2 items with zero hassle.
Searchine is a fast and easy to use Site Search solution that integrates well with Umbraco. We deliver ready made widgets to add to your site and provide you with this Umbraco package to further integrate Umbraco with Searchine. Any change made to Umbraco content will notify the Searchine Crawlers to re-index your content. More information about Searchine can be found at https://www.searchine.net
Here we need to wr0ite A short sentence explaining what your API does.
Rawler is programing framework for web crawler by XAML.RawlerView is Rawler extend for Wpf.
AngleSharp IHtmlParser Extension for Mechanize.NET
MessagePipe dataflow for the ScrapeAAS library.
Crawl and index your (static) asp.net website for searching using an MSBuild target. Simple search query interface. Uses Lucene.Net and NCrawler.
Default tooling for the ScrapeAAS library
This is the forked custom repo from official C#/.NET client library for the IPinfo.io IP address API. You can get rid of dirty data, at least to some extent, with the IsCrawler property. You must send crawler names from appsettings.json in the following format.
Default AngleSharp page loader extensions for the ScrapeAAS library.
PuppeteerSharp page loader for the ScrapeAAS library.
HttpClient static page loader for the ScrapeAAS library.
A .NET Standard web crawling library similar to WebMagic. It is a lightweight, modular, efficient and fast high-level web crawling. scraping framework for .NET
Polly page loader retry options for the ScrapeAAS library.
Basic in-memory implementations of cookie storage and proxy providers for the ScrapeAAS library.
AI visibility for Umbraco — request log + dashboard, robots.txt audit, content negotiation, llms.txt manifests, and *.md page rendering. Auto-generates per-page Markdown at {url}.md, /llms.txt, and /llms-full.txt; surfaces an AI Traffic dashboard so editors can see who's reading the site; audits robots.txt against the AI-crawler list. Dual-strategy renderer (in-process Razor with HTTP loopback fallback) works on clean Umbraco installs and agency-built sites that hijack Razor with custom view models. Built for Umbraco v17+.
Web data providers (crawler, sitemap, RSS) for Rag.NET
A .net core library to handle the login to Backpack.tf. Backpack.tf is a trading site for Team Fortress 2, Counter-Strike: Global Offensive, and Dota 2. Community item pricing, item trading and stats, and much more.
Site-wide SEO & content audit for Umbraco (17 and 18). Crawls every Content & Media URL (plus sitemap.xml), checks for broken links/images/resources, and audits each page for SEO, Open Graph/Twitter, technical SEO, content quality, a static accessibility subset and carbon emissions — with a composite SEO score, a run health score, grouped issues with priority, orphaned/duplicate detection, incremental scans and CSV export, all in a dedicated backoffice section. Runs as a recurring background job (auto-discovered by uTPro Job Monitor). Built on the framework-agnostic T4VN.Seo.Core engine.
A .NET Standard web crawling library similar to WebMagic. It is a lightweight, modular, efficient and fast high-level web crawling. scraping framework for .NET
MVC Search allows developers to easily add Search capabilities to their ASP.NET MVC Razor applications. Supports MVC version 3.0 to 6.0. MvcSearch works by crawling your site content and indexing it. Features catalog caching, binary and XML catalogs, word stemming, support for synonyms, stop words, file and folder filtering, content images, autocomplete search, and more. ▪ Responsive design allows views to automatically adjusts for smaller screens to support most devices from PC to tablet to smartphone. ▪ A mini search box allows you to implement unobtrusive search in a menu or toolbar, while a full-page search view allows users to perform more advanced searches. ▪ Built in crawler optionally creates thumbnails for search results allowing users to get a visual representation of each result. ▪ Paged search results allow users to easily navigate search results without having to scroll endlessly. ▪ Autocomplete search allows users to see instant results as they type without having to submit their search. Feature can easily be turned on or off. ▪ Multiple language support allows non-english users visiting your site to search in their native language. ▪ Support for synonyms allows you to specify alternate words for common search terms.
URL Viewer for Umbraco (17 and 18): a backoffice tool in the "SEO Audit" section to fetch any URL as Googlebot/Bingbot (or a browser), follow the full redirect chain, view headers and HTML source, and run a static analysis — on-page SEO, Open Graph/Twitter, technical SEO, content and accessibility metrics, spam/hidden-CSS/JavaScript detection and cloaking — with a composite SEO score. Also exposes an authenticated API to fetch a URL and return the analysis as JSON. Server-side and SSRF-safe. Built on the framework-agnostic T4VN.Seo.Core engine. (For the site-wide crawler, install uTPro.Feature.SEOAudit.)
A .NET Core 9 library for parsing robots.txt files, supporting Allow/Disallow, Crawl‐delay, and Sitemap directives.
Answer Engine Optimization for Umbraco. Exposes your published content to LLMs and AI answer engines by dynamically serving an /llms.txt index, an /llms-full.txt full-content dump, and a Markdown rendering of any page when the URL ends in ".md" — all generated from the live content cache, no editor work required. Also auto-injects schema.org JSON-LD (Organization, WebSite, Article, breadcrumbs, FAQ) into every page and ships an "AI Crawlers" backoffice dashboard that tracks visits from GPTBot, ClaudeBot, PerplexityBot and 150+ other AI crawlers (privacy-safe: analytics store no IPs by default — truncated/hashed storage is opt-in). Classifies crawler intent (training, AI search, user-triggered, scanner), records AI referrals — humans clicking through from claude.ai, chatgpt.com, perplexity.ai — and correlates them into "questions answered" conversions on an AI Referrals dashboard. Verifies claimed crawlers against vendor-published IP ranges and reverse DNS, blocks spoofers (vulnerability scanners wearing AI user agents) with an escalating in-memory-to-durable block list, and ships a Blocking dashboard to block or allow any crawler permanently. Zero-config install via an Umbraco pipeline filter; fully configurable through appsettings.