The set of classes to write a program for a data crawling with state in DB.
Conholdate.Total for .NET is a complete package to work with a large number of file formats from Microsoft Word®, Excel®, PowerPoint®, Outlook®, Project®, Visio®, Adobe Acrobat®, Illustrator®, Photoshop®, AutoCAD®, OpenOffice® and many more. Conholdate.Total for .NET allows you to use any API released under Aspose and GroupDocs for .NET in order to create, convert, read, edit, update and print popular document formats. Moreover, you may view, annotate, watermark, assemble, classify, search, redact, parse, merge and compare documents without needing to install the native applications. It helps you in file format manipulation and document automation via simple API. Conholdate.Total for .NET also includes specialized APIs to read and create barcodes, extract text from images using OCR as well as extract human marked data from questioners, surveys, quizzes, MCQ papers and feedback forms.
Aspose.HTML.Drawing for .NET is a cross-platform class library that functions as a headless browser, enabling seamless integration into .NET applications, including C#, VB.NET, ASP.NET, WCF, WinForms, and .NET Core. It supports both 32-bit and 64-bit architectures and runs reliably on Windows and Linux environments. The library supports parsing of HTML5, CSS3, SVG, and Canvas, constructing a DOM based on the WHATWG standard. Aspose.HTML.Drawing for .NET helps you create, modify, extract, copy, delete, and replace HTML document content; navigate HTML documents by using document traversal, XPath queries, CSS selector, and JavaScript; extract data from HTML; configure a document sandbox; perform asynchronous operations; load, read, and manage SVG files; load, read, and render Markdown, MHTML, and EPUB documents. Aspose.HTML.Drawing for .NET offers comprehensive format conversion support, allowing your applications to convert from HTML, SVG, EPUB, MHTML, and Markdown to various supported formats. You can also populate an HTML template from XML or JSON data sources. Aspose.HTML.Drawing can easily work with large and complex HTML documents.
HttpClient extension class, and the .Net Core version of the HttpHelper class. Simple and flexible crawler base class library. JsHttpClient类是 .Net Core 下的一个简单灵活的爬虫基础类库
dcsoup is a .NET library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods. This library is basically a port of jsoup, a Java HTML parser library. see also: http://jsoup.org/ API reference is available at: https://raw.githubusercontent.com/matarillo/dcsoup/master/sandcastle/Help/dcsoup.chm
ClearScript V8 JavaScript-rendering backend for SimpleCrawler: crawls client-rendered SPAs by executing page scripts against a pure-JS DOM. Native V8 binaries for all platforms are included.
Goes through http://www.smogon.com/forums/ and saves all teams it finds.
HtmlMonkey is a lightweight HTML/XML parser written in C#. It allows you to parse an HTML or XML string into a hierarchy of node objects, which can then be traversed or queried using jQuery-like selectors. The library also supports creating node objects from code and producing HTML or XML from those objects.
A foundational library for building, extending, and integrating custom crawler agents within the KamiYomu ecosystem. Provides essential abstractions, utilities, and lifecycle hooks to streamline agent development and ensure seamless interoperability across the platform.
This is a contoler class, by calling action -CrawlSite it will generate sitemap xml of the current site and will show the xml content in browser. For dynamic sitemap this is very useful.
DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Engine-agnostic core of SimpleCrawler: parallel single-domain crawling, robots.txt/sitemap parsing, and the pluggable HTML/rendering backend abstractions.
Advanced HTTP request analysis for device detection, browser identification, and geolocation services in ASP.NET Core applications. Provides comprehensive user agent parsing to detect device types (mobile, tablet, desktop), operating systems, browser versions, and bot detection. Includes IP-based geolocation with country, city, timezone detection using MaxMind GeoIP2 database. Features crawler/bot identification, proxy detection, and detailed request information extraction.
Requires Sitecore 10.3 Requires .NET Framework 4.8 Constellation is a collection of utilities for .NET CMS implementers. The Foundation.SitemapXml namespace contains Http Handlers which intercept the request before it hits Sitecore. Robots.TXT Handler: Returns a basic document that automatically applies to all search engine agents. The stock configuration has disallows for /sitecore urls to prevent the login page from being indexed. Additional default or site specific rules can be added through the supplied configuration file. See instructions within the config file. Assuming a given Site's robots.txt does not deny the root of a site, the robots.txt file will include a link to the site's sitemap.xml file. Sitemap.XML Handler: Returns an on-the-fly document (there are cache options available) for each configured Site in your installation. The document is created by having one or more "crawlers" inspect the Content Tree generating the necessary Item facts and URLs required to produce the Sitemap. You can specify a set of default crawlers for your installation, as well as override the crawler list on a site-by-site basis. There are also provisions for excluding sites from returning a sitemap.xml document if it is not appropriate. Default Functionality: The system will function on a basic level "out of the box". There is a default Content Tree crawler that will start at the root of the current Site and include Items that have presentation details. The crawlers operate in the extranet\anonymous role and thus security can be used to "hide" Items or branches from the crawler. Crawler output has extensive support for "validating" whether an Item should be included in the sitemap or not. Facts available through the API include: - Does the Item have presentation? - Does the Item have explicit rules to prevent Search Indexing? - Does the Item represent a "Page"? It is up to the developer to supply meaningful decisions behind these facts, but answering "no" to any of them will result in the Item not being included. This library outputs substantial amounts of troubleshooting warnings to the Sitecore log file. If you're having trouble, search for Constellation.Foundation.SitemapXml in your log file. Multi-Language Support: For every "site" definition element in your Sitecore configs, add a "supportedLanguages" attribute that is a comma-delimited list of all languages available on your site (example: supportedLanguages="en-US,fr-FR,pt-PT") Doing this will force the sitemap.xml file to include "alternate" links for each language the site supports. Installation Warning: Your web.config must have a system.webServer/handlers section, and that section must have at least one handler in it (an "add" element) in order to install this package successfully. If you're using the default Sitecore web.config this won't be an issue. The handlers provided in this package must be inserted at the top of the stack of handlers.
Crawler-Lib NHunspell is a spell check, hyphenation, word stemming and thesaurus library based on the Open Office spell check library Hunspell. NHunspell can use the vast amount of OpenOffice dictionaries. It is an alternative to NetSpell, GNU Aspell, ISpell, PSpell and Enchant. It wraps the native libraries for Hunspell and Hyphen and contains a fully managed version of MyThes. This version of the nuget package automatically copies the native binaries to the output directory. NHunspell is licensed under: GPL/LGPL/MPL. Free use in commercial applications is permitted according to the LGPL and MPL licenses. Your commercial application can link against the NHunspell DLLs.
Shared JavaScript-rendering pipeline for SimpleCrawler: the pure-JS DOM prelude and renderer used by the Jint and V8 engine backends.
xBudget CEI Crawler is a crawler to capture investment data for a user with an account on B3 (Brazilian stock exchange) O Canal Eletrônico do Investidor (CEI) é o portal que possibilita ao investidor a consulta consolidada de informações sobre os seus investimentos. xBudget CEI Crawler é um crawler que consulta as informações disponíveis no CEI sem a necessida de navegar pelo site.
Pacote privado para Kernel do projeto Caça Mantos
AppStoresScraper is library for downloading app metadata from Steam, Google, Apple and Windows app stores
The Rask browser host: runs a Rask app's C# components client-side on .NET WebAssembly, with the client halves of the batteries — the mediator and query cache, remote dispatch, accounts, validation and the component kit — on by default. Ships the JS boot scripts, the page shell, and the MSBuild integration that stages them into the published bundle — and, opted in, prerenders every route to static HTML with a sitemap.xml at publish, so a static host serves real pages to visitors and crawlers. Depends on Rask, the shared core.
Jint JavaScript-rendering backend for SimpleCrawler: crawls client-rendered SPAs by executing page scripts on the managed Jint engine (no native dependencies).
A dedicated crawler agent for accessing public data from MangaDex. Built on KamiYomu.CrawlerAgents.Core, it enables efficient search, metadata extraction, and integration with the KamiYomu platform.