@mintlify/scraping
Advanced tools
Scrape documentation frameworks to Mintlify docs.
mintlify-scrape section [url] - scrapes multiple pages in a sitemintlify-scrape page [url] - scrapes a single page in a siteWe currently support:
The commands will automatically detect the framework.
One time use:
npx @mintlify/scraping@latest section [url]
npx @mintlify/scraping@latest page [url]
GitBook, ReadMe, and Docusaurus sites are rendered with Puppeteer, which is an optional peer dependency. Include it when scraping those sites:
npx -p @mintlify/scraping@latest -p puppeteer mintlify-scrape section [url]
Global installation:
npm install @mintlify/scraping@latest -g
Global usage:
mintlify-scrape section [url]
mintlify-scrape page [url]
Provide the relative path or URL to the OpenAPI file to generate frontmatter files for each endpoint.
mintlify-scrape openapi-file [openApiFilename]
-w, --writeFiles Whether or not to write the frontmatter files [boolean] [default: true]
-o, --outDir The folder in which to write any created frontmatter files [string]
Join our Discord community if you have questions or just want to chat:
Built with 💚 by the Mintlify community.
A fast HTML parsing and querying library with jQuery-like selectors. It is a lower-level option: unlike a documentation-oriented scraper, you generally fetch pages and implement the extraction and cleanup yourself.
A browser automation library that can render JavaScript-heavy pages and interact with them. It offers more control over browser behavior, but requires writing more of the scraping workflow yourself.
A Chrome/Chromium automation library suitable for rendering and scraping dynamic pages. It is a general-purpose browser tool, whereas @mintlify/scraping is aimed at extracting web content with less browser-automation setup.
A JavaScript implementation of web standards for parsing and working with HTML in Node.js. It is useful for DOM-based extraction, but does not itself provide a complete website crawling or documentation-scraping workflow.
An HTTP client for making requests. It can retrieve page HTML, but you must handle parsing, content selection, and cleanup separately; @mintlify/scraping is a more specialized choice for extracting page content.
FAQs
Scrape documentation frameworks to Mintlify docs
The npm package @mintlify/scraping receives a total of 0 weekly downloads. As such, @mintlify/scraping popularity was classified as not popular.
We found that @mintlify/scraping demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 7 open source maintainers collaborating on the project.
Threat Research
/Security News
319 browser extensions installed by 13.4 million users hijack searches, capture queries, and collect referral payments from Yahoo and Bing.
Threat Research
Malicious Firefox extensions mimic Phantom, Jupiter, and Bitget wallets to steal recovery phrases and private keys, with one fetching its payload remotely.
Product
Socket Threat Intel now shows which malware campaigns were blocked or detected in your organization, with links to scans and Firewall events.