scrapyyy

Scrapyyy

A web scraping tool built with TypeScript, utilizing Cheerio and Axios to fetch and filter HTML content from web pages.

Features

• Fetch and parse HTML from web pages.
• Include or exclude specific HTML tags during scraping.
• Filter content based on CSS selectors.
• Optionally return only the data (text content) or full HTML.

Installation

npm install scrapyyy

Usage

First, import the scraper module and use it to scrape web pages.

import { scraper } from "scrapyyy";

// Example usage
(async () => {
  const result = await scraper("https://example.com", {
    onlyData: true,
    includeTags: ["div", "span"],
    excludeTags: ["script", "style"],
    selector: ["#content", ".main"],
  });

  console.log(result);
})();

API

scraper(url, options):

Parameters:

• url (string): The URL of the webpage to scrape.
• options (object): Optional configuration to modify the scraping process.
  • onlyData (boolean): If true, returns only the text content of the page (default: false).
  • includeTags (string[]): List of tag names to include in the result.
  • excludeTags (string[]): List of tag names to exclude from the result.
  • selector (string[]): CSS selectors to filter the content (e.g., #id, .class, tagName).

Returns:

• Promise: A promise that resolves with the scraped content (either HTML or plain text).

Example:

Scrape HTML Content

const html = await scraper("https://example.com");
console.log(html); // Full HTML content of the page

Scrape with Tag Inclusion/Exclusion

const filtered = await scraper("https://example.com", {
  includeTags: ["html", "body", "head", "title", "p", "div"],
  excludeTags: ["script"],
});
console.log(filtered); // HTML content with only <p> and <div>, excluding <script>

Note : If you are including a certain tag but not its parent tag, it won't be present in the result.

eg. If you include p tag but not include body, no p tags will actually be included.

Scrape Using CSS Selectors

const selectedContent = await scraper("https://example.com", {
  selector: [".header", "#main"],
});
console.log(selectedContent); // Text content from elements matching the provided selectors

Scrape Only Data (Text Content)

const data = await scraper("https://example.com", { onlyData: true });
console.log(data); // Plain text content of the page

License

MIT