flipkart-scraper

Flipkart Scraper

This package will help us to scrape all Flipkart products through Flipkart affiliate API.

Please refer API Documentation here.

See the Examples here

Installation

Install using 'npm'

npm i flipkart-scraper

Install using 'yarn'

yarn add flipkart-scraper

Usage

import { FlipkartScraper } from "flipkart-scraper";

const scraper = new FlipkartScraper(
  "<Affiliate-Id-Here>",
  "<Affiliate-Token-Here>"
);

// 'data' event handler
scraper.on("data", (data) => {
  console.log(data.products.length);
});

// Start the scraper
scraper.start();

Class: FlipkartScraper

• extends EventEmitter

This module will help us to scrape all the Flipkart products.

constructor(affiliateId, affiliateToken, [options])

This will create instance of the FlipkartScraper class with required authentication params.

• options {object}
  • concurrency? {number} Number for parallel processing in the queue, Default set to 2
  • maxRequest? {number} Maximum request to Flipkart affliate server, Default set to 0 - means unlimited
  • maxPage? {number} Maximum number of pages to scrape per category, Default set to 0 - means unlimited

Example

const scraper = new FlipkartScraper(
  "<Affiliate-Id-Here>",
  "<Affiliate-Token-Here>",
  {
    /**
     * It will make 5 parallel request to Flipkart.
     * This is optional param, default is set to 2
     **/
    concurrency: 5,
    /**
     * It will make only 500 request to Flipkart. After that program ends.
     * This is optional param, default is set to 0 means unlimited
     **/
    maxRequest: 500,
    /**
     * Maximum 3 request per category
     **/
    maxPage: 3,
  }
);

scraper.start([categoriesToScrape])

This method will start scraping through Flipkart affiliate API.

• categoriesToScrape? {string[]} Pass the list of categories that you want to scrape. Default set to [] which means all categories.

Example

scraper.start(["telivision", "mobiles"]); // It will scrape only specified categories

scraper.stats(showAsNumbers?=false)

This method will show the stats of scraper. By default stats will come as numerals like (3.1k, 1.45GB)

• showAsNumbers? {boolean} Stats will return as number instead of numerals.

Sample Stats

/*
{
  "startTime": "2021-02-21T08:14:29.445Z",
  "endTime": undefined, // endTime will be available once scraping finished
  "status": "inprogress",
  "concurrency": 30,
  "waitingRequests": 0,
  "productsCount": "19.54k",
  "elapsed": "0:00:06 10ms",
  "durationPerMillionProducts": "0:05:08 558ms",
  "productsPerSec": "3.25k products/sec",
  "avgResponseTime": "147ms",
  "requestPerSec": "7/sec",
  "requestedCount": "43.00",
  "processedCount": "41.00",
  "errorCount": "0.0",
  "retryCount": "0.0",
  "retryHaltCount": "0.0",
  "pendingCategory": 2,
  "completedCategory": 1,
  "downloadSize": "84.78MB",
  "downloadedSpeed": "14.11MB/sec",
  "info": {
    "pendingCategories": [
      {
        "category": "mens_clothing",
        "startTime": "2021-02-21T08:14:29.602Z",
        "noOfPages": 18,
        "elapsed": 5668,
        "totalProducts": 9000
      },
      {
        "category": "mobiles",
        "startTime": "2021-02-21T08:14:29.603Z",
        "noOfPages": 13,
        "elapsed": 5771,
        "totalProducts": 6500
      }
    ],
    "completedCategories": [
      {
        "category": "laptops",
        "noOfPages": 10,
        "totalProducts": 4500,
        "elapsed": 3878
      }
    ]
  }
}
*/

events: 'response'

Emitted when successful HTTP response from the Flipkart Affiliate server.

Example

// 'response' event handler
scraper.on("response", (response) => {
  console.log(response);
});

events: 'data'

Emitted when products returned from Flipkart affiliate API.

Example

// 'data' event handler
scraper.on("data", (data) => {
  console.log(data.apiData.products);
});

events: 'categoryCompleted'

Emitted when all products scraped for the category.

Example

// 'categoryCompleted' event handler
scraper.on("categoryCompleted", (completedCategoryInfo) => {
  console.log(completedCategoryInfo);
});

events: 'finished'

Emitted when scraper finished.

Example

// 'finished' event handler
scraper.on("finished", (info) => {
  console.log(info);
});

events: 'error'

Emitted if any errors occured.

Example

// Triggered if any error occured
scraper.on("error", (error) => {
  console.error(error);
});

events: 'retry'

Emitted if any retry occured.

Example

// Triggered if any retry occured
scraper.on("retry", (retryInfo) => {
  console.log(retryInfo);
});

events: 'retryHalted'

Emitted if any retries failed for 10 times.

Example

// Triggerd when retry failed 10 times
scraper.on("retryHalted", (retryHaltInfo) => {
  console.error(retryHaltInfo);
});