Socket
Socket
Sign inDemoInstall

@scrapingant/scrapingant-client

Package Overview
Dependencies
13
Maintainers
2
Versions
6
Alerts
File Explorer

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

@scrapingant/scrapingant-client

ScrapingAnt API client for JavaScript


Version published
Maintainers
2
Weekly downloads
109
decreased by-3.54%

Weekly downloads

Readme

Source

ScrapingAnt API client for JavaScript

@scrapingant/scrapingant-client is the official library to access ScrapingAnt API from your JavaScript applications. It runs both in Node.js and browser and provides useful features like automatic retries and parameters encoding to improve the ScrapingAnt usage experience.

Quick Start

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the example.com site.
client.scrape('https://example.com')
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

API key

In order to get API key you'll need to register at ScrapingAnt Service

Retries with exponential backoff

Network communication sometimes fails, that's a given. The client will automatically retry requests that failed due to a network error, an internal error of the ScrapingAnt API (HTTP 500+). By default, it will retry up to 8 times. First retry will be attempted after ~500ms, second after ~1000ms and so on. You can configure those parameters using the maxRetries and minDelayBetweenRetriesMillis options of the ScrapingAntClient constructor.

API Reference

All public classes, methods and their parameters can be inspected in this API reference.

ScrapingAntClient

ScrapingAntClient is the official library to access ScrapingAnt API from your JavaScript applications. It runs both in Node.js and browser.


new ScrapingAntClient(options)
ParamTypeDefault
[options]object
[options.maxRetries]number8
[options.minDelayBetweenRetriesMillis]number500
[options.timeoutSecs]number60
[options.apiKey]string

scrapingAntClient.scrape(url, [parameters])ScrapingAnt API response

https://docs.scrapingant.com/request-response-format#available-parameters

ParamType
urlstring
[parameters]object
[parameters.browser]boolean
[parameters.cookies]string
[parameters.headers]object
[parameters.js_snippet]string
[parameters.proxy_type]string
[parameters.proxy_country]string
[parameters.wait_for_selector]string
[parameters.return_text]boolean

IMPORTANT NOTE: parameters.js_snippet will be encoded to Base64 automatically by the ScrapingAnt JS client library.


ScrapingAntApiError

An ScrapingAntApiError is thrown for successful HTTP requests that reach the API, but the API responds with an error response. Typically, those are internal errors, which are automatically retried, or validation errors, which are thrown immediately, because a correction by the user is needed.

Properties

NameTypeDescription
messagestringError message returned by the API.
statusCodenumberHTTP status code of the error.
httpMethodstringHTTP method of the API call.

Examples

Using residential proxy

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Get the residential IP info using httpbin.org
client.scrape('https://httpbin.org/ip', { proxy_type: 'residential' })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Sending custom cookies

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the httpbin.org site and get all the cookies sent before
client.scrape('https://httpbin.org/cookies', { cookies: 'cookieName1=cookieVal1;cookieName2=cookieVal2' })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Adding custom headers

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the httpbin.org site and get all the headers that would be sent before
client.scrape('https://httpbin.org/headers', { headers: { scraping: "is cool!" } })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Executing custom JS snippet

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the httpbin.org site and replace all the content with "Hello, world"
const customJsSnippet = "var str = 'Hello, world!';\n" +
    "var htmlElement = document.getElementsByTagName('html')[0];\n" +
    "htmlElement.innerHTML = str;"
client.scrape('https://httpbin.org/cookies', { js_snippet: customJsSnippet })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Keywords

FAQs

Last updated on 06 May 2022

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc