scraper-api
Advanced tools
Readme
Interface to call ScraperAPI.com easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.
$ npm install --save scraper-api
const scraperAPI = require('scraper-api')({
// ... options
});
scraperAPI.get('http://httpbin.org/ip')
.then(result => {
// result => '<!doctype html> ...'
})
.catch(error => {
console.error(error);
//=> 'Internal server error'
});
ScraperAPI ⏏Creates new instance of ScraperAPI with the provided options.
Kind: Exported function
Returns: ScraperAPI - New instance of ScraperAPI.
| Param | Type | Default | Description |
|---|---|---|---|
| options | Object | Optional configuration options to pass into ScraperAPI. | |
| [options.apiKey] | String | process.env.SCRAPER_API_KEY | API key for Scraper API. Defaults to pulling API Key from environment variable. |
| [options.renderJs] | Boolean | false | Render JavaScript on the page before scraping the HTML for the page. |
| [options.keepHeaders] | Boolean | false | Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in options.gotOptions.headers. |
| [options.geoCode] | String | 'us' | Geo code in which to use proxies for when scraping. See documentation for more information. |
| [options.premium] | Boolean | false | Whether to use premium proxies. Caution: This will cost 10-25 times more than standard proxies. |
| [options.sessionId] | Number | A numeric session id to use to maintain the same proxy. See ScraperAPI.session() for more information. | |
| [options.gotOptions] | Object | {} | Additional options to pass into got for requests to ScraperAPI. |
Example
const scraperAPI = require('scraper-api')({
// options...
});
Access to the uninstantiated ScraperAPI class.
Kind: static property of scraperAPI
Example
const ScraperAPI = require('scraper-api').ScraperAPI;
const scraperAPI = new ScraperAPI({
// options...
});
Kind: global class
ScraperAPIPromise.<String>Promise.<Object>Constructor for the ScraperAPI class.
| Param | Type | Default | Description |
|---|---|---|---|
| [options] | Object | {} | Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information. |
ScraperAPICreates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through the same proxy. See documentation for more information.
Kind: instance method of ScraperAPI
Returns: ScraperAPI - A new instance of ScraperAPI with sessionId set to the provided id.
| Param | Type | Default | Description |
|---|---|---|---|
| id | Number | Session ID in which to use for the given session. Must only contain numbers. | |
| [options] | Object | {} | Options to override for all subsequent requests to Scraper API. Same as the global options. |
Example
const session = scraperAPI.session(1234);
let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'
Promise.<String>Calls Scraper API with a GET request to the provided url.
Kind: instance method of ScraperAPI
Returns: Promise.<String> - Promise that resolves with HTML source from requested URL.
| Param | Type | Default | Description |
|---|---|---|---|
| url | String | The URL in which to scrape. | |
| [options] | Object | {} | Options to override for this specific request. |
Example
let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'
Promise.<Object>Calls Scraper API with a POST request to the provided url with the provided data.
Kind: instance method of ScraperAPI
Returns: Promise.<Object> - Promise that resolves with object response from Scraper API. See documentation.
| Param | Type | Default | Description |
|---|---|---|---|
| url | String | The URL in which to scrape. | |
| data | Object | form-data | Data in which to post to the provided URL. Must be either a plain object or instance of form-data. | |
| [options] | Object | {} | Options to override for this specific request. May be any of the global options and any additional options below. |
| [options.form] | Boolean | false | Set true if provided data is form data and should be sent as such. By default, data will be sent as JSON. |
Example
let result = await scraperAPI.post('https://example.com/endpoint', {
hello: 'world',
some: 'data'
});
If you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in options.gotOptions.headers and enabling this option.
let result = await scraperAPI.get('https://google.com', {
keepHeaders: true,
gotOptions: {
headers: {
'My-Custom-Header': 'some value'
}
}
});
Tests are written and provided as part of the module. You may run the tests by calling:
$ npm run test
MIT License. See License in the repository.
FAQs
Interface for calling ScraperAPI from Node
The npm package scraper-api receives a total of 4 weekly downloads. As such, scraper-api popularity was classified as not popular.
We found that scraper-api demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
UnitedHealth Group disclosed that the ransomware attack on Change Healthcare compromised protected health information for millions in the U.S., with estimated costs to the company expected to reach $1 billion.
Security News
GitHub is susceptible to a CDN flaw that allows attackers to host malware on any public repository.
Security News
At Node Congress, Socket CEO Feross Aboukhadijeh uncovers the darker aspects of open source, where applications that rely heavily on third-party dependencies can be exploited in supply chain attacks.