
Security News
Open Source CAI Framework Handles Pen Testing Tasks up to 3,600× Faster Than Humans
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.
proxycrawl
Advanced tools
Dependency free module for scraping and crawling websites using [ProxyCrawl](https://proxycrawl.com) API
Dependency free module for scraping and crawling websites using ProxyCrawl API
Install using npm
npm i proxycrawl
Require the necessary API class in your project.
You can get your ProxyCrawl free token from here.
const { CrawlingAPI, ScraperAPI, LeadsAPI, ScreenshotsAPI } = require('proxycrawl');
Initialize with one of your account tokens, either normal or javascript token. Then make get or post requests accordingly.
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });
Pass the url that you want to scrape plus any options from the ones available in the API documentation.
api.get(url, options);
Example:
api.get('https://www.facebook.com/britneyspears').then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
You can pass any options from ProxyCrawl API.
Example:
api.get('https://www.reddit.com/r/pics/comments/5bx4bx/thanks_obama/', {
userAgent: 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
format: 'json'
}).then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
Pass the url that you want to scrape, the data that you want to send which can be either a json or a string, plus any options from the ones available in the API documentation.
api.post(url, data, options);
Example:
api.post('https://producthunt.com/search', { text: 'example search' }).then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
You can send the data as application/json instead of x-www-form-urlencoded by setting options postType
as json.
api.post('https://httpbin.org/post', { some_json: 'with some value' }, { postType: 'json' }).then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
Pass the url that you want to scrape, the data that you want to send which can be either a json or a string, plus any options from the ones available in the API documentation.
api.put(url, data, options);
Example:
api.put('https://producthunt.com/search', { text: 'example search' }).then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
If you need to scrape any website built with Javascript like React, Angular, Vue, etc. You just need to pass your javascript token and use the same calls. Note that only .get
is available for javascript and not .post
.
const api = new CrawlingAPI({ token: 'YOUR_JAVASCRIPT_TOKEN' });
api.get('https://www.nfl.com').then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
Same way you can pass javascript additional options.
api.get('https://www.freelancer.com', { pageWait: 5000 }).then(response => {
if (response.statusCode === 200) {
console.log(response.body);
}
}).catch(error => console.error);
You can always get the original status and proxycrawl status from the response. Read the ProxyCrawl documentation to learn more about those status.
api.get('https://craiglist.com').then(response => {
console.log(response.originalStatus, response.pcStatus);
}).catch(error => console.error);
Initialize the Scraper API and use it in the same way as the Crawling API (see above). Use it with your normal token.
const api = new ScraperAPI({ token: 'YOUR_TOKEN' });
api.get('https://www.amazon.com/Halo-SleepSack-Swaddle-Triangle-Neutral/dp/B01LAG1TOS').then(response => {
if (response.statusCode === 200) {
console.log(response.json);
}
}).catch(error => console.error);
Initialize with your Leads API token and call the getFromDomain
method.
const api = new LeadsAPI({ token: 'YOUR_TOKEN' });
api.getFromDomain('somesite.com').then(response => {
console.log(response.leads);
});
Initialize with your Screenshots API token and call the get
method, then do whatever you need with the binary content. For example save it in a file.
You can pass any of the available parameters
const api = new ScreenshotsAPI({ token: 'YOUR_TOKEN' });
api.get('https://www.amazon.com').then(response => {
fs.writeFileSync('amazon.jpg', response.body, { encoding: 'binary' });
});
// Example with parameters
api.get('https://www.amazon.com', { device: 'mobile' }).then(response => {
fs.writeFileSync('amazon-mobile.jpg', response.body, { encoding: 'binary' });
});
If you have questions or need help using the library, please open an issue or contact us.
Copyright 2022 ProxyCrawl
FAQs
Dependency free module for scraping and crawling websites using [ProxyCrawl](https://proxycrawl.com) API
The npm package proxycrawl receives a total of 270 weekly downloads. As such, proxycrawl popularity was classified as not popular.
We found that proxycrawl demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.
Security News
Deno 2.4 brings back bundling, improves dependency updates and telemetry, and makes the runtime more practical for real-world JavaScript projects.
Security News
CVEForecast.org uses machine learning to project a record-breaking surge in vulnerability disclosures in 2025.