puppeteer-page-proxy

puppeteer-page-proxy

Additional Node.js module to use with puppeteer for setting proxies per page basis.

Forwards intercepted requests from the browser to Node.js where it handles the request then returns the response to the browser, changing the proxy as a result.

Features

• Proxy per page and per request
• Supports ( http, https, socks4, socks5 ) proxies
• Authentication
• Cookie handling internally

Installation

npm i puppeteer-page-proxy

API

PageProxy(pageOrReq, proxy)

• pageOrReq <object> 'Page' or 'Request' object to set a proxy for.
• proxy <string> Proxy to use in the current page.
  • Begins with a protocol e.g. http://, https://, socks://

PageProxy.lookup(page[, lookupService, isJSON, timeout])

• page <object> 'Page' object to execute the request on.
• lookupService <string> External lookup service to request data from.
  • Fetches data from api.ipify.org by default.
• isJSON <boolean> Whether to JSON.parse the received response.
  • Defaults to true.
• timeout <number|string> Time in milliseconds after which the request times out.
  • Defaults to 30000 ms.
• returns: <Promise> Promise which resolves to the response of the lookup request.

NOTE: By default this method expects a response in JSON format and JSON.parse's it to a usable javascript object. To disable this functionality, set isJSON to false.

Examples

Proxy per page:

const puppeteer = require('puppeteer');
const useProxy = require('puppeteer-page-proxy');

(async () => {
    const site = 'https://example.com';
    const proxy = 'http://host:port';
    const proxy2 = 'https://host:port';
    
    const browser = await puppeteer.launch({headless: false});

    const page = await browser.newPage();
    await useProxy(page, proxy);
    await page.goto(site);

    const page2 = await browser.newPage();
    await useProxy(page2, proxy2);
    await page2.goto(site);
})();

Changing proxy if a page is already using one:

await useProxy(page, proxy);
await page.goto(site, {waitUntil: 'networkidle2'});

await useProxy(page, proxy2);
await page.reload();

Proxy per request:

const puppeteer = require('puppeteer');
const useProxy = require('puppeteer-page-proxy');

(async () => {
    const site = 'https://example.com';
    const proxy = 'socks://host:port';

    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    await page.setRequestInterception(true);
    page.on('request', async req => {
        await useProxy(req, proxy); // 'req' as argument
    });
    await page.goto(site);
})();

When changing proxies this way, the request object itself is passed as the first argument. Now 'proxy' can be changed every request. Leaving it as is will have the same effect as useProxy(page, proxy), meaning that the same proxy will be used for all requests within the page.

NOTE: It is necessary page.setRequestInterception() to true, otherwise the function will fail.

Using with other request listeners:

await page.setRequestInterception(true);
page.on('request', async req => {
	if (req.resourceType() === 'image') {
		req.abort();
	} else {
		await useProxy(req, proxy);
	}
});

Internally puppeteer-page-proxy handles the request which was passed in, then responds back to the browser using the interception method request.respond(). Calling this method will fulfill the request with a given response and set the proxy.

NOTE: Since all requests can be handled exactly once, it's not possible to call other interception methods e.g. request.abort(), request.continue() after calling useProxy(), without getting a 'Request is already handled!' error message.

Authentication:

const proxy = 'https://login:pass@host:port';

Lookup IP used by proxy -> Useful in headless environment:

const puppeteer = require('puppeteer');
const useProxy = require('puppeteer-page-proxy');

(async () => {
    const site = 'https://example.com';
    const proxy1 = 'http://host:port';
    const proxy2 = 'https://host:port';
    
    const browser = await puppeteer.launch({headless: false});

    /**1*/
    const page1 = await browser.newPage();
    await useProxy(page1, proxy1);
    let data = await useProxy.lookup(page1); // Waits until done, 'then' continues
        console.log(data.ip);
    await page1.goto(site);
    
    /**2*/
    const page2 = await browser.newPage();
    await useProxy(page2, proxy2);
    useProxy.lookup(page2).then(data => {   // Executes and 'comes back' once done
        console.log(data.ip);
    });
    await page2.goto(site);
})();

Dependencies

• Got
• http-proxy-agent
• https-proxy-agent
• socks-proxy-agent
• tough-cookie

Changelog

[1.2.0] - 2020-02-09

Major changes

• Added capability to change proxy per request.
• Added capability of changing the proxy if a page is already using one.
• Replaced Request with Got for forwarding requests.
• Added tough-cookie for handling cookies
• Now communicates directly with Chrome DevTools Protocol when getting cookies

Minor changes

• Added a simple type enforcer and proxy validator.
• Removed some redundant code.
• Removed obsolete 'cache' parameter