puppeteer-page-proxy - npm Package Compare versions

Comparing version 1.2.4 to 1.2.5

changelog.md

 # Change log
+### [1.2.5] - 2020-05-21
+#### Changes
+- Added ability to override requests
+- Increase redirect restriction ([#17](https://github.com/Cuadrix/puppeteer-page-proxy/issues/17))
 ### [1.2.4] - 2020-05-18
@@ -3,0 +7,0 @@ #### Changes

package.json

 {
   "name": "puppeteer-page-proxy",
   "description": "Additional Node.js module to use with 'puppeteer' for setting proxies per page basis.",
-  "version": "1.2.4",
+  "version": "1.2.5",
   "author": "Cuadrix <[e-mail]> (https://github.com/Cuadrix)",
@@ -6,0 +6,0 @@ "homepage": "https://github.com/Cuadrix/puppeteer-page-proxy",

README.md

@@ -21,4 +21,7 @@ # puppeteer-page-proxy <img src="https://i.ibb.co/kQrN9QJ/puppeteer-page-proxy-logo.png" align="right" width="150" height="150">
 - `pageOrReq` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> 'Page' or 'Request' object to set a proxy for.
-- `proxy` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Proxy to use in the current page.
+- `proxy` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)|[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> Proxy to use in the current page.
   * Begins with a protocol (e.g. http://, https://, socks://)
+  * In the case of [proxy per request](https://github.com/Cuadrix/puppeteer-page-proxy#proxy-per-request), this can be an object with optional properites for overriding requests:\
+`url`, `method`, `postData`, `headers`\
+See [request.continue](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestcontinueoverrides) for more info about the above properties.
   
@@ -38,3 +41,3 @@ #### PageProxy.lookup(page[, lookupService, isJSON, timeout])
     
-## Examples
+## Usage
 #### Proxy per page:
@@ -61,3 +64,3 @@ ```js
 ```
-To remove a proxy set this way, simply pass a falsy value (e.g `null`) instead of the proxy;
+To remove proxy, omit or pass in falsy value (e.g `null`):
 ```js
@@ -80,4 +83,4 @@ await useProxy(page, null);
     await page.setRequestInterception(true);
-    page.on('request', req => {
-        useProxy(req, proxy);
+    page.on('request', async req => {
+        await useProxy(req, proxy);
     });
@@ -88,19 +91,32 @@ await page.goto(site);
 The request object itself is passed as the first argument. The proxy can now be changed every request.
-Leaving it as is will have the same effect as applying a proxy for the whole page by passing in the page object as an argument. Basically, the same proxy will be used for all requests within the page.
 
-Using it with other interception methods is straight forward aswell:
+Using it along with other interception methods:
 ```js
 await page.setRequestInterception(true);
-page.on('request', req => {
+page.on('request', async req => {
     if (req.resourceType() === 'image') {
         req.abort();
     } else {
-        useProxy(req, proxy);
+        await useProxy(req, proxy);
     }
 });
 ```
-All requests can be handled exactly once, so it's not possible to intercept the same request after a proxy has been applied to it. This means that it will not be possible to call (e.g. [request.abort](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestaborterrorcode), [request.continue](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestcontinueoverrides)) on the same request without getting a *'Request is already handled!'* error message. This is because `puppeteer-page-proxy` internally calls [request.respond](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestrespondresponse) which fulfills the request.
 
-**NOTE:** It is necessary to set [page.setRequestInterception](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pagesetrequestinterceptionvalue) to true when setting proxies this way, otherwise the function will fail.
+Overriding requests:
+```js
+await page.setRequestInterception(true);
+page.on('request', async req => {
+    await useProxy(req, {
+        proxy: proxy,
+        url: 'https://example.com',
+        method: 'POST',
+        postData: '404',
+        headers: {
+            accept: 'text/html'
+        }
+    });
+});
+```
 
+**NOTE:** It is necessary to set [page.setRequestInterception](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pagesetrequestinterceptionvalue) to true when setting proxies per request, otherwise the function will fail.
 
@@ -142,12 +158,17 @@ #### Authentication:
 ## FAQ
-#### How does puppeteer-page-proxy work?
+#### How does this module work?
 
-It takes over the task of requesting resources from the browser to instead do it internally. This means that the requests that the browser is usually supposed to make directly, are instead intercepted and made indirectly via Node using a requests library. This naturally means that Node also receives the responses that the browser would have normally received from those requests. For changing the proxy, the requests are routed through the specified proxy server using ***-proxy-agent**'s. The responses are then forwarded back to the browser as mock/simulated responses using the [request.respond](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestrespondresponse) method, making the browser think that a response has been received from the server, thus fulfilling the request and rendering any content from the response onto the screen.
+It takes over the task of requesting content **from** the browser to do it internally via a requests library instead. Requests that are normally made by the browser, are thus made by Node. The IP's are changed by routing the requests through the specified proxy servers using ***-proxy-agent's**. When Node gets a response back from the server, it's forwarded to the browser for completion/rendering.
 
-#### Why does the browser show _"Your connection to this site is not secure"_ when connecting to **https** sites?
+#### Why am I getting _"Request is already handled!"_?
 
-This is simply because the server and the browser are unable perform the secure handshakes for the connections due to the requests being intercepted and effectively blocked by Node when forwarding responses to the browser. However, despite the browser alerting of an insecure connection, the requests are infact made securely through Node as seen from the connection property of the response object:
+This happens when there is an attempt to handle the same request more than once. An intercepted request is handled by either [request.abort](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestaborterrorcode), [request.continue](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestcontinueoverrides) or [request.respond](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestrespondresponse) methods. Each of these methods 'send' the request to its destination. A request that has already reached its destination cannot be intercepted or handled.
 
 
-```
+#### Why does the browser show _"Your connection to this site is not secure"_?
+
+Because direct requests from the browser to the server are being intercepted by Node, making the establishment of a secure connection between them impossible. However, the requests aren't made by the browser, they are made by Node. All `https` requests made through Node using this module are secure. This is evidenced by the connection property of the response object:
+
+
+```json
 connection: TLSSocket {
@@ -164,3 +185,3 @@ _tlsOptions: {
 ```
-While a proxy is applied, the browser is just an empty drawing board used for rendering content on the screen. All the network requests and responses, both secure and non-secure, are made by Node. Because of this, it makes no difference whether the site in the browser is shown as insecure or not.
+You can think of the warning as a false positive.
 
@@ -167,0 +188,0 @@ ## Dependencies

src/core/lookup.js

@@ -33,7 +33,6 @@ const lookup = async (page, lookupService = "https://api.ipify.org?format=json", isJSON = true, timeout = 30000) => {
     } catch(error) {
-        if (error.message === "Execution context was destroyed, most likely because of a navigation." || error.message === "Execution context was destroyed.") {
+        if (error.message === "Execution context was destroyed, most likely because of a navigation." || error.message === "Execution context was destroyed.")
             return await XMLHttpRequest();
-        }
     }
 };
 module.exports = lookup;

src/core/proxy.js

@@ -1,32 +0,30 @@
-const {setHeaders, setAgent, request} = require("../lib/request");
-const cookies = require("../lib/cookies");
+const request = require("got");
+const {type} = require("../lib/types");
+const {getCookies, cookieStore} = require("../lib/cookies");
+const {setOverrides, setHeaders, setAgent} = require("../lib/options");
 
-const pageProxy = async (param, proxy) => {
-    let page, req;
-    if (param.constructor.name === "Request") {
-        req = param;
-    } else if (param.constructor.name === "Page") {
-        page = param;
-        await page.setRequestInterception(true);
-    }
-	// Responsible for forward requesting using proxy
-    const $puppeteerPageProxyHandler = async req => {
-		endpoint = req._client._connection._url;
-		targetId = req._frame._id;
-        const cookieJar = cookies.store(
-			await cookies.get(endpoint, targetId)
-		);
+const useProxy = async (target, proxy) => {
+    // Listener responsible for applying proxy
+    const $puppeteerPageProxyHandler = async req => {       
+        endpoint = req._client._connection._url;
+        targetId = req._frame._id;
+        const cookieJar = cookieStore(
+            await getCookies(endpoint, targetId)
+        );
         const options = {
             cookieJar,
             method: req.method(),
+            body: req.postData(),
+            headers: setHeaders(req),
+            agent: setAgent(proxy),
             responseType: "buffer",
-            headers: setHeaders(req),
-            body: req.postData(),
-            followRedirect: false,
             throwHttpErrors: false
         };
         try {
-            options.agent = setAgent(req.url(), proxy);
             const res = await request(req.url(), options);
-            await req.respond(res);
+            await req.respond({
+                status: res.statusCode,
+                headers: res.headers,
+                body: res.body
+            });
         } catch(error) {
@@ -36,7 +34,7 @@ await req.abort();
     };
-	// Remove existing listener for reassigning proxy of current page
-    const removeRequestListener = () => {
+    // Remove existing listener for reassigning proxy of current page
+    const removeRequestListener = (page, listenerName) => {
         const listeners = page.listeners("request");
         for (let i = 0; i < listeners.length; i++) {
-            if (listeners[i].name === "$puppeteerPageProxyHandler") {
+            if (listeners[i].name === listenerName) {
                 page.removeListener("request", listeners[i]);
@@ -46,13 +44,23 @@ }
     };
-    if (req) {
-        $puppeteerPageProxyHandler(req);
-    } else {
-        removeRequestListener();
+    // Proxy per request
+    if (target.constructor.name === "Request") {
+        if (type(proxy) == "object") {
+            target = setOverrides(target, proxy);
+            proxy = proxy.proxy;
+        }
+        await $puppeteerPageProxyHandler(target);
+    // Page-wide proxy
+    } else if (target.constructor.name === "Page") {
+        if (type(proxy) == "object") {
+            proxy = proxy.proxy;
+        }
+        await target.setRequestInterception(true);
+        removeRequestListener(target, "$puppeteerPageProxyHandler");
         if (proxy) {
-            page.on("request", $puppeteerPageProxyHandler);
+            target.on("request", $puppeteerPageProxyHandler);
         } else {
-            await page.setRequestInterception(false);
+            await target.setRequestInterception(false);
         }
     }
 };
-module.exports = pageProxy;
+module.exports = useProxy;

src/index.d.ts

@@ -14,3 +14,3 @@ export = puppeteer_page_proxy;
  */
-declare function puppeteer_page_proxy(page: object, proxy: string): Promise<any>;
+declare function puppeteer_page_proxy(page: object, proxy: string | object): Promise<any>;
 declare namespace puppeteer_page_proxy {
@@ -17,0 +17,0 @@ /**

src/lib/cdp.js

@@ -24,5 +24,3 @@ const cdp = {
             })).result;
-            if (result) {
-                return result.sessionId;
-            }
+            return (result) ? result.sessionId : undefined;
         }
@@ -37,5 +35,3 @@ },
             })).result;
-            if (result) {
-                return result.cookies;
-            }
+            return (result) ? result.cookies : undefined;
         }
@@ -42,0 +38,0 @@ }

src/lib/cookies.js

@@ -6,3 +6,3 @@ const WebSocket = require("ws");
 const cookies = {
-    async get(endpoint, targetId) {
+    async getCookies(endpoint, targetId) {
         const ws = new WebSocket(endpoint, {
@@ -17,6 +17,5 @@ perMessageDeflate: false,
     },
-    store(cookies) {
-        if (!cookies) {
+    cookieStore(cookies) {
+        if (!cookies)
             return;
-        }
         return CookieJar.deserializeSync({
@@ -23,0 +22,0 @@ version: 'tough-cookie@4.0.0',

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

21509

increased by7.37%

Number of package files

11

increased by10%

Lines of code

282

increased by12.8%

Number of lines in readme file

187

increased by12.65%

No dependency changes