How can I obtain the original encoded response size when intercepting requests with Puppeteer?

Question

I'm using this code to log the encoded response size when loading a page in Chrome:

const puppeteer = require("puppeteer");

(async function() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  page._client.on("Network.loadingFinished", data => {
    console.log("finished", { encodedDataLength: data.encodedDataLength });
  });

  // await page.setRequestInterception(true);
  // page.on("request", async request => {
  //   request.continue();
  // });

  await page.goto("http://example.com");
  await browser.close();
})();

This is the output:

finished { encodedDataLength: 967 }

However, if I uncomment the four lines in the code snippet the output changes to:

finished { encodedDataLength: 0 }

This does make some sense, since the intercepted request could have been modified in some way by the client, and it would not have been gzipped again afterwards.

However, is there a way to access the original gzipped response size?

The Chrome trace also doesn't include the gzipped size:

"encodedDataLength": 0, "decodedBodyLength": 1270,

If the answer worked for you, please choose as accepted answer so it doesn't show as unanswered. :) — Md. Abu Taher, Oct 18 '18 at 21:55

Md. Abu Taher · Answer 1 · 2018-10-18T11:40:22.713

1

We can use Content-Length header value for such case.

The good guys at google decided they won't fix some weird bugs closely related to encodedDataLength.

Check the code and result below to see proof.

page.on("request", async request => {
  request.continue();
});

// Monitor using _client
page._client.on("Network.responseReceived", ({ response }) => {
  console.log("responseReceived", [
    response.headers["Content-Length"],
    response.encodedDataLength
  ]);
});

page._client.on("Network.loadingFinished", data => {
  console.log("loadingFinished", [data.encodedDataLength]);
});

// Monitor using CDP
const devToolsResponses = new Map();
const devTools = await page.target().createCDPSession();
await devTools.send("Network.enable");

devTools.on("Network.responseReceived", event => {
  devToolsResponses.set(event.requestId, event.response);
});

devTools.on("Network.loadingFinished", event => {
  const response = devToolsResponses.get(event.requestId);
  const encodedBodyLength =
    event.encodedDataLength - response.headersText.length;
  console.log(`${encodedBodyLength} bytes for ${response.url}`);
});

Result without setRequestInterception:

responseReceived [ '606', 361 ]
loadingFinished [ 967 ]
606 bytes for http://example.com/

Result with setRequestInterception:

responseReceived [ '606', 0 ]
loadingFinished [ 0 ]
-361 bytes for http://example.com/

Tested with multiple gzip tool. Same result everywhere.

The Content-Length Header is far more reliable in every sense.

edited Oct 18 '18 at 11:40

answered Oct 18 '18 at 11:25

Md. Abu Taher

17,395
5
49
73

Thanks for your answer, but a lot of response headers don't include the Content-Length. – Matt Zeunert Oct 20 '18 at 09:58
Can you share one example url? – Md. Abu Taher Oct 20 '18 at 10:13
This file for example: https://fonts.googleapis.com/css?family=YT%20Sans%3A300%2C500%2C700 – Matt Zeunert Oct 20 '18 at 14:25
Are you sure it's gzipped? – Md. Abu Taher Oct 20 '18 at 14:30
Opps, realized that's not the problem. Silly me. – Md. Abu Taher Oct 20 '18 at 14:35
Content-Length is not used when the response header Transfer-Encoding is used, usually set to "chunked", or, if the server is HTTP/1.0-compliant, when the "Connection: close" header is given (the body is then supposed to be read until the connection is closed). These are the 3 message framing mechanisms used in HTTP. https://tools.ietf.org/html/rfc7230#section-3.3 – Renato Apr 23 '20 at 08:42

How can I obtain the original encoded response size when intercepting requests with Puppeteer?

1 Answers1

Linked