0

I'm going to download using GitHub url.
However, the value received by urlopen is not always constant.

If you run the code below,

import urllib.request 

url = "https://github.com/PixarAnimationStudios/USD/archive/v20.11.zip"
u = urllib.request.urlopen(url)

meta = u.info()
print(u.headers.keys())
print(u.headers.get("Content-Type"))
print(u.headers.get("Transfer-Encoding"))
print(u.headers.get("Content-Length"))

case1

['Date', 'Content-Type', 'Content-Length', 'Connection', 'Access-Control-Allow-Origin', 'Content-Disposition', 'Content-Security-Policy', 'ETag', 'Strict-Transport-Security', 'Vary', 'X-Content-Type-Options', 'X-Frame-Options', 'X-XSS-Protection', 'X-Varnish', 'Age', 'Via', 'X-Cache', 'X-Cache-Hits', 'Accept-Ranges', 'Vary', 'X-GitHub-Request-Id']
application/zip
None
30301735

case2

['Date', 'Content-Type', 'Transfer-Encoding', 'Connection', 'Access-Control-Allow-Origin', 'Content-Disposition', 'Content-Security-Policy', 'ETag', 'Strict-Transport-Security', 'Vary', 'X-Content-Type-Options', 'X-Frame-Options', 'X-XSS-Protection', 'X-Varnish', 'Age', 'Via', 'X-Cache', 'X-Cache-Hits', 'Accept-Ranges', 'Vary', 'X-GitHub-Request-Id']
application/zip
chunked
None

What is the difference between Content-Length and Transfer-Encoding?

i used python3.9

chowooseoung
  • 85
  • 7

1 Answers1

1

What is the difference between Content-Length and Transfer-Encoding?

One tells you how long the response's content is (and may be absent), the other tells you whether the response is being sent using something other than directly sending the data (and may be absent).

In the first case, you're seeing that not using a chunked Transfer-Encoding means the server has to send Content-Length up front so your client knows it's received the whole file.

In the second, the file is being sent with chunked, so there's no upfront declaration of the complete length, and Content-Length is omitted. The length of the content can be determined from the sum of each chunk.

Joe
  • 29,416
  • 12
  • 68
  • 88
  • Thank you for your answer. Then, is there any way I can always include content-length? Or how can i getl the size by transfer-encoding? – chowooseoung Jan 08 '21 at 12:30
  • I've added a link to https://stackoverflow.com/questions/4929484/how-to-determine-content-data-length-from-chunked-encoding-if-http-header-not-se . The answer is: download the file, and see how much you got. You'll need to ask a more specific question if that's not an option. – Joe Jan 09 '21 at 02:45