why some website is utf-8 some are not (python)

Question

import urllib.request

#url="http://www.fafu.edu.cn"
url="http://www.zhihu.com"
m=urllib.request.urlopen(url).read()
#m.decode("utf-8").encode('')
f=open('/Users/HYN/Desktop/url.txt','wb')
f.write(m)
f.close()
#print(m)

I am using python3.4.3 Mac osx 10.10.4

This is my code.When I tried first website i can get the right thing like this :

<link rel="stylesheet" type="text/css" href="/page/main151/style.css" />

<style type="text/css">
<!--
body {font-size:9pt}
td {font-size:9pt}
-->
</style>
<style TYPE="text/css">
<!--
A:link{text-decoration:none; color:#000000}
A:visited{text-decoration:none; color:#000000}
A:hover {color: #ff00ff;text-decoration:underline}
 
#bar a:link {font-size: 10.5pt; font-weight: 700
}
#bar a:visited {font-size: 10.5pt; font-weight: 700
}
#bar a:hover {font-size: 10.5pt; font-weight: 700
}

But when i tried the second website,I can only get this:

1f8b 0800 4e8b c055 02ff ed3c 6b73 d456
96df f32b 14cd 546a a70a b9f5 7e10 eca9
541e 3599 5466 c84e b676 b3a9 94eb 4aba
722b 744b 8da4 f683 a9ad 0236 010c 06c2
f0c8 069c 00d9 1098 2418 4fb2 09c6 364e
d5fe 95b4 badb 9ff8 0b7b ae5e ad7e 896e
db24 ec30 1476 bbaf ce3d e7de 73cf f3de
7375 e0f9 57fe f8f2 dbef 1c7c 952a 07d5
cad4 7307 c807 5541 cecc 247d a4cc bcfc
079a b461 64c2 4715 0788 32ca c8f3 7130
49d7 038b 51e9 b4d9 4155 3c49 a35a ad82
193b a83b d867 e00b 4d19 ae13 6027 881e
31b6 3929 0a3c af88 82ca 663d cb41 5063
f0e1 ba3d 3b49 ff1b f32f 2f31 2fbb d51a
0a6c bd82 73dd 5f7f 7512 9b33 789f 51f6
5ca0 c4d1 54a9 9bb4 871d 137b d8cb f599
c3fa 213b e883 34b1 6f78 762d b05d 2707
dc58 3bda 58fb aab5 7c3d 5cf9 ac75 f583
d6c3 0bad 8de5 ed8f 575a 772f b5be d80a
97d6 1f6d 2e85 6b2b e1e9 bf36 1ede 08ef

How could i get the right thing?

perhaps you could take a look at this http://stackoverflow.com/questions/436220/python-is-there-a-way-to-determine-the-encoding-of-text-file — Henrik, Aug 04 '15 at 10:16

Padraic Cunningham · Accepted Answer · 2015-08-04T10:36:31.143

1

You can use gzip to decompress the gzipped data and decode to utf-8:

import gzip
m = urllib.request.urlopen(url,).read()
data = gzip.decompress(m).decode("utf-8")
with open('/Users/HYN/Desktop/url.txt','w') as f:
    f.write(data)

Or use requests to do it for you:

import requests

r = requests.get(url)

data = r.content.decode("utf-8")

If you print(r.headers) you can see 'content-encoding': 'gzip' to verify the data is gzipped, both websites charset is utf-8, the difference is one is gzipped and the other is not.

edited Aug 04 '15 at 10:36

answered Aug 04 '15 at 10:30

Padraic Cunningham

176,452
29
245
321

thanks ,but there is an error,i used the first way.....UnicodeEncodeError: 'ascii' codec can't encode characters in position 265-289: ordinal not in range(128) – R.hui Aug 04 '15 at 10:45
@R.hui, I tested the code above and it works for me, are you using it exactly as above? – Padraic Cunningham Aug 04 '15 at 10:46
I run it in Sublime Text2 and got the error, but i run it in terminal it works ! – R.hui Aug 04 '15 at 10:55
You might want `encoding="utf-8"` when writing in sublime – Padraic Cunningham Aug 04 '15 at 10:56
`with open('/Users/HYN/Desktop/url.txt','w',encoding="utf-8")` – Padraic Cunningham Aug 04 '15 at 11:14
1

with open('/Users/HYN/Desktop/url.txt','w',encoding="utf-8") it works .thinks a lot – R.hui Aug 04 '15 at 11:18

why some website is utf-8 some are not (python)

1 Answers1