Write a CSV from Urlib and manage encoding properly

Question

I need to put the content of a vector/list into a CSV. I'm getting trouble with the "python encoding problem" obviously.

Here is the code we're taklin' about :

import pdb
#pdb.set_trace()
import sys
sys.version_info
import csv
from bs4 import *
import urllib.request
rows=list()


def parse_csv(content, delimiter = ';'):
  csv_data = []
  for line in content.split('\n'):
    csv_data.append( [x.strip() for x in line.split( delimiter )] ) # strips spaces also
  return csv_data



list_url=parse_csv(open('url.csv','rU').read())

for i in range (0,len(list_url)):
    url=str(list_url[i][0]) ## read URL from an array coming from an Url-CSV
    page=urllib.request.urlopen(url)
    soup = BeautifulSoup(page.read(),"html.parser")

    for h1 in soup.find_all('h1'):
        rows.append(h1.get_text().encode('utf-8').strip()) #looks for title

    for h2 in soup.find_all('h2'):
        rows.append(h2.get_text().encode('utf-8').strip()) #looks for title

    restricted_webpage= soup.find( "div", {"id":"ingredients"} )
    readable_restricted=str(restricted_webpage)

    soup2=BeautifulSoup(readable_restricted,"html.parser")

    for td in soup2.find_all('td'):
        rows.append(td.get_text().encode('utf-8').strip())

print(rows)

if sys.version_info >= (3,0,0):
    f = open('FB.csv', 'w', newline='')
else:
    f = open('FB.csv', 'wb')


wr = csv.writer(f)
wr.writerows(rows)

If i do that. My CSV looks like that : (Which is not fine at all)

65,100,117,108,116,32,83,109,97,108,108,32,68,111,103
68,101,32,112,108,117,115,32,100,101,32,56,47,32,49,48,32,109,111,105,115,46
67,101,110,100,114,101,115,32,98,114,117,116,101,115,32,40,37,41
55,46,52
67,101,108,108,117,108,111,115,101,32,98,114,117,116,101,32,40,37,41
49,46,54
70,105,98,114,101,115,32,97,108,105,109,101,110,116,97,105,114,101,115,32,40,37,41
54,46,54
77,97,116,105,195,168,114,101,32,103,114,97,115,115,101,32,40,37,41
49,54,46,48
65,99,105,100,101,32,108,105,110,111,108,195,169,105,113,117,101,32,40,37,41
51,46,49
69,110,101,114,103,105,101,32,109,195,169,116,97,98,111,108,105,115,97,98,108,101,32,40,99,97,108,99,117,108,195,169,101,32,115,101,108,111,110,32,78,82,67,56,53,41,32,40,107,99,97,108,47,107,103,41
51,54,53,50,46,53
69,110,101,114,103,105,101,32,109,195,169,116,97,98,111,108,105,115,97,98,108,101,32,40,109,101,115,117,114,195,169,101,41,32,40,107,99,97,108,47,107,103,41
51,57,48,48,46,48
72,117,109,105,100,105,116,195,169,32,40,37,41
57,46,53
69,120,116,114,97,105,116,32,110,111,110,32,97,122,111,116,195,169,32,40,37,41
52,48,46,53
79,109,195,169,103,97,32,54,32,40,37,41
51,46,49,56
80,114,111,116,195,169,105,110,101,32,98,114,117,116,101,32,40,37,41
50,53,46,48
65,109,105,100,111,110,32,40,37,41
51,53,46,53
67,104,108,111,114,101,32,40,37,41
49,46,52,51
67,117,105,118,114,101,32,40,109,103,47,107,103,41
49,53,46,48
73,111,100,101,32,40,109,103,47,107,103,41
50,46,57
70,101,114,32,40,109,103,47,107,103,41
49,54,55,46,48
77,97,110,103,97,110,195,168,115,101,32,40,109,103,47,107,103,41
54,56,46,48
90,105,110,99,32,40,109,103,47,107,103,41
50,52,50,46,48
66,105,111,116,105,110,101,32,40,109,103,47,107,103,41
51,46,49,51
67,104,111,108,105,110,101,32,40,109,103,47,107,103,41
49,54,48,48,46,48
65,99,105,100,101,32,102,111,108,105,113,117,101,32,40,109,103,47,107,103,41
49,51,46,57
86,105,116,97,109,105,110,101,32,65,32,40,85,73,47,107,103,41
51,50,48,48,48,46,48
86,105,116,97,109,105,110,101,32,66,49,32,84,104,105,97,109,105,110,101,32,40,109,103,47,107,103,41
50,55,46,53
86,105,116,97,109,105,110,101,32,66,50,32,82,105,98,111,102,108,97,118,105,110,101,32,40,109,103,47,107,103,41
52,57,46,54
86,105,116,97,109,105,110,101,32,66,51,32,78,105,97,99,105,110,101,32,40,109,103,47,107,103,41
52,57,48,46,48
86,105,116,97,109,105,110,101,32,66,53,32,65,99,105,100,101,32,112,97,110,116,111,116,104,195,169,110,105,113,117,101,32,40,109,103,47,107,103,41
49,52,55,46,56
86,105,116,97,109,105,110,101,32,66,54,32,80,121,114,105,100,111,120,105,110,101,32,40,109,103,47,107,103,41
55,55,46,49
86,105,116,97,109,105,110,101,32,67,32,40,109,103,47,107,103,41
50,48,48,46,48
86,105,116,97,109,105,110,101,32,68,51,32,40,85,73,47,107,103,41
56,48,48,46,48
86,105,116,97,109,105,110,101,32,69,32,40,109,103,47,107,103,41
54,48,48,46,48
65,114,103,105,110,105,110,101,32,40,37,41
49,46,53,51
76,117,116,195,169,105,110,101,32,40,109,103,47,107,103,41
53,46,48
77,195,169,116,104,105,111,110,105,110,101,32,67,121,115,116,105,110,101,32,40,37,41
49,46,49,56
84,97,117,114,105,110,101,32,40,109,103,47,107,103,41
50,57,48,48,46,48
66,101,97,103,108,101,32,65,100,117,108,116
66,101,97,103,108,101,32,97,100,117,108,116,101,44,32,195,160,32,112,97,114,116,105,114,32,100,101,32,49,50,32,109,111,105,115
65,99,105,100,101,32,97,114,97,99,104,105,100,111,110,105,113,117,101,32,40,37,41
48,46,48,55
67,101,110,100,114,101,115,32,98,114,117,116,101,115,32,40,37,41
54,46,49
66,105,111,116,105,110,101,32,40,109,103,47,107,103,41
50,46,57,50
70,105,98,114,101,115,32,97,108,105,109,101,110,116,97,105,114,101,115,32,40,37,41
49,49,46,48
68,76,45,109,195,169,116,104,105,111,110,105,110,101,32,40,37,41
48,46,53
69,80,65,47,68,72,65,32,40,37,41
48,46,51
77,97,116,105,195,168,114,101,32,103,114,97,115,115,101,32,40,37,41
49,50,46,48
67,101,108,108,117,108,111,115,101,32,98,114,117,116,101,32,40,37,41
51,46,55
67,104,108,111,114,117,114,101,32,100,101,32,103,108,117,99,111,115,97,109,105,110,101,32,40,109,103,47,107,103,41
52,57,53,46,48
71,108,117,99,111,115,97,109,105,110,101,32,112,108,117,115,32,99,104,111,110,100,114,111,195,175,116,105,110,101,32,40,109,103,47,107,103,41
53,48,48,46,48
76,45,99,97,114,110,105,116,105,110,101,32,40,109,103,47,107,103,41
49,48,48,46,48
65,99,105,100,101,32,108,105,110,111,108,195,169,105,113,117,101,32,40,37,41
50,46,52,50
76,117,116,195,169,105,110,101,32,40,109,103,47,107,103,41
53,46,48
69,110,101,114,103,105,101,32,109,195,169,116,97,98,111,108,105,115,97,98,108,101,32,40,99,97,108,99,117,108,195,169,101,32,115,101,108,111,110,32,78,82,67,56,53,41,32,40,107,99,97,108,47,107,103,41
51,52,50,53,46,48
69,110,101,114,103,105,101,32,109,195,169,116,97,98,111,108,105,115,97,98,108,101,32,40,109,101,115,117,114,195,169,101,41,32,40,107,99,97,108,47,107,103,41
51,53,56,52,46,48
77,195,169,116,104,105,111,110,105,110,101,32,67,121,115,116,105,110,101,32,40,37,41
48,46,57,50
72,117,109,105,100,105,116,195,169,32,40,37,41
57,46,53
69,120,116,114,97,105,116,32,110,111,110,32,97,122,111,116,195,169,32,40,37,41
52,49,46,55
79,109,195,169,103,97,32,51,32,40,37,41
48,46,54,52
79,109,195,169,103,97,32,54,32,40,37,41
50,46,54,54
80,104,111,115,112,104,111,114,101,32,40,37,41
48,46,55
80,114,111,116,195,169,105,110,101,32,98,114,117,116,101,32,40,37,41
50,55,46,48
65,109,105,100,111,110,32,40,37,41
51,52,46,52
84,97,117,114,105,110,101,32,40,109,103,47,107,103,41
51,53,48,48,46,48
86,105,116,97,109,105,110,101,32,65,32,40,85,73,47,107,103,41
50,57,48,48,48,46,48
86,105,116,97,109,105,110,101,32,67,32,40,109,103,47,107,103,41
51,48,48,46,48
86,105,116,97,109,105,110,101,32,69,32,40,109,103,47,107,103,41
54,48,48,46,48
67,97,108,99,105,117,109,32,40,37,41
48,46,57
80,111,108,121,112,104,195,169,110,111,108,115,32,100,101,32,116,104,195,169,32,118,101,114,116,32,101,116,32,100,101,32,114,97,105,115,105,110,115,32,40,109,103,47,107,103,41
49,53,48,46,48
67,104,108,111,114,101,32,40,37,41
48,46,54,51
67,117,105,118,114,101,32,40,109,103,47,107,103,41
49,53,46,48
73,111,100,101,32,40,109,103,47,107,103,41
52,46,56
70,101,114,32,40,109,103,47,107,103,41
50,49,50,46,48
77,97,103,110,195,169,115,105,117,109,32,40,37,41
48,46,48,56
77,97,110,103,97,110,195,168,115,101,32,40,109,103,47,107,103,41
55,49,46,48
80,111,116,97,115,115,105,117,109,32,40,37,41
48,46,55
83,195,169,108,195,169,110,105,117,109,32,40,109,103,47,107,103,41
48,46,50,57
83,111,100,105,117,109,32,40,37,41
48,46,52
90,105,110,99,32,40,109,103,47,107,103,41
50,48,49,46,48
67,104,111,108,105,110,101,32,40,109,103,47,107,103,41
50,50,48,48,46,48
65,99,105,100,101,32,102,111,108,105,113,117,101,32,40,109,103,47,107,103,41
49,50,46,57
86,105,116,97,109,105,110,101,32,66,49,32,84,104,105,97,109,105,110,101,32,40,109,103,47,107,103,41
50,53,46,55
48,46,49,51
86,105,116,97,109,105,110,101,32,66,50,32,82,105,98,111,102,108,97,118,105,110,101,32,40,109,103,47,107,103,41
52,54,46,50
86,105,116,97,109,105,110,101,32,66,51,32,78,105,97,99,105,110,101,32,40,109,103,47,107,103,41
52,53,56,46,55
86,105,116,97,109,105,110,101,32,66,53,32,65,99,105,100,101,32,112,97,110,116,111,116,104,195,169,110,105,113,117,101,32,40,109,103,47,107,103,41
49,51,55,46,57
86,105,116,97,109,105,110,101,32,66,54,32,80,121,114,105,100,111,120,105,110,101,32,40,109,103,47,107,103,41
55,50,46,48
86,105,116,97,109,105,110,101,32,68,51,32,40,85,73,47,107,103,41
56,48,48,46,48
65,114,103,105,110,105,110,101,32,40,37,41
49,46,53,50
76,45,108,121,115,105,110,101,32,40,37,41
49,46,48,56
74,97,99,107,32,82,117,115,115,101,108,108,32,65,100,117,108,116
7
53,46,57
66,105,111,116,105,110,101,32,40,109,103,47,107,103,41

This implies i would have to decode the CSV to use it in the future. This doesn't work for me. I need something "visual".

If i try to replace every

   encode('utf-8').strip()

by :

strip()

I receive the error :

wr.writerows(rows) UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 8: ordinal not in range(128)

strip

is very useful here because it helps me get rid of the '\n' i don't really like

Here is a second thing i've tried : to get rid of those é'èàù% and other trouble maker by turning to ASCI with that one :

yourstring = yourstring.encode('ascii', 'ignore').decode('ascii')

The results was not bad but not enough :

"
",A,d,u,l,t, ,S,m,a,l,l, ,D,o,g
D,e, ,p,l,u,s, ,d,e, ,8,/, ,1,0, ,m,o,i,s,.
"
", , , , , , , , , , , , , , , , ,C,e,n,d,r,e,s, ,b,r,u,t,e,s, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,7,.,4,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,e,l,l,u,l,o,s,e, ,b,r,u,t,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,6,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,F,i,b,r,e,s, ,a,l,i,m,e,n,t,a,i,r,e,s, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,6,.,6,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,a,t,i,r,e, ,g,r,a,s,s,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,6,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,l,i,n,o,l,i,q,u,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,.,1,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,n,e,r,g,i,e, ,m,t,a,b,o,l,i,s,a,b,l,e, ,(,c,a,l,c,u,l,e, ,s,e,l,o,n, ,N,R,C,8,5,), ,(,k,c,a,l,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,6,5,2,.,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,n,e,r,g,i,e, ,m,t,a,b,o,l,i,s,a,b,l,e, ,(,m,e,s,u,r,e,), ,(,k,c,a,l,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,9,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,H,u,m,i,d,i,t, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,9,.,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,x,t,r,a,i,t, ,n,o,n, ,a,z,o,t, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,0,.,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,O,m,g,a, ,6, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,.,1,8,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,P,r,o,t,i,n,e, ,b,r,u,t,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,m,i,d,o,n, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,5,.,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,h,l,o,r,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,4,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,u,i,v,r,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,I,o,d,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,.,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,F,e,r, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,6,7,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,a,n,g,a,n,s,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,6,8,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,Z,i,n,c, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,4,2,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,B,i,o,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,.,1,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,h,o,l,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,6,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,f,o,l,i,q,u,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,3,.,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,A, ,(,U,I,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,2,0,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,1, ,T,h,i,a,m,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,7,.,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,2, ,R,i,b,o,f,l,a,v,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,9,.,6,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,3, ,N,i,a,c,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,9,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,5, ,A,c,i,d,e, ,p,a,n,t,o,t,h,n,i,q,u,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,4,7,.,8,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,6, ,P,y,r,i,d,o,x,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,7,7,.,1,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,C, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,D,3, ,(,U,I,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,8,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,E, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,6,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,r,g,i,n,i,n,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,5,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,L,u,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,t,h,i,o,n,i,n,e, ,C,y,s,t,i,n,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,1,8,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,T,a,u,r,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,9,0,0,.,0,"
", , , , , , , , 
"
",B,e,a,g,l,e, ,A,d,u,l,t
B,e,a,g,l,e, ,a,d,u,l,t,e,",", , ,p,a,r,t,i,r, ,d,e, ,1,2, ,m,o,i,s
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,a,r,a,c,h,i,d,o,n,i,q,u,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,0,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,e,n,d,r,e,s, ,b,r,u,t,e,s, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,6,.,1,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,B,i,o,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,.,9,2,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,F,i,b,r,e,s, ,a,l,i,m,e,n,t,a,i,r,e,s, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,l,i,n,o,l,i,q,u,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,.,4,2,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,L,u,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,n,e,r,g,i,e, ,m,t,a,b,o,l,i,s,a,b,l,e, ,(,c,a,l,c,u,l,e, ,s,e,l,o,n, ,N,R,C,8,5,), ,(,k,c,a,l,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,4,2,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,n,e,r,g,i,e, ,m,t,a,b,o,l,i,s,a,b,l,e, ,(,m,e,s,u,r,e,), ,(,k,c,a,l,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,5,8,4,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,t,h,i,o,n,i,n,e, ,C,y,s,t,i,n,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,9,2,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,H,u,m,i,d,i,t, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,9,.,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,x,t,r,a,i,t, ,n,o,n, ,a,z,o,t, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,1,.,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,O,m,g,a, ,3, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,6,4,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,O,m,g,a, ,6, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,.,6,6,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,P,h,o,s,p,h,o,r,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,P,r,o,t,i,n,e, ,b,r,u,t,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,7,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,m,i,d,o,n, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,4,.,4,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,T,a,u,r,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,5,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,A, ,(,U,I,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,9,0,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,C, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,E, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,6,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,a,l,c,i,u,m, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,P,o,l,y,p,h,n,o,l,s, ,d,e, ,t,h, ,v,e,r,t, ,e,t, ,d,e, ,r,a,i,s,i,n,s, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,5,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,h,l,o,r,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,6,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,u,i,v,r,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,I,o,d,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,.,8,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,F,e,r, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,1,2,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,a,g,n,s,i,u,m, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,0,8,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,a,n,g,a,n,s,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,7,1,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,P,o,t,a,s,s,i,u,m, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,S,l,n,i,u,m, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,2,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,S,o,d,i,u,m, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,4,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,Z,i,n,c, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,0,1,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,h,o,l,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,2,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,f,o,l,i,q,u,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,2,.,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,1, ,T,h,i,a,m,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,5,.,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,1,2, ,C,y,a,n,o,c,o,b,a,l,a,m,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,1,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,2, ,R,i,b,o,f,l,a,v,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,6,.,2,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,3, ,N,i,a,c,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,5,8,.,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,5, ,A,c,i,d,e, ,p,a,n,t,o,t,h,n,i,q,u,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,3,7,.,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,B,6, ,P,y,r,i,d,o,x,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,7,2,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,V,i,t,a,m,i,n,e, ,D,3, ,(,U,I,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,8,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,r,g,i,n,i,n,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,5,2,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,L,-,l,y,s,i,n,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,0,8,"
", , , , , , , , 
"
",J,a,c,k, ,R,u,s,s,e,l,l, ,A,d,u,l,t
J,a,c,k, ,R,u,s,s,e,l,l, ,T,e,r,r,i,e,r, ,a,d,u,l,t,e,",", , ,p,a,r,t,i,r, ,d,e, ,1,0, ,m,o,i,s
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,a,r,a,c,h,i,d,o,n,i,q,u,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,0,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,e,n,d,r,e,s, ,b,r,u,t,e,s, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,5,.,9,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,B,i,o,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,.,0,7,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,F,i,b,r,e,s, ,a,l,i,m,e,n,t,a,i,r,e,s, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,7,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,D,L,-,m,t,h,i,o,n,i,n,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,6,5,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,P,A,/,D,H,A, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,0,.,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,M,a,t,i,r,e, ,g,r,a,s,s,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,6,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,e,l,l,u,l,o,s,e, ,b,r,u,t,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,.,3,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,C,h,l,o,r,u,r,e, ,d,e, ,g,l,u,c,o,s,a,m,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,4,9,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,G,l,u,c,o,s,a,m,i,n,e, ,p,l,u,s, ,c,h,o,n,d,r,o,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,5,0,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,L,-,c,a,r,n,i,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,1,5,0,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,A,c,i,d,e, ,l,i,n,o,l,i,q,u,e, ,(,%,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,2,.,8,8,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,L,u,t,i,n,e, ,(,m,g,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,5,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,n,e,r,g,i,e, ,m,t,a,b,o,l,i,s,a,b,l,e, ,(,c,a,l,c,u,l,e, ,s,e,l,o,n, ,N,R,C,8,5,), ,(,k,c,a,l,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,7,1,6,.,0,"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,E,n,e,r,g,i,e, ,m,t,a,b,o,l,i,s,a,b,l,e, ,(,m,e,s,u,r,e,), ,(,k,c,a,l,/,k,g,),"
", , , , , , , , 
"
", , , , , , , , , , , , , , , , ,3,9,6,4,.,0,"
", , , , , , , ,

When i try to add a strip in addition. The CSV produced is empty.

Now just a strip produce an error :

UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 8: ordinal not in range(128)

I read and tried most of this stack-topic

The ASCII way isn't very nice because i lose informations. And i do need them.

How can i put properly my list into this FB.csv ?

Ohhh noo infinite looping code. It should be shorter I guess. — juanmajmjr, Jul 20 '15 at 22:53
what about [unicodecsv](https://github.com/jdunck/python-unicodecsv)? — NightShadeQueen, Jul 20 '15 at 23:02

score 0 · Answer 1 · edited May 23 '17 at 12:30

0

Test yout code with these instructions in the beggining of your code:

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

You should take a look here

edited May 23 '17 at 12:30

Community

1
1

answered Jul 21 '15 at 01:48

dot.Py

5,007
5
31
52

the `setdefaultencoding` is not available anymore in python 3. And when i ask my IDLE what is the default encoding, answer is UTF8. So i guess this isn't required, is it ? – BoobaGump Jul 21 '15 at 08:34

score 0 · Accepted Answer · answered Jul 21 '15 at 09:38

It looks like you are using Python 3. BeautifulSoup returns Unicode strings and csv takes Unicode strings, so don't encode them.

writerows takes a list of lists of column text. It looks like you are passing just a list of text, so the strings are being enumerated and giving one character at a time.

Here's what happens giving a list of encoded strings to csv.writerows:

>>> import csv
>>> with open('test.csv','w',newline='') as f:
...  w=csv.writer(f)
...  rows = ['abc,def'.encode('utf8'),'ghi,jkl'.encode('utf8')]
...  w.writerows(rows)
...
>>>

C:\>type test.csv
97,98,99,44,100,101,102
103,104,105,44,106,107,108

And here's what happens when you give a list of Unicode strings instead of a list of lists of Unicode strings. Note to specify an encoding for the Unicode strings, it is specified in the open:

>>> import csv
>>> with open('test.csv','w',encoding='utf8',newline='') as f:
...  w=csv.writer(f)
...  rows=['\xe8abc,def','\xe8def,ghi']
...  w.writerows(rows)
...
>>>

Note: Code page 65001 is UTF-8. Switching displays the characters correctly:

C:\>chcp 65001
Active code page: 65001

C:\>type test.csv
è,a,b,c,",",d,e,f
è,d,e,f,",",g,h,i

Here's how to get correct output:

>>> import csv
>>> with open('test.csv','w',encoding='utf8',newline='') as f:
...  w=csv.writer(f)
...  rows = [['abcd\xe8','d\xe8fg'],['ghijk','lmnop']]
...  w.writerows(rows)
...
>>>

C:\>type test.csv
abcdè,dèfg
ghijk,lmnop

Thank you for your clear answer. I can't get it work though. I edited my post withe url.csv if some of you want to give it a try with the correction you suggested. — BoobaGump, Jul 21 '15 at 09:53
Sorry. It works but not the way i was expecting. The thing is i keep having those "," between every characters of my outputs. When i open the CSV in excel it looks like : h,e,l,l,o (a line below) i,s,u,c,k,a,t,c,s,v. I don't understand why. Because when i `print(rows)` it appears in the shell quiet properly. — BoobaGump, Jul 21 '15 at 10:01
You are still passing a "list of strings" instead of a "list of lists of strings", giving you the problem in my second example. `['abc','def']` vs. `[['abc','def']]`. `writerow` takes the former while `writerows` takes the latter. — Mark Tolonen, Jul 21 '15 at 13:45

Write a CSV from Urlib and manage encoding properly

2 Answers2