python unicode csv export using pyramid

Question

I'm trying to export mongodb that has non ascii characters into csv format. Right now, I'm dabbling with pyramid and using pyramid.response.

from pyramid.response import Response
from mycart.Member import Member

@view_config(context="mycart:resources.Member", name='', request_method="POST", permission = 'admin')
def member_export( context, request):
     filename = 'member-'+time.strftime("%Y%m%d%H%M%S")+".csv"
     download_path = os.getcwd() + '/MyCart/mycart/static/downloads/'+filename

     member = Members(request)

     my_list = [['First Name,Last Name']]

     record = member.get_all_member( )             
     for r in record:

         mystr = [ r['fname'],  r['lname']]

         my_list.append(mystr)

     with open(download_path, 'wb') as f:
         fileWriter = csv.writer(f, delimiter=',',quotechar='|', quoting=csv.QUOTE_MINIMAL)
         for l in my_list:
             print(l)
             fileWriter.writerow(l)

     size = os.path.getsize(download_path)
     response = Response(content_type='application/force-download', content_disposition='attachment; filename=' + filename)
     response.app_iter = open(download_path , 'rb')
     response.content_length = size

     return response

In mongoDB, first name is showing 王, when I'm using print, it too is showing 王. However, when I used excel to open it up, it shows random stuff - ç¾…

However, when I tried to view it in shell

$ more member-20130227141550.csv

It managed to display the non ascii character correctly.

How should I rectify this problem?

May be you can try something mentioned in this [question](http://stackoverflow.com/questions/4182603/python-how-to-convert-a-string-to-utf-8) — Brandon Gao, Feb 27 '13 at 06:48
Or you can use unicodecsv to do the job, [unicodecsv](https://github.com/jdunck/python-unicodecsv) since python's internal csv module doesn't support unicode. — Brandon Gao, Feb 27 '13 at 07:01
Excel interprets CSV files according to the current language settings of the computer. It doesn't recognize UTF-8 and instead will happily mis-interpret the data as a different encoding. Solution: target your CSV encoding to whomever is going to use it. — Martijn Pieters, Feb 27 '13 at 14:53

score 1 · Answer 1 · edited May 23 '17 at 12:03

I'm not a Windows guy, so I am not sure whether the problem may be with your code or with excel just not handling non-ascii characters nicely. But I have noticed that you are writing your file with python csv module, which is notorious for headaches with unicode.

Other users have reported success with using unicodecsv as a replacement for the csv module. Perhaps you could try dropping in this module as a csv writer and see if your problem magically goes away.

python unicode csv export using pyramid

1 Answers1