file with Japanese name throws error

Question

class FileUpload(models.Model):
    cstr = models.CharField(max_length=50)
    file_nm = models.CharField(max_length=250,editable=False,verbose_name='Name')
    file_type_cd = models.CharField(max_length=10,editable=False, verbose_name='Type')
    file_size_val = models.CharField(max_length=25,editable=False, verbose_name='Size in MB')
    file_upld_dts = models.DateTimeField(default =datetime.datetime.now() ,editable=False, verbose_name='Uploaded Date & Time')
    fileupload = models.FileField(upload_to=fs,verbose_name='File to Upload')
    file_title = models.CharField(max_length=250,verbose_name='Title')
    fileretention = models.IntegerField(default=18,verbose_name = '# months of Retention')
    def __unicode__(self):
        return unicode(self.file_nm)




 Traceback:
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
      111.                         response = callback(request, *callback_args, **callback_kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/contrib/admin/options.py" in wrapper
      366.                 return self.admin_site.admin_view(view)(*args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/utils/decorators.py" in _wrapped_view
      91.                     response = view_func(request, *args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
      89.         response = view_func(request, *args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/contrib/admin/sites.py" in inner
      196.             return view(request, *args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/utils/decorators.py" in _wrapper
      25.             return bound_func(*args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/utils/decorators.py" in _wrapped_view
      91.                     response = view_func(request, *args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/utils/decorators.py" in bound_func
      21.                 return func(self, *args2, **kwargs2)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/transaction.py" in inner
      209.                 return func(*args, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/contrib/admin/options.py" in add_view
      955.                 self.save_model(request, new_object, form, False)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/contrib/admin/options.py" in save_model
      709.         obj.save()
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/base.py" in save
      463.         self.save_base(using=using, force_insert=force_insert, force_update=force_update)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/base.py" in save_base
      551.                 result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/manager.py" in _insert
      203.         return insert_query(self.model, objs, fields, **kwargs)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/query.py" in insert_query
      1593.     return query.get_compiler(using=using).execute_sql(return_id)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/sql/compiler.py" in execute_sql
      909.         for sql, params in self.as_sql():
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/sql/compiler.py" in as_sql
      872.                 for obj in self.query.objs
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/fields/files.py" in pre_save
      249.             file.save(file.name, file, save=False)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/db/models/fields/files.py" in save
      86.         self.name = self.storage.save(name, content)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/core/files/storage.py" in save
      44.         name = self.get_available_name(name)
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/core/files/storage.py" in get_available_name
      70.         while self.exists(name):
    File "/opt/venv/django-cmdb/lib/python2.6/site-packages/django/core/files/storage.py" in exists
      230.         return os.path.exists(self.path(name))
    File "/usr/lib64/python2.6/genericpath.py" in exists
      18.         st = os.stat(path)

Exception Type: UnicodeEncodeError at /admin/allinone/fileupload/add/
Exception Value: 'ascii' codec can't encode character u'\u524d' in position 76: ordinal not in range(128)

This is my django model for File upload functionality, when a try to upload a file which has a japanese name i'm getting below error.

I'm able to upload normal files, but files with non English names i get error.

I tried setting the LANG variable but i still get this error.

Can someone please help on how to overcome this.

What is `fs` (as in `models.FileField(upload_to=fs,verbose_name='File to Upload')`) set to? — Martijn Pieters, Oct 21 '13 at 07:07
And what does `manage.py shell`, then `import sys; print sys.getfilesystemencoding()` say your filesystem encoding is? — Martijn Pieters, Oct 21 '13 at 07:11
Weird; I'd have expected *that* to be `ASCII` or `None`, for the `os.stat()` call to fail. Clearly `path` is a unicode value, or else you'd not get a encode exception; but Python at that point thinks the locale is **not** UTF-8. How do you set the `LANG` environment variable? — Martijn Pieters, Oct 21 '13 at 07:20
But *normally* I'd expect having set `LANG=en_US.UTF-8` correctly this problem would go away. — Martijn Pieters, Oct 21 '13 at 07:20
export LANG="en_US.UTF-8" i had put this line in bash_profile, that's how i set the LANG variable — Raajsupra, Oct 21 '13 at 07:22
through wsgi from django.core.wsgi import get_wsgi_application application = get_wsgi_application() — Raajsupra, Oct 21 '13 at 07:25
@Raaj: Possible duplicates [http://stackoverflow.com/questions/3715865/unicodeencodeerror-ascii-codec-cant-encode-character][1] [1]: http://stackoverflow.com/questions/3715865/unicodeencodeerror-ascii-codec-cant-encode-character — Darknight, Oct 21 '13 at 07:44
@PSivachandran Went through most of them but none helped, that's why posted a new one — Raajsupra, Oct 21 '13 at 08:18

score 2 · Accepted Answer · edited Apr 13 '17 at 12:13

2

You need to set your locale correctly for Apache; without a correct locale, Python falls back to using ASCII for filenames.

Set the LANG environment variable to LANG=en_US.UTF-8 to the Apache envvars file. On Debian and Ubuntu systems, you find that in /etc/apache2/envvars, on RedHat, in /etc/sysconfig/httpd.

Add the line:

EXPORT LANG=en_US.utf8

If you have trouble locating the envvars file, see this question on ServerFault for some more hints about how to find that file.

edited Apr 13 '17 at 12:13

Community

1
1

answered Oct 21 '13 at 07:27

Martijn Pieters

1,048,767
296
4,058
3,343

How do you know this stuff? – Games Brainiac Oct 21 '13 at 07:51
I had set LANG variable both in /etc/sysconfig/httpd and /etc/init.d/httpd but still i'm not out of the error – Raajsupra Oct 21 '13 at 07:58
Django debug output show environment variables on error; verify that `LANG` is set there. Create a debug view that verifies `sys.getfilesystemencoding()` too, perhaps. – Martijn Pieters Oct 21 '13 at 08:02
@MartijnPieters 20 years eh? Don't worry, I'll catch up soon enough :) – Games Brainiac Oct 21 '13 at 08:07
It's weird, i get the below value when u get it through a view ANSI_X3.4-1968 – Raajsupra Oct 21 '13 at 08:16
Where can i change this?, should i direct put it before the model? – Raajsupra Oct 21 '13 at 08:17
Hate to state the obvious, but you did restart (not reload) Apache right? – Burhan Khalid Oct 21 '13 at 08:44
@Raajsupra: That means an incorrect locale is *still* set. In Django, are there any environment variables starting with `LC_`? If so, what are their values? – Martijn Pieters Oct 21 '13 at 08:54

score 0 · Answer 2 · answered Oct 21 '13 at 08:40

0

Maybe you run Apache with the LANG=C locale by default, which breaks uploading files with special characters in their names at least when running with mod_wsgi.

Activating a UTF-8 locale in /etc/apache2/envvars should resolve the issue.

In httpd.conf add (or change if it's already there):

AddDefaultCharset utf-8

answered Oct 21 '13 at 08:40

Vivek S

5,384
8
51
72

Setting the characterset for Apache will not help here; it's Python that determines the locale from env vars, and `AddDefaultCharset` has no influence on that. – Martijn Pieters Oct 21 '13 at 08:53
@MartijnPieters The developer is runniing the site in apache with mod_wsgi. So, Charset need to be added at both ends, I guess. – Vivek S Oct 21 '13 at 09:00
1

`mod_wsgi` still runs a Python interpreter (either in the Apache process or in a separate daemon). Python uses `sys.getfilesystemencoding()` which on UNIX is *exclusively* based on the locale (`LANG` and `LC_*` vars). – Martijn Pieters Oct 21 '13 at 09:03

file with Japanese name throws error

2 Answers2