3

I am on Google App Engine 2.5 with Django Template and Webapp Frame.

The db.TextProperty and UTF-8 and Unicode and Decode/Encode have confused me so much. I would really appreciate some experts can offer some suggestions. I have googled for the whole night and still have so many questions.

What I am trying to do:

[utf-8 form input] => [Python, Store in db.TextProperty] => [When Needed, Replace Japanese with English] => [HTML, UTF-8]

According to this answer Zipping together unicode strings in Python

# -*- coding: utf-8 -*-

and all .py files saved in utf-8 format

Here is my code:

#Model.py
class MyModel(db.Model):
  content = db.TextProperty()

#Main.py
def post(self):
    content=cgi.escape(self.request.get('content'))
    #what is the type of content? Unicode? Str? or Other?
    obj = MyModel(content=content)
    #obj = MyModel(content=unicode(content))
    #obj = MyModel(content=unicode(content,'utf-8'))
    #which one is the best?
    obj.put()

#Replace one Japanese word with English word in the content
content=obj.content
#what is the type of content here? db.Text? Unicode? Str? or Other?
#content=unicode(obj.content, 'utf-8') #Is this necessary?
content=content.replace(u'ひと',u'hito')

#Output to HTML
self.response.out.write(template.render(path, {'content':content})
#self.response.out.write(template.render(path, {'content':content.encode('utf-8')})

Hope some Google App Engine engineer can see this question and offer some help. Thanks a lot!

Community
  • 1
  • 1
Susan Mayer
  • 335
  • 1
  • 3
  • 12
  • Hi Susan. Interesting problem, but you may want to include more details like what encoding you're using to store the data and for what language, English or Japanese? If it's Japanese, are you using UTF-8 or something else? Do you always take input in UTF-8 format or is it mix and match? Any more details you can provide can help you get an answer. Good luck! – jamesmortensen May 26 '12 at 13:23
  • 1
    As a starting point, you could use the built-in function `type` to get `content` variable type: `type(content)` will tell you the type returned by `cgi.escape` – Zeugma May 26 '12 at 13:26
  • @jmort253 I intend to keep everything in `UTF-8` format. The HTML form is in `UTF-8` with `charset=utf-8` and no other encodings are used. – Susan Mayer May 26 '12 at 23:29
  • 1
    You shouldn't need to (or want to) call `cgi.escape`. Ever. – Nick Johnson May 28 '12 at 07:10
  • @Nick Huh? No need to use `cgi.escape`? so please tell me: whats this `self.response.out.write(cgi.escape(self.request.get('content'))) ` from https://developers.google.com/appengine/docs/python/gettingstarted/handlingforms – Susan Mayer May 28 '12 at 11:40
  • @SusanMayer A poor alternative to using a template library with built in escaping. The only reason it's used there is to make the example more compact, rather than referencing an external template. – Nick Johnson May 28 '12 at 13:23

2 Answers2

1

First, read this. And this.

In a nutshell, whenever you're dealing with a text string in your app, it should be a unicode string. You should encode into a byte string (an instance of 'str' instead of 'unicode') when you want to send data as bytes - for instance, over HTTP, and you should decode from a byte string when you receive bytes that represent text (and you know their encoding). The only operations you should ever be doing on a byte string that contains encoded text are to decode or encode them.

Fortunately, most frameworks get this right; webapp and webapp2, for instance (I can see you're using webapp) should return unicode strings from all the request methods, and encode any strings you pass to them appropriately. Make sure all the strings you're responsible for are unicode, and you should be fine.

Note that a byte string can store any sort of data - encoded text, an executable, an image, random bytes, encrypted data, and so forth. Without metadata, such as the knowledge that it's text and what encoding it's in, you cannot sensibly do anything with it other than store and retrieve it.

Don't ever try to decode a unicode string, or encode a byte string; it will not do what you expect, and things will go horribly wrong.

Regarding the datastore, db.Text is a subclass of unicode; to all intents and purposes it is a unicode string - it's only different so the datastore can tell it shouldn't be indexed. Likewise, db.Blob is a subclass of str, for storing byte strings.

Nick Johnson
  • 100,655
  • 16
  • 128
  • 198
0

Try 

db.Text("text", encoding="utf-8")

it helps me to save the utf-8 text into the TextProperty()

for the details, please refer to the following link : https://developers.google.com/appengine/docs/python/datastore/typesandpropertyclasses?hl=en#Text

Namh
  • 11
  • 1