Google App Engine - Data being stored in a weird way

Question

I'm using Java. This is the pure data that gets inserted in the datastore:

<p>Something</p>\n<p>That</p>\n<p> </p>\n<p>Should.</p>\n<p> </p>\n
<p>I have an interesting question.</p>\n<p>Why are you like this?</p>\n
<p> </p>\n<p>Aren't you fine?</p>

This is how it gets stored:

<p>Something</p> <p>That</p> <p>�</p> <p>Should.</p> <p>�</p> 
<p>I have an interesting question.</p> <p>Why are you like this?</p> 
<p>�</p> <p>Aren't you fine?</p>

What's up with the weird symbols? This happens only live, not on my local dev_appserver.

EDIT

Here's the code that inserts the data:

String content = ""; // this is where the data is stored
try {
    ServletFileUpload upload = new ServletFileUpload();
    FileItemIterator iter = upload.getItemIterator(request);
    while(iter.hasNext()) {
        FileItemStream item = iter.next();
        InputStream stream = item.openStream();

        if(item.isFormField()) {
            String fieldName = item.getFieldName();
            String fieldValue = new String(IOUtils.toByteArray(stream), "utf-8");
            LOG.info("Got a form field: " +fieldName+" with value: "+fieldValue);
            // assigning the value
            if(fieldName.equals("content")) content = fieldValue;
        } else {
            ...
        }
    }
} catch (FileUploadException e){
}

...
// insert it in datastore
Recipe recipe = new Recipe(user.getKey(), title, new Text(content), new Text(ingredients), tagsAsStrings);
pm.makePersistent(recipe);

It's a multipart/form-data form so I have to do that little item.isFormField() magic to get the actual content, and construct a String. Maybe that's causing the weird encoding issue? Not sure.

To retrieve the data I simply do:

<%=recipe.getContent().getValue()%>

Since content is of type Text (app engine type) I use the .getValue() to get the actual result. I don't think it's an issue with retrieving the data, since I can see the weird characters directly in the online app-engine datastore viewer.

\n is the indicator for a new line. Is that what you are referring to? — Kevin P, Apr 18 '11 at 19:58
It's 99% an encoding issue; how are you dealing with encoding in your code? Can you post some relevant snippet? — systempuntoout, Apr 18 '11 at 20:00
@systempuntoout: I'm not dealing with it. Is there something special I *need* to do? — Luca Matteis, Apr 18 '11 at 20:10
How do you know it gets stored like that? Where are you reading that? Datastore viewer or Jsp? — systempuntoout, Apr 18 '11 at 21:05
@systempuntoout: what i'm showing there is directly from the datastore viewer. The JSP just shows a plain question mark symbol. — Luca Matteis, Apr 18 '11 at 21:15
@DataNucleus, @systempuntoout: I update the code to show you how I store and retrieve the data. — Luca Matteis, Apr 19 '11 at 07:27
@Luca are you using the `<%@ page pageEncoding="UTF-8" %>` in your JSP files? — systempuntoout, Apr 20 '11 at 19:16
@systempuntoout: no I'm not. which jsp should this be in? because as i stated before, it's not a rendering issue since the data is stored that way in the datastore. — Luca Matteis, Apr 20 '11 at 21:14
@systempuntoout: added that, same output. Unicode isn't being stored :( — Luca Matteis, Apr 21 '11 at 07:11

score 0 · Answer 1 · answered Apr 20 '11 at 04:38

0

Are you using eclipse ? if yes check under File > Properties > Text File encoding that your file is UTF-8 encoding.

I would guess not.

So, change it to UTF-8 and your issue should get fixed.

regards

didier

answered Apr 20 '11 at 04:38

Didier Durand

627
7
16

This will only take care of the .java source code encoding, which is a separate issue. – Luca Matteis Apr 20 '11 at 07:31

score 0 · Accepted Answer · edited May 23 '17 at 12:18

0

Followed this page to create a Servlet Filter so that all my pages were being encoded in utf8:

How to get UTF-8 working in Java webapps?

After creating the filter, everything works!

edited May 23 '17 at 12:18

Community

1
1

answered Apr 21 '11 at 07:42

Luca Matteis

29,161
19
114
169

Google App Engine - Data being stored in a weird way

2 Answers2