eclipse vs tomcat deployment - exported war (partially) fails while the project runs in eclipse

Question

I have a webapp in eclipse juno - when I hit Run on server runs fine - either inside eclipse's browser (I am on windows) or in FF.

Right click > export war > dump this into $CATALINA_HOME/webapps > all is working fine (got unpacked alright) EXCEPT

• my custom tags - I had a WEB-INF\functions.tld file which is apparently not read. The only difference between the auto-generated eclipse server.xml (in Servers project) and the default Tomcat server.xml was the line :

  <Context docBase="ted2012" path="/ted2012" 
  reloadable="true"source="org.eclipse.jst.jee.server:ted2012"/>
  

source being a WTP specific attribute.
This I managed to solve - see my answer

• Tomcat won't get the Url correctly through - see the pics in my answer.

Questions :

1. (Unsolved) Why Tomcat does not decode the Url correctly - while eclipse does ? Where is the failure ? Do see my specific question for this for extensive details on the call stack and where exactly tomcat fails
2. Why did not tomcat see the tld in the first place while eclipse did ? Why did I have to edit the web.xml ? (worked around in my answer, should be another question)

The code is in github - in the file INSTRUCTIONS.txt there are detailed instructions to set the project up and reproduce the bug pictured in my answer below.

Tomcat 7.0.32, eclipse 4.2, java 1.7.9

Answer 1

For decoding URIs correctly, you need URIEncoding connector attribute in Tomcat:

<connector ... URIEncoding="UTF-8" ... />

See my rant https://stackoverflow.com/a/15587140/995876

So it doesn't come with normal code, you need it in the application server configuration separately or use an application server that defaults to UTF-8. There is no way to affect this from code unfortunately.

Drop the decodeRequest and never use new String/getBytes without explicit encoding argument.

---

Alternative.

If you can't edit the server connector configuration, you can fix your code by providing the encoding explicitly to new String:

public static String decodeRequest(String parameter) {
     return new String(parameter.getBytes("iso-8859-1"), "UTF-8");
}

Answer 2

One thing that helped was to add to the web-xml :

<jsp-config>
    <taglib>
        <taglib-uri>
            functions
        </taglib-uri>
        <taglib-location>
            functions.tld
        </taglib-location>
    </taglib>
</jsp-config>

Now tomcat (7.0.30) sees my taglib which is used to encode URIs.

---

Strange thing : when I print the username with system out I get ???? in tomcat's console instead of hieroglyphs. Maybe this points to the issue ? In my controller I have :

final String username = Helpers.decodeRequest(request
                .getParameter("user"));
System.out.println("ProfileController.doGet() user name DECODED : "
                                + username);

where :

private static final String CHARSET_FOR_URL_ENCODING = "UTF-8";

public static String decodeRequest(String parameter)
        throws UnsupportedEncodingException {
    System.out.println(Charset.defaultCharset()); // EDIT: suggested by @Esailija
    if (parameter == null)
        return null;
    System.out.println("decode - request.getBytes(\"iso-8859-1\"):"
            + new String(parameter.getBytes("iso-8859-1")));
    System.out.println("decode - request.getBytes(\"iso-8859-1\") BYTES:"
            + parameter.getBytes("iso-8859-1"));
    for (byte iterable_element : parameter.getBytes("iso-8859-1")) {
        System.out.println(iterable_element);
    }
    System.out.println("decode - request.getBytes(\"UTF-8\"):"
            + new String(parameter.getBytes(CHARSET_FOR_URL_ENCODING))); // UTF-8
    return URLDecoder.decode(new String(parameter.getBytes("iso-8859-1")),
            CHARSET_FOR_URL_ENCODING);
}

So tomcat :

windows-1252 // EDIT: suggested by @Esailija
decode - request.getBytes("iso-8859-1"):╬╡╬╗╬╗╬╖╬╜╬▒╧?╬▒
decode - request.getBytes("iso-8859-1") BYTES:[B@d171825
-50
-75
-50
-69
-50
-69
-50
-73
-50
-67
-50
-79
-49
-127
-50
-79
decode - request.getBytes("UTF-8"):├Ä┬╡├Ä┬╗├Ä┬╗├Ä┬╖├Ä┬╜├Ä┬▒├?┬?├Ä┬▒
ProfileController.doGet() user name DECODED : ╬╡╬╗╬╗╬╖╬╜╬▒╧?╬▒
???????? // user Dao System.out.println("ελληναρα");
com.mysql.jdbc.JDBC4PreparedStatement@67322bd9: SELECT * FROM users WHERE username='╬╡╬╗╬╗╬╖╬╜╬▒╧?╬▒'
ProfileController.doGet() user : null

Eclipse :

UTF-8 // EDIT: suggested by @Esailija
decode - request.getBytes("iso-8859-1"):ελληναρα
decode - request.getBytes("iso-8859-1") BYTES:[B@44c353ae
-50
-75
-50
-69
-50
-69
-50
-73
-50
-67
-50
-79
-49
-127
-50
-79
decode - request.getBytes("UTF-8"):ελληναÏα
ProfileController.doGet() user name DECODED : ελληναρα
ελληναρα // user Dao System.out.println("ελληναρα");
com.mysql.jdbc.JDBC4PreparedStatement@73aae7c6: SELECT * FROM users WHERE username='ελληναρα'
ProfileController.doGet() user : com.ted.domain.User@4b22015d

EDIT : if I change the eclipse encoding in prefs > workspace > text file encoding and set the default (Cp1252)

windows-1252
decode - request.getBytes("iso-8859-1"):λαλακης
decode - request.getBytes("iso-8859-1") BYTES:[B@5ef1946a
-50
// same bytes ....
decode - request.getBytes("UTF-8"):λαλακη�‚
ProfileController.doGet() user name DECODED : λαλακης
ελληναÏ?α
com.mysql.jdbc.JDBC4PreparedStatement@4646ebd8: SELECT * FROM users WHERE username='λαλακης'
ProfileController.doGet() user : null

and Eclipse also fails

---

NB : Tomcat does print the correct url in the address bar

Eclipse is fine :

Notice that Firefox automatically decodes the Url (to my bewilderment)