4

Is there a way to convert an ANSI string to UTF using Java.

I have a custom serializer that uses readUTF & writeUTF methods of the DataInputStream class to deserialize and serialze string. If i receive a string encoded in ANSI and is too long, ~100000 chars long i get the error;

Caused by: java.io.UTFDataFormatException: encoded string too long: 106958 bytes

However in my Junit tests i'm able create a string with 120000 'a's and it works perfectly

I have checked the following posts but still having errors;

Community
  • 1
  • 1
n002213f
  • 7,805
  • 13
  • 69
  • 105

4 Answers4

6

This error is not caused by character encoding. It means the length of the UTF data is wrong.

EDIT: Just realized this is a writing error, not reading error.

The UTF length is only 2 bytes so it can only hold 64K UTF-8 bytes. You are trying to writing 100K, it's not going to work.

This limit is hardcoded and no way to get around this,

if (utflen > 65535)
    throw new UTFDataFormatException(
            "encoded string too long: " + utflen + " bytes");
ZZ Coder
  • 74,484
  • 29
  • 137
  • 169
  • Interesting, but why do all my tests with more characters pass? – n002213f Sep 23 '09 at 14:50
  • You have to show me your test cases. They are wrong. See my edits. – ZZ Coder Sep 23 '09 at 14:52
  • i used the following code to generate the test string; StringBuffer sb2 = new StringBuffer(); for (int i=0; i < 120000;i++) { sb2.append("a"); } String longString2 = sb2.toString(); – n002213f Sep 23 '09 at 14:59
  • You can create long strings, until memory is out. You just can't write long strings using writeUTF(). Write it your own way with a 4 byte length header. – ZZ Coder Sep 23 '09 at 15:28
3
byte[] asciiBytes = ...;
String unicode = new String(asciiBytes, "US-ASCII");
byte[] utfBytes = unicode.getBytes("UTF-8");
iammichael
  • 9,477
  • 3
  • 32
  • 42
  • It seems I misread the original question regarding ASCII vs. ANSI, and with the latest question edits, my answer is not really relevant. – iammichael Sep 24 '09 at 13:42
2

Which ANSI codepage? There are lots of different character encodings which all refer to "ANSI". The DOS codepage is 437 (without the drawing symbols). If you use codepage 850, this will work:

String unicode = new String(bytes, "IBM850");

(where bytes is an array with the ANSI characters). After that, you can convert this string into a byte array with any encoding using unicode.getBytes(encoding).

Windows often uses the codepage 1252 (use "windows-1252" for that).

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • tried it but does not work, i get the same error. Is there a way to check the encoding in a string so that i can be sure its ANSI? – n002213f Sep 23 '09 at 15:02
  • this will convert ANSI from telnet, like a mud game, to a "regular" String? – Thufir Sep 05 '13 at 00:01
  • This will convert bytes from any source to a Unicode string. But for it to work properly, you need to know exactly which encoding the source is using. It doesn't matter if that's a file, a remote service or a hardware device. – Aaron Digulla Sep 05 '13 at 08:04
1

ZZ Coder already answered the question, but I have written a more detailed explanation and suggesting a workaround on this blog. Basically, the problem is in DataOutputStream, because it restricts the writeable String to 64KB. There are other possible workarounds to bystep the issue, some might work without breaking the actual binary data format one is using...

István
  • 518
  • 1
  • 5
  • 17