Setting the default Java character encoding

Question

How do I properly set the default character encoding used by the JVM (1.5.x) programmatically?

I have read that -Dfile.encoding=whatever used to be the way to go for older JVMs. I don't have that luxury for reasons I wont get into.

I have tried:

System.setProperty("file.encoding", "UTF-8");

And the property gets set, but it doesn't seem to cause the final getBytes call below to use UTF8:

System.setProperty("file.encoding", "UTF-8");

byte inbytes[] = new byte[1024];

FileInputStream fis = new FileInputStream("response.txt");
fis.read(inbytes);
FileOutputStream fos = new FileOutputStream("response-2.txt");
String in = new String(inbytes, "UTF8");
fos.write(in.getBytes());

Excellent comments guys - and things I was already thinking myself. Unfortunately there is an underlying String.getBytes() call that I have no control over. The only way I currently see to get around it is to set the default encoding programmatically. Any other suggestions? — , Dec 12 '08 at 05:50
maybe irrelevant question but, is there difference when UTF8 is set with "UTF8", "UTF-8" or "utf8". Recently I found that IBM WAS 6.1 EJB and WEB containers differently treats (in way of case-sensitivity) strings used to define encoding. — igor.beslic, May 29 '11 at 20:27
Setting or reading the `file.encoding` property is [not supported](http://bugs.sun.com/view_bug.do?bug_id=4163515). — McDowell, Mar 24 '12 at 17:26
Just a detail but: prefer UTF-8 to UTF8 (only the former is standard). This still applies in 2012... — Christophe Roussy, Mar 22 '12 at 13:16
@erickson Am still not clear with the query, Is it not true that, "file.encoding" is relevant when character based I/O streams are used(all subclasses of `class Reader` & `class Writer`)? Because `class FileInputStream` is byte based I/O stream, so why one should care about character set in byte-based I/O stream? — overexchange, Dec 21 '14 at 04:54

score 359 · Accepted Answer · edited May 23 '17 at 11:54

359

Unfortunately, the file.encoding property has to be specified as the JVM starts up; by the time your main method is entered, the character encoding used by String.getBytes() and the default constructors of InputStreamReader and OutputStreamWriter has been permanently cached.

As Edward Grech points out, in a special case like this, the environment variable JAVA_TOOL_OPTIONS can be used to specify this property, but it's normally done like this:

java -Dfile.encoding=UTF-8 … com.x.Main

Charset.defaultCharset() will reflect changes to the file.encoding property, but most of the code in the core Java libraries that need to determine the default character encoding do not use this mechanism.

When you are encoding or decoding, you can query the file.encoding property or Charset.defaultCharset() to find the current default encoding, and use the appropriate method or constructor overload to specify it.

edited May 23 '17 at 11:54

Community

1
1

answered Dec 12 '08 at 05:56

erickson

265,237
58
395
493

12

For completeness I would like to add that with a bit of trickery you can get to the actually used default encoding (as is cached), thanks to Gary Cronin: byte [] byteArray = {'a'}; InputStream inputStream = new ByteArrayInputStream(byteArray); InputStreamReader reader = new InputStreamReader(inputStream); String defaultEncoding = reader.getEncoding(); http://lists.xcf.berkeley.edu/lists/advanced-java/1999-October/001995.html – Stijn de Witt Mar 11 '11 at 12:11
2

[JDK-4163515](http://bugs.java.com/view_bug.do?bug_id=4163515) has some more info on setting the `file.encoding` sysprop after JVM startup. – Caspar Aug 27 '14 at 04:00
4

I was scratching my head cause that command was not working on Windows, linux and mac perfectly... then i put " around the value like this: java -D"file.encoding=UTF-8" -jar – cabaji99 Sep 22 '17 at 15:35
check my answer in case of Java Spring Boot: https://stackoverflow.com/a/48952844/986160 – Michail Michailidis Feb 23 '18 at 17:04

dwardu · Answer 2 · 2009-03-08T04:46:37.847

191

From the JVM™ Tool Interface documentation…

Since the command-line cannot always be accessed or modified, for example in embedded VMs or simply VMs launched deep within scripts, a JAVA_TOOL_OPTIONS variable is provided so that agents may be launched in these cases.

By setting the (Windows) environment variable JAVA_TOOL_OPTIONS to -Dfile.encoding=UTF8, the (Java) System property will be set automatically every time a JVM is started. You will know that the parameter has been picked up because the following message will be posted to System.err:

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8

edited Mar 08 '09 at 04:46

answered Mar 08 '09 at 04:31

dwardu

2,240
1
14
10

Do you know that "Picked up..." statement would be printed in Tomcat logs? – thatidiotguy Aug 15 '12 at 13:59
1

Hi Edward Grech I thank you for your solution. It was resolved my problmem in another forum post. http://stackoverflow.com/questions/14814230/the-arabic-input-parameter-passed-as-like-a-junk-input-jasper/14825894#14825894 – Smaug Feb 12 '13 at 05:21
1

@Tiny Java understands both. http://stackoverflow.com/questions/6031877/jvm-property-dfile-encoding-utf8-or-utf-8 – DLight Apr 13 '16 at 12:37

score 80 · Answer 3 · edited Mar 22 '13 at 10:16

80

I have a hacky way that definitely works!!

System.setProperty("file.encoding","UTF-8");
Field charset = Charset.class.getDeclaredField("defaultCharset");
charset.setAccessible(true);
charset.set(null,null);

This way you are going to trick JVM which would think that charset is not set and make it to set it again to UTF-8, on runtime!

edited Mar 22 '13 at 10:16

Lukas Eder

211,314
129
689
1,509

answered Feb 20 '13 at 19:09

naskoos

925
6
6

3

NoSuchFieldException for me – SparK Mar 13 '13 at 19:56
12

For the hack to work, you need to assume the security manager is off. If you don't have a way to set a JVM flag, you might (probably) have a security manager enabled system as well. – Yonatan Aug 24 '13 at 17:20
Though i haven't understood what it is, it works fine for me! Thanks. Hope it doesn't create any new issues to my app. Cheers! – Krish Nakum R Oct 05 '16 at 11:50
This worked for me, but the underlying issue was the ssh connections to spin up or jars had its LC_* set wrong (in the profile). – Fish Biscuit Apr 18 '17 at 15:24
8

JDK9 does *not* approve of this hack anymore. `WARNING: An illegal reflective access operation has occurred • WARNING: Illegal reflective access by [..] • WARNING: Please consider reporting this to the maintainers of [..] • WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations • WARNING: All illegal access operations will be denied in a future release` – dotwin Jan 14 '18 at 16:05
2

@Enerccio: That's not a good answer, that's a dirty hack, and a problem waiting to happen. That should only be used as an emergency measure. – sleske Mar 07 '18 at 18:20
@sleske problem is that java should have a way to override this but alas they don't, so this is a good answer because it is the ONLY answer – Enerccio Mar 08 '18 at 12:24
1

@Enerccio: It's arguable whether Java "should" have a way to set this - one could also argue that developers "should" explicitly specify encoding whenever it's relevant. At any rate, this solution has the potential to cause serious trouble in the longer run, hence the "for emergency use only" caveat. Actually, even emergency use is questionable, because there _is_ a supported way of doing it, setting JAVA_TOOL_OPTIONS as explained in another answer. – sleske Mar 08 '18 at 13:10
@sleske all other solutions can't change during the runtime... and if you have library that uses default encoding and you might not have sources and you must use that library, this is only working solution... – Enerccio Mar 10 '18 at 13:55
@Enerccio: If this solution works, using JAVA_TOOL_OPTIONS should work too, and is actually a supported solution. – sleske Mar 11 '18 at 21:31
For me, just setting the system property helped fix my encoding problem where the IDE used `UTF-8` and the `JAR` file the default system encoding which bugged my resource bundle strings. – BullyWiiPlaza May 05 '18 at 18:41
I confirm this work on JRE 1.8 and chinese windows 10! – Cheung Jun 11 '20 at 07:48

score 40 · Answer 4 · edited Mar 30 '17 at 14:15

I think a better approach than setting the platform's default character set, especially as you seem to have restrictions on affecting the application deployment, let alone the platform, is to call the much safer String.getBytes("charsetName"). That way your application is not dependent on things beyond its control.

I personally feel that String.getBytes() should be deprecated, as it has caused serious problems in a number of cases I have seen, where the developer did not account for the default charset possibly changing.

score 20 · Answer 5 · answered Dec 12 '08 at 05:36

20

I can't answer your original question but I would like to offer you some advice -- don't depend on the JVM's default encoding. It's always best to explicitly specify the desired encoding (i.e. "UTF-8") in your code. That way, you know it will work even across different systems and JVM configurations.

answered Dec 12 '08 at 05:36

Marc Novakowski

44,628
11
58
63

9

Except, of course, if you're writing a desktop app and processing some user-specified text that does not have any encoding metadata - then the platform default encoding is your best guess as to what the user might be using. – Michael Borgwardt Mar 11 '09 at 15:29
@MichaelBorgwardt "then the platform default encoding is your best guess" you seem to be advising that wanting to *change* the default is not such a good idea. Do you mean, use an explicit encoding wherever possible, using the supplied dafault when nothing else is possible? – Raedwald Feb 10 '12 at 13:33
1

@Raedwald: yes, that's what I meant. The platform default encoding is (at least on an end user machine) what users in the locale the system is set to are typically using. That is information you should use if you have no better (i.e. document-specific) information. – Michael Borgwardt Feb 10 '12 at 14:00
1

@MichaelBorgwardt Nonsense. Use a library to auto-detect the input encoding, and save as Unicode with BOM. That is the only way to deal with and fight encoding hell. – Aleksandr Dubinsky Dec 16 '13 at 14:24
I think you two are not in the same page. Michael talks about decoding while Raedwald you talk about processing after decoding. – WesternGun Jan 21 '16 at 11:37

score 15 · Answer 6 · answered Jan 20 '12 at 18:09

15

Try this :

    new OutputStreamWriter( new FileOutputStream("Your_file_fullpath" ),Charset.forName("UTF8"))

answered Jan 20 '12 at 18:09

Emmanuel.B

361
3
9

score 7 · Answer 7 · edited Jul 02 '19 at 14:49

We were having the same issues. We methodically tried several suggestions from this article (and others) to no avail. We also tried adding the -Dfile.encoding=UTF8 and nothing seemed to be working.

For people that are having this issue, the following article finally helped us track down describes how the locale setting can break unicode/UTF-8 in Java/Tomcat

http://www.jvmhost.com/articles/locale-breaks-unicode-utf-8-java-tomcat

Setting the locale correctly in the ~/.bashrc file worked for us.

score 7 · Answer 8 · edited Jan 03 '16 at 16:15

7

I have tried a lot of things, but the sample code here works perfect. Link

The crux of the code is:

String s = "एक गाव में एक किसान";
String out = new String(s.getBytes("UTF-8"), "ISO-8859-1");

edited Jan 03 '16 at 16:15

randers

5,031
5
37
64

answered Jul 03 '14 at 09:33

Lavixu

1,348
15
20

score 6 · Answer 9 · answered Feb 23 '18 at 17:01

In case you are using Spring Boot and want to pass the argument file.encoding in JVM you have to run it like that:

mvn spring-boot:run -Drun.jvmArguments="-Dfile.encoding=UTF-8"

this was needed for us since we were using JTwig templates and the operating system had ANSI_X3.4-1968 that we found out through System.out.println(System.getProperty("file.encoding"));

Hope this helps someone!

score 4 · Answer 10 · edited Jun 20 '20 at 09:12

My team encountered the same issue in machines with Windows.. then managed to resolve it in two ways:

a) Set enviroment variable (even in Windows system preferences)

JAVA_TOOL_OPTIONS
-Dfile.encoding=UTF8

b) Introduce following snippet to your pom.xml:

 -Dfile.encoding=UTF-8

WITHIN

 <jvmArguments>
 -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8001
 -Dfile.encoding=UTF-8
 </jvmArguments>

score 2 · Answer 11 · answered Apr 24 '18 at 08:59

I'm using Amazon (AWS) Elastic Beanstalk and successfully changed it to UTF-8.

In Elastic Beanstalk, go to Configuration > Software, "Environment properties". Add (name) JAVA_TOOL_OPTIONS with (value) -Dfile.encoding=UTF8

After saving, the environment will restart with the UTF-8 encoding.

score 1 · Answer 12 · answered Dec 16 '08 at 03:59

Not clear on what you do and don't have control over at this point. If you can interpose a different OutputStream class on the destination file, you could use a subtype of OutputStream which converts Strings to bytes under a charset you define, say UTF-8 by default. If modified UTF-8 is suffcient for your needs, you can use DataOutputStream.writeUTF(String):

byte inbytes[] = new byte[1024];
FileInputStream fis = new FileInputStream("response.txt");
fis.read(inbytes);
String in = new String(inbytes, "UTF8");
DataOutputStream out = new DataOutputStream(new FileOutputStream("response-2.txt"));
out.writeUTF(in); // no getBytes() here

If this approach is not feasible, it may help if you clarify here exactly what you can and can't control in terms of data flow and execution environment (though I know that's sometimes easier said than determined). Good luck.

DataInputStream and DataOutputStream are special-purpose classes that should never be used with plain text files. The modified UTF-8 they employ is not compatible with real UTF-8. Besides, if the OP could use your solution, he could also use the right tool for this job: an OutputStreamWriter. — Alan Moore, Dec 25 '08 at 04:19

score 1 · Answer 13 · answered Mar 06 '18 at 08:28

mvn clean install -Dfile.encoding=UTF-8 -Dmaven.repo.local=/path-to-m2

command worked with exec-maven-plugin to resolve following error while configuring a jenkins task.

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Error occurred during initialization of VM
java.nio.charset.IllegalCharsetNameException: "UTF-8"
    at java.nio.charset.Charset.checkName(Charset.java:315)
    at java.nio.charset.Charset.lookup2(Charset.java:484)
    at java.nio.charset.Charset.lookup(Charset.java:464)
    at java.nio.charset.Charset.defaultCharset(Charset.java:609)
    at sun.nio.cs.StreamEncoder.forOutputStreamWriter(StreamEncoder.java:56)
    at java.io.OutputStreamWriter.<init>(OutputStreamWriter.java:111)
    at java.io.PrintStream.<init>(PrintStream.java:104)
    at java.io.PrintStream.<init>(PrintStream.java:151)
    at java.lang.System.newPrintStream(System.java:1148)
    at java.lang.System.initializeSystemClass(System.java:1192)

score 1 · Answer 14 · answered Aug 07 '20 at 13:52

Solve this problem in my project. Hope it helps someone.

I use LIBGDX java framework and also had this issue in my android studio project. In Mac OS encoding is correct, but in Windows 10 special characters and symbols and also russian characters show as questions like: ????? and other incorrect symbols.

Change in android studio project settings: File->Settings...->Editor-> File Encodings to UTF-8 in all three fields (Global Encoding, Project Encoding and Default below).
In any java file set:

System.setProperty("file.encoding","UTF-8");
And for test print debug log:

System.out.println("My project encoding is : "+ Charset.defaultCharset());

score 1 · Answer 15 · answered Nov 05 '21 at 06:36

Setting up jvm arguments while starting application helped me resolve this issue. java -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8.

file.encoding=UTF-8 - This helps to have the Unicode characters in the file.

sun.jnu.encoding=UTF-8 - This helps to have the Unicode characters as the File name in the file system.

score 0 · Answer 16 · answered Oct 05 '17 at 15:40

Following @Caspar comment on accepted answer, the preferred way to fix this according to Sun is :

"change the locale of the underlying platform before starting your Java program."

http://bugs.java.com/view_bug.do?bug_id=4163515

For docker see:

http://jaredmarkell.com/docker-and-locales/

score 0 · Answer 17 · answered Oct 14 '17 at 17:14

Recently I bumped into a local company's Notes 6.5 system and found out the webmail would show unidentifiable characters on a non-Zhongwen localed Windows installation. Have dug for several weeks online, figured it out just few minutes ago:

In Java properties, add the following string to Runtime Parameters

-Dfile.encoding=MS950 -Duser.language=zh -Duser.country=TW -Dsun.jnu.encoding=MS950

UTF-8 setting would not work in this case.

score 0 · Answer 18 · answered Jan 19 '12 at 19:23

0

We set there two system properties together and it makes the system take everything into utf8

file.encoding=UTF8
client.encoding.override＝UTF-8

answered Jan 19 '12 at 19:23

lizi

69
1
2

8

The client.encoding.override property seems to be WebSphere specific. – Christophe Roussy Aug 20 '12 at 09:29

Setting the default Java character encoding

18 Answers18

Linked