Using Hadoop Text Object toString() Method

Question

I understood the difference between String & Text. Difference between Text and String in Hadoop

Question is If we are saying that String maximum storage size is 32767 bytes.

Text t = new Text("Hadoo... 2GB of content");
...
String c = t.toString();

How "c" will hold 2GB of data if it has size limitation?

What am I missing here?

There is no maximum storage of 32k on those classes, how is that problem for you? — Thomas Jungblut, Aug 13 '14 at 09:50
@ThomasJungblut is right. Take a look at this post: http://stackoverflow.com/questions/12120002/is-there-any-limit-for-string-size-in-a-java-program — vefthym, Aug 13 '14 at 09:59

score 2 · Accepted Answer · edited May 23 '17 at 11:54

The maximum size of a Java String is not 32k bytes. It is the size needed to store Integer.MAX_VALUE characters, which is 2^31 - 1 (~2 Billion), which is around 4GB (see this post).

The post that you mention, refers to the size limit of the deprecated UTF-8 class, not Java's String class.

Anyway, if you need so much space for a single Text instance, I would advise you to reconsider your algorithm. As Peter Lawrey says in the afforementioned post "I suspect all the works of J K Rowling would fit into one string."

Using Hadoop Text Object toString() Method

1 Answers1