Memory concern for dealing with large String

Question

In my application, I have written a REST call which would be used by developers and support persons only, where it loads data of n size from the logs file and return it as text/plain, where n is a configurable number which cannot be more than fixed size.

    RandomAccessFile file = new RandomAccessFile(logFinalPath, "r");
    //chunk default is 100000 max is 500000
    chunk = chunk <=0l?100000:(chunk>500000?500000:chunk);
    long fileSize = file.length();
    //start position is either chunk size from the end or when the file is smaller then from the beginning of file
    long pos = fileSize>chunk?file.length()-chunk:0;    
    int maxSize = (int) (fileSize>chunk?chunk:fileSize);
    file.seek(pos);
    byte[] bytes = new byte[maxSize];
    file.read(bytes);
    file.close();
    logger.info("fileSize : "+fileSize);
    return new String(bytes);

Now here is my question, we know that String are immutable and any new String created goes into String pool never to be cleaned till JVM is on and running. So in that case every time this REST call is made would not it be a hit on memory in the sense that it would keep loading large text into String pool?

If yes, what are the alternatives, I can't pass the byte array as that would be unreadable. How can it be better?

Update : Please note that this question is not about new constructor of String or literal representation, It is about how it can be optimized to avoid the large String being stored into string pool with or without a reference from heap object.

Possible duplicate of [What is the difference between "text" and new String("text")?](http://stackoverflow.com/questions/3052442/what-is-the-difference-between-text-and-new-stringtext) — the8472, Jan 07 '17 at 08:39

Peter Lawrey · Accepted Answer · 2017-01-07T09:25:19.110

1

we know that String are immutable

yes.

and any new String created goes into String pool

no, unless you call String.intern()

never to be cleaned till JVM is on and running.

String literals can be cleaned up from Java 6 as String literals were moved to the heap.

So in that case every time this REST call is made would not it be a hit on memory in the sense that it would keep loading large text into String pool?

Large objects are added into the tenured space and to clean those up you need a major collection which you generally you want to avoid.

There is a small performance hit, but not as big as a say converting a byte[] into a String only to convert it back again into bytes later. You could save a lot of work (and memory) by keeping the cached data as a byte[]

I would be tempted to memory map your source file on start up (this uses almost no heap no matter how big the file is) and copy from the source file directory to the output of the request as needed and avoid taking an on heap copy as much as possible.

edited Jan 07 '17 at 09:25

answered Jan 07 '17 at 09:19

Peter Lawrey

525,659
79
751
1,130

Thanks for the response Peter, I got that my 'assumption' that all String goes into String pool was unfounded and wrong and ultimately those large String would be collected. I cannot cache the byte[] since the log files are constantly written on and I want say last 100 lines. Memory mapping is a thing I have heard only as concept and am not sure if could be applicable for rolling files, I mean log files are rolled as new files either daily or say once it becomes 200MB large. – Rahul Kumar Jan 07 '17 at 09:37
@RahulKumar we use memory mapping for rolling files but we control the producer and consumer. Memory mapping would be tricky if you are not the producer as well. – Peter Lawrey Jan 07 '17 at 09:57
The statement “*String literals can be cleaned up from Java 6 as String literals were moved to the heap*” is misleading. Strings were always subject to garbage collection like any other object. The issue with the “permanent generation” was that it had a fixed size, not that it was never GC’ed. It’s especially misleading as the question wasn’t about literals and calling `intern()` on a string object adds it to the pool, but doesn’t turn it into a literal. Since such manually added pool strings are not associated with a class, they can be always garbage collected (in every Java version). – Holger Jan 09 '17 at 17:15

Memory concern for dealing with large String

1 Answers1