I'm trying to implement multi-threading for some task parallelism in a program i am writing. The program is utilizes Spring framework and runs on Pivotal Cloud Foundry. It was crashing occasionally so i went in and viewed the logs and performance metrics; which is when i discovered it has a memory leak. after some testing i narrowed down the culprit to my threading implementation. My understanding of the GC in the JVM is that it will not dispose of a thread that is not dead, nor will it dispose of any object that is still being referenced by another object or later line of executable code. I'm not holding any reference to the thread at all however, and if i do it claims to put itself into a dead state once it finishes running so I dont know what is causing the leak.
I've written a clean PoC to demonstrate the leak. it uses a rest controller so i can control the number of threads, a runnable class because my real program requires parameters, and a string to take up arbitrary space in memory that would be held by other fields in the real program (makes the leak more apparent).
package com.example;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class LeakController {
@RequestMapping("/Run")
public String DoWork(@RequestParam("Amount") int amount, @RequestParam("Args") String args)
{
for(int i = 0; i < amount; i++)
new Thread(new MyRunnable(args)).start();
return "Workin' on it";
}
public class MyRunnable implements Runnable{
String args;
public MyRunnable(String args){ this.args = args; }
public void run()
{
int timeToSleep = Integer.valueOf(args);
String spaceWaster = "";
for (int i = 0; i < 10000; i ++)
spaceWaster += "W";
System.out.println(spaceWaster);
try {Thread.sleep(timeToSleep);} catch (InterruptedException e) {e.printStackTrace();}
System.out.println("Done");
}
}
}
Can anyone explain why this program leaks memory?
Edit: I've gotten a few responses about String assignment vs string building and the string pool so i changed my code to the following
int[] spaceWaster = new int[10000];
for (int i = 0; i < 10000; i ++)
spaceWaster[i] = 512;
System.out.println(spaceWaster[1]);
and it still leaks.
Edit: In acquiring some real numbers to respond to Voo with i noticed something interesting. calling new threads starts eating memory up but only to a point. after permanently growing about 60mb the new integer based program stops growing further regardless of how hard it is pushed. Does this have something to do with the way the spring framework allocates memory?
I also think there is merit in going back to the String example as it more closely relates the my real use case; which is to do regex operations on incoming JSON, several hundred such JSON a second. With that in mind i have changed the code to:
@RestController
public class LeakController {
public static String characters[] = {
"1","2","3","4","5","6","7","8","9","0",
"A","B","C","D","E","F","G","H","I","J","K","L","M",
"N","O","P","Q","R","S","T","U","V","W","X","Y","Z"};
public Random rng = new Random();
@RequestMapping("/Run")
public String GenerateAndSend(@RequestParam("Amount") int amount)
{
for(int i = 0; i < amount; i++)
{
StringBuilder sb = new StringBuilder(100);
for(int j = 0; j< 100; j++)
sb.append(characters[rng.nextInt(36)]);
new Thread(new MyRunnable(sb.toString())).start();
System.out.println("Thread " + i + " created");
}
System.out.println("Done making threads");
return "Workin' on it";
}
public class MyRunnable implements Runnable{
String args;
public MyRunnable(String args){ this.args = args; }
public void run()
{
System.out.println(args);
args = args.replaceAll("\\d+", "\\[Number was here\\]");
System.out.println(args);
}
}
}
This new application exhibits similar behavior as the integer example in that it grows about 50mb permanently (after 2000 threads), and tapers off from there until i cant notice any memory growth with each new batch of 1000 threads (about 85mb past original deploy memory).
if i change it to remove stringbuilder:
String temp = "";
for(int j = 0; j< 100; j++)
temp += characters[rng.nextInt(36)];
new Thread(new MyRunnable(temp)).start();
it leaks indefinitely; i assume that once all 36^100 strings have been generated once that it will stop.
Combining those findings i guess my real issue could be both an issue with the string pool and an issue with how spring allocates memory. What i am still failing to understand is that in my real application if i make a runnable and call run() on the main thread the memory doesnt seem to spike, but if i make a new thread and give it the runnable then the memory jumps. Heres what my runnable looks like currently in the application i am building:
public class MyRunnable implements Runnable{
String json;
public MyRunnable(String json){
this.json = new String(json);
}
public void run()
{
DocumentClient documentClient = new DocumentClient (END_POINT,
MASTER_KEY, ConnectionPolicy.GetDefault(),
ConsistencyLevel.Session);
System.out.println("JSON : " + json);
Document myDocument = new Document(json);
System.out.println(new DateTime().toString(DateTimeFormat.forPattern("MM-dd-yyyy>HH:mm:ss.SSS"))+">"+"Created JSON Document Locally");
// Create a new document
try {
//collectioncache is a variable in the parent restcontroller class that this class is declared inside of
System.out.println("CollectionExists:" + collectionCache != null);
System.out.println("CollectionLink:" + collectionCache.getSelfLink());
System.out.println(new DateTime().toString(DateTimeFormat.forPattern("MM-dd-yyyy>HH:mm:ss.SSS"))+">"+"Creating Document on DocDB");
documentClient.createDocument(collectionCache.getSelfLink(), myDocument, null, false);
System.out.println(new DateTime().toString(DateTimeFormat.forPattern("MM-dd-yyyy>HH:mm:ss.SSS"))+">"+"Document Creation Successful");
System.out.flush();
currentThreads.decrementAndGet();
} catch (DocumentClientException e) {
System.out.println("Failed to Upload Document");
e.printStackTrace();
}
}
}
Any ideas where my real leak is? Is there somewhere i need a string builder? does string just do memory funny and i need to give it a higher ceiling to stretch out into then it will be fine?
Edit: i did some bench-marking so i could actually graph the behavior in order to better understand what the GC is doing
00000 Threads - 457 MB
01000 Threads - 535 MB
02000 Threads - 545 MB
03000 Threads - 549 MB
04000 Threads - 551 MB
05000 Threads - 555 MB
2 hours later - 595 MB
06000 Threads - 598 MB
07000 Threads - 600 MB
08000 Threads - 602 MB
It seems asymptotic but what was most interesting to me is that while i was out attending meetings and eating lunch it decided to grow 40mb on its own. i checked with my team and nobody used the app during that time. Not sure what to make of that either
