Here is what Java Docs say:
The grouping separator is commonly used for thousands, but in some countries it separates ten-thousands. The grouping size is a constant number of digits between the grouping characters, such as 3 for 100,000,000 or 4 for 1,0000,0000. If you supply a pattern with multiple grouping characters, the interval between the last one and the end of the integer is the one that is used. So "#,##,###,####" == "######,####" == "##,####,####".
So in other words, you can only specify one grouping separator format at a time. Either a thousands separator or ten thousand separator or hundred separator. Even if you try to specify multiple formats in one, the method will use only the interval between the last one and the end of the integer. So in your case you are specifying "#,##,###" but the method will consider only the last comma and the number of digits afterwards which is 3 in your case. That's why the output is a thousand separator.
I was also hoping that selecting a proper locale should address this number formatting issue without needing to specify any pattern. E.g. in India, the numbers are in fact formatted the way you want. The first separator is thousand digit based and the rests are hundred digit based. So if I select Indian English locale, it should automatically address this requirement. But it doesn't.
NumberFormat nf6 = NumberFormat.getInstance(new Locale("en-IN"));
System.out.println(nf6.format(54123756.523d));//Still prints 54,123,756.523 and not 5,41,23,756.523
So unfortunately there is no ready made way to achieve your objective unless you go for the recommendation of using third party libraries in the other answer. However you can always use a heck. It is not neat but achieves the objective. Maybe you can encapsulate this hack in a utility method or something:
double number = 54123756.523d;
int prefix = (int)number / 1000;
NumberFormat nf1 = new DecimalFormat("#,##");
String newPrefix = nf1.format(prefix);
double remainder = number - (prefix * 1000);
NumberFormat nf2 = new DecimalFormat("#,###.##");
String newRemainder = nf2.format(remainder);
String finalNum = newPrefix + "," + newRemainder;
System.out.println("Formatted number is " + finalNum);
So see if you can live with the code sample suggested above. It is not ideal or efficient but will at least serve your purpose and won't need any third party libraries.