Is it possible to compile a large Java module in parallel?

Question

I know that multiple modules can be compiled using multiple threads where each thread compiles a single module but what if I have a single large module? Does Javac or the Eclipse Java Compiler support compiling single modules in parallel (using many threads)? Or is there any other Java compiler which supports it?

Update: I created a Java source file with ~50k simple methods (just for the purpose of this test) such as:

    static int add1(int a, int b, int c) {
        return 2 * a + 55 * b - c;
    }

   static int add2(int a, int b, int c) {
        return 2 * a + 55 * b - c;
    }

   static int add3(int a, int b, int c) {
        return 2 * a + 55 * b - c;
    }

These methods do not depend on each other so compilation could be done in parallel (at least in theory). Compiling this file with Javac on my 12 core + HT machine lead to an average 20% CPU usage with a really short spike of up to 50%. This leads me to believe that although there is some parralelization done inside Javac, it is really minor.

The interesting thing is that if I create 2, 3 or 4 classes with the same number of methods and compile them at the same time with a single Javac process, I cannot get a higher CPU usage. The compilation takes exactly 2x, 3x, 4x longer which shows that Javac doesn't compile these totally unrelated classes in parallel. But if I start separate Javac processes to compile these files separately, the CPU jumps to almost 100% when 4 files(=Javac processes) are used and the compilation time is just 5-10% higher than compiling a single file (compared to this, a single Javac process compiling all these 4 files, the compilation takes 400% longer).

So my opinion is that Javac does compile files using multiple threads but it is kind of limited to ~4 threads, it cannot fully utilize a 12 cores machine. Also to me it seems that Javac compiles multiple files in serial, it only uses cores/threads to compile a single file in parallel(I believe that when a single file is compiled, some parts can be done in parallel and this is what Javac does, but what about compiling multiple files in parallel? If I have 100 files which are independent I should be able to see my CPU jump to 100% which is not the case.)

`maven` supports parallel build https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 — Liquidpie, Dec 27 '20 at 14:10
@Liquidpie as far as I know it only works if the project has multiple modules — Nfff3, Dec 27 '20 at 14:15
There are only two compilers tracking the latest versions of java in wide use, which would be important to me. The openjdk one and the eclipse one. I would have a very close look at the eclipse one to see what it can do. — Thorbjørn Ravn Andersen, Dec 27 '20 at 14:29
@akuzminykh I don't. I just created it for the purpose of this test. — Nfff3, Dec 27 '20 at 15:44
You might want to check out the following post because it has details that could be interesting: [why parallel execution on java compile take linear growth in time](https://stackoverflow.com/q/30684904/12323248) — akuzminykh, Dec 27 '20 at 15:52
How would the compiler know that two Java files are independent of each other without first compiling them? (Remember, if class A depends on class B, and both classes are currently in source form, while compiling A the compiler will see that there is a B.java file without a B.class file, and compile B.java.) — k314159, Dec 29 '20 at 14:45
@k314159 this is a fair point, but I still think that `Javac` could do better, like it could compile files in multiple steps and during a very fast first step it could already know that the file is independent. — Nfff3, Dec 29 '20 at 14:50
now think about how many users are _really_ affected by this? and take it into a different perspective. 20 years in and `HashSet` _still_ uses a `HashMap` under the hood (I can make more examples like this). The idea is that such optimizations might not ever be a priority to the JDK team. Unless a VERY brave and smart enthusiast wants to help with a PR (which will never happen) — Eugene, Dec 31 '20 at 03:55

Stephen C · Accepted Answer · 2021-01-05T04:13:28.970

Yes it is possible to build Java code in parallel.

The Java compiler (javac) itself doesn't do this, but both Maven and Ant (and some versions of Make) can run multiple javac instances in parallel.

Furthermore, the Eclipse Java compiler is multi-threaded and you can tell Maven to use it instead of javac; see https://stackoverflow.com/a/3727542/139985

I note that your example involves compiling a single class with a huge number of methods. Parallel compiler instances won't help with that. The Eclipse compiler might help depending on how it is implemented.

However, I put it to you that that is an unrealistic example. People don't write code like that in real life¹, and code generators can (and should) be written to not emit source code like that.

^{1 - Their co-workers will rebel ...}

score 3 · Answer 2 · answered Dec 29 '20 at 23:08

3

javac runs always single-threaded. There is a case for improving javac performance JDK-4229449 : RFE: Please multithread javac for better performance however Oracle does not intend to change the compilation architecture.

answered Dec 29 '20 at 23:08

Ismail Durmaz

2,521
1
6
19

1

You're right. This was submitted in 1999!! and it was marked as `Closed`, `Resolution: Won't Fix`. What a pity! – Nfff3 Dec 29 '20 at 23:17
The way of mind is one. I also think multithreaded compilation can be useful – Ismail Durmaz Dec 29 '20 at 23:20
`This was submitted in 1999!! and it was marked as Closed`. Yes, but only because the issue was too specific how performance should be improved. From the comments: `Yes, the compiler should be faster. [...] But this RFE is a bit too specific in suggesting how we do that.`. I think Java compiler developers do consider compilation performance in general. – mihca Aug 08 '22 at 09:27
1

There is a newer issue here with status `closed/delivered` [JEP 139](https://openjdk.org/jeps/139) – mihca Aug 08 '22 at 09:52

Is it possible to compile a large Java module in parallel?

2 Answers2