Difference between template interpreter and JIT compiler

Question

Below is an excerpt from Wikipedia text on interpreters, specifically the section on template interpreter. The answer to my question is technically there, in bold. But I still do not quite understand the difference, probably because I have very low-level knowledge of low-level processes, pun intended (sorry).

It is saying that the JIT compiler creates sequences of CPU-executable instructions from the entire code segment. What do they mean by "from the entire code segment"? To me, this sounds like a normal compiler. I thought JIT compilation meant that an interpretor is run (parsing and executing statement-by-statement) and then repeated instructions are kept track of and translated into native machine code to be used next time it is needed, like a compiler does. But this sounds like the template interpreter to me, which I interpret as translating often-appearing statements and saving the statements and respective translations as key-value mappings in an array. So wherein lies the difference? Is it how the JIT saves the native machine instructions compiled from bytecode-sections (i.e. not in an array with mappings, then if so how is it saved)? Is it the way that the blocks of bytecode that are compiled into native machine code are partitioned; i.e. that they are translated and saved as mappings by opcode (if i understood this correctly) vs by something else? etc

"Making the distinction between compilers and interpreters yet again even more vague is a special interpreter design known as a template interpreter. Rather than implement the execution of code by virtue of a large switch statement containing every possible bytecode, while operating on a software stack or a tree walk, a template interpreter maintains a large array of bytecode (or any efficient intermediate representation) mapped directly to corresponding native machine instructions that can be executed on the host hardware as key value pairs,[15][16] known as a "Template". When the particular code segment is executed the interpreter simply loads the opcode mapping in the template and directly runs it on the hardware.[17][18] Due to its design, the template interpreter very strongly resembles a just-in-time compiler rather than a traditional interpreter, however it is technically not a JIT due to the fact that it merely translates code from the language into native calls one opcode at a time rather than creating optimized sequences of CPU executable instructions from the entire code segment. Due to the interpreter's simple design of simply passing calls directly to the hardware rather than implementing them directly, it is much faster than every other type, even bytecode interpreters, and to an extent less prone to bugs, but as a tradeoff is more difficult to maintain due to the interpreter having to support translation to multiple different architectures instead of a platform independent virtual machine/stack. To date, the only template interpreter implementation of a language to exist is the interpreter within the HotSpot/OpenJDK Java Virtual Machine reference implementation.[15]"

Answer 1

The powerful advantage of Just-in-Time compilers is that they possess access to the runtime data and therefore can generate machine code tailored to the specific needs. The path to performance promise JIT compilers hold is being specific: the more specific and situational your case is, the more tailored code you can generate that could squeeze more performance. This is the reason why big projects like Chromium reject standard containers in the STL but create their own: the more general your code needs to be and serve multitude of cases, the least assumptions you can make about it that would allow you to perform better optimisations. If you are more curious what a difference JITs can make, I recommend this video that describes how an engineer accelerated hashmaps by 100 times compared to standard C++ and Rust hashtables by making more and more assumptions about it.

So, the whole purpose of JIT compilers is to generate code that will accomodated to the very situational conditions in the runtime. To do this, they gather profiling data as the code is executed and use it to perform a verity of profile-guided optimisations that can yield quite impressive boosts. In case of template interpreters, they don't really do any of these but they are aimed to reduce the overhead of instruction dispatching: normally if you would pass each instruction through a switch statement, you get a high degree of branch unpredictability. Over time, it accumulates and results in substential linear overhead of dispatching instrucitons alone.

You may say that compilers can optimise switch statements into jumptables by grouping the code into the cases as function pointers/labels and plugging the value into the array, and that's indeed what template interpreters do. They contain all of the implementations of their instructions as host's machine code that they retrieve using the instruction itself as a key. If instructions are represented as enum values, this allows to efficiently plugging it into the array as the index essentially taking only 2 CPU instructions to dispatch the code. However, template interpreters are preferred over switch statements because they are explicit and you don't depend on compiler optimising it, and they allow you to control the process of interpretation on the low level. As an example, if your functions that you call for run the instructions are stored as the raw machine code object, then they make take different sizes, and then you could make your own hashing function to access the proper fragment irregularly, allowing template intepreters to be more flexible and independent off specific compiler.

So far it seems that templates are alike reusable compilers, but you can notice a significant difference between them: they don't do any optimisations and only call pre-compiled handlers for each instruction. JIT compilers, aside from allocating and prepolating the memory with the same code, would also see profiling data and transform this code dynamically based on the runtime needs. On top of that, since the code is stored separately in the memory, you could adapt it as the conditions change and transform on the fly, which you cannot do with interpreters.

In conclusion, template interpreters are used only to run individula bytecode/IR instructions on themselves, and due to their simplicity they cannot do much aside from that. The task of interpreter is to minimise the overhead of itnerpreting the code and call the next handler as quickly as possible. On the other hand, JIT compilers are intended to be able to optimise the code on the fly, hence they allocate the memory where it's stored and can be manipulated.