26

I am trying to analyze the QEMU source code.

I know its huge and till date there is no official documentation for it.

My main areas of concern are the Instruction cache management and TCG operation.

Any pointers to them would be helpful ?

Dexter
  • 1,299
  • 3
  • 20
  • 38
  • Also highly recommend that you start with a simple freestanding hello world that you fully control and understand, e.g. [this](https://github.com/cirosantilli/linux-kernel-module-cheat/blob/5b7094fb68e36c2c4af73e8a209a979ebbf32eae/userland/arch/x86_64/freestanding/linux/hello.S). Or baremetal examples [like this](https://github.com/cirosantilli/linux-kernel-module-cheat/tree/5b7094fb68e36c2c4af73e8a209a979ebbf32eae#baremetal-setup). – Ciro Santilli OurBigBook.com Oct 15 '19 at 16:02
  • Also have a look at QEMU's logging options, starting obviously with [the output instruction dumping](https://stackoverflow.com/questions/13005303/how-does-native-android-code-written-for-arm-run-on-x86/44505097#44505097). And of course, use a proper IDE to GDB step the code which allows going to definitions and usages of things, e.g. Eclipse. – Ciro Santilli OurBigBook.com Oct 15 '19 at 16:02
  • Oh, and obviously read the [TCG README](https://github.com/qemu/qemu/blob/3af78db68176a049e2570822f64604e0692c1447/tcg/README). – Ciro Santilli OurBigBook.com Oct 15 '19 at 16:12

2 Answers2

47

I know full answer would be much longer, but for start I just want to bring to your attention this diagram: (now, it would be useful for you to play with gdb running QEMU, set breakpoints in functions you see in the diagram, follow code execution, etc.)

qemu source code flow design implementation diagram

VividD
  • 10,456
  • 6
  • 64
  • 111
  • Thanks, VividD, i had to eventually take the hard route to find out, as did not saw your post earlier. My mistake, that i did not share the findings here. Almost most of the things you have covered here. Would like to point that tb_find_fast and tb_find_slow both search in the cache, only difference i could found was that tb_find_fast uses hash func value to find the index of TB. If the TB found i snot valid it does a sequential search using tb_find_slow. – Dexter Jan 23 '14 at 13:47
  • 4
    your illustration is very well done, what software was used here? – Peter Teoh Mar 18 '14 at 16:26
  • Seconding what Peter said. Did you generate that image from sources using some tool? – Martin Jul 18 '14 at 14:25
  • 4
    @Phil - No, a source code analysis tool was not used. Only plain drawing program, LibreOffice Draw. The diagram is somewhat informal, but I found it conveys execution flow well, helps locate main source code files etc... – VividD Jul 18 '14 at 16:52
  • @Dexter @VividD I am trying to understand how the TB size can be manipulated ... Am I right to run `ddd qemu-system-arm` and place breakpoints in the code say in `gen_intermediate_code()`? Then I run the program say, `-M versatilepb -kernel ...` Doing this however hangs the program because of a glib error that fails to allocate a huge amount of memory. Could someone point me in the right direction? – Arjun Jul 19 '16 at 13:25
  • qemu is much different now. Hope someone can do an updated diagram – PPP Apr 11 '20 at 20:18
1

Yes, the QEMU code flow has changed a good bit. I'm not going to do a libreoffice presentation but here's a couple TCG stack traces of the QEMU 5.1 MTTCG code. The first is the TCG Front-end (FE) taking the guest code and converting it to internal intermediate code in a Translation Block (TB). The TB has a max size of 512 instructions.

#0  0x00005555559ce81f in disas_insn (s=0x7fffe933d450, cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/target/i386/translate.c:4476
#1  0x00005555559dd471 in i386_tr_translate_insn (dcbase=0x7fffe933d450, cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/target/i386/translate.c:8569
#2  0x00005555558c4222 in translator_loop (ops=0x5555565ae9a0 <i386_tr_ops>, db=0x7fffe933d450, cpu=0x555556b2d990, tb=0x7fffac099900 <code_gen_buffer+134846675>, max_insns=512) at /opt/distros/qemu-5.1.0/accel/tcg/translator.c:102
#3  0x00005555559dd643 in gen_intermediate_code (cpu=0x555556b2d990, tb=0x7fffac099900 <code_gen_buffer+134846675>, max_insns=512) at /opt/distros/qemu-5.1.0/target/i386/translate.c:8631
#4  0x00005555558c2258 in tb_gen_code (cpu=0x555556b2d990, pc=18446744071591428680, cs_base=0, flags=4244144, cflags=-16252928) at /opt/distros/qemu-5.1.0/accel/tcg/translate-all.c:1743
#5  0x00005555558be77a in tb_find (cpu=0x555556b2d990, last_tb=0x0, tb_exit=0, cf_mask=524288) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:407
#6  0x00005555558bf18e in cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:748
#7  0x00005555559846eb in tcg_cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1356
#8  0x0000555555984f41 in qemu_tcg_cpu_thread_fn (arg=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1664
#9  0x0000555555e5ec8d in qemu_thread_start (args=0x555556b5e0e0) at /opt/distros/qemu-5.1.0/util/qemu-thread-posix.c:521
#10 0x00007ffff3c406db in start_thread (arg=0x7fffe933e700) at pthread_create.c:463
#11 0x00007ffff396971f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The second is the TCG Back-end (BE) translating the intermediate code into host instructions.

Thread 3 "qemu-system-x86" hit Breakpoint 9, tcg_out_op (s=0x7ffe9c000b20, opc=INDEX_op_ld_i32, args=0x7fffe933d530, const_args=0x7fffe933d4f0) at /opt/distros/qemu-5.1.0/tcg/i386/tcg-target.inc.c:2259
2259        int c, const_a2, vexop, rexw = 0;
#0  0x00005555558447d2 in tcg_out_op (s=0x7ffe9c000b20, opc=INDEX_op_ld_i32, args=0x7fffe933d530, const_args=0x7fffe933d4f0) at /opt/distros/qemu-5.1.0/tcg/i386/tcg-target.inc.c:2259
#1  0x000055555584fb40 in tcg_reg_alloc_op (s=0x7ffe9c000b20, op=0x7ffe9c00a418) at /opt/distros/qemu-5.1.0/tcg/tcg.c:3803
#2  0x000055555585078e in tcg_gen_code (s=0x7ffe9c000b20, tb=0x7fffac129880 <code_gen_buffer+135436371>) at /opt/distros/qemu-5.1.0/tcg/tcg.c:4244
#3  0x00005555558c22f1 in tb_gen_code (cpu=0x555556b2d990, pc=94290869746347, cs_base=0, flags=4244147, cflags=-16252928) at /opt/distros/qemu-5.1.0/accel/tcg/translate-all.c:1766
#4  0x00005555558be77a in tb_find (cpu=0x555556b2d990, last_tb=0x7fffac0ea700 <code_gen_buffer+135177939>, tb_exit=1, cf_mask=524288) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:407
#5  0x00005555558bf18e in cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:748
#6  0x00005555559846eb in tcg_cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1356
#7  0x0000555555984f41 in qemu_tcg_cpu_thread_fn (arg=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1664
#8  0x0000555555e5ec8d in qemu_thread_start (args=0x555556b5e0e0) at /opt/distros/qemu-5.1.0/util/qemu-thread-posix.c:521
#9  0x00007ffff3c406db in start_thread (arg=0x7fffe933e700) at pthread_create.c:463
#10 0x00007ffff396971f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Once the TB is generated then tb_find returns with it and cpu_tb_exec runs it.

As prior answers indicate, there is A LOT more to the TCG.

Note that I used host and guest rather than native and target. TCG has the terms somewhat reversed: the source is the guest code that is converted to run on the target, which is the host. In other words, the TCG target is the QEMU host (which makes sense for the VM code generation.)

Nimantha
  • 6,405
  • 6
  • 28
  • 69
dturvene
  • 2,284
  • 1
  • 20
  • 18