I find one very strange probelm with io_submit latency.
If I write a loop invoke io_submit 5 times, like following:
for (int i = 0; i < 5; i++) {
gettimeofday(&start);
io_submit(...);
gettimeofday(&end);
}
The latency of io_submit are all very small except the first one
io_submit cost: 9 us
io_submit cost: 2 us
io_submit cost: 2 us
io_submit cost: 3 us
but if I sleep after every invoke of io_submit, just like following:
for (int i = 0; i < 5; i++) {
gettimeofday(&start);
io_submit(...);
gettimeofday(&end);
sleep(1);
}
The latency of io_submit are all very large:
io_submit cost: 9 us
io_submit cost: 8 us
io_submit cost: 9 us
io_submit cost: 7 us
The block device is a nvme ssd. I have tried to used blktrace, but it seems that blktrace has some problems with nvme, only event 'Q' and 'A' are catched, this is not enough to figure out this question. I have also tried to use systemtap to trace some point in io_submit's code, but that make io_submit's latency become too large nearly 50us which makes the diff between the upper situatios not apparently.
Does anyone know why about this or give some advices to figure out this question.
NEW PROGRESS:
using systemptap, i found that increase of latency came from many places of the code path, not in one place. Two releted things. first, the cpu cache miss that come from context switch, sleep situation lead to more context switch; second, the code path alloc then free memory, if run without sleep, the memory free this turn can be reused by next turn. while with sleep, the memory just free may be used by other threads.
