I'm running a sysbench OLTP benchmark with TiDB on GKE local SSD disk. But I'm getting poor performance compared to GKE persistent SSD disk. How can I get the expected IOPS performance on GKE local SSD disk by default?
I've run TiDB OLTP benchmark and fio benchmark with psync engine but the results both shows IOPS on local SSD disk is poorer than on persistent SSD disk. And I've also run a thorough blktrace analysis. The fio command I ran is:
fio -ioengine=psync -bs=32k -fdatasync=1 -thread -rw=write -size=10G -filename=test -name="max throughput" -iodepth=1 -runtime=60 -numjobs=4 -group_reporting
The fio benchmark result for local SSD disk and persistent disk is:
| disk type | iops | bandwidth |
|---------------------+------+-----------|
| local SSD disk | 302 | 9912kB/s |
| persistent SSD disk | 1149 | 37.7MB/s |
And the blktrace btt result is:
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000002 0.003716416 14.074086987 34636
Q2G 0.000000236 0.000005730 0.005347758 25224
G2I 0.000000727 0.000005446 0.002450425 20575
Q2M 0.000000175 0.000000716 0.000027069 9447
I2D 0.000000778 0.000003197 0.000111657 20538
M2D 0.000001941 0.000011350 0.000431655 9447
D2C 0.000065510 0.000182827 0.001366980 34634
Q2C 0.000072793 0.001181298 0.023394568 34634
==================== Device Overhead ====================
DEV | Q2G G2I Q2M I2D D2C
---------- | --------- --------- --------- --------- ---------
( 8, 48) | 0.3532% 0.2739% 0.0165% 0.1605% 15.4768%
---------- | --------- --------- --------- --------- ---------
Overall | 0.3532% 0.2739% 0.0165% 0.1605% 15.4768%
According to the optimization guide, I’ve manually remounted the disk with the nobarrier option and the blktrace btt result looks as normal.
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000006 0.000785969 12.031454829 123537
Q2G 0.000003929 0.000006162 0.005294881 94553
G2I 0.000004677 0.000029263 0.004555917 94553
Q2M 0.000004069 0.000005337 0.000328930 29019
I2D 0.000005166 0.000020476 0.001078527 94516
M2D 0.000012816 0.000056839 0.001113739 29019
D2C 0.000081435 0.000358712 0.006724447 123535
Q2C 0.000113965 0.000415489 0.006763290 123535
==================== Device Overhead ====================
DEV | Q2G G2I Q2M I2D D2C
---------- | --------- --------- --------- --------- ---------
( 8, 48) | 1.1351% 5.3907% 0.3017% 3.7705% 86.3348%
---------- | --------- --------- --------- --------- ---------
Overall | 1.1351% 5.3907% 0.3017% 3.7705% 86.3348%
However, according to RedHat’s document, the nobarrier mount option should only have a very small negative impact on performance (about 3%) and it is not recommended to use it on storage configured on virtual machines.
The use of nobarrier is no longer recommended in Red Hat Enterprise Linux 6 as the negative performance impact of write barriers is negligible (approximately 3%). The benefits of write barriers typically outweigh the performance benefits of disabling them. Additionally, the nobarrier option should never be used on storage configured on virtual machines.
In addition to the nobarrier option, the local SSD disk optimization guide also suggests installing the Linux Guest Environment but states that it’s already installed on newer VM image. However, I found that it’s not installed on the GKE node.
So I manually installed the Linux Guest Environment and tested again, this time the btt result looks as expected:
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000001 0.000472816 21.759721028 301371
Q2G 0.000000215 0.000000925 0.000110353 246390
G2I 0.000000279 0.000003579 0.003997348 246390
Q2M 0.000000175 0.000000571 0.000106259 54982
I2D 0.000000609 0.000002635 0.004064992 246390
M2D 0.000001400 0.000005728 0.000509868 54982
D2C 0.000051100 0.000451895 0.009107264 301372
Q2C 0.000054091 0.000458881 0.009111984 301372
==================== Device Overhead ====================
DEV | Q2G G2I Q2M I2D D2C
---------- | --------- --------- --------- --------- ---------
( 8, 80) | 0.1647% 0.6376% 0.0227% 0.4695% 98.4778%
---------- | --------- --------- --------- --------- ---------
Overall | 0.1647% 0.6376% 0.0227% 0.4695% 98.4778%
So how can I get the expected IOPS performance on GKE local SSD disk by default without extra tuning?
