4

Running sysbench against volumes I'm finding the gp3 volumes are much slower. Even when I provision a volume with 16000IOPS after waiting for optimization I'm getting a cap at 1000 IOPS when monitoring through Percona PMM, New Relic CloudWatch.

Instance:

  • C5.4xlarge Ubuntu 18.04

Volumes:

  • gp2 3000GB (gives 9000 IOPS)
  • gp3 3000GB 9000 IOPS 250/s

Sysbench results below:

sysbench --file-total-size=15G --file-num=16 fileio prepare
sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --time=600 fileio run
sysbench --file-total-size=15G --file-num=16 fileio cleanup

gp3 | 9000 | 3000 | 250/s

File operations:
reads/s: 576.37
writes/s: 384.24
fsyncs/s: 153.70

Throughput:
read, MiB/s: 9.01
written, MiB/s: 6.00

General statistics:
total time: 600.0333s
total number of events: 668612

Latency (ms):
min: 0.00
avg: 0.90
max: 337.40
95th percentile: 3.89
sum: 599693.33

Threads fairness:
events (avg/stddev): 668612.0000/0.00
execution time (avg/stddev): 599.6933/0.00

gp2 | 9000 | 3000 | gp2

File operations:
reads/s: 1523.68
writes/s: 1015.79
fsyncs/s: 406.33

Throughput:
read, MiB/s: 23.81
written, MiB/s: 15.87

General statistics:
total time: 600.0064s
total number of events: 1767487

Latency (ms):
min: 0.00
avg: 0.34
max: 70.10
95th percentile: 1.06
sum: 599390.12

Threads fairness:
events (avg/stddev): 1767487.0000/0.00
execution time (avg/stddev): 599.3901/0.00

Percona PMM gp3 > gp2 comparison:

Percona PMM gp3 > gp2 comparison

TylerH
  • 20,799
  • 66
  • 75
  • 101
tippingpints
  • 49
  • 1
  • 2
  • You might get a better response at: https://www.reddit.com/r/aws – John Rotenstein Jan 10 '21 at 09:21
  • I'm doing benchmarks for `gp3` volumes as well. However, I'm observing the opposite: `gp3` with less latency than `gp2`. In which region are you operating? – Andreas Jan 11 '21 at 12:49
  • Amazon EBS volumes take slightly longer when using a block for the first time, due to the allocation of storage space. Before performing performance tests, it is recommended to zero-out the contents of the disk (eg with `dd`). This will then provide consistent performance. See: [Initializing Amazon EBS volumes - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html) – John Rotenstein Jan 11 '21 at 23:04

1 Answers1

2

My initial enthusiasm for gp3 was dampened by inferior performance to gp2, however, when I set the size, IOPS, and bandwidth the same on C5, M5, and M5A instance type I got similar performance measurements from fio: https://github.com/axboe/fio

I'm testing on CentOS 7.8, fio-3.7-2.el7.x86_64. Region us-east-1.

Are you sure sysbench is the right benchmarking tool? The fileio options don't seem greatly documented, and I can get wildly different results when changing the sync mode, flags, etc.

Your example command equivalent to this because of the default values:

sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --file-io-mode=sync --file-rw-ratio=1.5 --file-fsync-freq=100 --max-requests=0 --time=600 fileio run

When I run a different sysbench command, notably using direct access and fsync=off:

sysbench fileio --file-total-size=15G --file-test-mode=rndrw --time=600 --max-requests=0 --file-io-mode=async --file-extra-flags=direct  --file-num=16 --file-rw-ratio=1 --file-fsync-freq=0 run

I get equivalent performance from gp2 and gp3. (Actually, gp3 is slightly better).

Here's a summary of my fio data, using the command

fio --rw=TEST --direct=1 --ioengine=libaio --bs=16k --numjobs=8 --size=1G --group_reporting

with TEST={read,write,randread,randwrite,randrw}

gp2:

gp2-randread-16k.out:   read:   IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170117msec)
gp2-randrw-16k.out:     read:   IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170121msec)
gp2-randrw-16k.out:     write:  IOPS=1541,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170121msec)
gp2-randwrite-16k.out:  write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170116msec)
gp2-read-16k.out:       read:   IOPS=3090,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169642msec)
gp2-write-16k.out:      write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170115msec)

gp3:

gp3-randread-16k.out:   read:   IOPS=3080,  BW=48.1MiB/s  (50.5MB/s)(8192MiB/170188msec)
gp3-randrw-16k.out:     read:   IOPS=1539,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170238msec)
gp3-randrw-16k.out:     write:  IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170238msec)
gp3-randwrite-16k.out:  write:  IOPS=3082,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170073msec)
gp3-read-16k.out:       read:   IOPS=3089,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169719msec)
gp3-write-16k.out:      write:  IOPS=3083,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170053msec)
Matt L
  • 39
  • 3
  • Adding to my confusion, tests with `ioping` consistently show `gp3` to have twice the average latency as `gp2`. I don't see any difference in latency when testing with `fio`, nor any other metric (IOPS, throughput). – Matt L Jan 12 '21 at 21:32
  • Hi Matt, Yep, I did equivalent tests with fio and got more consistent results, though the GP3s with 16000IOPS didn't perform that much better than the GP2s with 9000IOPS. I originally used sysbench since these are MySQL servers and I like to run checks with sysbench to get ideas of MySQL. In the end I think they are a suitable replacement but do not offer the performance boost stated. NOTE: AWS support said they are looking into inconsistencies with latency on GP3s, so hopefully that get's fixed. – tippingpints Jan 15 '21 at 23:17