Disclaimer: I'm no expert on (My)SQL performance at all, just commenting on the AWS aspects of your use case.
With that out of the way, there are several questions to address, first and foremost:
Would moving MySQL data to ephemeral storage (/mnt) improve this?
I've provided an answer for the identical question Will moving data from EBS to ephemeral storage improve MySQL query performance?, please check it out for some important details - TL;DR: You most definitely don't want to do that if you have any durability needs (except if you exactly know what you are doing), and performance gains via ephemeral storage claimed in the past are also dubious at best, if not plain wrong from today's perspective.
Would using Provisioned IOPS help with?
Absolutely, Provisioned IOPS Volumes are specifically designed to meet the needs of I/O-intensive workloads, particularly database workloads, that are sensitive to storage performance and consistency in random access I/O throughput, see the post Fast Forward - Provisioned IOPS for EBS Volumes for a general introduction.
Please note that these ideally (but not necessarily) go hand in hand with EBS-Optimized Instances, which use an optimized configuration stack and provides additional, dedicated capacity for EBS I/O. This optimization provides the best performance for your EBS volumes by minimizing contention between EBS I/O and other traffic from your Amazon EC2 instance.
Specifically you'll want to read through the dedicated section Increasing EBS Performance, which addresses how to look at the I/O performance you require and your options for increasing EBS performance to meet those requirements with RAID and/or Provisioned IOPS depending on your use case.
My question: > 500 ms, > 1000ms is a reasonable latency for a query that just retrieves rows by PRIMARY KEY? Even in a 42M table? Even when all rows are in disk? It seems too much for me.
As mentioned I can't judge the values as such, however, given your specification you seem to have memory contention, insofar the m2.2xlarge instance features 'only' 34.2 GiB of memory and you are allocating ~30GB for the innodb_buffer_pool_size
already - this seem to be a bit high to me given other memory requirements of the OS and/or MySQL, so there might already be swapping involved, which would perfectly explain the cache/memory warming behavior you are experiencing.
- As a general recommendation for database workloads it seems to be the biggest bang for the buck by far these days to simply ensure your dataset fits entirely into RAM, which is easier than ever with the plethora of instance types (if at all feasible in the first place).
Finally I recommend to read the very recent post about Improving PostgreSQL performance on AWS EC2 - the recommendations there primarily address the AWS side of things as well and do apply to MySQL too accordingly; section Durable databases pretty much summarizes my suggestions above:
For a durable database where you care about your data, what you want instead of a high I/O instance is an EBS Optimized instance, which has guaranteed network bandwidth to the EBS storage servers. Use EBS volumes with provisioned IOPs and, for best results, stripe a group of EBS volumes into a RAID10 array. See increasing EBS performance.