This blog on Why IOPS Don’t Matter was originally published in 2014. Some figures may have changed.
The commonly accepted measure of performance for storage systems has long been IOPS (input/output operations per second). The first storage system I ever installed at a customer site had controllers rated for 20K IOPS and we all thought it was awesome.
Over the years, I’ve developed an extensive knowledge of storage IO benchmarking tools and have demonstrated hero IOPS numbers many times on a number of different storage systems.
Today, I’ve learned that IOPS don’t matter.
Who needs all those IOPS anyway?
These days, you would struggle to find a storage system on the market that cannot deliver at least 100K IOPS. Some vendors brag about million IOPS numbers or even multimillion IOPS numbers.
At Pure, we’ve rated our FA-400 systems for 400K IOPS. Do many customers need this? Really? No…
In the generic, multipurpose enterprise storage market (a.k.a. “Tier 1”) used for your typical mix of enterprise applications and physical and virtual servers, my experience is that finding a customer who needs even 100K IOPS is very rare. I used to ask customers how many IOPS their current array was doing at peak time and the answer was invariably 30K to 40K IOPS. It was rarely more than that—the exception being VDI environments and some very specific database workloads, of course. I’ve stopped asking. I know that any one of our systems can do more IOPS than what 99% of the customers need.
How do you measure IOPS?
When testing a storage system, the standard practice has long been to use an industry-standard benchmark tool such as Iometer or Vdbench to find out how many IOPS a system can deliver with different IO profiles.
Unfortunately, these IO profiles are usually based on outdated assumptions. My personal opinion is that they aren’t realistic.
Why is that? Because most of the profiles used in these benchmarks are based on small, 4KB or 8KB IOPS, whereas the average block size commonly observed on customer arrays in mixed workload environments is 32KB to 64KB.
The following chart shows the average block size across all the Pure Storage® systems dialing home as of April 2014. As you can see, there’s actually a very small percentage (<5%) of systems with an average block size of less than 10KB and less than 15% of systems with an average block size below 20KB.
Even a single application rarely does just one type of IO—just one single database instance will have different IO profiles for the different components of the engine (e.g., data files, logs, indexes).
So these synthetic benchmark tools will allow you to extract a number. Unfortunately, this number has no relationship to what you can expect in your environment.
And what about latency?
Can you use the latency measured with these benchmark tools to evaluate and compare storage systems? Not really.
Even if we ignore the fact that these tools tend to measure average latencies and miss outliers (one single IO taking longer than the other ones in the same transaction can slow down the whole transaction), the latency of an IO will vary depending on the block size. Since the block size isn’t realistic for IOPS benchmarking, the latency measured during these benchmarks is also pretty much useless.
While IOPS undoubtedly offers valuable insights into how quickly a storage device can handle read and write operations, it doesn’t provide a holistic view of real-world performance. The obsession with achieving higher IOPS figures can lead to a tunnel vision that overlooks other crucial factors such as latency, throughput, and overall system architecture. Modern applications and workloads are becoming increasingly complex, demanding a more nuanced approach to evaluating storage performance beyond a single metric.
So if neither IOPS nor latency are a good measure of the performance of a storage system, what is then?
Run the app, not a benchmark tool
The only real way to understand how fast an application will run on a given storage system is to run the application on this storage system. Period.
When you run a synthetic benchmark tool such as Iometer, the only application you’ll measure is Iometer.
Ideally, move your production applications to the storage system you’re evaluating. If you can’t move the app, move a copy of this app or the test/dev server/instances, and run the exact same tasks your users would run on your production app.
Then, measure how this app behaves with your real data and your real workload.
Measure application metrics, not storage metrics
What’s the point of measuring IOPS and latency anyway? After all, these are metrics that are relevant only to the storage admin.
Will your application owner and end users understand what IOPS means to them? Does your CIO care about storage latency?
No. Outside of the storage team, these metrics are useless; the real metrics that application owners and users care about are metrics that relate to these apps. It’s application and user metrics that should be measured.
- How long does this critical daily task take to execute in the application?
- How fast can your BI make data available to decision makers?
- How often can you refresh the test and dev instances from the production database?
- How long does it take to provision all of these virtual servers the dev team needs every day?
- How many users can you run concurrently without them complaining about performance issues?
- How quickly can this OLAP cube be rebuilt? Can it now be rebuilt every day instead of every week?
Take the time to test properly and measure what really matters
Testing a storage system in your environment with your applications is the only responsible way of evaluating it. Don’t just believe spec sheets or vendor claims. Test a real system in a proof of concept, in your environment, with your data.
But just as important is measuring the correct metrics. At the end of the day, a storage system’s job is to serve data to applications. It’s the impact on these applications that should be measured.
If you want to evaluate a great all-flash array, contact your Pure representative today.
We’d love to show you that our systems do what we say they do and to work with you to understand what really matters to your users, application owners, and ultimately, your business.
