8 Incredibly Useful Tools for Storage I/O Benchmarking


 8 Incredibly Useful Tools for Storage I/O Benchmarking

By Reduxio Systems for Beyond The Blocks - Tuesday, March 21, 2017

Storage Pros know that IOPS in spec sheets can sometimes be misleading, and real discussions about storage needs should never be only about the maximum number of IOPS, but also about actual latency and application requirements in terms of SLAs.

We decided to put together the following list to describe the favorite tools our own pros use on a day to day basis for Storage I/O benchmarking. Most of them can help you generate synthetic IO loads that put some form of stress on different environments (DBs, VMs, etc.) or let you replay real application workloads and trace back their performance.


Storage Performance Testing Basics

Read this first if you are new to running application benchmarks. For seasoned professionals skip ahead to our list of utilities.

To begin with, you must know whether you are going to test block or file storage. Sounds basic, but you’d be surprised at the times we’ve come across people comparing apples to donuts (yes. The difference between file and block is so tremendous we think the comparison doesn’t merit an ‘orange’ but rather a donut).

On which OS will you be running the tests. This will likely dictate the type of tools you’ll use, although many of the utilities listed below work both for Windows as well as Linux. Still, you’ll find that depending on the OS there are important considerations when running the benchmarks.

Finally, consider the applications you’ll be supporting. Are they running some form of database, of so which? As there are utilities that only work on a single type of DB. The following reference table will help you choose from the list:

Utility OS Type of Storage / Application
DiskSPD Windows Filesystem   
IOMeter Windows Linux  Filesystem   Block device
FIO Windows  Linux  Filesystem  Block device 
IOZone Linux  Filesystem 
VDBench Windows  Block device 
BenchmarkFactory Windows  All major DB   
HammerDB Windows  Linux  OracleDB/MySQL/SQL   
SwingBench Windows  Linux  DB  Oracle only 


General Tips on Testing

We can speak for hours on this topic. We’ve actually done this in front of large audiences, you can learn more about running application benchmarks here. For now, we’ve compiled a few tips.

    • Tip #1 Work on large datasets

      At least twice above the RAM of the OS, for example, if the OS RAM is 32GB, test 64GB datasets multiplied by number of CPU cores you have.
    • Tip #2 Golden Rule of 70/30

      Although it really depends on your workloads, typically the rule of thumb is that there are 30% writes and 70% reads on the dataset.
    • Tip #3 Test from small to large blocks of I/O

      Consider testing small blocks of I/O up to large blocks of I/O in the following order: 512 bytes, 4K, 16K, 64K, 1MB to get proper measurement that can be the visualized as a histogram. This makes it easier to interpret.
    • Tip #4 Test several patterns

      A good SysAdmin knows that not everything is sequential read/write. So test for: read / write; write only; read only; random read / random write; random read only; random write only.

Now it’s time to get into the list itself.

For Microsoft Windows:

We start with Windows because there’s a very important consideration to put into practice for all utilities. You must disable the OS cache so the benchmark doesn’t take into consideration the system’s current RAM as part of the test.

1. Microsoft DiskSpd & VMFleet (Windows)

Diskspd is Microsoft-built and recommended tool for testing and measuring your storage performance. VMFleet is a set of scripts that let you run a sort of ‘Fleet of VM’s which can then have DISKPD workloads while controlling IO patterns, quantities and other behaviors. You can get VMFleet on GitHub.

Why We like it

      • DiskSpd is very simple to use.
      • DiskSpd can generate many workload scenarios.
      • Several builds and well maintained.
      • Adopted Open-Source and can be found on GitHub.

Cons

      • For Windows only and takes some time to get the hang of it. For Windows environments we rather go with the next tool.
      • VMFleet is rather complex to understand and setup.

Find DISKSPD/VMFleet here

2. IOMeter (Windows & Linux*)

An ‘old-time’ favorite of Storage Pro’s, originally developed by Intel and and now distributed under  GNU Public License. The tool allows you to run stress tests sessions for I/O Performance Analysis.

IO Meter.png

Why we like it

      • Nice UI that’s easy to use, you can learn quickly how to run tests
      • It can produce graphs fairly easily

Cons

      • Limited command-line parameters. Advanced CLI fans don’t like this.
      • Even its full random load generation turns up entirely deduped in Reduxio - makes it hard to simulate real-life data.
      • Although pretty stable and we use it continously, development for IOMeter ended long time ago and there are no new builds. As Storage industry keeps advancing, new media and features come out, this can have a toll on how tests come out.

Tip for Storage Pro’s

      • Remember to disable OS Cache.

Find IOMeter here

For Linux:

2. IOMeter (Windows & Linux*)

We like it so much we decided to mention it twice. SourceForge gives it 4.9 out of 5 stars. However, it’s fair to say it has limited support for Linux.

3. FIO - Flexible IO Tester (Windows & Linux)

This is by far, our all-time favorite, originally written by Jens Axboe, current blq-mq developer for Linux kernel. Axboe got tired of writing specific test application and developed FIO which set the world record in 2012 for highest IOPS in a single system. Axboe left Fusion IO and is currently employed by Facebook.

FIO allows you to simulate different types of IO loads and tweak several parameters including the write/read mix, amount of processes, etc.

Why we like (LOVE) it

      • Batch mode & very extensive set of parameters.
      • It's still being developed. Our friend Jens keeps fine-tuning it.
      • Multiple OS support: Although we mostly run it on Linux, it also supports Windows.

Cons

      • No sophisticated GUI or Graphics features. This is for CLI hardliners only.
      • Sophisticated syntax, it’ll take some time to get the hang of it. But definitely worth the effort if you are going to run multiple tests.

Tip for Storage Pro’s

      • Here’s an FIO example for Windows command:

Find FIO here

4. IOZone (Linux only)

A filesystem benchmark tool that enables you to get a more broad view of filesystem performance. IOZone can let you benchmark.

IOZone.jpg

Why we like it

      • Graphics are built-in feature.
      • Their lab has a poster of Star Trek

Find IOZone here

5. VDBench (Linux only)

A CLI utility developed by Oracle which lets you generate a wide variety of I/O workloads and has plenty of control over workload parameters, include rate, LUN/file sizes, read/write ratios, which as mentioned before, is important to test different ratios, among other parameters.

VDbench.png

Why we like it

      • Lets you analyze response time by buckets (i.e. 0.1-0.2ms, 0.2-0.3ms etc.).
      • If you run histograms on the performance results you end up with quite nice looking graphics.Reports are HTML accessible as well.
      • Supported & maintained by Oracle: There’s an active community that can solve questions you might have.

Tip for Storage Pro’s

      • Check out SNIA’s presentation on how to VDBench Features and how to use it here

Find VDBench here 

For Databases Stress Testing

6. Benchmark Factory for Databases (Linux only)

A commercial, state-of-the-art load generator for databases. Developed by Quest Software, the company behind TOAD, Foglight and many other database management and monitoring tools, and owned a few years by Dell after its acquisition of Quest (Quest is now independent again).

Benchmark Factory provides out-of-the-box database workloads - OLTP, DSS with the very familiar TPC-C, TPC-H workloads and options to replay captures etc. It supports all major databases, and can similar users, transactions etc. 

Why we like it

      • Supports all major DBs.
      • Provides out of the box method to run TPC-C type workload with minimum effort.
      • Supports multiple agents.

Cons

      • Expensive - per-user licenses, and to reach 6-digit IOPS you need thousands of user licenses.

Find Quest Benchmark Factory for Databases here

7. Hammer DB (Windows & Linux)

An open source stress testing tool for multiple types of Databases. Originally called Hammerora and only supported Oracle DB, popular demand made it expand its capacity to multiple DB.

perframp.png

Why we like it

      • Supports multiple DBs and plenty of documentation including a well-thought FAQ.
      • They have their own Benchmarks section where you can compare your results.
      • Did we say Open Source?

Cons

      • Does not provide all the customizations that Quest provides for monitoring response time etc.

Find HammerDB here

Tip for Storage Pro’s

8. SwingBench (Linux)

A similar tool but designed only for Oracle DB Stress Testing. Developed by Dominic Giles who’s come up with other tools for Oracle environments.

swingbench.png

Why we like it

  • Well maintained by the original developer.
  • If you are a heavy Oracle user,

Cons

  • Works only with Oracle.

Find SwingBench here

Honorable Mentions: For Virtualized Environments

Although not exactly testing the storage itself, the following list is also useful after you’ve run benchmarks with the above tools:

  • VMWare VMMark

Developed by VMWare, a free resource that can enable you to measure the performance, scalability, and power consumption of virtualization platforms. Since many of the tools listed above were developed to test specific single-workload performance (in non-virtualized environments), with the rise of server consolidation using virtual machines, VMMark allows you to run benchmarks on multi-server virtualization platforms.

A cool feature with their 2.5 version of VMmark is the option to measure both the power consumption and the power efficiency of the servers and storage used in the benchmark tests.

vmware-vmark-4a.jpg

Find VMmark here

  • VMWare I/O Analyzer

I/O Analyzer is offered as a virtual appliance that can be used to configure and deploy benchmarks and at the same time view graphical results for the tests. Comes from VMWare Labs and considered as one of their ‘Flings’, which are apps and tools built by VMWare engineers that are intended to be played with and explored.

I/O Analyzer actually uses IOMeter for Linux under the hoods.

Find I/O Analyzer here

  • VMWare HCIBench

Another tool that comes from VMWare ‘Flings’ lab. This one is intended to work on Hyper-Converged Infrastructure, based on Oracle’s VDBench mentioned above. Useful for running tests on Virtual SANs.

Find HCIBench here

Which one is the best?

As we’ve seen, it comes down to your environment, OS, type of storage and applications you are running. Depending on who you ask at Reduxio you’ll get IOMeter or FIO as an answer (which will probably spark a debate as well). It all comes down to having the right tool for the job.

Which one is your favorite? We probably miss a few when we put together this list. Share your thoughts below.

And in case you are interested in learning how to actually conduct application benchmarks, check out the webinar with VMUG.

New Call-to-action

Reduxio Systems

Written by Reduxio Systems

There has been no fundamental innovation in data management for primary storage for the last two decades. In 2012, a group of storage industry veterans founded Reduxio with the vision to redefine data management and protection by taking advantage of new processing, networking and media technologies.





Want to comment on this blog post?

Want to share this?

    


Subscribe to our blog.