criterion-0.5.0.10: Robust, reliable performance measurement and analysis

Portability | GHC
Stability | experimental
Maintainer | bos@serpentine.com

Criterion.Main

Contents

- How to write benchmarks
  - Benchmarking IO actions
  - Benchmarking pure code
  - Fully evaluating a result
- Types
- Constructing benchmarks
- Running benchmarks
- Other useful code

Description

Wrappers for compiling and running benchmarks quickly and easily. See `defaultMain` below for an example.

Synopsis

- class Benchmarkable a where
  - run :: a -> Int -> IO ()
- data Benchmark
- data Pure
- bench :: Benchmarkable b => String -> b -> Benchmark
- bgroup :: String -> [Benchmark] -> Benchmark
- nf :: NFData b => (a -> b) -> a -> Pure
- whnf :: (a -> b) -> a -> Pure
- nfIO :: NFData a => IO a -> IO ()
- whnfIO :: NFData a => IO a -> IO ()
- defaultMain :: [Benchmark] -> IO ()
- defaultMainWith :: Config -> Criterion () -> [Benchmark] -> IO ()
- defaultOptions :: [OptDescr (IO Config)]
- parseArgs :: Config -> [OptDescr (IO Config)] -> [String] -> IO (Config, [String])

# How to write benchmarks

The `Benchmarkable` typeclass represents the class of all code that can be benchmarked. Every instance must run a benchmark a given number of times. We are most interested in benchmarking two things:

- `IO` actions. Any `IO` action can be benchmarked directly.
- Pure functions. GHC optimises aggressively when compiling with `-O`, so it is easy to write innocent-looking benchmark code that doesn't measure the performance of a pure function at all. We work around this by benchmarking both a function and its final argument together.

## Benchmarking IO actions

Any `IO` action can be benchmarked easily if its type resembles this:

     IO a

## Benchmarking pure code

Because GHC optimises aggressively when compiling with `-O`, it is potentially easy to write innocent-looking benchmark code that will only be evaluated once, for which all but the first iteration of the timing loop will be timing the cost of doing nothing.

To work around this, we provide a special type, `Pure`, for benchmarking pure code. Values of this type are constructed using one of two functions.

The first is a function which will cause results to be evaluated to head normal form (NF):

     nf :: NFData b => (a -> b) -> a -> Pure

The second will cause results to be evaluated to weak head normal form (the Haskell default):

     whnf :: (a -> b) -> a -> Pure

As both of these types suggest, when you want to benchmark a function, you must supply two values:

- The first element is the function, saturated with all but its last argument.
- The second element is the last argument to the function.

Here is an example that makes the use of these functions clearer. Suppose we want to benchmark the following function:

     firstN :: Int -> [Int]
     firstN k = take k [(0::Int)..]

So in the easy case, we construct a benchmark as follows:

     nf firstN 1000

The compiler will correctly infer that the number 1000 must have the type `Int`, and the type of the expression is `Pure`.

## Fully evaluating a result

The `whnf` harness for evaluating a pure function only evaluates the result to weak head normal form (WHNF). If you need the result evaluated all the way to normal form, use the `nf` function to force its complete evaluation.

Using the `firstN` example from earlier, to naive eyes it might appear that the following code ought to benchmark the production of the first 1000 list elements:

     whnf firstN 1000

Because in this case the result will only be forced until it reaches WHNF, what this would actually benchmark is merely the production of the first list element!

# Types

class Benchmarkable a where

A benchmarkable function or action.

Methods

run

Arguments

:: a | The function or action to benchmark.
-> Int | The number of times to run or evaluate it.
-> IO () |

Run a function or action the specified number of times.

Instances

Benchmarkable Pure |
Benchmarkable (IO a) |

data Benchmark

A benchmark may consist of either a single `Benchmarkable` item with a name, created with `bench`, or a (possibly nested) group of `Benchmark`s, created with `bgroup`.

Instances

Show Benchmark |

data Pure

A container for a pure function to benchmark, and an argument to supply to it each time it is evaluated.

Instances

Benchmarkable Pure |

# Constructing benchmarks

bench

Arguments

:: Benchmarkable b |
=> String | A name to identify the benchmark.
-> b |
-> Benchmark |

Create a single benchmark.

bgroup

Arguments

:: String | A name to identify the group of benchmarks.
-> [Benchmark] | Benchmarks to group under this name.
-> Benchmark |

Group several benchmarks together under a common name.

nf :: NFData b => (a -> b) -> a -> Pure

Apply an argument to a function, and evaluate the result to head normal form (NF).

whnf :: (a -> b) -> a -> Pure

Apply an argument to a function, and evaluate the result to weak head normal form (WHNF).

nfIO :: NFData a => IO a -> IO ()

Perform an action, then evaluate its result to head normal form. This is particularly useful for forcing a lazy IO action to be completely performed.

whnfIO :: NFData a => IO a -> IO ()

Perform an action, then evaluate its result to weak head normal form (WHNF). This is useful for forcing an IO action whose result is an expression to be evaluated down to a more useful value.

# Running benchmarks

defaultMain :: [Benchmark] -> IO ()

An entry point that can be used as a `main` function.

     import Criterion.Main

     fib :: Int -> Int
     fib 0 = 0
     fib 1 = 1
     fib n = fib (n-1) + fib (n-2)

     main = defaultMain [
            bgroup "fib" [ bench "10" $ whnf fib 10
                         , bench "35" $ whnf fib 35
                         , bench "37" $ whnf fib 37
                         ]
                        ]

defaultMainWith

Arguments

:: Config |
-> Criterion () | Prepare data prior to executing the first benchmark.
-> [Benchmark] |
-> IO () |

An entry point that can be used as a `main` function, with configurable defaults.

Example:

     import Criterion.Config
     import qualified Criterion.MultiMap as M
     import Criterion.Main

     myConfig = defaultConfig {
                  -- Always display an 800x600 window with curves.
                  cfgPlot = M.singleton KernelDensity (Window 800 600)
                }

     main = defaultMainWith myConfig (return ()) [
              bench "fib 30" $ whnf fib 30
            ]

If you save the above example as `"Fib.hs"`, you should be able to compile it as follows:

     ghc -O --make Fib

Run `"Fib --help"` on the command line to get a list of command line options.

# Other useful code

defaultOptions :: [OptDescr (IO Config)]

The standard options accepted on the command line.

parseArgs :: Config -> [OptDescr (IO Config)] -> [String] -> IO (Config, [String])

Parse command line options.