7c. Benchmarking Execution Time

Time Metrics

Julia uses the same time metrics described below, regardless of whether you use BenchmarkTools or Chairmarks. For quick reference, these metrics can be accessed at any point in the left bar under "Notation & Hotkeys".

Alongside execution times, each package also reports the amount of memory allocated on the heap, typically referred to simply as allocations. These allocations can play a major role in overall performance, and usually indicate suboptimal coding practices. As we'll explore in later sections, monitoring allocations tends to be crucial for achieving high performance.

@time

Julia comes with a built-in macro called @time, allowing you to get a quick sense of an operation's execution time. The results provided by this macro, nonetheless, suffer from several limitations that make it unsuitable for rigorous benchmarking.

First, a measurement based on just a single execution is often unreliable, as runtimes can fluctuate significantly due to background processes on your computer. Second, if that run represents a function's first call, the measurement will include compilation overhead. The extra time Julia spends generating machine code inflates the reported runtime, making it unrepresentative of subsequent calls.

Moreover, while running @time multiple times can address these issues, the macro still has a major flaw when used for rigorous benchmarking: it evaluates code in the global scope. We'll show in upcoming sections that non-constant global variables have a marked detrimental effect on performance. Consequently, the time reported doesn't accurately reflect how the function would perform in practice.

The following example illustrates the use of @time, highlighting the difference in execution time between the first and subsequent runs.

x = 1:100

@time sum(x)         # first run                     -> it incorporates compilation time 
@time sum(x)         # time without compilation time -> relevant for each subsequent run

  0.002747 seconds (3.56 k allocations: 157.859 KiB, 99.36% compilation time)
  0.000003 seconds (1 allocation: 16 bytes)

Package "Chairmarks"

A new alternative for benchmarking code is the Chairmarks package. Its notation closely resembles that of BenchmarkTools, with the macros @b and @be providing a similar functionality to @btime and @benchmark respectively. The main benefit of Chairmarks is its speed, as it can be orders of magnitude faster than BenchmarkTools.

As with BenchmarkTools, measuring the execution time of functions appropriately requires prepending function arguments with $.

using Chairmarks
x = rand(100)

display(@b sum($x))        # provides minimum time only

6.550 ns

using Chairmarks
x = rand(100)

display(@be sum($x))       # analogous to `@benchmark` in BenchmarkTools

Benchmark: 3856 samples with 3661 evaluations
 min    6.679 ns
 median 6.815 ns
 mean   6.785 ns
 max    14.539 ns

Benchmarks In Perspective

When evaluating approaches for performing a task, execution times are often negligible, typically on the order of nanoseconds. Yet, this doesn't mean that the choice of method is without practical consequence.

While it's true that operations in isolation may have an insignificant impact on a program's overall runtime, the relevance of benchmarks emerges when these operations are performed repeatedly. This includes cases where the operation is called in a for-loop or embedded in iterative computations (e.g., solving systems of equations or optimizing functions). In these situations, small differences in timing are amplified, as they are replicated hundreds, thousands, or even millions of times.

An Example

To illustrate this matter, let's consider a specific example. Suppose we want to double each element of a vector x, and then calculate their sum. In the following, we'll compare two different approaches to accomplish this task. Both methods produce the same result.

The first method will be based on sum(2 .* x), with x treated as a global variable. As we'll discuss in later sections, this approach is inefficient. A more performant alternative is given by sum(a -> 2 * a, x), where x is passed as a function argument. The measured runtimes of each approach are as follows.

x      = rand(100_000)

foo()  = sum(2 .* x)

  35.519 μs (5 allocations: 781.37 KiB)

x      = rand(100_000)

foo(x) = sum(a -> 2 * a, x)

  6.393 μs (0 allocations: 0 bytes)

The results reveal that the second approach achieves a significant speedup, requiring less than 15% of the time consumed by the slower method. However, even the "slow" approach is remarkably fast when interpreted in absolute terms, completing in under 0.0001 seconds.

This pattern will be recurring in our benchmarks, where execution times are often negligible. In such cases, the relevance of our conclusions depends heavily on context. If the operation is performed only once, readability should guide the choice of implementation. Instead, if the operation is executed repeatedly, small performance differences accumulate and might become meaningful, making the faster approach a more suitable choice.

To make this point concrete, let's place these functions inside a for-loop that runs 100,000 times. Since our sole goal is to repeat the operation, the iteration variable itself plays no role. In such cases, it's a common practice to employ a throwaway variable: a placeholder that exists only to satisfy syntax requirements, without ever being referenced. This convention signals to other programmers that the variable's value can be safely ignored. In our example, _ serves this purpose within the for-loop. It reflects that each iteration performs exactly the same operation, without depending on the for-loop index.

x      = rand(100_000)
foo()  = sum(2 .* x)

function replicate()
   for _ in 1:100_000
      foo()
   end
end

  5.697 s (500000 allocations: 74.52 GiB)

x      = rand(100_000)
foo(x) = sum(a -> 2 * a, x)

function replicate(x)
   for _ in 1:100_000
      foo(x)
   end
end

  677.130 ms (0 allocations: 0 bytes)

The example starkly reveals the consequences of calling these functions within a for-loop. The execution time of the slow version now jumps to more than 5 seconds, while the fast version finishes in under one second. Such a stark contrast underscores the importance of optimizing functions that are executed repeatedly: even seemingly minor improvements can accumulate into pronounced performance gains.