11d. Thread-Safe Operations

Thread-Unsafe Operations

In a multithreaded environment, an operation is deemed unsafe if executing it concurrently can lead to incorrect behavior, data corruption, or even program crashes. These issues typically arise when tasks exhibit some degree of dependency, either in terms of operations or shared resources. The examples below illustrate how these dependencies can introduce serious issues when multithreading is applied without proper safeguards.

Writing on a Shared Variable

One of the simplest examples of a thread-unsafe operation is writing to a shared variable. To illustrate, consider a scalar variable output initialized to zero. This is then updated in a for-loop with two iterations, where the i-th iteration sets output to the value i.

To starkly show the challenges of concurrent execution, we'll deliberately introduce a decreasing delay before each update of output. This is implemented with sleep(1/i), causing the first iteration to pause for 1 second and the second iteration for half a second. Although this delay is artificially induced via sleep, it mimics the kind of timing gaps that real computations can introduce. Such gaps can cause output to be updated in an unexpected order.

function foo()
    output = 0

    for i in 1:2
        sleep(1/i)
        output = i
    end

    return output
end

2

function foo()
    output = 0

    @threads for i in 1:2
        sleep(1/i)
        output = i
    end

    return output
end

1

The delay is inconsequential for a sequential procedure, where output takes on the values 0, 1, and 2 as the program progresses. However, when executed concurrently, the first iteration completes after the second iteration has finished. As a result, the sequence of values for output is 0, 2, and 1.

Sequential

Parallel

While the problem may seem apparent in this simple example, it can manifest in more complex and subtle ways in real-world applications. The core issue is that the order of execution isn't guaranteed in a multithreaded environment. Thus, when multiple threads modify the same shared variable, the final value depends on which thread executes last.

In fact, the issue can be exacerbated when each iteration additionally involves reading a shared variable. Next, we consider such a scenario.

Reading and Writing a Shared Variable

Reading and writing shared data doesn't necessarily yield incorrect results. For instance, a parallel for-loop could safely mutate a vector: even if multiple threads are simultaneously modifying the same shared object, each thread would be operating on distinct elements of the vector. Since no two threads would interfere with one another, the updates would remain independent.

Problems arise, however, when the correctness of reading and writing shared data depends on the order in which threads execute. This situation is known as a race condition. The term reflects that the final output may change from one run to the next, depending on which thread finishes and updates the data last.

To demonstrate the issue, let's modify our previous example by introducing a variable temp, whose value is updated in each iteration. This variable will be shared across threads and used to mutate the i-th entry of the vector output. By introducing a delay before writing each entry of output, a race condition is introduced, where all threads end up using the last value of temp (in this case, 2).

function foo()
    output = Vector{Int}(undef,2)
    temp   = 0

    for i in 1:2
        temp      = i; sleep(i)
        output[i] = temp
    end

    return output
end

2-element Vector{Int64}:
 1
 2

function foo()
    output = Vector{Int}(undef,2)
    temp   = 0

    @threads for i in 1:2
        temp      = i; sleep(i)
        output[i] = temp
    end

    return output
end

2-element Vector{Int64}:
 2
 2

function foo()
    output = Vector{Int}(undef,2)
    

    @threads for i in 1:2
        temp      = i; sleep(i)
        output[i] = temp
    end

    return output
end

2-element Vector{Int64}:
 1
 2

Sequential

Parallel

In this specific scenario, the issue can be easily circumvented: instead of defining temp outside the for-loop, we simply define it as variable local to the for-loop. In this way, each thread would work with its own private copy of temp, thereby eliminating the data race entirely.

Beyond any solution proposed, the example highlights the subtleties of parallelizing operations. Even simple patterns can conceal dependencies that lead to unsafe behavior when executed concurrently. To illustrate this further, we now turn to a more common scenario where data races occur: reductions.

Embarrassingly Parallel Programs

In the context of thread-safe programming, the simplest case occurs when tasks are completely independent of one another. These cases are known as embarrassingly parallel problems. embarrassingly parallel problems. In these problems, the computation can be decomposed into many independent tasks, each of which can be executed in parallel without communication, synchronization, or shared state. Because no task influences any other, the program has full freedom to schedule and execute them in any order, making parallelization highly efficient.

In the context of for-loops, a simple way to parallelize embarrassingly parallel problems is through the macro @threads from the package Threads. This is a form of thread-based parallelism, where the distribution of work is based on the number of threads available. In particular, @threads attempts to balance the workload by dividing the iterations as evenly as possible. Unlike @spawn, @threads automatically schedules the tasks and waits for their completion before execution proceeds. In the next section, we'll present a detailed comparison between @threads and @spawn. For now, we simply demonstrate the simplicity of @threads through the following example.

x_small  = rand(    1_000)
x_medium = rand(  100_000)
x_big    = rand(1_000_000)

function foo(x)
    output = similar(x)

    for i in eachindex(x)
        output[i] = log(x[i])
    end

    return output
end

  3.319 μs (3 allocations: 7.883 KiB)

  332.609 μs (3 allocations: 781.320 KiB)

  3.396 ms (3 allocations: 7.629 MiB)

x_small  = rand(    1_000)
x_medium = rand(  100_000)
x_big    = rand(1_000_000)

function foo(x)
    output = similar(x)

    @threads for i in eachindex(x)
        output[i] = log(x[i])
    end

    return output
end

  9.092 μs (125 allocations: 20.508 KiB)

  42.710 μs (125 allocations: 793.945 KiB)

  336.284 μs (125 allocations: 7.642 MiB)