11d. Thread-Safe Operations

Unsafe Operations

We start by presenting some operations that aren't thread-safe. The examples highlight the need for caution when tasks exhibit some degree of dependency, either in terms of operations or shared resources.

Writing on a Shared Variable

The following example highlights the potential pitfalls of writing to a shared variable in a concurrent environment. The scenario considered is such that a scalar variable output is initialized to zero. Then, this value is updated within a for-loop that iterates twice, with output set to i in the i-th iteration. The script is as follows.

function foo()
    output = 0

    for i in 1:2
        sleep(1/i)
        output = i
    end

    return output
end

2

function foo()
    output = 0

    @threads for i in 1:2
        sleep(1/i)
        output = i
    end

    return output
end

1

To illustrate the challenges of concurrent execution, we've deliberately introduced a decreasing delay before updating output. This delay is implemented using sleep(1/i), causing the first iteration to pause for 1 second and the second iteration to pause for half a second. Although this delay is artificially introduced through sleep, it represents the potential delays caused by intermediate computations, which could prevent an immediate update of output.

The delay is inconsequential for a sequential procedure, with output taking on the values 0, 1, and 2 as the program progresses. However, when executed concurrently, the first iteration completes only after the second iteration has finished. As a result, the sequence of values for output are 0, 2, and 1.

While the problem may seem apparent, it can manifest in more complex and subtle ways. In fact, the issue can be exacerbated when each iteration additionally involves reading a shared variable. Next, we consider a scenario like this.

Reading and Writing a Shared Variable

Reading and writing shared data doesn't necessarily cause problems. For instance, we'll demonstrate that a parallel for-loop can safely mutate a vector, despite that multiple threads are simultaneously modifying a shared object (the vector). However, in scenarios where reading and writing shared data is sensitive to the specific order of thread execution, it can give rise to a data race (also known as race condition). The name reflects that the final output will change in each execution, depending on which thread finishes and modifies the data last.

To illustrate the issue, let's keep using an example similar to the outlined above. We modify the example by introducing the variable temp, whose value is updated in each iteration. Moreover, this is a variable shared across threads, and is used to mutate the i-th entry of a vector output. By introducing a delay before writing each entry of output, the example shows that all threads end up using the last value of temp, which is 2.

function foo()
    out  = zeros(Int, 2)
    temp = 0

    for i in 1:2
        temp   = i; sleep(i)
        out[i] = temp
    end

    return out
end

2-element Vector{Int64}:
 1
 2

function foo()
    out  = zeros(Int, 2)
    temp = 0

    @threads for i in 1:2
        temp   = i; sleep(i)
        out[i] = temp
    end

    return out
end

2-element Vector{Int64}:
 1
 1

function foo()
    out  = zeros(Int, 2)
    

    @threads for i in 1:2
        temp   = i; sleep(i)
        out[i] = temp
    end

    return out
end

2-element Vector{Int64}:
 1
 2

As the last tab shows, the issue can be easily circumvented in this case. The solution simply requires defining temp as a local variable, which is achieved by avoiding its initialization out of the for-loop. By doing so, each thread will refer to its own local copy of temp.

Beyond this specific solution, the example aims to highlight the subtleties of parallelizing operations. To further illustrate it, we next examine a more common scenario where data races occur: reductions.

Embarrassingly Parallel Problems

The simplest scenario in which multithreading can be applied is known as an embarrassingly parallel problem. The term highlights the ease with which code can be divided for parallel execution. It comprises programs consisting of multiple independent and identical subtasks, not requiring interaction with one another to produce the final output. This independence allows for seamless parallelization, providing complete flexibility in the order of task execution.

In for-loops, one straightforward way to parallelize these problems is given by the macro @threads. This is a form of thread-based parallelism, where the distribution of work is based on the number of threads available. Specifically @threads attempts to evenly distribute the iterations, in an effort to balance the workload. The approach contrasts with @spawn, which is a task-based parallelism where iterations are divided according how the user has manually defined tasks. Unlike @spawn, which requires a manual synchronization of the tasks, @threadsautomatically schedules the tasks and waits for their completion before proceeding with any further operations. This is demonstrated below.

x_small  = rand(    1_000)
x_medium = rand(  100_000)
x_big    = rand(1_000_000)

function foo(x)
    output = similar(x)

    for i in eachindex(x)
        output[i] = log(x[i])
    end

    return output
end

  3.043 μs (1 allocations: 7.938 KiB)

  315.751 μs (2 allocations: 781.297 KiB)

  3.326 ms (2 allocations: 7.629 MiB)

x_small  = rand(    1_000)
x_medium = rand(  100_000)
x_big    = rand(1_000_000)

function foo(x)
    output = similar(x)

    @threads for i in eachindex(x)
        output[i] = log(x[i])
    end

    return output
end

  10.139 μs (122 allocations: 20.547 KiB)

  42.044 μs (123 allocations: 793.906 KiB)

  340.589 μs (123 allocations: 7.642 MiB)

In the next section, we provide a thorough analysis of the differences between @threads and @spawn.