11c. Task-Based Parallelism: @spawn

Introduction

The previous section explained the basics of multithreading. In particular, we've shown that operations can be computed either sequentially (Julia's default) or concurrently. The latter approach enables multiple operations to be processed simultaneously, with operations running as soon as a thread becomes available. When Julia's session is initialized with more than one thread, this implies that computations can be executed in parallel.

This section will focus on Julia's native multithreading mechanisms, a topic that will span several sections. Our primary goal here is to demonstrate how to write multithreaded code, rather than exploring how and when to apply the technique.

We've deliberately structured our explanation in this way to smooth subsequent discussions. However, a crucial caveat at this point remains necessary: while multithreading can offer significant performance advantages, it's not applicable in all scenarios. In particular, multithreading demands extreme caution in handling dependencies between operations, as mismanagement can lead to silent catastrophic bugs. We'll defer the topic of unsafe-thread operations for now, as identifying them presupposes a basic understanding of parallelism techniques.

Task-Based Parallelism: @spawn

The first approach we'll cover is implemented through the macro @spawn, which streamlines the application of the previous section's techniques. Specifically, by prepending any operation with @spawn, we create a (non-sticky) task that's scheduled right away for its execution. Recall that once a task is scheduled, it'll immediately start its computation if there's a thread available.

Unlike other approaches that we'll present, @spawn requires explicitly instructing Julia to wait for the task to complete. The way to do this depends on the nature of the output. For tasks that perform computation and additionally return an output, we have the function fetch. This waits for calculations of a task to finish and then returns its output. Since parallel computation requires spawning multiple tasks, the function argument of fetch should comprise all the tasks spawned and fetch be broadcasted.

In the following, we illustrate fetch with two spawned tasks that return vectors as their output.

x = rand(10); y = rand(10)

function foo(x)
    a = x .* -2
    b = x .*  2

    a,b
end

x = rand(10); y = rand(10)

function foo(x)
    task_a = @spawn x .* -2
    task_b = @spawn x .*  2

    a,b = fetch.((task_a, task_b))
end

It's important to distinguish between task_a and a: while a refers to the vector created (i.e., the task's output), task_a denotes the task creating the vector a. The distinction is essential since the function fetch only takes a task as its input.

Alternatively, for operations that don't return any output, we can use either the function wait or the macro @sync. The function wait is applied similarly to fetch. Instead, the macro @sync requires wrapping all operations to be synchronized, which is done by enclosing the operations with the keywords begin and end.

For the demonstration, let's consider a mutating function. Mutating functions are suitable as an example, since they only modify values of a collection, without returning any output.

x = rand(10); y = rand(10)

function foo!(x,y)
    @. x = -x
    @. y = -y
end

x = rand(10); y = rand(10)

function foo!(x,y)
    task_a = @spawn (@. x = -x)
    task_b = @spawn (@. y = -y)

    wait.((task_a, task_b))
end

x = rand(10); y = rand(10)

function foo!(x,y)
    @sync begin
        @spawn (@. x = -x)
        @spawn (@. y = -y)
    end    
end