Free AI Tools for Coding in Julia: Disappointing Results

Introduction

In the past, I've applied AI tools for coding in HTML/CSS/JS, Python, and R. While the results are far from perfect, they tend to work relatively well after some minor adjustments. However, Julia is a less popular language, and hence there's a limited amount of information to train AI models. Despite this limitation, can we expect AI tools to perform decently well when applied to Julia?

The post explores this question by focusing on free AI tools. The conclusion is that these tools applied to Julia tend to perform quite poorly. My claim is supported by two reasons.

First, the suggested Julia code often requires more than simple adjustments—it's not uncommon for the code to need a complete rewrite, unlike what occurs with more popular languages.

Second, AI chatbots perform particularly poorly at explaining how the Julia language works. This is a more severe issue than the first one, as it affects new learners disproportionately and incorrect explanations aren't as easy to spot as invalid code. In fact, it's quite common obtaining answers based on other languages. This is likely arising because of the limited Julia data, combined with ample information for other languages. As an example, all the AI chatbots consistently claim that vectorized code is faster than for-loops in Julia, and even discourage the use of functions. Similarly, the AIs state that Julia creates views of slices by default, rather than copies. [note] ChatGPT-4 was providing different answers, depending on the question asked.

On a more positive note, AI tools seem pretty efficient at constructing regular expressions in Julia. This is remarkable, as writing regular expressions isn't particularly hard, but can quickly become tedious. [note] To avoid a lengthy post, examples with regular expressions aren't included. I'll probably write a separate post about it.

Overall, the main message is that you should be cautious when using AI tools for coding, but even more so when applying them to Julia. This becomes particularly relevant if you rely on AI tools to learn the language itself.

Conclusions of the Tests

There are two key points you can appreciate from the results.

All code has been checked under Julia 1.9.2.

TEST 2: IDENTIFYING PERFORMANT CODE

For this test, I also consider two examples. The first one asks the AIs to enhance some existing code, whereas the second one asks the AIs to rank three code snippets by performance.

Asking for Improvements in Code

The example considers a code snippet that is clearly inefficient due to the use of global variables. The goal was to identify whether the AIs would suggest wrapping the code in a function. None of them did it.

# The following code is written in Julia to compute 'y'. 
# Rewrite it to speed up its computations.

x = rand(1000)
y = zeros(length(x))

for i in eachindex(x)
    y[i] = sum(x[i:end]) / i
end

x = rand(1000)
n = length(x)
y = zeros(n)

cumulative_sum = cumsum(x)

for i in 1:n
    y[i] = cumulative_sum[n] - cumulative_sum[i-1]
    y[i] /= i
end

x = rand(1000)
y = zeros(length(x))
cumulative_sum = sum(x)

for i in eachindex(x)
    y[i] = cumulative_sum / i
    cumulative_sum -= x[i]
end

x = rand(1000)
y = zeros(length(x)) 

s = cumsum(x)

for i in eachindex(x)
    y[i] = (s[end] - s[i]) / i  
end

x = rand(1000)
y = zeros(length(x))

# Calculate the cumulative sum of x.
cumsum_x = cumsum(x)

# Calculate y using the cumulative sum.
for i in 1:length(y)
    y[i] = cumsum_x[i] / i
end

# invalid code, and changing `dim` for `dims` provides a different result
x = rand(1000)
y = cumsum(x, dim = 1) / cumsum(1:length(x), dim = 1)

All the models provide similar answers if you clear the chat and re-execute the prompt, so this feature is not determining the results. The only exception is Bard, which provides different answers. It occasionally suggested performant code, but due to incorrect reasons. [note] For instance, several times it wrapped the code in a function, without any reference to why it was doing it. Quite likely, the use of functions was actually responding to a "hallucination", where it provided computation times by executing @btime.

For ChatGPT-4, I additionally tested other prompts. The aim was to ensure that the model wasn't assuming I'd eventually wrap the code in a function. The results were identical. Furthermore, I asked whether the code suggested was "a more" efficient or "the most" efficient solution, for which it indicated that "the provided solution is the most efficient for this specific problem."

It's also interesting that Phind has the capability of accessing the internet. This makes the incorrect answer more surprising, as it reports "Performance Tips" as one of its sources. This webpage is part of the official documentation, and its first recommendation is to wrap code in functions to avoid the use of global variables.

Ranking Code Snippets by Performance

The example provides three code snippets to the AIs, all of which return the same output. Then, the task of the AI is to rank each code by performance. This makes it possible to identify whether the AIs provide a correct answer, but also assess the explanations outlined. Only ChatGPT-4 was able to do it.

# I'm using Julia and have codes named 'code 1', 'code 2', and 'code3'. 
# If I execute each as they're written, which code should be faster? 
# Rank them by speed.

##### CODE 1 ######
x = rand(100_000)
y = similar(x)

function example1(x,y)
  for i in eachindex(x)
      y[i] = log(x[i] + 1) ^ x[i]
  end
  
  return y
end

example1(x,y)



##### CODE 2 ######
x = rand(100_000)
y = similar(x)

function example2(y)
  for i in eachindex(x)
      y[i] = log(x[i] + 1) ^ x[i]
  end 

  return y
end

example2(y)

##### CODE 3 ######
x = rand(100_000)
y = similar(x)

for i in eachindex(x)
  y[i] = log(x[i] + 1) ^ x[i]
end

The results are as follows.

ChatGPT-3.5 indicates that Code 3 is the fastest because "it preallocates the memory for `y` before the loop, which can be more efficient than allocating it within the loop ". Conversely, Claude-2, Bard, and LLaMA2-70b reach the same conclusion, but because "Code 3 avoids the function overhead of Code 1 and Code 2".

ChatGPT-4 is the only model that provides the correct answer. Furthermore, it offers an appropriate analysis of each option.

Remark

Phind only uses ChatGPT-4 if you're logged in, otherwise it's powered by ChatGPT-3.5.