The recent explosion in Artificial Intelligence (AI) has not left compilers
untouched. In many cases, for a good reason. For example, cost models are one
plausible application of machine learning. For simplicity, here’s how a cost
model works: it is given a space of possible solutions and it chooses one them.
For example, let’s say you have a loop that can be tiled be a factor of 16, 32,
or 64. You may, then, have a cost model that is asked to choose among the four
options: not tile at all, or tile by any of the three factors.
For the longest time, cost models have been written and maintained by hand.
However, hand-tuned cost models are often boring to make, hard to scale
(especially as the number of architectures increases), and anyway hardware
vendors tell us so little about their architectures that hand-tuned cost models
are (as machine-learning-based ones) black magic; they are not interpretable or
intuitive. In other words, a great case for machine learning and neural
networks because they can automate the process. Indeed, a lot of research on machine-learning cost models has appeared the last decade, and some have even made into production. For example, recently Google adopted a neural network cost model
for its XLA compiler.
But there’s a catch. Cost models are good application of neural networks because
you don’t need any strict guarantees. In particular, the way to think about it
is that cost models are given a set of correct programs to choose from. As such, no matter
which choice the cost model makes, the generated code will be correct! So, even though neural networks currently have this “minor” issue that we
have no idea how they work and they have no guarantees, this is fine e.g., for cost models.
However, it is not fine for code generation! This used to be obvious, but recently my
impression is that it’s not anymore. For example, recently we got a comment from
one of the reviewers of Dias which
basically asked “can’t you just use an LLM?” No! Please allow me to take a
moment to explain why using LLMs for code generation is not (at least currently)
ok at all.
Here’s the question this boils down to: Is it ok if the code generated from the
compiler is correct, say, 99% of the time? We need to clarify this a bit. Does
that mean that out of 100 generated programs, 99 are fully correct and the one
is near-total garbage? Obviously not, this not realistic. A more realistic
version would be something like this: Out of 100 lines of code, the compiler
generates correct code for 99 of them. I know this is still pretty simplistic
but I think it’s good enough for this argument. So, is that ok?
I hope that your answer is no. From a compiler developer’s standpoint, this
absolute garbage. Basically such a compiler is unusable. At best, it is some
kind of research artifact that helps you explore an idea. But forget production.
It’s not even ok for debugging. To see why, consider a small project with say
5000 lines of code. With 99% correctness rate, this means that in every
compilation, 50 lines of code are incorrect. Fifty! And the worst part is: you
don’t know which and they can be different with every code change. You probably
have had the experience of tracking down a bug in a single line of code, which
can be both frustrating and time-consuming. Image how it is debugging 50
changing lines of code! Now, imagine moving this to a large-scale project, with
possibly millions of lines of code. No, thanks.
A final note: Even if LLMs were producing correct code, they are currently
extremely slow compared to compilers. I know, I said compilers are not that
fast in previous sections. But, compared to LLMs, mainstream compilers are
blazingly fast. So, even though LLMs have a promising future, and already a
bright present, there are some things they currently don’t solve. One of them is
(online) code generation.
