“AI Models Don’t Understand, They Just Predict”

“Generative AI models don’t understand, they just predict the next token.” You’ve probably heard a dozen variations of this theme. I certainly have. But I recently heard a talk by Shuchao Bi that changed the way I think about the relationship between prediction and understanding. The entire talk is terrific, but the section that inspired this post is between 19:10 and 21:50.

Saying a model can “just do prediction,” as if there were no relationship between understanding and prediction, is painting a woefully incomplete picture. Ask yourself: why do we expend all the time, effort, and resources we do on science? What is the primary benefit of, for example, understanding the relationship between force, mass, and acceleration? The primary benefit of understanding this relationship is being able to make accurate predictions about a huge range of events, from billiard balls colliding to planets crashing into each other. In fact, the relationship between understanding and prediction is so strong that the primary way we test people’s understanding of the relationship between force, mass, and acceleration is by asking them to make predictions. “A 100kg box is pushed to the right with a force of 500 N. What is its acceleration?” A student who understands the relationships will be able to predict the acceleration accurately; one who doesn’t, won’t.

If a person was provided with a prompt like “10 grams of matter are converted into energy. How much energy will be released?,” and they made the right prediction, would you believe they “understand” the relationship between energy, matter, and the speed of light? What if, when given ten variations on the exercise, they made the correct prediction ten times out of ten? You would likely decide that they “understand” the relationship, and if these ten exercises happened to comprise a quiz, you would certainly give them an A.

And it would never occur to you to be concerned about the fact that you can’t crack open the learner’s skull, shove in a microscope or other instrument inside, and directly observe the specific chemical, electrical, and other processes happening inside their brain as they produce their results. As we always do with assessment of learning, you would happily accept their observable behavior as a proxy for their unobservable understanding.

If a model can make accurate predictions with a high degree of consistency and reliability, does that means it understands? I don’t know. But when a person can make accurate predictions with a high degree of consistency and reliability, we award them a diploma and certify their understanding to the world.

“LLMs Just Compress Language, They Don’t Understand It”

Along the same lines as the prediction argument, you may have heard people say that generative AI models “simply compress” language instead of truly understanding it. “They just exploit patterns in the statistical structure of language.” I’ve heard some version of that dozens of times, too. But coming back to our science analogy, consider this: scientific experiments are conducted in order to generate data. Scientists examine the resulting data for patterns, and sometimes those patterns can be compressed into exquisitely elegant forms, like f = ma. What are equations like f = ma and e = mc² if not ways of compressing the outcomes of an infinite number of possible events into a compact form? A compact form that allows us to make accurate predictions?

Do the fundamental equations of physics “simply compress” the behavior of the physical universe by “just exploiting patterns” in the way the universe behaves without really understanding? Do large language models “simply compress” language without really understanding it? I don’t know. Everything hinges on your definition of the word “understand.” But I do know that one of the primary reasons I would want to achieve understanding in either case is so that I can make accurate predictions.

—

Previously Published on opencontent.org with Creative Commons License

The post “AI Models Don’t Understand, They Just Predict” appeared first on The Good Men Project.

View the full article