1 Motivation

Autonomous systems, by definition, act without immediate human control. When engineering such systems, we require assurance that bad behaviors cannot occur. To achieve such assurance, we combine careful engineering, rigorous testing, and, sometimes, formal methods. It has become clear, however, that many of the autonomous systems we are trying to deploy, such as self-driving cars and autonomous robots, cannot work reliably enough without employing AI techniques such as deep neural networks (DNNs).

The engineering of DNN-based AI systems is, in the words of Janelle Shane, more like educating a child than like traditional software engineering [30]. Traditional methods of “careful engineering” prove inadequate. Moreover, rigorous testing is extremely challenging because the scenarios we most care about, which are typically anomalous ones, are difficult to anticipate and extremely difficult to test for. And while formal methods have shown some progress for small AI systems, their applicability to the kinds of systems being deployed is questionable. They certainly can play a role, for example by systematically synthesizing scenarios for training and testing [9]. But they show little promise of providing the sorts of “proofs of correctness” that formal methods strive for, except at rather small scales.

Consider the example where, in San Francisco in October, 2023, a pedestrian was hurdled in front of a self-driving taxi by another accident and ended up under the taxi. The taxi detected the pedestrian and braked hard, but then decided to move out of traffic, dragging the pedestrian pinned under the car some 20 feet and stopping with the rear wheel on top of the pedestrian. The California DMV suspended Cruise’s driverless permits, and Cruise recalled all its driverless vehicles for redesign.

There are many subtly different conceptions of intelligence. AI researchers Legg and Hutter emphasize instrumental rationality [15], saying that “intelligence measures an agent’s ability to achieve goals in a wide range of environments” [23]. City roads certainly qualify as a “wide range of environments.” Legg and Hutter assume that each environment can be modeled as a computable function, they assign a probability proportional to the complexity involved in computing that function, and then define intelligence as the average degree to which goals are met in a given environment weighted by the probability of that environment occurring.

Here, Legg and Hutter are relying on a foundational assumption, pervasive today among scientists and engineers, that real-world environments are, at a fundamental level, computable. I have previously shown that this hypothesis is untestable by experiment [21], and that it leads to models of the physical world that are awkward and inconvenient [22]. In this paper, I go further and adopt the perspective of the philosopher Nancy Cartwright, who asserts that true facts about the world may not even be formalizable, much less computable [5].

The intuition behind Legg and Hutter’s model is nevertheless valid, that most conceptions of intelligence require adaptability to a variety of environments. Adaptability is a form of learning, an ability to improve based on observations. Cruise’s recall represents a form of learning, and, here, the AIs have a distinct advantage over humans. The experience of a single autonomous system can improve the performance of a whole fleet.

The requirement that intelligence include adaptability and learning depends on an assumption that it is impossible to know everything, in the sense of having a true, justified belief regarding every fact of the world and being able to predict future facts. If such an all-knowing being were possible, then under this conception of intelligence, that being would not be intelligent. Wolpert has proven that no mechanism performing perfect prediction of the physical world can exist in the physical world [42], so this contradiction cannot arise.

I will show two ways in which adaptability is incompatible with certainty. The first way is rather trivial, where I use Bayes’ law to show that certainty makes it impossible to learn. The second way is much more subtle. I will make the case that the mechanisms we use to build certainty, specifically logic, mathematics, and algorithms, are not, as I myself previously thought, the foundations of intelligence, but are rather a consequence of intelligence. They are mechanisms invented by an evolutionary process, not facts that humans have discovered. The real foundational mechanism is prediction, a phenomenon exhibited by even the most primitive animal nervous system. Prediction also happens to be the foundational mechanism behind today’s revolutionary AIs.

Intelligence, as we commonly understand it, is a phenomenon that, until recently, has only been exhibited in the natural world. We have now made machines that unmistakably exhibit behaviors that are indistinguishable from human intelligence. Even the experts have been surprised by these behaviors. My claim here is that these behaviors may not be explainable in the way that, for example, Newton’s laws explain gravitational motion. These behaviors may be fundamentally incompatible with the mathematical world of certainty.

I will begin with the trivial argument, that certainty prevents learning. I will then build a case that the mechanisms of certainty, reasoning with logic, mathematics, and algorithms, are the result of intelligence rather than its precursor.

2 Probability

Learning can be modeled formally using probability theory. Here, I take a distinctly Bayesian approach, where probability is a model of what we don’t know, not a model of some intrinsic randomness in a system (see [21, Chapter 11]). Suppose that A and B are events. For example, suppose that A is the event that there is a pedestrian under the car, and B is the event that the car hits a pedestrian. Suppose our sensors have told us that the car has hit a pedestrian, i.e., that B has occurred. Then we are interested in P(A|B), the probability that there is a pedestrian under the car given that we have hit a pedestrian. According to Bayes’ rule,

$$ p(A | B) = \frac{p( B | A ) p(A)}{p(B)} . $$

p(A) is called the “prior probability” or, more simply, the “prior.” It gives our estimate, before we make an observation, of the likelihood of A occurring. Given an observation of event B, Bayes’ rule gives us a way to update that prior to a “posterior probability” P(A|B). The posterior is the probability that there is a pedestrian under the car (A) given that we have observed hitting a pedestrian (B).

In a Bayesian approach, probabilities are subjective. But notice a key feature of Bayes’ rule: if we are certain that there is no pedestrian under the car, then our prior probability \(P(A) = 0\). Perhaps we have formally verified this, proving for example that no pedestrian would fit under the car. Then the posterior probability P(A|B) is also zero (unless \(P(B) = 0\), i.e. we are certain that the car could never hit a pedestrian, an unreasonable choice). Certainty, in this case, makes it impossible to update our prior! The posterior is always equal to the prior when we are certain, regardless of observations. Certainty makes it impossible to learn, impossible to adapt, and hence impossible to be intelligent.

In applying Bayes’ rule, certainty could also appear in the form of a prior where \(p(A) = 1\). I.e., we are certain that A will occur. In this case, according to the rules of probability, \(p(B|A) = p(B)\), which again leads to \(p(A|B) = p(A) = 1\). The posterior is equal to the prior. Once again, certainty makes it impossible to learn from observations.

Through Bayesian learning, we can steadily improve our models of the environments in which our systems operate. Does this lead, in the limit, to a model in which we can reach a reasonable level of confidence? The optimism that it does is based on an assumption that many of us take as an axiom and rarely question. We assume that the physical world can be modeled to arbitrary precision. Models play a central role in any effort to build certainty. We therefore examine the role of models next.

3 Models

Formal verification promises certainty about a model, not about a physical system [20]. The usefulness of that certainty, of course, depends on the likelihood that the physical system will actually conform with the model. But more importantly, it requires us to have a model of the environments in which the system will operate.

Here, we have to distinguish between two uses of models [21]. For a “scientific model,” we demand that the model conform with the physical system. For an “engineering model,” we demand that the physical system conform with the model. When Box famously said, “all models are wrong, but some are useful” [3], he was referring to scientific models. The corresponding statement for engineering models might be “all physical systems are wrong, but some are useful” [20].

A piece of software is usually an engineering model. Ultimately, it specifies how we would like a machine to behave, and the goal of the electronics industry is to deliver that behavior with very high (but never perfect) reliability. Models of an uncertain environment, however, are necessarily scientific models. We usually cannot design those environments, at least not completely. Because such models will always be approximate and incomplete, it seems we have to incorporate learning into our systems. This means that the system will evolve. It will change in the field, further undermining our certainty that it will not do bad things.

The discipline of formal verification is all about providing assurance in a mathematically rigorous way. A model that is amenable to formal verification is a formal system with clearly stated axioms and assumptions. Assurance takes the form of a proof that a certain well-defined class of bad behaviors is inconsistent with the model and the assumptions. The proof is based on the axioms and rules of logic. Such a proof provides certainty, meaning that if the assumptions hold, and the logic and axioms are sound, then the model cannot exhibit a behavior in the defined class of bad behaviors.

Certainty, as expressed here, has a plethora of caveats. What makes us believe in the model, the assumptions, the axioms, and the rules of logic? An axiom, by definition, is something we take to be self evidently true. And most of us trust at least some set of rules of logic. If we accept these, we are left with the assumptions and the model as sources of doubt.

4 Computation

If software is a model, to what extent can certainty about software provide certainty about systems? Digital electronics has proven astonishingly reliable at carrying out logical operations specified by software. A modest microprocessor performs billions of operations per second and can work flawlessly for years. The software that we write for autonomous systems executes on such microprocessors, and itself can be abstracted and analyzed using rules of logic. A computer program is a model of what we would like the electronics to do. It is possible, and often reasonable, to be certain about a program. Under the assumption that the electronics executes the program correctly, which it almost always does, we can then have absolute confidence in the result. Since the AIs today that seem to exhibit intelligence are realized on computers, shouldn’t certainty be achievable?

Some efforts to regulate AI demand “algorithmic transparency.” An algorithm is a sequence of rule-based transformations of data that terminates with a conclusion. The computer programs that realize the training and operation of today’s AIs specify algorithms. The hope with such regulation is that if you know the algorithm, you will know how the behavior of the AI comes about. Unfortunately, this has not proven to be the case. Even the experts and the programmers themselves are mystified by the behavior exhibited.

The programs that realize the AIs are relatively simple. They consist of straightforward arithmetic calculations repeated a vast number of times. Unfortunately, even knowing exactly what arithmetic calculations are carried out offers very little insight into why the AIs behave the way they do. The belief that such insight will emerge is a bit like the following sentiment: If you assume that everything that happens in the world is a consequence of the laws of physics, then you should be able to understand the emergence of Russia’s war against the Ukraine in terms of the laws of physics. This is clearly absurd. In fact, the laws of physics explain little of what happens in the world. For example, although metabolic pathways in biology depend on chemistry, which depends on physics, very few biological processes are usefully explained in terms of the laws of physics. This has led several thinkers, such as Cartwright [5], to conclude that while these laws sometimes provide useful approximations and abstractions, they should not be conflated with true facts about the world.

Some efforts to regulate AI demand a “right to an explanation.” Explanations can take many forms, and some forms have proved useful for justifying certain AI decisions. For example, Ribeiro et al. [28] pioneered a technique for explaining the results of a classifier by identifying the portions of the input data that most influence a classification decision. Such an explanation, it was hoped, would amount to being able to make statements like, “it is a cat because it has four paws, pointy ears, and whiskers.” Such a statement loosely has the form of a logical deduction. Ribeiro, et al. showed, however, that AI decisions could be based on spurious patterns in the training data. They constructed an (admittedly artificial) training data set that led to the classifier tending to classify an image as a wolf if the background had snow in it and a dog otherwise. In fact, such region-of-interest explanations have proved to be rather crude, like old-fashioned taxonomy in botany. They are more useful for identifying defects in a training dataset than for explaining the classifications performed by an AIs. And the principle has proven stubbornly difficult to apply to the large-language models.

When we view AIs as computer programs, we are unlikely to find explanations for their actions adequate. The very same explanations might be perfectly adequate for actions taken by a human actor. For human actors, adequate explanations usually involve beliefs, attitudes, desires, and intentions [7]. But “the AI denied the loan because it disliked the applicant” would not be an adequate explanation in the way it could be for a human banker. For computer programs, we seek instead explanations in the form of logical deductions governed by laws of physics or logic. Davidson [7] makes a distinction between causal explanations and explanations of actions (or rationalizations), and what we seek for machines are causal explanations based on laws rather than desires.

The European GDPR (General Data Protection Regulation) and the 2024 EU AI Act (Art. 86) both include a “right to an explanation” when a decision is made by a machine. Some legal scholars argue that these clauses are unenforceable [37]. In my opinion, it would be a mistake to attempt to enforce such a clause. Humans are actually very good at explaining decisions made by humans. Ask any historian why Russia started a war with Ukraine in 2022, and they will give an explanation (albeit not in terms of the laws of physics). Taleb calls the human mind an “explanation machine,” saying that we are masters at explaining any decision [34].

Today’s large-language models can similarly provide convincing explanations for their decisions. Together with a Fulbright Scholar visiting my research group, Moez Ben Haj Hmida, we performed some experiments with ChatGPT, giving it legal cases and asking for explanations for a variety of decisions that could be made. In most cases, it was able to provide convincing explanations for any decision. I expect that as soon as there is enforcement of a right to an explanation, a new generation of LLMs will be trained specifically to provide such explanations. Hence, such enforcement will backfire. Once we have artificial explanation machines, it will make it even harder to contest the decisions of an AI.

One general form of “explanation” is itself an algorithm. It is a sequence of logical deductions that lead to the conclusion. It may seem that an explanation for the actions of a machine should take this form. For an explanation to be useful to a human, however, the sequence needs to be quite short. As proposed by Simon, humans have “bounded rationality,” a limited capacity to reason via logical deductions and a limited ability to handle large data sets [31, 32]. If we think of the computer operations performed by an AI when answering a query as logical deductions, then the sequence is so vastly longer than a human can comprehend that it becomes useless as an explanation. “Your loan application was denied because these trillion arithmetic calculations led to a number less than one.”

Formal methods such as automated theorem proving and model checking have an advantage over humans in that their rationality is much less bounded. The number of steps in the sequence and the size of the set of alternatives is only constrained by the memory capacity and processing speed of the computer. If a human trusts the methods used, then achieving certainty becomes possible with less-bounded rationality. However, these formal methods do not appear to be anywhere near being able to handle the scale of modern AIs. They can handle only very small neural networks. The gap is so huge that it seems delusional to expect it to be closed. Perhaps the questions we need answered about the AIs simply cannot be answered by these methods.

The dramatic progress of science and engineering over the last century or so has been deeply rooted in the formal structures of logic and mathematics. It is hard to imagine achieving what we have achieved without these structures. But these same successes have made such structures into a god, a Kuhn-style paradigm [16] that is so deeply entrenched that we are blind to its defects. Many thinkers have concluded that reality itself is mathematics and computation. See, for example, Lloyd [24], Gleick [11], Tegmark [35], Domingos [8], and Wolfram [41]. Thoughtful counterpoints are more skeptical about binding reality with logic and math. See for example Cartwright [5], Hossenfelder [12], Smolin [33], and Wilson [39].

I have previously argued that computation is far from universal (see Lee [21, chapter 8]). Similar limitations have cropped up in logic and mathematics (see Kline [14]). The new LLMs seem to have an ability with logic, mathematics, and computation that arises not from the intrinsic capabilities of the machines they run on, but rather as an emergent property from prediction. I examine this phenomenon next.

5 Prediction

Bubeck et al. [4] report an extensive set of experiments where they compare GTP-3.5 (the version that shook the world in late 2022) against its successor GPT-4, both from OpenAI. The underlying technology behind both is token prediction. Given a sequence of tokens representing words or symbols that might have been uttered or written by a human, they predict the next sequence of tokens. They are trained on a vast amount of textual data harvested from the Internet. The difference between GPT-3.5 and GPT-4 is the size of the training data set and the number of parameters that get adjusted during training.

A truly remarkable phenomenon that emerged from these models is their ability to reason about numbers, do arithmetic, manipulate algebraic expressions, and write computer programs.Footnote 1 All of these lie in the world of the certain, and they have long been within the capabilities of computers, albeit in a mechanical way. Calculators do well with numbers and arithmetic, symbolic algebra programs do well with algebraic expressions, and compilers have written computer programs since the 1950s. But the GPTs are not using any of this prior technology, except that they have “read about it” during training.

The GPTs do not use the (flawless) arithmetic capabilities of the machines they are running on when performing arithmetic.Footnote 2 Indeed, they make errors that the machines would not make. The errors they make look remarkably human-like, and the progress between GPT-3.5 and GPT-4 is astonishing. To me, the errors made on math problems in Bubeck et al. [4] by GPT-3.5 look like those that a smart high-school student might make, whereas those made by GPT-4 look like errors a graduate student might make.

Moreover, Bubeck et al. show that rephrasing a question can yield better answers. They show an example where, when asked to simplify a complicated algebraic expression, GPT-4 gets it wrong, but when asked to break down the steps of the simplification, it gets it right. This is remarkably human-like. In fact, the response is indistinguishable from reasoning based on an understanding of algebra. I will not attempt to define “understanding,” but these behaviors might give us some clues about how understanding works in humans. Humans, after all, also learn to do math by reading about it or hearing about it from other humans.

Prediction underlies even the most primitive biological nervous system. The c. elegans worm, one of the most thoroughly mapped organisms on the planet, has 302 neurons with about 8000 synapses that control 96 muscles [6]. One of the mechanisms that has been identified in this relatively simple nervous system is an efference feedback where signals from the motor neurons (which drive muscles) are fed back into the sensory neurons so that the nervous system can distinguish sensations caused by its own actions from sensations caused by external events [38]. This feedback provides the worm with a primitive proprioception and builds into the wiring a sense of causation. Motor efferences cause sensory inputs that can be safely ignored. This phenomenon apparently occurs in all animal nervous systems, including humans.Footnote 3 When you wave your hand next to your face, your brain does not get alarmed by the motion detected by your peripheral vision system. Put another way, based on the efference feedback, the sensory system predicts sensations that result from the organism’s own actions. Prediction, therefore, is wired into even the most primitive animal nervous systems.

Although the neural networks of AIs do not closely resemble biological neural networks, Lechner et al. have shown that artificial neural network systems can be given architectures inspired by the well-studied neural structure of c. elegans to greatly reduce the size of the networks needed to accomplish certain tasks [17]. This at least implies that we can continue to learn from the biological mechanisms to improve the artificial ones. Despite the considerable distance between the mechanisms of these two classes of systems, the centrality of prediction in both is remarkable. What if prediction is the foundation of intelligence, and logic, math, and physics are all phenomena that evolved in humans because of their ability to enhance prediction? I am suggesting that rationality is an emergent phenomenon driven by prediction, and that logic and mathematics are invented by an evolutionary process rather than discovered fundamental truths. If this is the case, intelligence goes well beyond logic, mathematics, and algorithms and is capable of things that are inexplicable with the methods of the hard sciences.

6 Causation

Many readers will find my radical thesis here disturbing and will likely reject it. As an educator myself, I have spread the dogma that everything that happens has a cause that can fundamentally be explained scientifically, even if we do not yet know how to do that. The modeling we do in science is all about associating causes with results. The strongest forms of this assumption reduce to a deterministic model of the physical world [19]. We refuse to accept results without causes. Is this a faith or a truth? I am suggesting that we consider this idea to be a scientific paradigm that we are clinging to despite evidence to the contrary. What is that evidence?

I have previously shown that any set of deterministic models that is rich enough to include Newton’s laws and also allows discrete behaviors is incomplete [18]. This is shown by constructing a sequence of Newtonian models of elastic collisions (hence the discrete behaviors), where each model in the sequence is deterministic, the sequence is Cauchy in a metric space, and the limit of the sequence does not exist within the set of deterministic models. As a consequence, such a set of deterministic models has “holes” that exhibit nondeterministic behavior. In this context, such nondeterministic behaviors have no Newtonian cause.

If we accept these models as revealing some truth about the world, then an immediate corollary is that we have to admit the possibility of actions that have no cause. The “holes” can only be filled in with models where any single cause can lead to any of a multiplicity of behaviors. Although this result is only about models, not about reality, it suggests that our models of reality should, in fact, admit the possibility of uncaused action. Yet we resist this conclusion, even in the face of everyday empirical evidence. The reality is that we cannot identify causes for much of what happens, and yet we dogmatically assume that there is a cause.

The doubt about causation is confirmed by recent theories and experiments towards reconciling quantum mechanics and relativity [40]. The effect is called “indefinite causality.” In a system subject to both quantum mechanics and relativity, one can construct a scenario where an event A causes another event B, and, simultaneously, B causes A. Causality is not as simple as we thought. And yet, causality is required for certainty.

Judea Pearl has shown that it is impossible to reason about causation objectively [25, 26]. Here, “objective” reasoning means to let the data speak for itself. I.e., it means to draw conclusions about whether some phenomenon A causes B using observations alone of A and B. Instead, Pearl argues, causation can only be inferred based on prior assumptions about causation or by first-person interaction with (rather than just observation of) the system. Interaction is a feedback process, like efference copies, and (stable) feedback processes fundamentally involve prediction. A prediction is compared against observation to choose an action which then affects the observation. I have previously shown that such interaction can accomplish things that no computation (in the Turing-Church sense) can accomplish [22]. I am suggesting that the phenomenon of intelligence is rooted in such interaction and is therefore out of reach for algorithmic processes.

But wait! Today’s large-language models are realized on computers, so to the extent that they seem to be exhibiting intelligence, intelligence must be realizable by computation. When you use ChatGPT, you are truly interacting with the system only if your own behavior has persistent effects on the system. However, GPT stands for “generative pre-trained” models. The model parameters do not get updated based on the model’s interaction with clients. Perhaps this lack of interaction is the only remaining gap between ChatGPT and true intelligence.

Such interaction, however, has been tried before with disastrous consequences. In 2016, Microsoft unleashed a chatbot on Twitter that learned from its interactions with users. Vincent writes, “Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day” [36]. In other words, an AI that is allowed to learn from its interactions with humans can exhibit remarkably human-like intelligence.

There is extensive use of feedback in the training process for any modern AI. The back-propagation training strategy is fundamentally a prediction-based feedback process, as are reinforcement-learning approaches. OpenAI (wisely) breaks the feedback loop when putting ChatGPT online because the resulting behavior becomes impossible to control or predict. I suspect, however, that AI researchers will learn how to mitigate these risks and will reintroduce interaction. When the AIs have true interaction with their environment, I believe there is a chance they will become indistinguishable from intelligent beings.

7 Conclusion

We demand certainty from engineered, safety-critical systems. We also demand intelligence, that they should behave reasonably even in unanticipated scenarios. These two requirements may be fundamentally incompatible. My (admittedly radical) stance here is that certainty is achievable only at the expense of intelligence. It requires us to reject learning and to reject adaptability.

At a trivial level, Bayes’ law tells us that if we are certain about something, then no observation can change our mind. But the more radical thesis I am asking you to consider is that rational processes based on logic and math may be emergent from the phenomenon of intelligence rather than its foundation. Moreover, I am asking you to consider that algorithmic computation, with all its possibilities for certainty, is not usefully considered a foundation for intelligence either.

Abelson and Sussman, in the preface to their classic introduction to computer science [1], describe the computer revolution as a “procedural epistemology,” knowledge through procedure. I am suggesting here that we have entered a second computer revolution, where prediction trumps procedure. Knowledge through prediction is what today’s AIs exhibit. As their interaction with their environment increases, I predict that it will no longer be useful to think of them as computer programs in the classical sense. They will become interactive systems with behaviors that are persistently inexplicable in terms of procedural epistemology. And logic, mathematics, and computer programming will be things they do rather than what they are, just like humans.