🎲

On Probabilism and Determinism in AI

Words
Words
🧠Summary🎧Podcast
image

Introduction

An AI development framework characterized by shifts between deterministic and probabilistic approaches in computing. It outlines four stages:

  • Where it started: Deterministic → Deterministic → Deterministic (e.g. spreadsheets, databases; companies like Microsoft, Oracle)
  • Where we were: Deterministic → Probabilistic → Deterministic (e.g. targeted ads; companies like Google, Meta)
  • Where we are now: Probabilistic → Probabilistic → Probabilistic (e.g. Generative AI; companies like OpenAI, Anthropic)
  • Where we want to get to: Probabilistic → Probabilistic → Deterministic (a future ideal state)

We will analyze the historical accuracy of this framework, its logical coherence, and the future implications of reaching the final stage. We’ll also identify companies and technologies working toward that future (Probabilistic → Probabilistic → Deterministic).

Evolution of AI Paradigms

Early Era: Fully Deterministic Systems (Deterministic → Deterministic → Deterministic)

In the earliest stage of modern computing, everything was deterministic. Both the input data, the programmed algorithm, and the output were fixed and predictable. In practical terms, this era includes tools like spreadsheets and relational databases. For example, a spreadsheet (like Microsoft Excel) will always produce the same result given the same set of inputs and formulas – if you input X and apply a set formula, you deterministically get Y every time. Classic databases (Oracle, IBM DB2, etc.) similarly execute queries with deterministic algorithms (e.g. SQL joins, filters) to return precise, unchanging results for a given query. As one source explains, traditionally “computers have been deterministic machines, providing consistent outputs for the same input.” (Did we accidentally make computers more human? — Algorithma) In other words, early software was engineered such that if you ran the same operation twice with identical inputs, you’d get identical outputs. This property was (and still is) essential for applications like financial calculations or transaction processing.

Historical accuracy: Describing the start of computing as deterministic is accurate. From the mid-20th century through the 1980s-1990s, most software followed explicit rules and logic. Even early “AI” in that era (expert systems) were rule-based – essentially a series of hardcoded deterministic if-then statements. Companies like Microsoft (with Excel) and Oracle (with database systems) dominated this deterministic computing era. These systems had no randomness involved: their behavior was fully determined by their code and input, reflecting the Deterministic → Deterministic → Deterministic model.

Big Data and ML Era: Probabilistic Algorithms with Deterministic Outputs (Deterministic → Probabilistic → Deterministic)

As data grew and problems became more complex in the 2000s-2010s, we entered an era where probabilistic algorithms (machine learning models) were introduced into otherwise deterministic systems. In this stage, the inputs were often well-defined (e.g. a user’s profile or a search query – which we can consider “deterministic” in the sense that the input is a fixed piece of data), but the processing was done by probabilistic models. However, the system would still produce a single decided output – giving the appearance of determinism in the final result.

A prime example is online advertising and content recommendation. Companies like Google and Meta (Facebook) led this shift. Google’s search and ad products traditionally took a user’s query (deterministic input) and then, starting in the mid-2000s, used machine-learned models to rank results or choose the best advertisement. These models compute things like the probability of a user clicking each ad or finding a result relevant. The highest-scoring result is then shown to the user as the answer or ad – a single, discrete output. The user doesn’t see probabilities; they just see one ad or one set of search results. In other words, internally the system is probabilistic, but externally it delivers a fixed outcome. This fits the Deterministic → Probabilistic → Deterministic pattern (deterministic input and one final choice, with probabilities in between).

For instance, Facebook’s news feed and ads are heavily driven by ML algorithms: “Meta is already highly reliant on machine learning for its ads product… Meta will work across their vast troves of data in a way that is only possible using machine learning-derived algorithms to find the right targets for an ad and deliver exactly the business goals requested.” (Meta’s AI Abundance – Stratechery by Ben Thompson) Here, the advertiser’s desired outcome and user data are the given inputs; a probabilistic model predicts which user is likely to convert; then the platform deterministically shows an ad to that user. Similarly, Google Search introduced the RankBrain algorithm in 2015, which is a “machine learning-based search engine algorithm” used to help rank search results (RankBrain - Wikipedia). RankBrain would interpret a query and probabilistically adjust rankings, but the search engine still returns a fixed ordered list of results – again deterministic output to the user.

Historical accuracy: Characterizing the 2000s-2010s as adding a probabilistic middle to otherwise deterministic systems is largely accurate. Advertising technology is a clear example – over time it moved from deterministic rules (show ads based on fixed keywords or demographics) to probabilistic predictions (show ads based on likelihood of engagement). By the late 2010s, virtually all major internet platforms were using machine learning models under the hood. This was not limited to ads: recommendation engines (Netflix, Amazon, YouTube) and spam filters, for example, all use probabilistic models but give the user a single result or action. Even where user data was “deterministic” like a known user ID or query, the decision process became statistical. A Meta executive noted a “shift in digital advertising to probabilistic models in place of deterministic ones” in recent years (Meta’s AI Abundance – Stratechery by Ben Thompson). That shift was driven by both the need for better prediction accuracy and, more recently, privacy changes that forced companies to infer missing data probabilistically. Overall, the framework’s second stage aligns with the industry’s trajectory of that period: deterministic inputs, ML-based processing, deterministic outputs delivered.

Current Era: Generative AI and End-to-End Probabilistic Systems (Probabilistic → Probabilistic → Probabilistic)

Today, with the rise of Generative AI (large language models, diffusion models for images, etc.), we have probabilistic behavior at every stage. The input might be an open-ended prompt from a user – for example, “Write a story about a dragon.” This input is not a structured, unambiguous piece of data; it’s a probabilistic or wide-ranging input in the sense that it could be interpreted in many ways (the AI has to decide which direction to take the story, what style, etc. based on learned probabilities). The model (like OpenAI’s GPT-4 or Anthropic’s Claude) then probabilistically generates text by selecting likely next words according to its training. Finally, the output itself is probabilistic: it’s not a single fixed answer but one of many possible valid outputs. If you ask the same question multiple times, a generative AI might give different phrasing or even entirely different answers, especially if any randomness (“temperature”) is involved in generation. In short, the process can produce non-deterministic outputs – the hallmark of this era.

For example, large language models (LLMs) “can produce slightly different responses for the same prompt due to the use of randomness and probabilities during generation.” (Did we accidentally make computers more human? — Algorithma) If you prompt GPT-4 or ChatGPT twice with exactly the same request, you may get variations in the reply. This stands in contrast to traditional software which would give the exact same output every time. An AI commentary notes that this “controlled fuzziness” mimics human-like variability and creativity (Did we accidentally make computers more human? — Algorithma). It’s a feature, not a bug, for many creative applications – it allows AIs to draft essays, create art, or have conversations that aren’t canned.

However, having probabilistic behavior at all stages also introduces challenges. The output might be plausible yet incorrect – what we call AI “hallucinations” or errors. Stephen Wolfram illustrated this by asking ChatGPT to compute a factual question (“How far is it from Chicago to Tokyo?”). ChatGPT produced a convincing-sounding answer, but “it turns out that it’s wrong,” whereas a deterministic computation via Wolfram|Alpha got the correct distance (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings) (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings). In Wolfram’s words, ChatGPT’s responses are always “statistically plausible” but not guaranteed true (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings). The model will confidently “trot out” facts or figures that aren’t necessarily correct (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings) because it’s drawing on patterns, not a grounded database. This encapsulates the Probabilistic→Probabilistic→Probabilistic condition: the query wasn’t a fixed database lookup, the reasoning wasn’t a fixed algorithm, and the answer wasn’t certain or unique.

Generative AI companies like OpenAI and Anthropic exemplify this stage. Their systems are trained on vast, probabilistic patterns of data (e.g. predicting the next word in text). The upside is they can handle tasks we don’t know how to explicitly program (writing an article, coding from a spec, conversing naturally). The downside is the lack of guaranteed correctness or consistency. Another analysis puts it succinctly: “LLMs challenge [the deterministic computing] notion by introducing probabilistic and variable outputs.” (Did we accidentally make computers more human? — Algorithma) We’ve essentially made computers less predictable in exchange for greater capability – a shift that indeed makes them seem “more human” in their flexibility.

Historical accuracy: Describing the current state of AI as Probabilistic → Probabilistic → Probabilistic is valid. Around 2016–2023, deep learning models grew in ability, and by late 2022 the public saw ChatGPT’s emergence, highlighting the paradigm of fully probabilistic AI. It’s worth noting that not all current software is like this – for instance, a banking system still runs on deterministic code. But the frontiers of AI – from image generators like DALL-E/Stable Diffusion to chatbots like ChatGPT/Bard – do fit this triple-probabilistic model. The framework’s characterization of “where we are now” aligns with how generative AI operates. Major players (OpenAI, Anthropic, Google with Bard, etc.) are indeed using probabilistic neural networks to handle ambiguous inputs and produce open-ended outputs.

Logical coherence of progression so far: The evolution from the earlier stages to now is logical. As tasks became too complex to hand-code (e.g. recognizing images, translating languages, anticipating user preferences), we introduced statistical methods (stage 2). As we pushed further to generating content and handling vague goals, we allowed even the outputs to vary and not be strictly predetermined (stage 3). Each stage increased AI’s flexibility and scope, but also its unpredictability.

Future Outlook: Toward Probabilistic Inputs/Algorithms with Deterministic Outcomes

The proposed final stage is Probabilistic → Probabilistic → Deterministic. This implies an AI system that can accept the messy, probabilistic inputs from the real world, process them with advanced (likely probabilistic or learned) techniques – but still guarantee a reliable, deterministic output. In essence, it’s the best of both worlds: we want AIs that can understand and reason about uncertain, complex situations (like humans can, using probability and intuition), yet deliver results that are as dependable and verifiable as traditional software.

Why do we “want to get to” this stage? One big reason is to combat the flaws of current generative AI, such as hallucinations and inconsistency. If an AI’s final output can be made deterministic (correct and the same every time for the same query), we can trust it in high-stakes applications. An industry report looking toward 2030 notes that “the risks of hallucinating fictional summaries will be better managed as algorithms evolve from probabilistic to deterministic outcomes.” (AI 2030: Provocation Makes the Difference) In other words, to avoid AI making things up, the solution is to incorporate more determinism in how it produces answers. This doesn’t necessarily mean going back to rigid rules, but rather constraining or validating the output of probabilistic models so that the end result is grounded in facts or consistent logic.

Is such a framework logically coherent? Yes – it basically means adding a layer of control or verification at the end of a probabilistic pipeline. The system can still handle uncertainty internally (and we want it to, because that’s how it deals with complex tasks), but before presenting the answer, it would check or enforce correctness. This idea is already influencing AI research and development:

  • Neuro-symbolic AI (Hybrid systems): These combine neural networks (probabilistic learning) with symbolic reasoning or knowledge bases (deterministic logic). The goal is to let the neural part handle perception or language, then use a symbolic part to ensure consistency and apply rules. Proponents say “by integrating neural learning’s adaptability with symbolic AI’s structured reasoning, we are moving towards AI that can understand the world and explain its understanding in a way humans can comprehend and trust.” (What is Neuro-Symbolic AI? - AllegroGraph) Such systems could take an open-ended question, use an LLM to interpret it, but then query a deterministic database or reason over a knowledge graph to get a definitive answer – yielding a deterministic outcome backed by logic. This field is being explored by companies like IBM (which has researched neuro-symbolic AI for image and language understanding) and startups focusing on knowledge graphs (e.g. Franz Inc. with AllegroGraph (What is Neuro-Symbolic AI? - AllegroGraph) (What is Neuro-Symbolic AI? - AllegroGraph)).
  • Tool use and APIs: Another approach is having AI models call external tools (which are deterministic) to finalize answers. A simple example is the integration of Wolfram|Alpha (a computational knowledge engine) with GPT. ChatGPT might not reliably multiply two 10-digit numbers (it could guess or make an arithmetic mistake), but if it can call Wolfram|Alpha, the calculation result will be exact every time. OpenAI has moved in this direction by introducing function calling in their API, which allows developers to have the model return structured data or trigger code execution. This provides “a more deterministic and structured” interface for outputs (Function Calling: Integrate Your GPT Chatbot With Anything - Semaphore). In practice, that means an AI could turn a free-form request into a JSON result via a function – ensuring the output format and content follow certain rules (for example, calling a weather API to get the exact temperature rather than guessing). Wolfram itself demonstrated that hooking ChatGPT to Wolfram|Alpha corrects its mistakes: the LLM’s natural language is used to formulate a query, Wolfram|Alpha computes it, and the answer fed back causes ChatGPT to give the right result (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings) (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings). This synergy shows a probabilistic model (ChatGPT) yielding a deterministic outcome (a verified factual answer) when paired with the right tool.
  • Search and Knowledge Grounding: Companies are also grounding AI outputs in deterministic knowledge sources. Microsoft’s Bing Chat (which uses GPT-4) is coupled with the Bing search index via a system called Prometheus. This “combines the fresh and comprehensive Bing index and search results with the reasoning capabilities of OpenAI’s GPT models.” (Microsoft explains how Bing AI Chat uses ChatGPT and Search with Prometheus) The result is that Bing Chat responses come with citations and are based on real-time information, which Microsoft says it “can only do because of” this integration of a deterministic search component (Microsoft explains how Bing AI Chat uses ChatGPT and Search with Prometheus). By ensuring the final answer is supported by retrieved documents, the output is made more verifiable and consistent (deterministic in the sense of being tied to concrete sources). Google’s Bard is following a similar path, allowing the LLM to tap into Google Search for up-to-date facts. And many enterprise AI solutions now insist on a retrieval step (from a company’s knowledge base) before answering, to avoid off-the-cuff fabrications.

Who Is Working Toward This “Final Stage”?

Several companies and research efforts are effectively aiming for Probabilistic→Probabilistic→Deterministic systems, even if they don’t label it that way. Here are a few examples:

  • OpenAI and Microsoft: OpenAI’s latest models, when used via Microsoft’s Azure OpenAI Service, can integrate with external data and APIs. Microsoft has built Copilot products (for coding, Office apps, etc.) that use GPT-4 but constrain it with business data, retrieval of documents, and tools like calculators. These ensure that while the interface is natural language, the outcome (e.g. code generated or answer given) is checked against deterministic compilers or sources. The Bing Chat example above is one case where Microsoft explicitly combines probabilistic AI with a deterministic backbone to get grounded answers (Microsoft explains how Bing AI Chat uses ChatGPT and Search with Prometheus) (Microsoft explains how Bing AI Chat uses ChatGPT and Search with Prometheus).
  • Wolfram Research: As discussed, they provided a plugin for ChatGPT that brings the full deterministic power of Wolfram|Alpha into the loop. Stephen Wolfram described this as combining “the kind of thing Wolfram|Alpha is good at – precise computation on structured knowledge – with ChatGPT’s human-like flow of language”, resulting in answers that are both fluent and correct (Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT—Stephen Wolfram Writings). This is a clear move toward deterministic outputs from probabilistic prompts.
  • IBM and Academic Partners: IBM has been investing in neuro-symbolic AI, which is explicitly about merging statistical AI with symbolic deterministic reasoning. For instance, IBM Research (often in partnership with MIT) has developed systems that use neural networks to parse visual data or text, and then logic engines to reason about them (achieving high performance on tasks like answering questions that require understanding a scene or a story logically). While specific product names are not always public, IBM’s emphasis on trustworthy AI and products like Watsonx (a platform for building AI with governance) indicate a push for reliable, controllable AI outputs – aligning with this deterministic goal.
  • Anthropic and Others (AI Safety): Anthropic’s Claude is another LLM that, like OpenAI’s, can be steered to use tools. Their focus has been on a “Constitutional AI” approach (to keep the AI’s behavior safe), but ensuring factual accuracy is another domain. Startups in the AI safety and alignment space are building “AI guardrails” – essentially validators that check an LLM’s output against knowledge bases or constraints, and either fix or refuse outputs that don’t meet a deterministic truth criterion. While not yet perfect, this is another path to enforce determinism at the output stage (e.g. an AI assistant that will only answer a medical question by looking up an approved medical database and quoting it exactly). An example is research on “Deterministic Quoting” which forces an AI model to only quote verbatim from sources when giving answers in domains like healthcare (Deterministic Quoting: Making LLMs Safer for Healthcare | Matt Yeung).

Future implications: If we achieve Probabilistic → Probabilistic → Deterministic AI, it could revolutionize trust and adoption of AI in critical fields. Imagine an AI lawyer that can take a fuzzy description of a case (probabilistic input), use a learned model to interpret and reason about it (probabilistic processing), but then output a deterministically correct legal brief that cites all the right laws and never fabricates a case. Or consider healthcare: an AI could converse with a patient, analyze symptoms (probabilistic understanding), but its final diagnosis and treatment plan are validated against medical knowledge and proven guidelines (deterministic output), reducing error. Essentially, AI would handle ambiguity and creativity, but with a safety net that ensures correctness. This could unlock using AI for autonomous driving, financial advice, scientific research, etc., where today we are cautious because of unpredictability.

That said, reaching this stage is challenging. It requires advances in AI architecture (or clever combinations of systems) to marry the flexibility of neural networks with the rigor of symbolic logic or reliable data retrieval. It’s an active area of research and development. There is also an open question of whether complete determinism is achievable – some tasks may always have some uncertainty. But even in those cases, the aim is to maximize reliability (for example, an AI might give a distribution of outcomes but within known confidence bounds, rather than arbitrary guesses).

Conclusion

The proposed framework of AI development – from fully deterministic beginnings, through a hybrid probabilistic phase, into today’s fully probabilistic generative models, and onward to hopefully more deterministic outputs – holds up under analysis. Historically, it’s broadly accurate: early computing was indeed deterministic, the machine learning era injected probabilistic decision-making into products, and current generative AIs are probabilistic at heart. The framework’s timeline simplifies a complex history (not every system fits neatly, and the transitions were gradual), but as a high-level view it is valid. Logically, the progression makes sense: each stage addressed the limitations of the previous (more adaptability and intelligence) but introduced new issues (loss of predictability), setting the stage for the next evolution. The ultimate goal of Probabilistic → Probabilistic → Deterministic is essentially about keeping the intelligence and flexibility of modern AI while re-imposing the reliability of earlier software. Many companies and researchers are actively striving toward this goal, as seen in the trend of combining LLMs with tools, knowledge bases, and formal reasoning.

If successful, this convergence could yield AI systems that are both powerful and trustworthy – capable of human-like understanding and creativity, yet delivering answers and actions we can count on with the certainty we expect from a computer. As one analysis looking toward the future put it, algorithms will evolve from probabilistic to deterministic outcomes to manage the risks of AI’s creative freedom (AI 2030: Provocation Makes the Difference). Achieving that will likely define the next era of AI development, unlocking new applications and increasing societal trust in AI.

References:

This was created using ChatGPT Deep Research with the following prompt and Notebook LM for the podcast.

On Probabilism and Determinism in AI - PromptOn Probabilism and Determinism in AI - Prompt