A large language model (LLM) and when to replace it with a decision tree

Behind today’s AI chatbots stands a large language model, LLM for short. It is a powerful tool, but not a cure for everything. Let us explain what an LLM is and how it works, and then show when it is wiser to replace it with a simple, predictable decision tree.

What a large language model (LLM) is

A large language model is a type of artificial intelligence trained on a huge amount of text that can generate human-sounding language. It stands behind tools like ChatGPT, Claude or Gemini. It belongs to machine learning, more precisely to its deep learning branch with neural networks.

How an LLM works

The principle is surprisingly simple. From billions of sentences the model learned the patterns of language. When you write it a question, it predicts the most likely next word (more precisely a piece of a word, a so called token) and thus builds the answer word by word.

It is important to understand that the model does not think or understand in the human sense and does not even “know” facts. It works with the statistics of patterns and generates text that sounds plausible. That is why it sometimes confidently makes things up, which is called a hallucination.

What an LLM can do and what traits it has

An LLM is excellent at working with language: writing and editing text, summaries, translations, explaining, ideas and code. But it also has traits you need to know:

• It is non-deterministic. It can give a slightly different answer to the same question each time. That is great for creativity, but bad where you want certainty.
• It hallucinates. It can make things up, so important outputs need to be verified.
• It is demanding on performance and money. It usually runs in the cloud and every call costs something.
• It is not always explainable. It is hard to find out why it answered exactly that way.

There is more on using it properly in the article how to use AI chatbots.

What a deterministic decision tree is

A decision tree is a sequence of clear questions and conditions that branch to a result. Picture a flowchart or an “if this, then that” scheme.

When the tree is deterministic, it means that the same input always gives the same output. It is plain, pre-written logic (rules), not a statistical prediction.

For example: “Is the order over 100 euros? If yes, free shipping. If not, shipping is 4 euros.” No guessing, no hallucination, always the same and verifiable result.

(Note: in machine learning there is also a decision tree trained from data. Here we mean a hand-designed tree of rules, that is pure deterministic logic.)

LLM versus a decision tree

When to replace AI with a decision tree

This is the main message: not every problem needs AI. Many tasks people reach for an LLM for are actually clearly bounded and a simple deterministic tree solves them, being cheaper, faster and more reliable. A tree is the better choice when:

• The problem has clear and finite rules, for example calculating prices and discounts, checking conditions (eligibility, meeting criteria), validating a form, routing a query to the right department, a step-by-step diagnostic procedure.
• You need predictability and explainability, for example in finance, law and compliance, where the decision must be auditable.
• You cannot afford a hallucination or a random answer.
• You want zero cost, speed, privacy and offline operation.

In practice, many “chatbots” for frequent questions can be replaced by a plain tree with fixed answers, and it works more reliably.

When you do need an LLM

A tree has its limits. An LLM is irreplaceable where the rules cannot be enumerated:

• Understanding free text and ambiguous inputs.
• Creating and editing content, summaries, translations.
• An enormous variety of cases that you would never write into a tree.

A hybrid approach: the best of both

Often it is ideal to combine both. You use the LLM to understand and extract the essence from the text, and the deterministic tree then makes the actual decision. That way you get the flexibility of language and the certainty and explainability of rules. There is an article on agents that combine models with tools, what an AI agent is.

Conclusion

A large language model is an AI that composes text by predicting the next word. It is great at language, but it is non-deterministic, hallucinates, costs money and is hard to explain. A deterministic decision tree is, in contrast, simple rule logic that is predictable, cheap and auditable. Before reaching for AI, it pays to ask: can this be solved by a plain tree? Often yes. And when you need both, their combination works best. As with any technology, AI is a good servant but a bad master, more in the article AI, a good servant and a bad master.

Want to deploy automation or AI in your company so that it makes sense and does not miss the mark? Get in touch, we will design a tailored solution.

This article is part of our Artificial intelligence overview.