Why some predictions are more reliable than others

A prediction that a coin lands heads about half the time is extremely reliable. A prediction about how a once-in-a-generation event resolves is barely a prediction at all. Same tool, wildly different trustworthiness — and the difference isn't random. There are specific, knowable reasons some predictions deserve your confidence and others deserve a raised eyebrow. Knowing them is how you avoid the two failure modes: trusting a shaky prediction, and ignoring a solid one.

This post is about prediction reliability itself — what drives it, and how to read a prediction's own honesty about its limits.

Reliability tracks the density of analogues

The biggest single driver of a prediction's reliability is how many relevant analogues exist for the situation. AI predicts well when it has effectively seen many similar situations play out — repeated, well-documented patterns. It predicts badly when the situation is genuinely novel, because there's little to reason from and it ends up extrapolating into the dark.

This is why a prediction about a routine business move (a price change on an existing base) is far more reliable than one about a singular geopolitical shock. Not because the tool tries harder on one; because one has a thick base of analogues and the other has almost none. When you read a prediction, ask: how common is this situation? Common situations get trustworthy predictions.

Reliability tracks the quality of your inputs

The second driver is you. A prediction is built on the scenario you describe plus the assumptions the AI fills in for the gaps. The more of the scenario you specify with real, specific detail, the less the AI has to invent, and the less invented material there is to be wrong about.

A scenario described in one vague sentence produces a prediction that's mostly the AI's guesses about your situation — and those guesses, not the world, become the main source of error. A scenario described with the actual numbers and context produces a prediction grounded in your reality. Garbage in isn't quite garbage out, but vague in is definitely vague out. The craft of good inputs is its own topic: how to write a scenario question.

Reliability tracks how reflexive the system is

A third, subtler driver: how much the thing being predicted reacts to being predicted (or to the actions of many other predictors). Physical and procedural systems are stable — they don't change their behavior because you modeled them. Markets, by contrast, are reflexive: everyone is predicting and acting on predictions, which moves the very thing being predicted. That reflexivity is why crypto and market scenarios are inherently less reliable than, say, predicting how a hiring decision plays out — the market fights back, the new hire mostly doesn't.

How to read a prediction's own confidence

A well-built prediction tells you how much to trust it, and you should take that signal seriously. Watch for three things:

• Does it hedge appropriately? A prediction that's all confident specifics on a novel, reflexive situation is overreaching. One that explicitly widens its range when data is thin is being honest.
• Are the branch probabilities spread or concentrated? Tightly concentrated probabilities signal a situation the model finds predictable. A near-even spread across branches is the model telling you it genuinely doesn't know — which is information, not failure.
• Does it name a load-bearing assumption? The best predictions end by saying "this whole thing hinges on X." If X is something you can verify, the prediction just told you exactly how to make itself more reliable.

The practical rule

Trust a prediction in proportion to: how common the situation is, how specific your inputs were, and how non-reflexive the system is — and read the prediction's own hedging as signal, not noise. A prediction that admits low confidence on a novel, data-poor, reflexive scenario isn't a weak tool; it's an honest one, and honesty is the foundation of reliability.

This reliability lens applies to everything else on the site, from predicting policy changes to predicting your financial future. And once you understand reliability, the highest-leverage way to improve it is on your end — which is exactly how to write a scenario question to get a useful prediction.