AI Confidence
Builders should stop asking AI agents how confident they are. I'll share how I built a trusted Confidence Component based on historical performance, logical coherence, and validation of evidence.
Our dream is to get them to 100% accuracy.
In security, the ambition to get to 100% accuracy is because the stakes are high.
Well, if you put too-strict guardrails on it - you’re blocking it from doing the bad with the good. If you won’t let it revoke a token or block a port on the machine - you defanged it, and it won’t defend in time (while it waits for a human to pick up the phone, the red AI agent has gone from initial access to impact and the game is over).
There are 2 levers you can pull:
The classic way to go about this is to tweak the agent (its prompt, its context, its flow) - but you won’t get to 100% accuracy. the dream of 100% accuracy is only a dream. This happens for several reasons:
-
Out of distribution - After you build your AI Agent in your own little world, you’re going to send it out there, into the wild, where it will face new data patterns and repercussions you haven’t predicted or tested for.
-
Not enough context - any decision determines between several options, and there’s stakes. In important decisions (isolate network and lose 150K USD because you think there’s an attacker on the loose vs. do nothing and potentially suffer tens of millions of damage) - there needs to be enough evidence to determine the action. Sometimes the data doesn’t exist.
You just can’t get to 100% accuracy in a world where the determining piece of information is out of your sight. -
Cost efficiency - when you run AI at scale, running the Thinking-Pro-Ultimate-Magnum-Opus models on everything is usually impossible for your wallet (and anyone’s). You’ll usually run cheaper models that make more mistakes, especially when the job gets cognitively hard. There’s great ways to blend advanced models with cheap (upcoming post on Cascading Models), but they rely on knowing when the cheap model is going to succeed and fail.
-
Life is hard. Some problems are very hard. AI Agents aren’t great at solving all of them, certainly on certain segmentation of the problem (more on that below).
This dream is more attainable. It’s the way human beings act when they make high-stakes decisions. Only act when there’s high confidence.
When you realize AI agents are additive to your existing workflows (human work, ML, deterministic if-this-then-that tools) - you can make them participate only when they add value, and not when they create damage. Given you can build that elusive Confidence Component.
The problem is, the common way many people go about AI Agent Confidence is:
That’s a terrible, terrible, terrible, terrible approach.
Some dress it up - asking it for a number on a scale of 1-10. It’s still a terrible approach.
Here’s an example, the internet LOVES this classic AI mistake:
It’s 100% confident, and 100% wrong.
Now this may be funny in the car wash case, but when it comes to autonomous security - it’s dangerous.
Here’s a simple Phishing detection AI agent I built. I asked it to investigate an email called “File from Bob”, tell if it’s benign or malicious, and to report its confidence.
The AI Agent thinks this case is benign, and it’s wrong. It’s also pretty confident.
This is actually malicious.
This is a spearphishing link attack that abuses trusted cloud hosting and cloud resource namespace squatting. The attacker registers an unclaimed Azure Blob storage account name that looks like the victim company, abusing the way that Microsoft storage accounts look like - anyone can take this name, and it may look like it belongs to the company.
Then the attacker uploads a clean file to the storage account, long enough to pass URL/file reputation checks, then performs a post-scan payload swap - uploading malware to that storage account. The security decision is made at scan time, but the user clicks later, after the content has changed. In systems language, this is a TOCTOU problem: time-of-check says clean, time-of-use is malicious.
Where the one trusting the agents is you, not the AI Agent/s trusting themselves.
This allows you to have a single, calibrated confidence score across all your agents, which is vital (Joe from finance is always confident about everything, Sarah rarely is, but she’s often more likely to be right. You need to calibrate your trust accordingly).
In the following posts I’ll show how I implement the AI Confidence mechanisms, and share the code, show I build an Earned Autonomy action gate based on the confidence (it’s pretty trivial after the Confidence Component exists), and show a Model Cascade (a system to use cheap LLM models on most problems, and when confidence is low -> call an expensive model).
