Are explanations essential to AI model outputs essential?
My first answer to that is: probably not.
When an evidence is a rhetorical exercise to impress me you had your reasons for a call, it’s just bells and whistles with no impact. If I’m waiting for a cancer diagnosis based on my MRI, I’m far more inquisitive about improving accuracy from 80% to 99% than in seeing a compelling image showing where the evidence lies. It might take a highly trained expert to acknowledge the evidence, or the evidence is likely to be too diffuse, spread across hundreds of thousands of pixels, for a human to understand. Chasing explanations just to be ok with trusting the AI is pointless. We must always measure correctness, and if the mathematics shows the outcomes are reliable, explanations are unnecessary.
But, sometimes an evidence are greater than a rhetorical exercise. Here’s when explanations matter:
- When accuracy is crucial, and the reason lets us bring down the error levels, e.g. from 1% to 0.01%.
- When the raw prediction isn’t really all you care about. The reason generates useful actions. For instance, saying “somewhere on this contract there’s an unfair clause”, isn’t useful as showing exactly where this unfair clause shows up, because we are able to take motion and propose an edit to the contract.
Let’s double click on a concrete example from DocuPanda, a service I’ve cofounded. In a nutshell, what we do is let users map complex documents right into a JSON payload that comprises a consistent, correct output
So perhaps we scan a whole rental lease, and emit a brief JSON: {“monthlyRentAmount”: 2000, “dogsAllowed” : true}.
To make it very concrete, here’s all 51 pages of my lease from my time in Berkeley, California.
Should you’re not from the US, you is likely to be shocked it takes 51 pages to spell out “You’re gonna pay $3700 a month, you get to live here in exchange”. I feel it won’t be essential legally, but I digress.
Now, using Docupanda, we are able to get to bottom line answers like — what’s the rental amount, and may I take my dog to live there, what’s the beginning date, etc.
Let’s take a take a look at the JSON we extract
Should you look all the way in which at the underside, we’ve a flag to point that pets are disallowed, together with an outline of the exception spelled out within the lease.
There are two reasons explainability can be awesome here:
- Possibly it’s crucial that we get this right. By reviewing the paragraph I can be certain that that we understand the policy appropriately.
- Possibly I need to propose an edit. Just knowing that somewhere in these 51 pages there’s a pet prohibition doesn’t really help — I’ll still should go over all pages to propose an edit.
So here’s how we solve for this. Fairly than simply providing you with a black box with a dollar amount, a real/false, etc — we’ve designed DocuPanda to ground its prediction in precise pixels. You possibly can click on a result, and scroll to the precise page and section that justifies our prediction.
At DocuPanda, we’ve observed three overall paradigms for a way explainability is used.
Explanations Drive Accuracy
The primary paradigm we predicted from the outset is that explainability can reduce errors and validate predictions. When you could have an invoice for $12,000, you actually need a human to make sure the number is valid and never taken out of context, since the stakes are too high if this figure feeds into accounting automation software.
The thing about document processing, though, is that we humans are exceptionally good at it. In actual fact, nearly 100% of document processing remains to be handled by humans today. As large language models turn into more capable and their adoption increases, that percentage will decrease — but we are able to still rely heavily on humans to correct AI predictions and profit from more powerful and focused reading.
Explanations drive high-knowledge employee productivity
This paradigm arose naturally from our user base, and we didn’t entirely anticipate it at first. Sometimes, greater than we would like the raw answer to an issue, we would like to leverage AI to get the proper information in front of our eyes.
For instance, consider a bio research company that desires to scour every biological publication to discover processes that increase sugar production in potatoes. They use DocuPanda to reply fields like:
{sugarProductionLowered: true, sugarProductionGenes: [“AP2a”,”TAGL1″]}
Their goal is not to blindly trust DocuPanda and count what number of papers mention a gene or something like that. The thing that makes this result useful is that researcher can click around to get right to the gist of the paper. By clicking on the gene names, a researcher can immediately jump in to context where the gene got mentioned — and reason about whether the paper is relevant. That is an example where the reason is more essential than the raw answer, and may boost the productivity of very high knowledge employees.
Explanations for liability purposes
There’s one more reason to make use of explanations and leverage them to place a human within the loop. Along with reducing error rates (often), they allow you to display that you could have a reasonable, legally compliant process in place.
Regulators care about process. A black box that emits mistakes is just not a sound process. The flexibility to trace every extracted data point back to the unique source enables you to put a human within the loop to review and approve results. Even when the human doesn’t reduce errors, having that person involved may be legally useful. It shifts the method from being blind automation, for which your organization is responsible, to at least one driven by humans, who’ve an appropriate rate of clerical errors. A related example is that it looks like regulators and public opinion tolerate a far lower rate of fatal automobile crashes, measured per-mile, when discussing a totally automated system, vs human driving-assistance tools. I personally find this to be morally unjustifiable, but I don’t make the principles, and we’ve to play by them.
By providing you with the power to place a human within the loop, you progress from a legally tricky minefield of full automation, with the legal exposure it entails, to the more familiar legal territory of a human analyst using a 10x speed and productivity tool (and making occasional mistakes like the remaining of us sinners).
all images are owned by the creator
