The black box problem: why legal AI needs glass walls
Picture this: your AI just flagged a critical contradiction across fifty thousand pages of discovery. It could change the case. But when you ask how it knows, the system offers the legal equivalent of “trust me.”
That’s the black box problem, and it’s quietly limiting what legal AI is allowed to be.
Why lawyers are right to be skeptical
The skepticism attorneys feel toward unexplainable AI isn’t technophobia — it’s professional responsibility. Legal practice operates under constraints most industries don’t:
- Duty of competence. You must understand the work you present to courts and clients.
- Accountability. You remain personally responsible for every filed document, however it was drafted.
- Explainability. Judges and clients deserve to understand the reasoning behind a position.
- Auditability. Work must be reviewable for quality control and ethics compliance.
An AI output you can’t trace, explain, or verify fails all four tests — no matter how often it’s right. The well-publicized sanctions cases over hallucinated citations are just the visible edge of the problem; the deeper issue is any workflow where “the system said so” stands in for reasoning.
The three kinds of transparency (you need all three)
Decision transparency — what the system concluded and why. “This document is relevant” is a black box. “This document is relevant because the witness statement on page 47 conflicts with the email on page 1,293, sent five days earlier” is evidence.
Process transparency — how the analysis was performed. What was extracted, what was compared, in what order. If the method can’t be described, it can’t be defended in a meet-and-confer.
Limitation transparency — what the system didn’t do. Which pages couldn’t be read, which extractions are low-confidence, what remains unreviewed. Honest systems disclose their gaps; dangerous ones stay silent.
Five questions to ask any legal AI vendor
- “Show me how you reached that conclusion.” If the answer is a shrug — or a confidence percentage with no reasoning behind it — walk away.
- “Can I trace this back to source documents?” Every extracted fact and every generated sentence should carry a pointer to the page it came from.
- “What did you not analyze?” Ask to see how the system reports unreadable pages, low-confidence extractions, and coverage gaps.
- “What happens when the model is wrong?” The honest answer describes a verification workflow — not a disclaimer at the bottom of the screen.
- “How do I explain this to a judge?” If you can’t, you shouldn’t file work built on it.
Glass walls, by construction
This is the standard we build FactMarshal against. Every extracted fact is pinned to the exact source page. Facts whose text is verbatim from the record are machine-verified against the document at ingest. Everything else goes through a human review queue built to be cleared at reading speed — and the system always shows you which parts of the record are verified and which aren’t. When the assistant answers a question, each claim cites its way back down to the page.
Transparency isn’t a feature we added to AI. It’s the constraint we designed under — because in litigation, an answer you can’t show your work for isn’t an answer. It’s a liability with good grammar.