Somewhere in every hospital, a piece of paper is the last place anyone actually checked.

A nurse opens an AI-generated after-visit summary and signs off on it in the time it takes to skim a paragraph — call it nine seconds, the exact number isn't the point. They're trusting it's accurate. The hospital's compliance team trusts their sign-off. The insurer trusts the hospital's audit log. The regulator trusts the insurer's paperwork.

Nobody in that chain re-checked the original claim. But it raises an honest question worth sitting with: if no single person in the chain verified the thing everyone is relying on, where did the trust actually go? It didn't disappear. It just moved one step further from the decision.

I can say this from personal attestation, not just theory: there have been stretches of my life where I've gone back over a stack of medical bills trying to work out which specific professional, system, policy, or insurance contract was actually responsible for a charge — and never landed on a straight answer. The chain doesn't only diffuse clinical judgment. It diffuses financial accountability too, and that version of the problem is arguably the more expensive one.

In software, this is the kind of thing we'd draw as a decision flow — not what got decided, but how the decision moved from one place to the next, and where it picked up or lost scrutiny along the way. A hospital's verification chain is the human version of the same diagram.

This shows up anywhere an AI-generated output feeds into a decision that nobody manually re-verifies — which, as these systems spread, is a growing number of decisions in medicine, lending, hiring, and government benefits. It's worth naming what kind of scaling this actually is. Every step in that chain is individually rational: checking costs money and time, and not checking is free right up until the day it isn't. Multiply that logic across every hospital, insurer, and regulator, and you get bureaucracy at its most capitalist — not evil, not even inefficient by its own accounting, just optimized one link at a time in a way that leaves no one holding the whole chain. It's tempting to think better technology closes the gap: a second model checking the first, a cryptographic signature, a committee vote. Each of those genuinely helps — and each one still ends with something, somewhere, that nobody re-checked. The trust has to land on someone eventually.

That's quietly becoming a policy question as much as a technical one. The EU's AI Act, for one, now formally requires a named human "in the loop" for certain high-risk systems — not because a person is more accurate than the system, but because a person is a place accountability can legally land. Similar requirements are being debated elsewhere. Where that person sits, how much they're actually able to check in that kind of window, and who answers for it when they're wrong, is going to be argued about in courtrooms and legislatures for years. It's worth more public attention than it currently gets.

This is also where the debate about how fast AI itself should advance stops being abstract. Anthropic's Dario Amodei has argued, in an essay called "We Must Pace the Frontier," that capability should advance deliberately enough for safety and alignment work to keep up with it — not because progress should stop, but because what the technology is built to do matters as much as how fast it gets built. That argument reads differently once you're picturing it applied to one specific hospital chart instead of to AI in the abstract.

For what it's worth, here's where I actually land on this, stated plainly rather than hedged: I think American companies should be aggressively questioning their own engineers and pushing ideas forward — as much to find out where the real limits are as to stay ahead. And at the same time, I'm in full agreement that there's a greater-than-10% chance AI ends up killing us all. I don't think those two positions are in tension. Pushing hard and taking the risk seriously are the same project, not opposite ones — and a verification chain that quietly loses track of who's accountable, whether it's in a hospital or somewhere much bigger, is the small-scale version of the same problem.