The better the AI got, the closer I had to watch it

The debate about what AI tools can now do persuaded me to test the question directly. Could they replace or eliminate the human? I decided to find out by building something real rather than by reading about it. The result is a browser-native, privacy-first platform aimed at small-business finance and data teams, now running in production. Eight tool engines, on-device AI, subscription billing, team and role management, client-side encrypted backup. Around 48,000 lines of application code and 8,000 lines of tests. That is 287 test cases across some 300 commits.

I wrote none of it by hand. My background is advisory and domain work. I cannot write code.

That is the context rather than the finding. The finding surprised me, and it took several months of watching the same thing happen before I trusted it.

As the tool got better, I had to supervise the output more closely, not less.

That is the wrong way round. It is worth explaining why it happens, because it has practical consequences for anyone putting these tools into an organisation.

The mechanical work really did disappear

The first thing to say is that the productivity story is true.

Over the course of the build, things that were impossible at the start became routine. Capabilities absent in one release turned up as features in the next. Early on I did a great deal of mechanical work myself, because the tool could not. By the later stages it was running test suites, driving a browser to reproduce test flows that had previously been manual, working across many files at once, and holding far more of the system in view than it could at the beginning.

Each of those shifts genuinely removed work. None of it came back. If you measure only the mechanical labour, the productivity gain is real and large, and it kept growing. Judging by the trend so far, it is not controversial to predict that more of what is still manual today will be automated by these tools.

I should be clear that the net effect was strongly positive. The point is not that the gains were illusory. It is that the gain arrived in one column while a new cost appeared in another, and only the first column tends to get reported.

The errors got quieter

Here is the mechanism, as plainly as I can put it.

Early in the build, the errors were loud. Code did not compile. Pages did not render. A step in a workflow simply failed. Loud errors have a useful property: they announce themselves. You do not need to be vigilant to find them, and finding them costs nothing, because they arrive at your door.

As the tool improved, the loud errors mostly stopped. What replaced them were quiet ones. The code compiled. The page rendered. The workflow completed. The test suite passed. And the behaviour was wrong.

Quiet errors do not announce themselves. They have to be hunted, and you only hunt what you already suspect. So the improvement in capability did not reduce the amount of attention required. It changed the kind of attention required, from noticing to actively looking, and looking is far more expensive than noticing.

That is the first half of it. The second half is harder to see.

What the green dashboard concealed

Late in the build I found that a critical email header had never been sent. Not once, for months, across every message the platform produced. A single wrong key in a configuration object, which the underlying library silently ignored rather than rejecting.

At that point the platform had 287 automated test cases and the suite was green. It stayed green while this was broken, and it would have gone on doing so indefinitely. Not one test asserted anything about that header. Nobody had thought to doubt it, so it was never in the frame.

This is the part worth generalising. Automated monitoring encodes a hypothesis about what can go wrong. Anything outside that hypothesis is not merely unmonitored. It is concealed, because the absence of a signal is indistinguishable from the absence of a problem. A green dashboard is not evidence that the system is healthy. It is evidence that the system is healthy in the specific ways somebody previously thought to check.

The failure was found because I decided to send a real message to a real address and read the raw headers that came back, rather than trust the green tick. That is not a clever act. It is an ordinary act of looking at the actual thing rather than at the instrument reporting on it.

And the same property that made the tool productive made this failure hard to see. The output was coherent across five different parts of the system. It was internally consistent, plausible, and consistent with the documentation. Interconnected work fails in interconnected ways, and a mistake that is uniformly applied looks a great deal like a convention.

None of this is new, which is the reassuring part

I thought I had found something peculiar to AI. I had not. I had rediscovered a result from 1983.

Lisanne Bainbridge's paper "Ironies of Automation" made the argument for process control and aviation. Automating a system does not simply remove human work. It removes the easy work and leaves the hard work. At the same time it degrades the operator's ability to do that hard work, because someone who is no longer hands-on no longer holds an accurate picture of the system. The better the automation, the rarer and stranger the situations the human is eventually called on to handle, and the less practised they are when it happens.

Charles Perrow supplies the other half in Normal Accidents. Systems that are interactively complex and tightly coupled fail in ways that are hard to foresee and hard to detect. The interactions that produce the failure were never designed, and are not visible in any single component. Perrow extended the argument to financial markets in a 1999 afterword. Tim Harford later made the same case about banking, on the grounds that anything melting down as often as finance is almost certainly described by the theory.

I find this genuinely reassuring rather than alarming. It means the problem is not novel, not specific to AI, and not a reason for caution about adoption. It is a known class of problem with forty years of accumulated practice attached to it. The mistake would be to treat AI-assisted work as unprecedented and therefore improvise the safeguards from scratch.

What actually helps

The response is not to slow down or to adopt less. It is to accept that the verification burden does not fall in proportion to the production burden, and to build practice around that.

Four things worked for me.

Verify at the boundary, not inside the frame. Test the system where it meets the world. Read the raw output. Send the real message. Look at the actual record in the actual database. Internal checks confirm the system is behaving as somebody imagined it should, which is a different claim from the system being right.

Assume your monitoring encodes yesterday's failures. A passing test suite tells you the known failure modes are absent. Periodically go looking outside it, on the assumption that the most dangerous problem is the one nobody thought to write a test for. The quiet failures in this project surfaced that way rather than through the suite.

Write things down far beyond what feels necessary. Because you are no longer building by hand, your mental model of the system decays, and it decays without your noticing. I kept a single running document recording what exists, what was decided, and why, including the reasoning behind choices that look arbitrary later. It is the only reliable way back into a system you did not type. It is also what stops a future decision quietly undoing an earlier one whose purpose has been forgotten.

Treat "it looks finished" as the moment to be suspicious, not the moment to relax. The most dangerous output is the one that is fluent, coherent and arrives when you are ready to be finished. That is precisely when checking feels least necessary and is most valuable.

None of these are technical controls. They are practice.

What this case does not show

The limits deserve the same plainness as everything else.

Lines of code are a crude proxy. They measure volume rather than value, and any engineer will say so. The figure I find more telling is the churn. Those 48,000 surviving lines took about 85,000 to produce, roughly 1.8 written for every one that shipped, with around 18,000 discarded. Several core files were revised between twenty and forty-four times. That is not one-shot generation. It is somebody returning to the same problem, judging the last answer inadequate, and going again.

I have kept elapsed time out of the claim deliberately. The work spanned several months broken by long stretches of nothing, and I did not track hours. Elapsed time would flatter me and tell you very little.

There is obvious survivorship bias. I am writing because it worked, and the people who tried this and abandoned it do not write articles.

And on the specialists. Work that would normally be spread across a full-stack engineer, a front-end specialist, an infrastructure engineer, a security engineer, a test engineer, an AI engineer, a technical writer and a designer was held together here by one non-technical operator and an AI model. But much of what those specialists provide is not visible at launch. It is what does not break in year two. A passing test suite on a young system is a snapshot rather than proof of durability. The specialisms were rented for the build. Whether they were genuinely replaced is a question only time answers.

The part that holds

Strip out everything uncertain and one thing survives.

The tool's capability grew steadily throughout. So did the attention it demanded. Those two facts sit together comfortably once you see that the work did not disappear. It moved. It moved from producing things to checking them, and then from checking that they run to checking that they are right. That last kind of checking is slower, less satisfying and much harder to schedule.

So I have an answer to the question I started with. The tools did not replace the human. They changed what the human is for. I built something with them that I could not otherwise have built, and I would do it again. What they did not do was remove the need for someone to go and look at the real thing, particularly at the moment when everything on the screen says there is no need to look.