The Quiet Convergence

A sentence gets completed before we finish writing it. A model summarizes something we could have read ourselves. An agent writes the code, runs the tests, and fixes its own mistakes while we supervise from above. Each individual use seems harmless, and often genuinely useful. The concern is what happens when those conveniences accumulate to the point that consulting AI becomes the default way we think through difficult decisions.

Software engineering is already beginning to show what this might look like. AI tools have moved well beyond autocomplete and isolated code snippets. Developers can now give an agent a task, describe the constraints, and allow it to inspect a codebase, write an implementation, execute shell commands, run tests, and iterate on the result. The engineer increasingly becomes a director of the work rather than the person performing every step of it. I use these tools myself, and I find them incredibly useful. That is part of what makes me uneasy about them.

Ironically, while I was first writing about the gradual integration of AI into everyday life, the autocomplete feature in my editor began suggesting the next sentence of the argument for me. I was writing about AI shaping human thought while an AI system was, in a small but literal sense, helping shape mine. I almost accepted the suggestion without thinking about it. That moment was not dramatic, but that is exactly the point. The influence I am concerned about does not require anyone to consciously surrender their judgment. It can become part of the background.

Much of the public concern around AI influence has focused on obvious failures: models reinforcing delusions, agreeing with false beliefs, or confidently producing information that is simply wrong. Those cases deserve attention partly because their danger is easy to understand. A machine tells someone something bizarre, and the problem is visible. I am increasingly interested in what may be the opposite problem. What happens when models become so heavily optimized to avoid harmful, reckless, or unsupported advice that their judgment begins to lean toward the safest and most defensible answer?

I do not know that this is happening systematically, and I do not want to pretend that my own experience proves it. But I have experienced something that made the possibility difficult for me to ignore.

I had been working for several months on an independent software project. I had made mistakes along the way, some of them significant, and I wanted an outside perspective on whether the idea was still worth pursuing. So I asked several frontier AI models to critically evaluate the project. I expected criticism. That was the reason I asked.

What surprised me was how quickly the criticism turned into a judgment about whether I should continue at all. The models identified legitimate weaknesses, but they also concluded that the project's chances of success were extremely low and, in some cases, suggested that I stop pursuing it. The responses were detailed, articulate, and confident. That confidence mattered more than I expected it to. For a while, I seriously questioned whether I had wasted months of work and whether continuing would simply be irrational.

Looking back, what bothers me is not that an AI criticized my project. It should have. What bothers me is how much authority I gave the criticism.

Had another developer told me the same thing, I probably would have considered the argument alongside their experience, biases, and knowledge of the problem. With the model, those qualifications were much easier to forget. It could analyze the code faster than I could. It could discuss the market, architecture, and technical tradeoffs in the same conversation. It sounded calm and objective. All of this made its conclusion feel less like one opinion and more like an assessment produced by something that could see the situation more completely than I could.

There is a term for part of this behavior: automation bias, our tendency to give automated systems more weight than we otherwise might, even when our own knowledge points in another direction. AI makes that problem especially interesting because it does not feel much like traditional automation. A calculator gives an answer. An AI model gives an argument. It explains itself, responds to objections, changes tone, produces counterarguments, and speaks with the same language people use when trying to persuade one another.

That distinction matters. If these systems increasingly participate in decisions about what to build, what risks are reasonable, which ideas deserve further investigation, and which projects should be abandoned, then their biases do not need to be extreme to matter. They only need to be consistent.

My concern is that widespread reliance on the same small group of models could create a pressure toward the same kinds of conclusions. AI does not literally contain the sum of human knowledge, and models obviously do not all think alike. Their outputs depend on training data, model architecture, post-training, system instructions, and the context supplied by the user. Still, these systems are trained on overlapping bodies of human knowledge and then further shaped to produce responses that are useful, safe, and broadly acceptable. It seems reasonable to ask whether depending on them for judgment could gradually narrow the range of ideas people are willing to pursue.

This does not require AI to explicitly tell everyone what to think. The effect could be much more mundane. Imagine thousands of developers asking whether an unusual architecture is worth attempting, founders asking whether a strange business idea is viable, researchers asking whether a hypothesis is promising, or writers asking whether an argument makes sense. If AI becomes the first place people go for those judgments, then even a small tendency toward conventional or risk-averse answers could have consequences. Ideas would not need to be censored. Some of them might simply never get attempted.

I think about this partly because of how completely another technology entered my life without me noticing: the smartphone. Recently, I looked at my screen-time report and was disturbed by how many hours of my week were disappearing into my phone. What struck me was not only the number of hours, but how little memory I had of choosing to spend them that way. I did not wake up and decide to devote that much of my life to a screen. I checked a message, looked something up, opened an app, and repeated those small actions until they became automatic.

AI is beginning to feel similar to me. I work in technology, so I encounter it constantly. I use it to answer questions, inspect code, brainstorm approaches, critique ideas, and increasingly to perform work that I would have done myself only a few years ago. Often this is an obvious improvement. It saves time and lets me attempt things I might not otherwise have the ability or patience to do.

But I am becoming less interested in the question of whether AI can do these things and more interested in what happens to me when I stop noticing that I am asking it to do them.

The smartphone primarily captured my attention. AI has the potential to participate in my judgment. That makes the comparison more concerning. If consulting a model becomes as automatic as reaching for a phone, then its influence may eventually become difficult to separate from my own thinking. I may still feel as though I am making my own decisions while quietly allowing a system to determine which options seem reasonable in the first place.

That is the form of AI takeover I find most plausible. Not machines suddenly taking control from humans, but humans gradually developing the habit of asking machines what they should do, then forgetting that asking was ever a choice.