The Decisions We Keep for Ourselves
I build software that rewrites itself, and I assumed the hard part would be the engineering. Seventy years of research told me otherwise.
Anyone who has worked with me will tell you the same thing: I am analytical, objective, and like to go by data.
I have talked myself out of instincts that turned out to be right, because I couldn’t find the evidence for them fast enough to act. It isn’t a virtue, necessarily. Perhaps closer to a reflex.
AI is now capable enough to write software, and I believe we’re on a path toward software that watches how people use it and continuously rewrites itself in response.
From the start, I knew the technical problems weren’t going to be the only ones. I expected people to be uneasy about handing over control. But I thought that unease was fundamentally about trust in the technology: make it good enough, show that it works — which, to be clear, is still a work in progress — and the rest would mostly take care of itself.
Looking for why the unease was stronger than I’d anticipated led me into seventy years of psychology research.
In 1954, a psychologist at the University of Minnesota named Paul Meehl published a book his own field would spend decades trying to explain away.
Meehl was a clinical psychologist, trained in the craft of reading a patient. The sort of judgment no questionnaire was supposed to replace. Then he went looking for the evidence that clinical judgment worked, and gathered every study he could find that put a clinician’s prediction against a plain statistical rule: which patients would relapse, which parolees would reoffend, which students would pass. Around twenty contests. The rule won or tied almost every one. The reception was mixed and many of his colleagues were insulted.
The evidence only grew. A meta-analysis done in the year 2000 worked through a hundred and thirty-six studies and landed in the same place: the formula matched or beat the expert in all but a handful.
None of it settled anything. People kept preferring the human.
You might think that as people became more familiar with computers and algorithms grew more sophisticated, they would become more comfortable trusting them. Perhaps they would be less inclined to abandon an algorithm when it made the occasional mistake.
This wasn’t the case. In 2015, Berkeley Dietvorst, Joseph Simmons, and Cade Massey ran Meehl’s contest as a laboratory experiment and titled the finding “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err.” They gave people a model that beat human forecasters, then let them watch it work through a run of forecasts and make the small, ordinary errors any forecaster makes. Afterward, people defected to human judgment, even when it cost them money, even if they had just watched humans do worse.
A person who errs gets a second chance. A machine that errs gets fired.
During the same period, a different set of experiments found the opposite. In 2019, Jennifer Logg, Julia Minson, and Don Moore reported what they called algorithm appreciation. Asked to estimate a stranger’s weight from a photograph, or to forecast an uncertain outcome, people moved their answers toward the algorithm’s.
So the research says both. People flee the machine and people defer to it. One explanation: the difference sits in the task, not the machine.
Noah Castelo, Maarten Bos, and Donald Lehmann asked, and published the answer in 2019 as “Task-Dependent Algorithm Aversion.” Aversion tracks one thing: how subjective the person believes the task to be. On the tasks people file as objective, like forecasting the weather, giving directions, or analyzing the numbers, they take the machine’s answer without a fight.
On the ones they file as subjective, such as which song to play next, whether a joke is funny, or who to set a friend up with, they dig in, even when the machine is measurably, repeatedly better. That intuition runs deep, even if it’s not always correct. The objection isn’t that a machine can’t do it, but the discomfort of handing something so human to a machine.
In one experiment, an algorithm predicted which jokes people would find funny better than their own friends and partners could — and people still preferred the human's picks. The machine wasn't worse. People just didn't want it deciding what was funny to them.
The dividing line was not how analytical you are, or how high the stakes were. It was whether the task felt like the kind of thing only a person should do. Taste, humor, the read on another human, and therefore like something of yours to keep.
That reflex is the exact shape of the fight coming for software.
We already hand machines the decisions we experience as objective, and mostly without noticing. A calculator does the arithmetic. A map app picks the route. None of it feels like a loss because those decisions never felt like ours in the first place.
Software is different because it contains both kinds of judgment. Teams are happy to let machines optimize retry logic, route traffic, or choose between acceptable variants. What they hesitate to hand over is the judgment of what the product should become: which features should exist, how onboarding should feel, what “good” even means.
The boundary is intuitive. An AI reworks the signup flow and lifts activation seven percent. One team sees a better funnel. Another sees the machine rewriting their product.
People don’t resist optimization. They resist surrendering authorship.
Software teams already have a name for the judgment they believe only humans can make: product sense. Sometimes taste, or knowing the user. It is the last thing most teams would agree to automate
For most of its history, software got to skip Meehl’s problem, because a machine could watch a product but not change it — and, more to the point, could not decide what to change it to. We had the analytics, the A/B tests, the dashboards; what we couldn’t build was the other half of the loop: something that could look at what it learned, decide what the product should become, and write the change itself.
So the question of whether to trust a machine to change the product never came up; there was no machine that could.
It is now becoming possible, and it is why the question is no longer avoidable. A model can read a system’s own code and rewrite it; it can watch how a product is used and adjust what it does. The loop can close.
It is the bet a number of companies, mine included, are making. If the research travels, the hard part of that bet will be where people feel the decision is theirs. That resistance will look like ego dressed up as caution; the same reaction Meehl ran into in 1954.
What I’d taken for a wall turned out to be a map.
Castelo’s team didn’t stop at naming the effect; they went looking for what moves it. The obvious lever was to show people the proof that the algorithm wins. Told outright that a model beat the human expert, participants warmed to the machine on subjective tasks just enough to reach indifference. Indifference sounds like a truce, but it’s a quiet defeat, because a person who feels no preference takes the default: the human they already had.
The levers that actually changed behavior were subtler than evidence. Reframing a task as one that rewards measurement rather than intuition moved people; so did showing that the algorithm could handle the emotional, creative work machines are assumed to fail at. None of what worked was proof of performance. They were levers on perception: what kind of task this was, and what kind of thing the machine was. The two beliefs the ownership reflex is built from.
Those levers are the raw material of something a better model can’t hand you: the design of the transition itself. The framing, the sequence, and the guardrails by which a person becomes willing to let a machine take a judgment they’d kept for themselves.
That shifts what companies building adaptive software are actually competing on.
Model quality will continue to improve across the industry. The harder thing to copy is the experience of handing over control without making people feel they’ve lost ownership. The companies that solve that transition will have an advantage no benchmark can capture.
The research didn’t make me less optimistic about adaptive software. It made me more specific about what has to be built, and clearer about what it’s for. The goal was never to remove human judgment. The authored calls, the ones people feel are theirs, are the part to protect, not the part to quietly take over.
What a machine is for is the rest: to make software capable of adapting at a scale no human team can. What stays human is the level above: deciding what the system is for, and where the line falls between what should adapt and what should hold still.
The research suggests that people hand over the decisions they experience as objective first, and hold onto the ones they experience as their own the longest. Systems that respect that will earn trust gradually, never taking away the ability to see what changed or take it back.
All of this isn’t a reason adaptive software won’t happen. It’s the map of how it will.
Adaptive software needs a target? Or is the target itself part of the adaptation? In the case of user retention, is the target always more retention? Or retention towards what the user surfaces?