Computer in the Loop
Wendy Mackay has spent four decades watching people bend software to their will, and she has a precise idea of where AI belongs in that process.
In 1988, Wendy Mackay spent a year and a half watching people use one of the first intelligent email systems, and found they had quietly repurposed it.
The system was called Information Lens. Built at MIT to fight the then-new problem of email overload, it let people write rules to sort, filter, and prioritize their mail. Its designers had built it around a particular assumption: that you would run those rules on incoming messages, before you read them, so the important ones rose to the top and the junk got filtered out. Some people used it exactly that way. But some did the opposite. They let the mail arrive unsorted, read it, and then ran rules to file it. The designers had built a prioritizer and a good share of users had turned it into a filing system.
Mackay drew a conclusion from this that she has been restating, in one form or another, ever since.
People do not simply receive software. They reinterpret it, and they adapt it to what they actually need, often in ways its designers never anticipated.
She would later call this phenomenon co-adaptation, and it became the spine of her career.
She is now a research director at Inria, runs a group at Université Paris-Saclay called ExSitu, holds the ACM SIGCHI Lifetime Research Award, and was promoted to Inria’s highest research rank. She spent a decade in industry, at Digital Equipment, where her group built some of the first interactive-video software products — just before the Macintosh shipped. She wrote the authoring language for multimedia educational software, coded the first two products herself, and ran a group that shipped more than thirty. The observation that organizes her work is the one she had watching people, in the late 1980s, bend an email tool into something its designers hadn’t drawn.
I called her because of a paper she wrote in 1991, “Triggers and Barriers to Customizing Software,” and because my company, Sky Valley Ambient Computing, is building on the same premise she has spent her career defending: that software should keep adapting after it ships, instead of hardening into a finished artifact.
We agree on that completely. Where the conversation got interesting was the next question down: who does the adapting. Her answer is the human, with the computer in the loop to assist. My company’s bet is that the machine can increasingly lead.
The setup for the 1991 paper was MIT’s Project Athena, which Mackay describes, with the precision of someone who has said it many times, as “an eight-year, $100 million experiment in educational computing” funded by Digital Equipment Corporation and IBM. Eleven hundred workstations across campus. Mackay studied fifty-one of the people using them, collecting interviews, questionnaires, and automatic records of what they changed in their configuration files.
The finding that everyone remembers is about who customizes software, and the answer is: most people don’t.
Not because they wouldn’t want it to fit them better. Because customizing is work, and they’re busy. Nearly two-thirds of her participants named lack of time as the barrier; a third said they didn’t know how. “Time spent customizing,” as the paper puts it, “is time spent not working.” People “satisfice”, they get the software to a state that’s good enough and go do their jobs. The wish for better-fitting software is real; what’s missing is anyone with the time to go produce it.
When I put a version of this to Mackay, that most of our users will be passive, that they won’t make requests, that we still think we can adapt the system to them, she agreed save for one nuance.
In every organization she has studied, Mackay finds what she calls a local guru: the self-appointed expert on Word, or on whatever the critical tool is. Sometimes genuinely technical, sometimes not. People bring this person their questions, which means this person learns more, which deepens the role. She watched one organization go through a reorganization and tracked what happened to the gurus. A group that lost its expert grew a new one. A group that ended up with two watched one quietly cede the role to the other.
She found the same structure in places with high stakes. Move an air traffic controller from one team to another, and they drop their personal way of doing things and pick up the local team’s conventions, because being legible to the people next to you matters more than being optimized for yourself. Personalization that is purely individual, she argues, breaks the moment people have to work together. Uniformity that is enforced for everyone is too rigid to fit how anyone actually works. What real organizations do is land somewhere in between, negotiated locally, mediated by the guru.
This was the moment in the conversation where her old work stopped being a historical footnote and became a design spec. “If you’re gonna build an adaptive system,” she said, “I would strongly recommend that you think about how those local people who are willing to help each other out can do so. Help the guru, and the guru customizes for the group. Some of them talk to other gurus, and customizations spread. It’s a finding from her thesis that she says she “keeps finding again.”
The implication is pointed. If adaptive software treats every user as an atomized individual to be personalized for, it is optimizing for a unit that her data suggests barely exists. The real unit is the group, and the real leverage is the handful of people inside it who are willing to do the adapting for everyone else.
Mackay’s own work has moved from observing this to building on it. Underneath the projects she’s most animated about now is a framework she and her longtime collaborator (and husband) Michel Beaudouin-Lafon developed two decades ago, called instrumental interaction.
The short version is a quarrel with the button. In most software, a command is an abstract operation you invoke. You find “draw” in a menu, click it, and it acts. Mackay wants the command turned into a thing you hold instead. The example she reached for in our conversation is a paintbrush: rather than invoking a draw command, you have a paintbrush instrument, and because it’s an object it has properties, so you can keep it, reuse it, make a red one and a blue one, modify them, and share them.
Instruments should be reusable and should work coherently across different kinds of objects. The aim, as she puts it, is “a physics of interaction.” Pick up an unfamiliar object in the real world and you can already guess much of what it does. A mug you’ve never seen still has a handle to grasp, a volume to fill, a mass you can use to pin down paper. You learned those properties once, as a child, and they’ve held for every rigid object since. Software rarely works this way: each application invents its own rules, so mastering one teaches you little about the next. Mackay wants the digital tool to behave like the physical one: learn it once, and the knowledge carries.
The reason this matters for adaptive software is the part about who gets to do the adapting. If a system is built from a small set of legible primitives then four different parties can operate on the same material with a shared understanding of what it is. Developers build interoperable components. Designers arrange them into something usable. The local guru tweaks a designer’s template into something that fits their group and shares it. And the ordinary user manipulates the result without having to understand the layer beneath.
Mackay’s group is testing this right now on a project that is its own small thesis about where software is going. The French government, wary of depending on American tools like Teams and Google, has commissioned an open-source office suite for its civil servants, La Suite Numérique. Mackay is working with them to build a coherent interaction layer, so that the whole thing behaves according to shared principles rather than as a pile of separate apps. It is, among other things, a bet that adaptive, malleable software is a thing that can be useful and implemented across organizations of all sizes including the government.
I asked Mackay the question that the most aggressive version of this future implies: when AI gets good enough, do we even need software providers? Do we just prompt the model and have it generate whatever we want?
“I really don’t think we’re gonna do prompt and tweak. No, thank you. No, no, no, no.”
Her case has two parts. The first: what made ChatGPT take over the world was the chat box bolted onto it, which let people who didn’t understand the model use it anyway. But chat is a black box, and a fragile one: ask the same question three different ways and you get three different answers, and the user has no idea why. The second part is more fundamental, and it is about what humans are good at. We are extraordinarily good at manipulating objects with our hands and our judgment. We are not, mostly, good at specifying in words what we want and then auditing whether a machine produced it. “If I’m an artist,” she said, “I don’t want to talk to my painting”
On this we agree. We need much more natural ways to interact with AI than a chat box. Prompting is a starting point, but the interface for AI hasn’t been invented yet.
Where our views diverge is how control is divided between human and AI. Mackay is against what she calls “human in the loop”, that is, the framing where the person is a checkpoint in the machine’s process, approving or correcting its output. She wants the opposite arrangement, “computer in the loop,” where the machine serves to make the human better at the thing the human is trying to do.
She pointed to a study that she thinks most people get backward. The study pooled a hundred-odd experiments comparing a person working alone, an AI working alone, and the two together. The seductive assumption is that the pair always wins. On average, it didn’t: the human-AI combination tended to do worse than the better of the two on its own. The gap ran in a particular direction. When the human was less expert than the AI, the pair often performed worse than either alone. The pairing tended to beat the parts only when the human was the stronger of the two, expert enough to recognize a possibility they hadn’t considered and to throw out a bad idea on sight.
The design conclusion she draws is that a system that hands an expert a finished answer wastes the expert, who now spends their time checking the machine instead of creating. A system that hands a novice a finished answer is actively dangerous, because the novice can’t check it. The useful system is the one that expands what the person can explore and decide — not the one that decides for them.
It would be easy to file Mackay as a skeptic, the researcher who has seen enough hype cycles to wave this one off. That is not where she lands, and it would be a misreading of someone who has spent four decades insisting software should never have frozen at deployment in the first place. She is not against software that changes. She is against software that changes on its own, invisibly, in ways the person can’t see, steer, or undo.
The through line from 1991 holds remarkably straight. Back then she found that customization is rare, that it’s overwhelmingly social, and that a few local experts carry it for everyone else. Every one of those findings is a constraint on what “software that adapts itself” can responsibly mean. Personalize per-individual and you break the group legibility that air traffic controllers and energy-grid operators depend on. Aim the whole apparatus at the passive majority and you’ve ignored the small number of people who do the real adapting on everyone’s behalf.
Mackay would let runtime-adaptive software through, I think, on three conditions: that the user can see what changed, that the user can redirect or reverse it, and that it makes them more capable rather than less.
There’s a failure mode worth naming. Mackay borrows the vocabulary for this from ecology, where co-adaptation comes from. Two species sharing an environment can settle into any of several relationships: symbiosis, where both benefit; commensalism, where one gains and the other is unaffected; on down through parasitism, where one gains at the other’s expense, to outright competition, where both lose. A human and an intelligent system, she argues, co-adapt along the same spectrum.
A system optimizing on its own is only as good as what it is told to optimize for. Point it at engagement and “adaptation” can become parasitic: software that gets better at serving its owner.
Now focused mostly on research, Mackay can hold the user’s interest as her core goal. But she is not naive about the other side of it. She spent a decade shipping products that had to sell. As a young woman in a male-dominated industry, the only reason her early work survived was that customers wanted it: she built examples, showed them to customers, and customers went to her bosses to demand what they saw. What made the work good and what made it sell were the same thing. She’s still working with industry today: La Suite Numérique, and a system headed for the control rooms of France’s national energy grid, RTE. “It’s on me to show what we mean,” is roughly how she puts it, “not just talk about it.”
It is easy to assume that businesses whose goal is to make money fall into the lower part of the chart and that “adaptation to improve conversion” is just parasitism with better marketing. But her own career is the counterexample: the work sold because it served the people using it, not in spite of them.
That leaves only a narrow disagreement. Mackay believes the person should be the one who sees the change and decides whether to keep it. We think a good enough model can carry most of that load by making the change, and making it good while leaving the person able to see and override it. Where she sees a decision being taken out of the user’s hands, we see a chore being taken off them.
Neither of us can say in advance where the line falls, because it almost certainly moves by domain. In finance, defense, or medicine Mackay’s caution is hard to argue with. A wrong adaptation is expensive or dangerous, and the user is often an expert; you want the person seeing and approving the change. In many other types of software, the case for the system adapting on its own gets stronger, because the alternative isn’t careful human control. It’s the one-size-fits-all default that fits no one in particular.
None of this means a business is fated to the parasitic row. Software that is genuinely better to use is what earns its keep: people stay, rely on it, pay for it. Dark patterns are the other strategy, like software that’s free because the user is the product. But where incentives are aligned, the symbiotic row is the profitable one.