In the tech world, AI distillation began as a benign research idea. It’s now morphed into a shadow economy that threatens the business model underpinning trillions of dollars in AI investment.
The technique involves training one AI model using the outputs of another. What’s allowed and what’s not is either unclear or hotly debated, with many AI companies using model outputs from rivals in different parts of their development process.
US AI giants spend billions on data, talent, and computing power to build leading models, hoping to charge premium prices. If those models can be replicated quickly and cheaply through distillation, returns on those investments could erode.
“Generally, AI companies distill other AI companies,” Elon Musk said during a legal battle with OpenAI this year.
Variations of this broad approach could cause the industry to compete away much of its profitability. Distillation helps rivals quickly develop models that are almost as good as, and much cheaper than, frontier systems from Anthropic, OpenAI, and Google.
“They hired the best talents, burned billions and built the newest model, only to have Chinese free models wiping out all your margins,” Xiaoyin Qu, a former senior product manager at Meta, wrote on X recently.
“That must feel shitty as hell,” Qu continued.
Distillation challenges AI’s economics
Anthropic recently framed the issue in exactly those terms when it accused Chinese tech giant Alibaba of using malicious distillation techniques, such as creating tens of thousands of fake Claude accounts to harvest its answers en masse.
“This inverts the economic logic that underwrites American AI leadership, turning billions of dollars worth of research and development, compute, and other US investments into a subsidy for our competitors,” Sarah Heck, Anthropic’s head of policy, wrote in a letter to top US politicians.
OpenAI has warned that distillation could eventually create models that outperform today’s frontier systems, saying that “by blending outputs” from multiple US models, “adversaries could replicate and even combine frontier capabilities in ways that surpass any single teacher model.”
Investors appear worried. AI stocks slumped in recent weeks after Chinese companies released new models, including GLM-5.2 from Z.ai, formerly known as Zhipu AI. Some AI researchers believe this model benefited from distilling knowledge from leading US systems.
“Yes, they distilled Claude and GPT 5.5,” Patrick Toulme, an AI chip software engineer at Google, wrote on X, referring to models from Anthropic and OpenAI.
AI distillation had a benign beginning
Distillation wasn’t always so divisive. In 2015, leading Google AI researchers described a technique in which a lab trained a smaller model using its own large model.
After ChatGPT’s 2022 release, AI became a global race. Distillation expanded beyond a company’s own models, becoming a way to accelerate development by leveraging competitors’ outputs.
Zhang Chi, an AI researcher who recently worked on ByteDance’s large language models, said on a recent podcast that many Chinese AI companies rely heavily on distillation rather than building their own high-quality training data. Rather than paying human experts to write detailed explanations and answers, companies can ask models like ChatGPT, Claude, or Gemini to generate them and then use those responses for training.
On another podcast in May, Google DeepMind researcher Yao Shunyu said Chinese companies, limited in their access to advanced AI chips, have turned distillation into a competitive advantage. He distinguished between “dumb distillation,” which simply copies another model’s answers, and “smart distillation,” in which multiple AI models generate, evaluate, and improve each other’s responses to create better training data.
According to a translation of the podcast, Yao suggested Chinese labs could become leaders in these more advanced techniques.
Earlier this year, a lawyer for OpenAI asked Musk whether xAI had ever distilled OpenAI’s models. Musk replied that it was common across the industry. Asked whether that meant xAI had done it, he answered, “Partly.”
Anthropic, OpenAI, and Google have repeatedly complained about what they describe as malicious distillation, accusing Chinese companies, including DeepSeek, Moonshot, MiniMax, and Alibaba, of abusing the technique while urging regulators to crack down.
Still, US companies have used each others’ work to improve their own models. Google, for example, paid Scale AI gig workers to generate ChatGPT answers and improve them in an attempt to catch up to OpenAI.
Many researchers don’t see that as distillation. AI labs’ terms of service, though, ban the use of their services to build competing models, regardless of how it’s done.
‘Distillation panic’
Not everyone agrees on where legitimate research ends and abuse begins.
AI researcher Nathan Lambert said the original form of distillation is being conflated with more aggressive practices. He worries that broad restrictions spurred by “distillation panic” will hurt smaller AI companies and academic researchers, who rely on the technique to build and study AI models without enormous budgets.
Stopping distillation may also prove difficult.
“It’s always a kind of a cat-and-mouse game,” Zilan Qian, a researcher at the Oxford China Policy Lab, told Business Insider. As long as AI model outputs are out in the world, “people will probably find a way to get access to it.”
Anthropic has tightened access to its models by blocking users in China, requiring overseas phone numbers and payment methods, and asking some users to verify their identity with government-issued ID and a live selfie.
China’s AI transfer stations
Qian found that Chinese developers have responded to such limitations by building a network of go-betweens known as “transfer stations.” These overseas proxy services let customers bypass account checks while paying as little as 10% of the official price.
Some operators rely on thousands of fake or recycled accounts, while others recruit people in lower-income countries to complete identity checks. Because every request passes through these services, operators can collect users’ prompts and AI-generated responses, creating valuable datasets that can later be used to train other AI models or sold to third parties.
Qian thinks tougher restrictions may simply make these workarounds more profitable.
“History teaches us that access blockage rarely stops determined users,” she wrote. “They raise the cost of access, which in turn creates profitable markets for anyone with the expertise to lower it.”
Anthropic’s recent efforts are backfiring
Some recent efforts appear to have backfired. Anthropic quietly degraded model responses to questions about AI development before reversing parts of the policy following developer backlash.
The Information also reported that the company abandoned spyware that tracked Chinese users.
Some AI researchers say the crackdown has had an unintended consequence: pushing more developers toward cheaper, distilled open-source models.
That’s the very outcome frontier labs are trying to stop.
Sign up for BI’s Tech Memo newsletter here. Reach out to me via email at [email protected].
Read the full article here
