Distillation Is the New Pretraining

蒸馏即预训练：当 AI 的智慧开始流动

"Education is the kindling of a flame, not the filling of a vessel."

教育不是灌满一桶水，而是点燃一把火。

—— Socrates，被知识蒸馏工程师反复引用

一、The Metaphor That Broke the Industry

一、一个隐喻，撬动了整个行业

The AI industry in 2026 runs on a quiet secret: the frontier is no longer built only by pretraining. It is built by a 175-billion-parameter teacher leaning over the shoulder of a 3-billion-parameter student, saying: here is how I think.

2026 年的 AI 行业藏着一个公开的秘密：前沿能力不再只靠预训练堆出来。它更常是这样一幅画面——一个 1750 亿参数的教师俯身在 30 亿参数的学生肩头，说：我是这样想的。

We call it knowledge distillation. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean named it in 2015; by 2026, it had become the default operation of the entire industry. The name is chemical: distillation​ is the process of heating a mixture, vaporizing the volatile part, and condensing it elsewhere — separating what you want from what you don't. In neural networks, the "mixture" is a cumbersome, expensive, over-parameterized teacher. The "vapor" is its probability distribution. The "condensate" is a small, fast student that fits in a phone.

我们叫它知识蒸馏（Knowledge Distillation）。Geoffrey Hinton、Oriol Vinyals 与 Jeff Dean 在 2015 年将其系统化；到了 2026 年，它已成为整个行业的默认工序。名字来自化学：蒸馏是加热混合物、让易挥发部分气化、再在别处冷凝的过程——把你想要的和不需要的分开。在神经网络里，"混合物"是笨重、昂贵、参数过剩的教师；"蒸气"是它的概率分布；"冷凝物"是能塞进手机的小而快的学生。

A 7B parameter LLM in FP32 is 14GB. In FP16, 7GB. A phone has 4–8GB of RAM, shared with the camera, the OS, and your 40,000 unread emails. Compression is not an optimization. It is a permission slip to exist.

一个 7B 模型在 FP32 下 14GB，FP16 下 7GB。而手机只有 4–8GB 内存，还要和相机、操作系统、你那四万封未读邮件共享。压缩不是优化，而是"被允许存在"的许可证。

This essay is about what distillation is, what it is not, and why the answer determines who gets to run AI — and who only rents it.

本文讲三件事：蒸馏是什么、不是什么，以及这个答案为何决定了谁能运行 AI、谁只能租用 AI。

二、The Thermodynamics of Teaching

二、教学的热力学

2.1 Hard labels are poor teachers

2.1 硬标签是糟糕的老师

Supervised learning gives a model a hard label: a one-hot vector like [cat: 1, dog: 0, bird: 0]. It says: this is the answer; everything else is wrong. The information content is one bit. It is the pedagogical equivalent of a drill sergeant.

监督学习给模型的是硬标签（hard label）：一个 one-hot 向量，如 [猫: 1, 狗: 0, 鸟: 0]。它只说：这是答案，其余皆错。信息量只有 1 比特。这相当于教学法里的教官。

Soft labels​ are different. A teacher model outputs a full probability distribution: [cat: 0.85, dog: 0.14, bird: 0.01]. Hidden inside the "wrong" answers is a confession: this looks a little like a dog, and almost nothing like a bird. Hinton called this "dark knowledge"​ — the structural information about how classes relate. It is the difference between memorizing a fact and understanding a subject.

软标签（soft label）则不同。教师模型输出完整的概率分布：[猫: 0.85, 狗: 0.14, 鸟: 0.01]。在那些"错误"答案里，藏着一个坦白：这东西有点像狗，几乎不像鸟。Hinton 称之为"暗知识"（dark knowledge）——关于类别之间如何关联的结构性信息。这是"背下一个事实"与"理解一门学问"的区别。

A hard label tells you the capital of France is Paris. A soft label tells you it is also France, and that Monaco is a plausible confusion, and that the answer space has geometry.

硬标签告诉你法国首都是巴黎。软标签还告诉你：法国本身就是答案，摩纳哥是容易混淆的选项，整个答案空间是有几何结构的。

2.2 Temperature: turning up the heat

2.2 温度：把火开大

The softening is done with a temperature parameter​ T in the softmax:

软化靠的是 softmax 里的温度参数​ T：

pi​(T)=∑j​exp(zj​/T)exp(zi​/T)​

where zi​ are the raw logits.

其中 zi​ 是原始 logits。

At T=4, a category with probability 0.001 has its relative weight amplified roughly tenfold. The standard sweet spot is T∈[2,5]. Too cold and the student learns nothing. Too hot and it drowns in noise.

当 T=4 时，概率仅 0.001 的类别其相对权重被放大约十倍。标准的甜蜜点是 T∈[2,5]。太冷，学生一无所获；太热，它又被噪声淹没。

This is the central metaphor of distillation, and of this essay: you must heat the system to make the useful part mobile.​ Intelligence that cannot be expressed as a probability distribution cannot be transferred. A teacher who only gives final answers produces students who are good at exams and helpless in life.

这正是蒸馏——也是本文——的核心隐喻：必须加热系统，才能让有用的部分流动起来。​ 无法表达为概率分布的那部分智能，是无法迁移的。一个只给最终答案的老师，教出的学生擅长考试，却不擅生活。

2.3 The loss function: learning from two voices

2.3 损失函数：向两个声音学习

The student is trained with a combined loss:

学生用组合损失训练：

Ltotal​=α⋅Ldistill​+(1−α)⋅Lstudent​

• Ldistill​: usually KL divergence​ between teacher and student soft distributions, scaled by T2 to keep gradients comparable.
• Lstudent​: ordinary cross-entropy against the true hard label.
• Ldistill​：教师与学生软分布之间的 KL 散度，乘以 T2 以保持梯度量级可比。
• Lstudent​：对真实硬标签的普通交叉熵。

The T2 factor deserves a moment. When you divide logits by T, the gradient shrinks by 1/T2; the scaling restores it. This is the difference between teaching and mumbling.​ A teacher's voice must be calibrated to the room.

这个 T2 因子值得停留一秒。当你把 logits 除以 T，梯度会缩小 1/T2；这个缩放把它补回来。这正是教学与喃喃自语的区别。​ 老师的音量必须适配教室。

The hyperparameter α is a moral choice disguised as engineering: how much should the student trust the teacher versus the ground truth?​ Set it too high and the student inherits the teacher's errors. Set it too low and you've wasted the distillation. The literature typically uses α∈[0.1,0.5], but the right answer is empirical, dataset by dataset, domain by domain.

超参数 α 是一个伪装成工程问题的道德选择：学生该多大程度相信老师，而非相信真相本身？​ 设太高，学生继承老师的错误；设太低，蒸馏白做。文献通常用 α∈[0.1,0.5]，但真正的答案只能靠实验——一个数据集、一个领域地试。

Teacher knows the shape of the answer; the ground truth knows which answer is right. A good student listens to both, and argues with neither.

老师知道答案的形状，真相知道哪个答案是对的。好的学生两者都听，又都不盲从。

三、From Compression to Civilization

三、从压缩到文明

3.1 The classical pipeline

3.1 经典流水线

In the canonical setup, the teacher is frozen​ — parameters fixed, gradients off. Same input goes to both models; only the student updates. This is efficient, but it creates a problem we will return to: the teacher is immortal, and the student is forever a copyist.

在经典设定里，教师被冻结——参数固定、不计算梯度。同一份输入同时喂给两个模型，只有学生更新。这很高效，却埋下一个我们后文要回到的问题：教师永生，学生永远是抄写员。

DeepSeek made this pipeline famous in early 2025 with R1​ and its three-stage post-training recipe: R1-Zero pure RL → cold-start SFT → reasoning RL → rejection-sampling SFT → final RL → distill to dense. The final step — distill to dense​ — is the one that changed the deployment economics of the entire industry. MoE models with hundreds of billions of total parameters were compressed into dense students from 1.5B to 70B, with the 7B version jumping 54.36% on mathematical reasoning and the 1.5B version improving by 178.74% over its base.

DeepSeek 在 2025 年初凭 R1​ 及其三阶段后训练配方让这套流水线声名大噪：R1-Zero 纯 RL → 冷启动 SFT → 推理 RL → 拒绝采样 SFT → 最终 RL → 蒸馏为 dense。最后那一步——distill to dense——改变了整个行业的部署经济学。总参数数千亿的 MoE 模型被压缩成 1.5B 到 70B 的 dense 学生，其中 7B 版本在数学推理上提升 54.36%，1.5B 版本相对基座提升了 178.74%。

The most cited number of 2025 was not a benchmark score. It was the compression ratio.

2025 年被引用最多的数字不是某个基准分，而是压缩比。

3.2 Offline vs. On-Policy: the great fork

3.2 离线 vs. 在线策略：关键分叉

There are two ways to distill, and they disagree about the nature of knowledge.

蒸馏有两种做法，它们对"知识是什么"持相反看法。

Offline distillation​ has the teacher generate a static dataset of answers; the student trains on it like any other corpus. It is cheap, reproducible, and suffers from distribution drift: during training the student sees teacher-written prefixes, but at inference it autoregressees on its own prefixes. Errors compound. This is the "privilege illusion" identified by Shanghai AI Laboratory in 2026: the teacher conditions on reference answers the student will never see, and the student learns to lean on a crutch that disappears in production. Their fix, Dual-Anchored Policy Distillation, closes the information gap; it reported roughly a 70% reduction in confidently wrong claims.

离线蒸馏（offline）让教师预先生成一份静态答案数据集，学生像训练普通语料一样学习。它便宜、可复现，却承受分布漂移：训练时学生看的是教师写的前缀，推理时它却要基于自己生成的前缀自回归。错误会累积。这正是上海人工智能实验室在 2026 年指出的"特权错觉（privilege illusion）"：教师在训练时能见到参考答案，而学生推理时见不到，于是学生学会依赖一根部署时消失的拐杖。他们的修复方案双锚定策略蒸馏（Dual-Anchored Policy Distillation）弥合了信息差，据报"过度自信的错误声明"减少了约 70%。

On-Policy Distillation (OPD)​ generates teacher traces on the fly, conditioned on the student's own rollouts. It is more expensive — the teacher must run during training — but the training distribution tracks the student's actual behavior. It is now a standard component of post-training at Qwen3, MiMo, GLM-5 and others.

在线策略蒸馏（OPD）让教师在训练过程中即时生成轨迹，且以学生自己的 rollout 为条件。它更贵——训练时教师必须一直跑——但训练分布跟着学生的真实行为走。如今它已是 Qwen3、MiMo、GLM-5 等后训练的标准组件。

3.3 The most surprising paper of September 2026

3.3 2026 年 9 月最令人意外的论文

A paper submitted to arXiv on September 4, 2026​ — What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection — did something almost absurd: it trained OPD on one example. One. Problem.

一篇于 2026 年 9 月 4 日提交 arXiv 的论文——《What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection》——做了一件近乎荒唐的事：它用 1 个样本训练 OPD。一个。

1-shot OPD worked.​ Across sampled examples it was consistently effective; models entered a plateau around step 300 and stayed there for the full 2,000 steps without collapse. The overlap ratio — how closely student and teacher agreed on the top-16 tokens — kept rising.

1-shot OPD 居然奏效了。​ 在多个采样样本上始终有效；模型约在第 300 步进入平台期，并在完整 2,000 步内保持稳定，没有崩溃。重叠率（学生与教师在 top-16 token 上的一致程度）持续上升。

Then came the real finding. Hard examples beat easy ones.​ The researchers split problems into easy / medium / hard. Hard examples often reached the full-dataset level faster — and "unsolvable" samples, where both teacher and student got essentially zero accuracy, still improved the validation score. The student was learning how to reason, not what the answer is.

接着是真正的发现：难题胜过简单题。​ 研究者把题目分为 easy / medium / hard。难题往往更快逼近全量数据集的水平——而 "不可解"样本（教师和学生正确率都近乎零）仍然提升了验证集分数。学生在学的是如何推理，而非答案是什么。

Why? Two reasons, cleanly demonstrated:

为什么？两个被干净地证实的原因：

1. Long reasoning horizons align better.​ Long-CoT training keeps the student close to the teacher across thousands of tokens; token-level KL stays low even past 4,000 tokens, where 2K and 4K models diverge sharply.
2. Hard problems expose structural reasoning patterns.​ The tokens with the largest KL drop were words like Alternatively, Wait, Perhaps, Because, Since — reflection, transition, causal reasoning.
3. 长推理程的对齐更好。​ 长思维链训练让学生在数千 token 上始终贴近教师；token 级 KL 即便超过 4,000 token 仍保持低位，而 2K、4K 模型在此处急剧发散。
4. 难题暴露出结构性推理模式。​ KL 下降最大的 token 是 Alternatively、Wait、Perhaps、Because、Since 这类词——反思、转折、因果推理。

The decisive experiment controlled for entropy: when rollout lengths were truncated to similar values, the performance gap between high-entropy and low-entropy samples shrank. The field's favorite data-selection heuristic — prefer uncertain, high-entropy examples — was wrong. Length, not entropy, is the variable that matters.

决定性实验对熵做了控制：当 rollout 长度被截断到相近值时，高熵与低熵样本之间的性能差距缩小了。该领域最爱用的数据选择启发式——优先选不确定、高熵的样本——是错的。重要的变量是长度，不是熵。

8 hard examples matched the 17,000-dataset baseline. Eight. Going from 8 → 16 → 64 added little. The paper's conclusion is a rebuke to an entire industry's data-hoarding instinct: the value of a distillation sample lies in the reasoning process it exposes, not in the answer it produces.

8 个难题就追平了 17,000 条数据集的基线。八个。从 8 → 16 → 64 增益有限。这篇论文的结论是对整个行业"囤数据"本能的当头一棒：一条蒸馏样本的价值，在于它暴露出的推理过程，而非它产出的答案。

This is the deepest sentence in the paper, and the one most worth tattooing on the office wall of every AI lab:

这是全篇最深刻的一句话，值得刻在每个 AI 实验室的墙上：

The model learns "how to reason," not merely "what the answer is."

模型学的是"如何推理"，而不只是"答案是什么"。

四、The Frontier of Distillation

四、蒸馏的前沿

4.1 Negative signals: learning from failure

4.1 负信号：向失败学习

The ACL 2026 paper Harnessing Negative Signals asked a heretical question: what if the teacher's wrong​ reasoning traces are the most useful data of all? Standard practice discards them.

ACL 2026 的论文《Harnessing Negative Signals》问了一个离经叛道的问题：如果教师错误的推理轨迹才是最有用的数据呢？标准做法把它们全扔掉了。

Their recipe: SFT on positive traces first, then a REINFORCE-style refinement using both positive and negative traces, dubbed REDI. The result: Qwen-REDI-1.5B, trained on just 131k traces​ from the open Open-R1​ dataset, scored 83.1% on MATH-500​ — matching DeepSeek-R1-Distill-Qwen-1.5B, which was trained on 800k proprietary samples.

他们的配方是：先在正轨迹上做 SFT，再用正负轨迹一起做 REINFORCE 式精炼，称为 REDI。结果：Qwen-REDI-1.5B​ 仅用开放数据集 Open-R1​ 的 131k 条轨迹，在 MATH-500 上拿到 83.1%——追平了用 80 万条专有数据训练的 DeepSeek-R1-Distill-Qwen-1.5B。

A 6× data efficiency gain​ from refusing to throw away the failures. In AI, as in life, the failures are where the structure lives.

仅仅因为不舍得扔掉失败样本，就换来 6 倍的数据效率。AI 如此，人生亦然：结构恰恰藏在失败里。

4.2 Multi-teacher and self-distillation

4.2 多教师与自蒸馏

Multi-teacher distillation​ routes a student to several domain experts — math, code, instruction following, tool use — and merges their supervision. The 2026 state of the art, MT-SDPO​ (Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation), admits a teacher's feedback only when its private answer passes an independent verifier. On Qwen3-8B it cut the domain gap from 20.96 to 5.30 points.

多教师蒸馏把学生路由给多个领域专家——数学、代码、指令跟随、工具调用——再合并它们的监督。2026 年的最先进水平 MT-SDPO（《Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation》）规定：只有当某教师的私有答案通过独立验证器时，才采纳它的反馈。在 Qwen3-8B 上，它把领域差距从 20.96 缩小到 5.30 分。

Self-distillation​ goes further: the model is its own teacher. Born-Again Networks (ICML 2018) showed that distilling into an identically-sized​ student could surpass the original teacher. Cursor's Composer 2.5 uses a contextual hint to turn the model into its own teacher; Thinking Machines distills from an earlier checkpoint to recover abilities erased during fine-tuning. The teacher and student are becoming the same model, separated only by time.

自蒸馏走得更远：模型就是它自己的老师。Born-Again Networks（ICML 2018）证明，蒸馏到同等大小的学生，竟能超越原教师。Cursor 的 Composer 2.5 用一条上下文提示把模型变成自己的老师；Thinking Machines 从更早的检查点蒸馏，以恢复微调中被抹掉的能力。教师与学生正合二为一，仅被时间分开。

There is a beautiful recursion here that the field has not yet fully digested: the student, having learned, becomes a teacher to a smaller student, who becomes a teacher in turn.​ Intelligence propagates downward through parameter space like heat through a metal rod — and like heat, some of it is always lost.

这里有一个行业尚未完全消化的美妙递归：学生学成之后，成为更小学生的老师，而后者又成为再下一个的老师。​ 智能像热量穿过金属棒一样，沿着参数空间向下传导——而像热量一样，它总会损失一部分。

4.3 Distillation beyond parameters: "skills"

4.3 参数之外的蒸馏："技能"

Microsoft's 2026 paper proposed a different target: distill expensive test-time reasoning into a Markdown skill document​ — 35 to 50 trajectories boiled down to reusable rules, then injected into a non-reasoning model's system prompt. On 4 agent benchmarks, skill-augmented GPT-5.4-mini recovered 55–100%+ of the gap​ between non-reasoning and reasoning modes while using 2.9–4.5× fewer output tokens; on ALFWorld and τ²-retail it beat​ the reasoning mode.

微软 2026 年的论文提出了另一个目标：把昂贵的测试时推理蒸馏成一份 Markdown 技能文档——35 到 50 条轨迹浓缩为可复用规则，再注入非推理模型的系统提示词。在 4 个智能体基准上，加了技能的 GPT-5.4-mini 补回了非推理与推理模式之间 55–100%+ 的差距，同时少用了 2.9–4.5 倍输出 token；在 ALFWorld 和 τ²-retail 上甚至击败了推理模式。

But it drew a sharp boundary: on telecommunications and SpreadsheetBench, reasoning mode still won, because those tasks contain instance-specific dependencies no fixed skill can capture.

但它划出了清晰的边界：在电信和 SpreadsheetBench 上，推理模式仍然胜出，因为那些任务含有固定技能无法捕捉的、每次都变的实例依赖。

Distill repeated structure into skills; reserve expensive reasoning for genuine novelty.​ It is the architectural principle of the next generation of agents — and, not incidentally, of every competent human being.

把重复出现的结构蒸馏成技能；把昂贵的推理留给真正的新鲜事物。​ 这是下一代智能体的架构原则——也顺便是每一个 competent 的人类的原则。

4.4 Distillation at inference time: speculative decoding

4.4 推理时的蒸馏：投机解码

Distillation is not just for training. Speculative decoding​ uses a small draft model to propose several tokens, then has the large target model verify them in a single pass. Acceptance is governed by distributional closeness — and the best draft heads are trained to match the target's output distribution, not to be good language models in their own right.

蒸馏不只用于训练。投机解码（speculative decoding）让小模型提议几个 token，再由大模型一次性验证。接受率取决于分布贴近度——而最好的 draft head 是被训练来匹配目标分布的，并非为了成为独立的语言模型。

EAGLE-3 is the 2026 production default: direct token prediction plus multi-layer feature fusion, 3–4× decode speedup. It is deployed in vLLM, SGLang, TensorRT-LLM. DeepSeek-V3 bakes multi-token prediction heads​ directly into pretraining.

EAGLE-3 是 2026 年的生产默认方案：直接 token 预测加多层特征融合，解码加速 3–4×。它已部署在 vLLM、SGLang、TensorRT-LLM 中。DeepSeek-V3 把多 token 预测头直接烘焙进预训练。

The lossless guarantee is subtle: identical sampling distribution, not bit-for-bit identical output. Two runs with the same seed can diverge — the distributions match, the strings need not. This distinction matters more than it sounds; it is the difference between reproducibility and determinism.

"无损"保证是微妙的：是采样分布相同，而非逐 bit 完全一致。相同种子的两次运行可能发散——分布相同，具体字符串不必相同。这个区别比听起来重要；它是"可复现性"与"确定性"之间的距离。

五、The Economics of Small

五、小的经济学

5.1 The numbers that changed the industry

5.1 改变行业的那些数字

DeepSeek-R1's architecture — MoE with 671B total / 37B active parameters, plus MLA and FP8 quantization — made a GPT-4-class model cheap to serve. Distillation did the rest. Reported figures:

DeepSeek-R1 的架构——671B 总参数 / 37B 激活参数的 MoE，加上 MLA 与 FP8 量化——让 GPT-4 级模型的推理成本大幅降低。蒸馏完成了其余部分。已披露数据：

At deployment, the arithmetic is brutal and real. A customer-service workload at 100k calls/day:

在部署侧，账是这样算的（日均 10 万次调用的客服场景）：

A reported 10-billion-request comparison: 67B at 0.12/request=∗∗12M; 3B at 0.015=∗∗1.5M​ — and an 8× reduction in GPU fleet size.

一份 100 亿次请求的对比：67B 模型每次 0.12 美元 = 1200 万美元；3B 模型每次 0.015 美元 = 150 万美元——GPU 集群规模缩减 8 倍。

These are vendor-reported figures and should be read as order-of-magnitude demonstrations, not procurement quotes. Your mileage will vary with token mix, context length, batching, and whether your "3B model" is any good. But the direction is unambiguous: inference cost is a size problem before it is an algorithm problem.

这些是厂商披露的数字，应视为数量级的演示，而非采购报价。你的实际结果会因 token 构成、上下文长度、批处理和"3B 模型到底行不行"而大不相同。但方向毫无歧义：推理成本首先是体积问题，其次才是算法问题。

5.2 Where the small model lives now

5.2 小模型如今住在哪里

2026 is being called the year of on-device AI. Flagship NPUs crossed 40–45 TOPS (INT8); claimed figures include Qualcomm Snapdragon 8 Gen5 at 45 TOPS and Apple M4 Ultra running 13B locally at 50+ tokens/s. The standard recipe is quantization + distillation + sparsification: INT4 shrinks a 7B model from ~14GB to ~3.5GB with <1% accuracy loss.

2026 被称为端侧 AI 落地元年。旗舰 NPU 突破 40–45 TOPS (INT8)；披露数据包括骁龙 8 Gen5 的 45 TOPS、苹果 M4 Ultra 本地跑 13B 达 50+ tokens/s。标准配方是量化 + 蒸馏 + 稀疏化：INT4 把 7B 模型从约 14GB 压到约 3.5GB，精度损失 <1%。

The applications are unglamorous and enormous:

应用朴素而巨大：

• Industrial quality inspection​ — cameras + edge inference; one cited deployment raised defect detection from 89% to 94% and cut per-device cost from 150kto28k.
• Financial risk control​ — a bank reportedly cut fraud-response time from 2s to 500ms, lowered false positives 19%, and saved over $2M/year.
• Healthcare and law​ — local deployment means sensitive data never leaves the premises; distillation retains ~87% of professional terminology coverage in one cited medical-QA setup.
• Smartphones​ — offline long-document parsing, on-device RAG over private notes, translation without a network.
• 工业质检——相机 + 边缘推理；一份被引用的部署把缺陷检测从 89% 提到 94%，单设备成本从 15 万美元降到 2.8 万美元。
• 金融风控——某银行据报把反欺诈响应从 2 秒降到 500 毫秒，误报降 19%，年省 200 万美元以上。
• 医疗与法律——本地部署意味着敏感数据不出门；一份医学问答设定里蒸馏模型保留了教师约 87% 的专业术语覆盖率。
• 手机——离线长文档解析、基于私有笔记的端侧 RAG、无网翻译。

The geopolitics of AI used to be about who has the most FLOPs. Increasingly, it is about who can fit the most intelligence into three watts.

AI 的地缘政治一度是关于谁拥有最多 FLOPs。如今它越来越关乎：谁能把最多的智能塞进三瓦特。

六、The Ceiling — And the Cracks In It

六、天花板——以及它上面的裂缝

6.1 The capability cliff

6.1 能力悬崖

A student cannot exceed its teacher.​ This is not a guideline; it is the structure of the operation. Distillation is imitation, not discovery. On tasks near the edge of the teacher's ability, students show capability cliffs​ — sharp drops concentrated on the hardest 15% of problems, precisely where reliability matters most.

学生无法超越教师。​ 这不是经验法则，而是操作的固有结构。蒸馏是模仿，不是发现。在教师能力边界附近的任务上，学生会遭遇能力悬崖——性能在最难的那 15% 题目上急剧下滑，而那恰恰是可靠性最关键之处。

Compression has a nonlinear knee. 400B → 70B retains most; 400B → 7B loses meaningfully; 400B → 1B may lack the parameter count to represent complex concepts at all. There is no free lunch, only a menu of expensive lunches.

压缩存在非线性拐点。400B → 70B 保留大部分；400B → 7B 明显失守；400B → 1B 可能根本缺乏表征复杂概念所需的参数量。天下没有免费午餐，只有一份份昂贵的午餐。

6.2 Distillation vs. Reinforcement Learning

6.2 蒸馏 vs. 强化学习

This is the central technical debate of 2026, and Nathan Lambert's framing is the clearest: RL and distillation are different in kind.

这是 2026 年核心技术争论，Nathan Lambert 的表述最清晰：RL 与蒸馏本质上不同。

• Distillation = imitation.​ Copy the answer shape from a strong model.
• RL = exploration.​ Generate, fail, iterate; derive the solution path yourself.
• 蒸馏 = 模仿。​ 从强模型复制答案的形状。
• RL = 探索。​ 生成、失败、迭代；自己摸索出解题路径。

DeepSeek's own updated R1 paper made the comparison explicit on Qwen2.5-32B-Base: over 10,000 steps of RL​ produced a model behind the distilled version — AIME 2024: 47.0% (RL) vs. 72.6% (distillation). The conclusion was carefully worded: for small and medium models, distillation is more cost-effective — but to break through the frontier, you still need larger base models and large-scale RL.

DeepSeek 更新后的 R1 论文在 Qwen2.5-32B-Base 上做了直接对比：超过 1 万步 RL​ 得到的模型仍落后于蒸馏版——AIME 2024：47.0%（RL）vs. 72.6%（蒸馏）。结论措辞谨慎：对中小模型，蒸馏更具性价比——但要突破前沿，仍需更大的基座模型和大规模 RL。

Meanwhile, pure RL has its own victories: Qwen2-Math-7B-Zero reached 22.3% on AIME 2024, ahead of Qwen2-Math-7B-Instruct (7.9%) and GPT-4o (9.3%), proving that RL can develop novel reasoning strategies rather than merely imitate demonstrations. But the same team found that 7B Dense and 16B MoE simply could not improve meaningfully on AIME regardless of RL budget.

与此同时，纯 RL 也有自己的胜利：Qwen2-Math-7B-Zero 在 AIME 2024 上达到 22.3%，超过 Qwen2-Math-7B-Instruct（7.9%）与 GPT-4o（9.3%），证明 RL 能发展出全新推理策略，而非仅模仿示范。但同一团队发现：7B Dense 与 16B MoE 无论投入多少 RL 预算，都无法在 AIME 上取得有意义的提升。

The honest synthesis: distillation is the fast path to competence; RL is the slow path to transcendence. One gives you the frontier cheaply. The other moves the frontier. No serious lab is choosing between them — they are stacking both.

诚实的综合：蒸馏是通往"够用"的快路，RL 是通往"超越"的慢路。一个廉价地给你前沿，另一个把前沿往前推。没有正经实验室在二选一——它们在把两者叠起来。

6.3 Cross-model interference

6.3 跨模型干扰

There is a less-discussed problem. Lambert points out that different models have subtly different data distributions. Feed a Claude-generated answer into a model of another architecture and it may not just fail to help — it may interfere, dragging the student toward a representation space it was not built for. Distillation is a research problem, not a drag-and-drop operation.

还有一个讨论较少的问题。Lambert 指出，不同模型的数据分布存在微妙差异。把 Claude 生成的回答喂进另一个架构的模型，它可能不仅没帮助——反而会造成干扰，把学生拽向一个它并非为之设计的表示空间。蒸馏是一个研究课题，而非拖放操作。

This is why raw "API scraping" results are almost always worse than a proper distillation pipeline. The word "distillation" covers at least four distinct operations:

这就是为什么原始的"爬 API"结果几乎总比规范的蒸馏流水线差。"蒸馏"一词至少覆盖了四种截然不同的操作：

Most public "distillation" is actually response distillation — a fancy name for training on synthetic data. It works, but it is not what Hinton meant.

多数公开的"蒸馏"其实是响应蒸馏——用合成数据训练的漂亮说法。它有效，但不是 Hinton 的本意。

七、The Shadow: Law, Bias, Homogeneity

七、阴影：法律、偏见、同质化

7.1 The legal question that will not go away

7.1 挥之不去的法律问题

In January 2025, OpenAI accused DeepSeek of distilling its models; the dispute pushed distillation onto the front pages. The core claim: a model's outputs encode protected "knowledge" — response patterns, decision boundaries, stylistic regularities — and systematically extracting them via API constitutes unauthorized copying or a derivative work.

2025 年 1 月，OpenAI 指控 DeepSeek 蒸馏其模型，争议把蒸馏推上了头条。核心主张是：模型输出编码了受保护的"知识"——响应模式、决策边界、风格规律——通过 API 系统性提取即构成未经授权的复制或演绎作品。

The counterargument, developed in detail in a 2026 paper in Nanjing University of Posts and Telecommunications Journal (Social Science Edition), uses a three-dimensional framework: training data, model architecture, output results. Its conclusions:

反驳意见在一篇 2026 年发表于《南京邮电大学学报（社会科学版）》的论文中被细致展开，采用三维框架：训练数据、模型架构、输出结果。其结论：

• Training data:​ a teacher's learned regularities are not copyrightable; using them is transformative fair use.
• Architecture:​ model architectures are generic; student implementation shows independent originality.
• Output:​ student outputs are non-directly-copying, independently generated, and objectively different.
• 训练数据：教师习得的规律不受版权保护；使用属转换性合理使用。
• 架构：模型架构具有通用性；学生的实现体现独立独创性。
• 输出：学生输出非直接复制、独立生成、客观上相异。

One cited study found that students reproduced only ~11.35%​ of samples memorized by the teacher — evidence both for​ the "no substantial similarity" defense and​ for the "memory inheritance" problem. The same mechanism that transfers knowledge also transfers memories.

一项被引用的研究发现，学生仅复现教师记忆样本的约 11.35%——这一数字既支持"无实质性相似"的抗辩，也暴露了"记忆遗传"问题。迁移知识的同一机制，也在迁移记忆。

The major labs have answered with their terms of service: OpenAI, Google (Gemini), and Anthropic all restrict using outputs to train competing models. Enforcement is hard — outputs are text, and model fingerprinting remains contested. Meanwhile, geopolitics: in September 2026, U.S. agencies alleged that six Chinese AI companies had systematically extracted outputs from American frontier models; China's foreign ministry called the claims "unfounded".

主流实验室以服务条款作答：OpenAI、Google（Gemini）、Anthropic 均限制将输出用于训练竞争模型。执行很难——输出就是文本，而模型指纹识别仍存争议。与此同时，地缘政治：2026 年 9 月，美国多个机构指控六家中国 AI 公司系统性提取美国前沿模型输出；中国外交部称指控"毫无根据"。

The technical question ("can knowledge be transferred?") and the legal question ("who may transfer it?") must not be confused. Distillation is agnostic about authorization; the authorization is the entire point.

技术问题（"知识能否迁移？"）与法律问题（"谁有权迁移它？"）绝不可混淆。蒸馏本身对授权无感；而授权，恰恰是全部要害。

7.2 Bias inheritance

7.2 偏见的遗传

A teacher's biases are latent; a student is a direct mapping. Whatever the teacher gets wrong — stereotypes, refusal patterns, factual hallucinations — the student learns as a feature, not a bug. Worse, it becomes harder to detect, because the student's errors are now your product's errors, and the original teacher is someone else's black box.

教师的偏见是隐性的，学生是它的直接映射。教师弄错的任何东西——刻板印象、拒绝模式、事实性幻觉——学生都把它当作特征而非缺陷来学。更糟的是，它更难被发现，因为学生的错误如今成了你产品的错误，而原始教师是别人的黑箱。

Alignment tax​ is real: a safety-aligned teacher that reliably refuses harmful requests can produce a student that complies, simply because refusals are underrepresented in the distillation data. This is why Anthropic invests specifically in safety-focused distillation.

对齐税（alignment tax）是真实的：一个可靠拒绝有害请求的对齐教师，可能产出顺从的学生，仅仅因为拒绝在蒸馏数据中被代表不足。这正是 Anthropic 专门投入安全聚焦蒸馏的原因。

7.3 The monoculture risk

7.3 同质化风险

ByteDance's internal ban on distilling open-source models is the cleanest statement of a serious position: "Distillation brings short-term rankings but not real technological moats". The listed costs:

字节跳动内部禁止蒸馏开源模型，这是一个严肃立场的最清晰表述："蒸馏能换来短期排名，但换不来真正的技术壁垒"。所列代价：

1. Ceiling dependence​ — hard to surpass the teacher
2. Loss of originality​ — long-term reliance kills foundational innovation
3. Homogenization​ — everyone distilling the same models produces indistinguishable products
4. Compliance exposure​ — some open-source licenses explicitly forbid output-based training
5. Black-box inheritance​ — teacher errors and biases replicate silently
6. 天花板依赖——难超教师
7. 原创性丧失——长期依赖扼杀底层创新
8. 同质化——大家蒸馏同一批模型，产品趋同
9. 合规风险——部分开源协议明禁基于输出的训练
10. 黑箱继承——教师的错误与偏见被静默复制

"The fast lane is easy to take, but its ceiling is visible to the naked eye."

"快车道好走，但天花板肉眼可见。" — on the route-choice war of 2026

"快车道好走，但天花板肉眼可见。"——论 2026 年的路线之争

Model homogenization may be more dangerous than model weakness.​ A diverse ecosystem fails gracefully; a homogeneous one fails in lockstep. When every on-device model is a student of the same three teachers, a shared blind spot becomes a civilizational blind spot.

模型的同质化，可能比模型不够强更危险。​ 多样的生态会优雅地失败；同质的生态会整齐地一起失败。当每个端侧模型都是同一三位教师的学生，共同的盲点就成了文明的盲点。

八、A Taxonomy of the Field

八、领域分类学

For reference, the established categories:

供参考，学界已建立的分类：

Key milestones worth remembering:

值得记住的关键节点：

• 2006​ — Buciluǎ et al.: a large ensemble's predictions can be approximated by a single small model.
• 2015​ — Hinton, Vinyals, Dean: Distilling the Knowledge in a Neural Network; temperature scaling.
• 2018​ — Deep Mutual Learning (CVPR): students teach each other, no fixed teacher.
• 2018​ — Born-Again Neural Networks (ICML): same-size student can beat the teacher.
• 2019​ — Relational Knowledge Distillation (CVPR): transferring sample relations.
• 2020​ — Knowledge Distillation: A Survey (IJCV): the field gets its taxonomy.
• 2025​ — DeepSeek-R1: distillation as a mass-deployment industry.
• 2026​ — OPD data efficiency, negative-trace REDI, multi-teacher MT-SDPO, skills distillation, privilege illusion.
• 2006​ — Buciluǎ 等：大模型集成的预测可被单个小模型近似。
• 2015​ — Hinton、Vinyals、Dean：《Distilling the Knowledge in a Neural Network》；温度缩放。
• 2018​ — Deep Mutual Learning（CVPR）：学生互相教，无固定教师。
• 2018​ — Born-Again Neural Networks（ICML）：同等大小学生可胜教师。
• 2019​ — Relational Knowledge Distillation（CVPR）：迁移样本间关系。
• 2020​ — Knowledge Distillation: A Survey（IJCV）：领域获得分类学。
• 2025​ — DeepSeek-R1：蒸馏成为大规模部署的产业。
• 2026​ — OPD 数据效率、负轨迹 REDI、多教师 MT-SDPO、技能蒸馏、特权错觉。

九、The Engineer's Notebook

九、工程师手册

A practical distillation recipe, assembled from the research above:

一份实用的蒸馏配方，由上述研究拼合而成：

1. Pick your teacher like you pick a co-author.

Use a teacher whose output distribution resembles your student's target domain. Cross-family distillation needs extra alignment work.

1. 选教师如选合著者。

选用输出分布接近学生目标领域的教师。跨族蒸馏需要额外对齐工作。

2. Start with response distillation, graduate to logits.

If you have API access to logits, use them (KL loss). If not, high-quality traces + SFT is a strong baseline.

2. 从响应蒸馏起步，进阶到 logits。

若有 logits 访问权限就用（KL 损失）；没有，高质量轨迹 + SFT 也是强基线。

3. Set temperature dynamically.

Common practice: τ=5 early, τ=1 late. The intuition: teach broadly first, then sharpen.

3. 动态设置温度。

常见做法：前期 τ=5，后期 τ=1。直觉：先广泛教，再 sharpen。

4. Choose hard, long examples — not high-entropy ones.

The 2026 evidence is strong: difficulty and CoT length matter; entropy does not. Eight carefully chosen hard examples can match 17,000 random ones.

4. 选难题、长样本——而非高熵样本。

2026 年的证据很强：难度与 CoT 长度才重要，熵不重要。8 个精心挑选的难题可追平 1.7 万条随机样本。

5. Keep the ground truth in the loss.

A pure distillation loss lets teacher errors become student convictions. The hard-label term is your emergency brake.

5. 把真实标签留在损失里。

纯蒸馏损失会让教师的错误变成学生的执念。硬标签项是你的紧急刹车。

6. Audit train/inference information asymmetry.

If your teacher sees reference answers or privileged context, use dual-anchored or distribution-corrected objectives — or your eval numbers are lying to you.

6. 审计训练/推理的信息不对称。

若教师能见到参考答案或特权上下文，请用双锚定或分布校正目标——否则你的评测数字在骗你。

7. Use negative traces.

Don't discard wrong reasoning. SFT on positives, then REDI-style refinement on both. 131k traces, 83.1% on MATH-500.

7. 使用负轨迹。

别扔掉错误推理。先在正轨迹上 SFT，再用 REDI 式方法同时精炼正负轨迹。131k 条轨迹，MATH-500 上 83.1%。

8. Bake in safety explicitly.

Safety behavior is underrepresented by default. Distill refusals as a first-class category, or pay the alignment tax.

8. 显式注入安全。

安全行为在数据中天然被代表不足。把拒绝当作一等类别来蒸馏，否则缴纳对齐税。

9. Pair with quantization and pruning.

Distillation preserves capability; quantization + pruning + sparsification​ deliver the actual deployment win. One cited path: FP16 → INT8 with <1.2% precision loss.

9. 与量化、剪枝搭配。

蒸馏保留能力；量化 + 剪枝 + 稀疏化才兑现部署收益。一份被引路径：FP16 → INT8，精度损失 <1.2%。

10. Evaluate on out-of-distribution tasks.

Distilled models overfit benchmarks inherited from the teacher. If it works on real tasks outside the teacher's training distribution, then you have something.

10. 在分布外任务上评测。

蒸馏模型会过拟合从教师那里继承来的基准。只有当它在教师训练分布之外的真实任务上也奏效，你才算真正拿到了点东西。

十、What Comes Next

十、接下来

Prediction 1: The "teacher" will increasingly be a committee.​ Multi-teacher on-policy distillation is already the post-training workhorse at the frontier labs; Nemotron 3 Ultra uses more than ten specialized teachers​ providing dense, token-level feedback. Convergence is roughly 10× faster than pure RL at 1/10 the GPU hours.

预测一："教师"将日益成为一个委员会。​ 多教师在线性蒸馏已是前沿实验室后训练的主力；Nemotron 3 Ultra 用十余个专精教师提供密集的 token 级反馈。收敛速度约为纯 RL 的 10 倍，GPU 小时仅为其 1/10。

Prediction 2: Self-distillation will erase the teacher/student boundary.​ When a model distills from its own earlier checkpoint, "teaching" becomes a temporal relation within one model. The pedagogy of the future is autobiography.

预测二：自蒸馏将抹去教师/学生的边界。​ 当模型从自己的早期检查点蒸馏，"教学"就成了单个模型内部的时间关系。未来的教育学是自传。

Prediction 3: The scarcest resource will be high-quality reasoning trajectories, not GPU-hours.​ If 8 hard examples ≈ 17,000 random ones, then curation is the new scaling law. Data moats will be built from carefully selected failures, not from web scrape volume.

预测三：最稀缺的资源将是高质量推理轨迹，而非 GPU 小时。​ 若 8 个难题 ≈ 1.7 万个随机样本，那么策展就是新的 scaling law。数据护城河将由精心挑选的失败筑成，而非由爬取量。

Prediction 4: On-device will split the market in two.​ A 3B–7B distilled model on a phone, plus a 70B–400B teacher in the cloud, with dynamic routing between them — this is already the de facto architecture. The interesting question is the routing policy, which itself will be distilled.

预测四：端侧将把市场撕成两半。​ 手机上的 3B–7B 蒸馏模型 + 云端的 70B–400B 教师，二者间动态路由——这已是事实上的架构。有趣的问题是路由策略，而它本身也会被蒸馏。

Prediction 5: The legal status will solidify around "transformative use," but with conditions.​ The three-dimensional framework (data / architecture / output) is coherent and likely influential. But the access question — did you obtain the outputs legitimately? — will become the central battleground, complete with model fingerprinting and API watermarking.

预测五：法律地位将围绕"转换性使用"固化，但附条件。​ 三维框架（数据/架构/输出）是自洽且可能具影响力的。但访问问题——你是否合法取得输出？——将成为主战场，伴随模型指纹识别与 API 水印。

十一、The Philosophical Turn

十一、哲学转向

Let us end with the part that is not technical.

最后谈点非技术的部分。

For most of computing history, software was designed. Then, for a brief intense period, intelligence was trained​ — a costly, once-per-generation affair requiring data centers and nation-state budgets. Distillation changes the unit of intelligence from a model​ to a transferable state.

在计算机史的大部分时间里，软件是被设计出来的。接着，在一段短暂而剧烈的时期内，智能被训练出来——一件成本高昂、一代一次的事，需要数据中心和国家级预算。蒸馏把智能的基本单元从一个模型改成了一种可迁移的状态。

We are moving from an economy of creation to an economy of transmission.

我们正在从"创造的经济"走向"传递的经济"。

This has precedents. Agriculture did not teach us to photosynthesize; it taught us to propagate​ what already worked. Writing did not invent language; it let language survive its speaker. Distillation is the same move at a new scale: take a capacity that was expensive to produce, and make it cheap to copy.

这有先例。农业没教我们光合作用，它教我们把已然有效的东西繁殖下去。文字没发明语言，它让语言在说话者死后存续。蒸馏是同一动作在新尺度上的重演：取一种生产成本高昂的能力，让它被廉价复制。

But every transmission loses something. A child of two fluent parents may grow up speaking neither language perfectly. A student of a brilliant teacher may master the teacher's gestures and miss the teacher's doubts. The history of distillation is the history of fidelity versus compression — and fidelity has always lost, a little, on purpose.

但每一次传递都会丢失些什么。双语父母的孩子可能两种语言都说不地道。 brilliant 教师的学生可能掌握了教师的姿态，却错过了教师的怀疑。蒸馏史即保真度与压缩的斗争史——而保真度总是有意地、一点点地落败。

The question is whether AI, unlike agriculture and writing, can distill upward​ — whether a student can become more than its teacher. Self-distillation and RL say maybe. The capability-cliff data says not yet. The truth is almost certainly in the middle, and the middle is where the next decade of AI will be fought.

问题在于：AI 能否不同于农业与文字，向上蒸馏——让学生超越教师。自蒸馏与 RL 说"或许"。能力悬崖的数据说"还不行"。真相几乎肯定在中间，而中间正是未来十年 AI 的战场。

In the beginning was the parameter. Then came the token. Now comes the transfer. And the transfer, like all transfers, asks a question it cannot answer: what was lost?

起初是参数，然后是 token。如今是迁移。而这次迁移，像所有迁移一样，提出了一个它自己无法回答的问题：丢掉了什么？

Glossary / 术语表

Author's note.​ The performance figures and deployment numbers cited in this essay come from vendor reports, academic preprints, and industry case studies; they should be treated as illustrative evidence, not a benchmark leaderboard. The underlying research is evolving weekly — the 1-shot OPD and REDI papers discussed here were published within weeks of this writing. Where numbers conflict, prefer primary papers and your own evaluation harness.

作者注：​ 本文引用的性能与部署数字来自厂商报告、学术预印本和行业案例研究，应视为说明性证据，而非基准排行榜。相关研究每周都在演进——本文讨论的 1-shot OPD 与 REDI 论文发表于写作的数周之内。数字冲突时，请优先采信原始论文和你自己的评测框架。

References / 参考来源

腾讯云，《知识蒸馏的基本原理是什么？》

百度智能云，《知识蒸馏：Deepseek模型优化的关键引擎？》

李宗辉、文捷，《大模型知识蒸馏版权侵权之否定》，《南京邮电大学学报（社会科学版）》2026(3)

《少量难题足以蒸馏推理能力：On-Policy Distillation 的数据效率解读》

《知识蒸馏（Knowledge Distillation, KD）详细介绍》

百度智能云，《DeepSeek本地化部署：轻量级蒸馏模型全解析》

Zhinan Hou et al., What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection, arXiv:2609.05198 (2026)

《AI向AI学习，模型蒸馏省下了什么，边界又在哪里？》

《推理成本只付一次？微软把推理能力蒸馏成技能，小模型反超大模型》

Knowledge Distillation, authoritative technical survey

Shuyao Xu et al., Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning, ACL 2026

新浪新闻，《字节跳动严禁蒸馏开源模型》

Shanghai AI Laboratory, Dual-Anchored Policy Distillation / Privilege Illusion, arXiv:2608.01735 (2026)

《DeepSeek-R1模型蒸馏技术原理》

《AI 泡沫什么时候会爆 · 深度研究报告》

Model Distillation, legal and ethical survey

Yumeng Zhang et al., Distribution Corrected Offline Data Distillation for Large Language Models, arXiv:2605.14071 (2026)

《一文读懂后训练：从 RLHF 到 MOPD》

Model Distillation in 2026: How AI Leaders Optimize Performance

《你的下一部手机，可能不需要联网就能跑大模型！》

AWS Documentation, Model Optimization Techniques

《DeepSeek突然更新R1论文：暴增64页，能公开的全公开了》

《2026年至今，前沿大模型都在怎么用蒸馏》

Sebastian Raschka，《从头开始推理》

Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs, arXiv:2609.02548 (2026)

《三重浪潮共振，谁能抢占新一轮AI"红利窗口"？》

Speculative Decoding, production methods survey

爱范儿，《让 Anthropic 破防的「蒸馏」风波》

《D³-MOPD：用训练中的 KL 信号做多教师蒸馏的动态数据调度》

Further reading

• Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network.
• Gou, J. et al. (2020). Knowledge Distillation: A Survey. IJCV.
• Furlanello, T. et al. (2018). Born-Again Neural Networks. ICML.
• Zhang, Y. et al. (2026). Distribution Corrected Offline Data Distillation for LLMs. arXiv:2605.14071.
• DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. (updated version, 64-page release).

继续阅读：Distillation   Pretraining   蒸馏   预训练   AI   智慧   知识蒸馏   distill   离线蒸馏

此文由 怡心湖 编辑，若您觉得有益，欢迎分享转发！：首页 > 常识论 » Distillation Is the New Pretraining 蒸馏即预训练：当 AI 的智慧开始流动