Perplexity Improves AI Self-Distillation: Training Not Only Students, but Also Teachers
2026-10-22 23:06:20
According to CoinMeta, Perplexity proposed a new method in the self-distillation training of AI, which not only trains students but also teachers. In this process, the same model plays both the roles of student and teacher; the student solves problems independently, while the teacher can see the reference answers in advance and provide step-by-step token guidance. The two roles share parameters, eliminating the need to run a larger teacher model separately, thus obtaining more detailed training signals. Methods like the one proposed by UCLA and others retain the reward for the final result, allowing only those problem-solving processes that perform above the group average to receive distillation guidance. The distillation weight is gradually reduced towards the end of training to avoid excessive imitation that could hinder the model's exploration. By combining this with the newly proposed JOLT method of Perplexity and NVIDIA, the teacher uses additional information to solve problems independently and receives rewards, while KL is used to constrain the output differences between teachers and students in terms of diversity. In the Qwen3-0.6b math test, the accuracy rate reached 72.14%, which is higher than the 61.20% of previous self-distillation methods.
Source:Internet
This content is for market information only and does not constitute investment advice.
Follow WalletJYS official accounts to stay updated

Hot Articles
Refresh

What Are Crypto Airdrops? Top 5 Projects to Watch in Nov 2026
5h ago

ZEC Price Today: $1,342 Surge on NU7, Can It Break $1,295?
11h ago

Bitcoin Price Today: ETF $527M Inflow Rebound or Bull Trap?
10-21 16:51

US Memory Chip Stocks Surge: Can SanDisk's $70B Rally Last?
10-20 12:24

How to Get US Stock News in China? 4 Free Channels
10-16 19:24



