Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
Hien Dang, Pratik Patil et al.
TLDR: This paper demonstrates that self-distillation can significantly improve ridge regression performance by optimally mixing teacher predictions, providing precise asymptotic analyses and a practical one-shot tuning method.