Tailored Temperature for Student in Knowledge Distillation

作者：Hongjin Chen, Tiantian Zeng, Kai Xiao, Shaozhi Wu, Xingang Liu, Han Su · 年份：2024 · DOI：10.1109/swc62898.2024.00132 · 被引用次数：1 · 研究领域：Experimental Learning in Engineering

In knowledge distillation, the temperature controls the shape of the model’s prediction probability distribution, thereby influencing the transfer of knowledge from the teacher. Most knowledge distillation methods simply fix the temperature, while a few studies employ a dynamic temperature improperly, both of which constrain the further progress of the distilled student model. In this paper, to address this issue, we present a method dubbed TTSD to distill with the temperature tailored for the student, which achieves label smoothing effectively according to the student’s performance. The tailored temperature consists of two parts: a learnable temperature (LT) and an adaptive temperature (AT), each functioning at different stages of student development (junior and senior). LT learns alongside the junior to find the optimal temperature, whereas in the senior stage, AT compensates for LT’s deficiencies through adaptation. Extensive experiments on CIFAR-100 ImageNet-2012, and Describable Textures Dataset (DTD) demonstrate that TTSD can effectively improve existing knowledge distillation frameworks.