INTERSPEECH2021

Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification

Code framework for FluentAI database.

An overview of the proposed STD model: (a) overall architecture and different loss computation, (b) knowledge distillation losses (i.e., computation of $Loss_{att}$ and $Loss_{hid}$) between two transformer layers of student and teacher model expanded from dotted box in (a).