This reply links to research on black-box, on-policy distillation. The paper’s Generative Adversarial Distillation approach pairs a student model with a discriminator that learns to distinguish student and teacher responses, providing adaptive feedback without access to the teacher’s internal parameters.
Source: @MrPeterLMorris. This note summarizes the linked material; images belong to their respective creators.