Perplexity AI unveils JOLT: A new method for advanced LLM training
Perplexity AI has introduced JOLT, a novel post-training method designed to significantly improve how large language models (LLMs) learn. Unlike traditional reinforcement learning methods like GRPO, which only provide a single reward at the end of a task, JOLT utilizes a unique teacher-student dynamic to give detailed, step-by-step feedback.
In this setup, a single model acts as both the teacher and the student. The teacher is trained with extra information, such as verified solutions, allowing it to provide granular, per-token feedback that guides the student's learning process. This self-distillation approach is far more informative than previous methods, helping the student learn exactly where it went wrong.
JOLT further enhances the learning process by having the teacher train on outcome rewards from its own attempts while simultaneously penalizing itself if its predictions deviate from the student's. The results are highly impressive: JOLT+ outperformed GRPO across five diverse benchmarks, including tasks related to math, coding, and tool use, achieving a notable 9.9 points on AppWorld.
Why it matters
- —It introduces a highly efficient, advanced training paradigm (JOLT) that improves model performance significantly.
- —By providing per-token feedback, it overcomes the limitations of older reinforcement learning methods (like GRPO).
- —The strong performance across diverse benchmarks (math, coding, tool use) validates the method's broad applicability.
Key facts
- JOLT is a post-training method where one model serves as both teacher and student.
- It provides detailed, per-token feedback, which is superior to single-reward systems like GRPO.
- JOLT+ outperformed GRPO across five benchmarks, including AppWorld (9.9 points).
- The teacher component is trained on outcome rewards and penalizes divergence from the student's predictions.
The full text is in the original source. Here we provide a brief summary and key facts.