Improving Language Models by Learning Enhancement Automatically
In a groundbreaking development, researchers have proposed a novel approach called Preference-based Iterative Tuning (PIT) that enables large language models (LLMs) to learn self-improvement from human preference data, without the need for explicit prompts. This advancement, detailed in a new paper, could pave the way for more scalable and flexible LLMs that better align with human values in real-world applications.
The PIT approach leverages human preference data instead of prompts for self-improvement. It employs curriculum reinforcement learning, starting with easy-to-improve references then switching to the LLM's own samples. This two-stage process significantly outperforms traditional prompting methods, as shown in comprehensive experiments on real-world dialog datasets and synthetic instruction-following datasets.
Across conditions, PIT improved response quality by 7-34% compared to the original LLM samples, as measured by third-party evaluator models. Furthermore, PIT was able to learn nuanced objectives like making responses more helpful, harmless, or relevant without prompts explicitly defining those criteria.
A key insight behind PIT is that human preference data provides implicit guidance on what constitutes an improvement in quality. The approach uses preferences as a form of reward to guide the model's updates, enabling it to improve alignment with human values and judgments without needing explicit task instructions each time.
Traditional fine-tuning often requires retraining or explicit annotated examples for each new prompt or task. PIT, however, continually adapts, reducing computational overhead and improving scalability in real-world interactive settings.
Moreover, PIT facilitates continual learning and prevents catastrophic forgetting. It incorporates human preferences as regularizing signals to maintain previously learned knowledge while adapting to new preference data, helping the model improve without losing older skills or behaviors.
Notably, PIT also significantly outperforms the prompting method Self-Refine. This work represents an important advance in enabling LLMs to refine themselves without direct human oversight, reducing reliance on human intervention.
In conclusion, the PIT approach offers a promising way forward to learn nuanced goals like improving helpfulness, harmlessness, and accuracy by tapping into the implicit guidance within training data. It represents a significant step towards creating more autonomous and adaptable LLMs that can better serve humanity in various real-world applications.
Technology and artificial-intelligence are integral components of the Preference-based Iterative Tuning (PIT) approach, a novel method proposed by researchers for large language models (LLMs) to learn self-improvement from human preference data. This AI-driven method leverages technology to enable the model to adapt and improve without the need for explicit prompts, thereby reducing reliance on human intervention.
