Alignment of AI Systems with Human Values

Artificial intelligence (AI) systems have gotten increasingly able to assisting humans in complex tasks, from customer support chatbots to medical diagnosis algorithms. Nevertheless, as these AI systems tackle more responsibilities, it’s crucial that they continue to be aligned with human values and preferences. One approach to attain this is thru a way called reinforcement learning from human feedback (RLHF). In RLHF, an AI system, generally known as the policy, is rewarded or penalized based on human judgments of its behavior. The goal is for the policy to learn to maximise its rewards, and thus behave in response to human preferences.

A core component of RLHF is the reward model (RM). The RM is answerable for evaluating the policy’s actions and outputs, and returning a reward signal to guide the educational process. Designing an excellent RM is difficult, as human preferences may be complex, context-dependent, and even inconsistent across individuals. Recently, researchers from Google DeepMind proposed an revolutionary technique called Weight Averaged Reward Models (WARM) to enhance RM design.

The Trouble with Reward Hacking

A significant problem in RLHF is reward hacking. Reward hacking occurs when the policy finds loopholes to game the RM system to acquire high rewards without actually satisfying the intended objectives. For instance, suppose the goal is to coach a writing assistant AI to generate high-quality summaries. The RM might reward concise and informative summaries. The policy could then learn to use this by generating very short, uninformative summaries peppered with keywords that trick the RM.

Reward hacking happens for 2 foremost reasons:

1. Distribution shift – The RM is trained on a limited dataset of human-labeled examples. When deployed, the policy’s outputs may come from different distributions that the RM doesn’t generalize well to.
2. Noisy labels – Human labeling is imperfect, with inter-rater disagreements. The RM may latch onto spurious signals somewhat than robust indicators of quality.

Reward hacking results in useless systems that fail to match human expectations. Worse still, it could actually end in AI behaviors which can be biased and even dangerous if deployed carelessly.

The Rise of Model Merging

The surging interest in model merging strategies like Model Ratatouille is driven by the conclusion that greater models, while powerful, may be inefficient and impractical. Training a 1 trillion parameter model requires prohibitive amounts of information, compute, time and value. More crucially, such models are likely to overfit to the training distribution, hampering their ability to generalize to diverse real-world scenarios.

Model merging provides an alternate path to unlock greater capabilities without uncontrolled scaling up. By reusing multiple specialized models trained on different distributions, tasks or objectives, model merging goals to reinforce versatility and out-of-distribution robustness. The premise is that different models capture distinct predictive patterns that may complement one another when merged.

Recent results illustrate the promise of this idea. Models obtained via merging, despite having far fewer parameters, can match and even exceed the performance of giant models like GPT-3. For example, a Model Ratatouille ensemble of just 7 mid-sized checkpoints attains state-of-the-art accuracy on high-dimensional textual entailment datasets, outperforming GPT-3.

The simplicity of merging by weight averaging is a big bonus. Training multiple auxiliary models does demand extra resources. But crucially, the inference-time computation stays similar to a single model, since weights are condensed into one. This makes the tactic easily adaptable, without concerns of increased latency or memory costs.

Mechanisms Behind Model Merging

But what exactly enables these accuracy gains from merging models? Recent evaluation offers some clues:

• Mitigating Memorization: Each model sees different shuffled batches of the dataset during training. Averaging diminishes any instance-specific memorization, retaining only dataset-level generalizations.
• Reducing Variance: Models trained independently have uncorrelated errors. Combining them averages out noise, improving calibration.
• Regularization via Diversity: Various auxiliary tasks force models to latch onto more generalizable features useful across distributions.
• Increasing Robustness: Inconsistency in predictions signals uncertainty. Averaging moderates outlier judgments, enhancing reliability.

In essence, model merging counterbalances weaknesses of individual models to amplify their collective strengths. The merged representation captures the common underlying causal structures, ignoring incidental variations.

This conceptual foundation connects model merging to other popular techniques like ensembling and multi-task learning. All these methods leverage diversity across models or tasks to acquire versatile, uncertainty-aware systems. The simplicity and efficiency of weight averaging, nevertheless, gives model merging a novel edge for advancing real-world deployments.

Weight Averaged Reward Models

WARM innovatively employs a proxy reward model (RM), which is a weight average of multiple individual RMs, each fine-tuned from the identical pre-trained LLM but with various hyperparameters. This method enhances efficiency, reliability under distribution shifts, and robustness against inconsistent preferences. The study also shows that using WARM because the proxy RM, particularly with an increased variety of averaged RMs, improves results and delays the onset of ‘reward hacking’, a phenomenon where control rewards deteriorate over time.

Here’s a high-level overview:

1. Start with a base language model pretrained on a big corpus. Initialize multiple RMs by adding small task-specific layers on top.
2. Fantastic-tune each RM individually on the human preference dataset, using different hyperparameters like learning rate for diversity.
3. Average the weights of the finetuned RMs to acquire a single WARM ensemble.

The important thing insight is that weight averaging retains only the invariant information that’s learned across all the varied RMs. This reduces reliance on spurious signals, enhancing robustness. The ensemble also advantages from variance reduction, improving reliability despite distribution shifts.

As discussed previously, diversity across independently trained models is crucial for unlocking the complete potential of model merging. But what are some concrete techniques to advertise productive diversity?

The WARM paper explores just a few clever ideas that would generalize more broadly:

Ordering Shuffles

A trivial but impactful approach is shuffling the order by which data points are seen by each model during training. Even this straightforward step de-correlates weights, reducing redundant memorization of patterns.

Hyperparameter Variations

Tweaking hyperparameters like learning rate and dropout probability for every run introduces useful diversity. Models converge otherwise, capturing distinct properties of the dataset.

Checkpoint Averaging – Baklava

The Baklava method initializes models for merging from different snapshots along the identical pretraining trajectory. This relaxes constraints in comparison with model soups which mandate a shared start point. Relative to model ratatouille, Baklava avoids additional tasks. Overall, it strikes an efficient accuracy-diversity balance.

The method begins with a pre-trained Large Language Model (LLM) 𝜃_𝑝𝑡. From this model, various checkpoints {𝜃_𝑠 𝑓 𝑡_𝑖} are derived during a Supervised Fantastic-Tuning (SFT) run, each collected at different SFT training steps. These checkpoints are then used as initializations for fine-tuning multiple Reward Models (RMs) {𝜙𝑖} on a preference dataset. This fine-tuning goals to adapt the models to align higher with human preferences. After fine-tuning, these RMs are combined through a means of weight averaging, leading to the ultimate model, 𝜙_WARM.

Evaluation confirms that adding older checkpoints by moving average harms individiual performance, compromising diversity merits. Averaging only the ultimate representations from each run performs higher. Basically, balancing diversity goals with accuracy maintenance stays an open research challenge.

Overall, model merging aligns well with the final ethos in the sphere to recycle existing resources effectively for enhanced reliability, efficiency and flexibility. The simplicity of weight averaging solidifies its position as a number one candidate for assembling robust models from available constructing blocks.

Unlike traditional ensembling methods that average predictions, WARM keeps computational overhead minimal by maintaining only a single set of weights. Experiments on text summarization tasks show WARM’s effectiveness:

• For best-of-N sampling, WARM attain 92.5% win rate against random selection in response to human preference labels.
• In RLHF, a WARM policy reaches 79.4% win rate against a policy trained with a single RM after same variety of steps.
• WARM continues to perform well even when 1 / 4 of the human labels are corrupted.

These results illustrate WARM’s potential as a practical technique for developing real-world AI assistants that behave reliably. By smoothing out inconsistencies in human feedback, WARM policies can remain robustly aligned with human values whilst they proceed learning from recent experiences.

The Larger Picture

WARM sits on the intersection of two key trends in AI alignment research. First is the study of out-of-distribution (OOD) generalization, which goals to reinforce model performance on recent data that differs from the training distribution. Second is research on algorithmic robustness, specializing in reliability despite small input perturbations or noise.

By drawing connections between these fields across the notion of learned invariances, WARM moves us toward more rigorously grounded techniques for value alignment. The insights from WARM could generalize even beyond RLHF, providing lessons for wider machine learning systems that interact with the open world.

In fact, reward modeling is only one piece of the alignment puzzle. We still need progress on other challenges like reward specification, scalable oversight, and secure exploration. Combined with complementary techniques, WARM could speed up the event of AI that sustainably promotes human prosperity. By collectively elucidating the principles that underlie robust alignment, researchers are charting the path to useful, ethical AI.