Introduction of Generative AI with Large Language Models (3): Reinforcement Learning and LLM-Powered Applications
In the previous section, we explored various techniques for fine-tuning LLMs. Building on that, incorporating human feedback is an effective method to enhance the 3H qualities (helpfulness, honesty, and harmlessness) of generated completions. This approach also helps in reducing toxicity and minimizing the generation of incorrect information.
In this section, we will discuss RLHF (Reinforcement Learning from Human Feedback), a popular technique for fine-tuning LLMs using human feedback. Additionally, we’ll cover a few optimization methods for deploying LLM-powered applications and explore the building blocks for creating these applications.
What is Reinforcement Learning?
Reinforcement learning is a type of machine learning where an agent learns to make decisions to achieve a specific goal by taking actions in an environment, with the objective of maximizing a cumulative reward. The agent continually learns from its experiences by taking actions, observing the resulting changes in the environment, and receiving rewards or penalties based on the outcomes. Through this iterative process, the agent refines its strategy to make better decisions and increase its chances of success.
Mechanism Behind RLHF
Initially, we select an instruct LLM with good performance in specific tasks such as summarization and Q&A. This LLM generates multiple completions from prompt samples. Human labelers then score these completions based on criteria like helpfulness, honesty, and harmlessness. Clear criteria are essential to ensure high-quality human feedback. The human rankings are converted into a pairwise training dataset, which is used to train a reward model.
The reward model, often a language model like BERT, uses supervised learning to predict the preferred completion, outputting a reward value. Higher reward values indicate more aligned responses. This reward value is then fed into a reinforcement learning algorithm to update the LLM’s weights, guiding it to generate more aligned, higher reward responses.
Proximal Policy Optimization (PPO) in RLHF
PPO is a popular RL algorithm used in RLHF. It optimizes a policy within a small, bounded region to maximize the reward, ensuring that the updated LLM remains close to the original version. A crucial concept in PPO is Kullback-Leibler Divergence (KL-Divergence), which measures the difference between two probability distributions. In PPO, KL-Divergence helps ensure that the updated policy does not deviate too much from the original policy, preventing drastic changes that could negatively affect learning.
Techniques for Model Optimization Before Deployment
- Distillation: Train a smaller student model from a larger teacher model. Use the student model for inference to lower storage and compute requirements.
- Quantization: Transform a model’s weights to lower precision (e.g., 16-bit floating point or 8-bit integer) to reduce memory footprint.
- Pruning: Remove redundant model parameters that contribute little to performance, focusing on weights close to zero.
Lifecycle of a Generative AI Project
- Pre-training: Requires high expertise and can take from days to months.
- Prompt engineering: Requires less technical expertise and no additional model training.
- Prompt tuning and fine-tuning: Varies based on use cases, performance goals, and compute budget. Fine-tuning can be completed in a day with a small training dataset.
- RLHF: Similar in time to fine-tuning if using an existing reward model. Training a reward model from scratch can take longer due to the effort required for gathering human feedback.
- Compression, optimization, deployment
Limitations of LLMs
LLMs struggle with math operations and complex reasoning, often providing incorrect answers due to knowledge cut-offs or hallucinations. To mitigate these issues, three frameworks are introduced:
- RAG (Retrieval Augmented Generation): Uses external data sources at inference time to enhance responses.
- PAL (Program-aided Language Models): Pairs LLMs with external code interpreters to carry out calculations.
- ReAct (Synergizing Reasoning and Action in LLMs): Integrates verbal reasoning and interactive decision-making to improve performance and interpretability.
Building Blocks for LLM-Powered Applications
- Infrastructure Layer: Comprises compute, storage, and network resources, available on-premises or as on-demand services.
- LLM Layer: Includes foundation models and task-specific models deployed on suitable infrastructure. May involve real-time interaction and external data sources for information retrieval.
- Additional Tools and Frameworks: Tools like LangChain, model hubs, and frameworks for techniques such as PAL and ReAct.
- User Interface Layer: A website or REST API for consuming the application, including necessary security components.
The model is just one part of building a comprehensive generative AI application, which also requires robust infrastructure, tools, and user interfaces.