An Overview of Transformer-based Language Models
In this article, we focus on transformer-based models that address previous limitations. We’ll explore the attention mechanism and transformer components and how they’re applied in models like USE, BERT, and GPT.
Attention Mechanism and Transformers
Attention mechanisms enable models to make predictions by considering the entire input and selectively focusing on specific parts. This focus is guided by learned weights. For example, in the sentence “The fox saw a rabbit. It was very hungry so it tried to grab it but it dodged just in time,” attention helps determine which “it” refers to the rabbit or the fox.
Transformers use an encoder-decoder structure leveraging the attention mechanism. The encoder processes the input by selectively focusing on different parts, passing these encodings to the decoder. Figure 1 shows a simplified encoder transformer architecture. Words are first converted to numbers via methods like one-hot encoding or Word2Vec embeddings. This sequence is fed to a self-attention layer, which uses parts of the input to predict another part. The results are passed to a feed-forward neural network, creating context-aware encoded information for the decoder.
The decoder is similar but includes an encoder-decoder attention layer to focus on important input parts. Unlike recurrent neural networks (e.g., LSTMs), transformers read all input words simultaneously, allowing for parallelization and training on larger corpora.
Universal Sentence Encoder (USE)
USE models address two main limitations: predicting the next/previous word limits application in downstream tasks, and word embeddings fail to capture contextual information of entire sentences. Google Research introduced USE models using transformers and Deep Averaging Networks (DAN) optimized for sentence embeddings across multiple tasks.
The transformer-based USE model stacks six transformer encoders, tokenizing inputs and computing outputs sequentially. The final outputs form the sentence embedding. DAN, on the other hand, averages vector representations of input words and their bigrams, passing them through feed-forward layers to form the sentence embedding.
USE models employ multi-task learning, combining unsupervised and supervised datasets like Wikipedia, web news, and discussion forums. They are trained on tasks such as skip-thought (predicting surrounding sentences), conversational input-response, and natural language inference (classifying sentence pairs as neutral, entailment, or contradiction). These models are fine-tuned for specific tasks like sentiment analysis and classification.
Despite excelling at sentence-level tasks, USE models are less effective for general text generation, which we’ll explore with other models.
Generative Pretraining Transformer (GPT)
OpenAI’s GPT, introduced in 2018, was groundbreaking for pretraining a general language model for various tasks. GPT uses a stack of 12 decoders, encoding input sequences with Byte Pair Encoding (BPE) and position encoding. It was trained on the BooksCorpus dataset for text prediction and classification.
GPT is fine-tuned on tasks like natural language inference, question answering, semantic similarity, and classification, outperforming previous models. However, its unidirectional nature is a limitation.
Bidirectional Encoder Representations from Transformers (BERT)
BERT, introduced by Google AI, is an encoder-based model trained bidirectionally. BERT_BASE is comparable in size to GPT, while BERT_LARGE has 340M parameters. It uses WordPiece tokenization, segment embeddings, and position embeddings. BERT is pretrained on masked language model and next sentence prediction tasks using datasets like BooksCorpus and Wikipedia.
BERT outperforms GPT on various tasks due to its bidirectional nature. Variants like ALBERT (light-weighted) and RoBERTa (optimized) offer different trade-offs. BERT’s bidirectionality improves performance but struggles with understanding negation.
GPT-2 and Task Conditioning
GPT-2, OpenAI’s 2019 model, varies in size, with the largest being 10x GPT’s size. Trained on 40 GB of Reddit data, GPT-2 uses task conditioning, producing different outputs based on task prompts. It outperforms BERT in most tasks but shows limitations in text summarization.
GPT-3
GPT-3, introduced in 2020, has 175B parameters and shares GPT-2’s architecture with more layers. Trained on a mix of datasets, GPT-3 excels in few-shot learning, producing coherent text and code. However, it struggles with long text coherence and repetition.
Conclusion
We reviewed transformer-based models like USE, GPT, and BERT, highlighting their advancements and limitations. While transformative for natural language processing, these models still face challenges, such as handling new concepts and domain-specific text analysis.