Unlocking the Mysteries of ChatGPT's Language Processing Power

Have you ever wondered how ChatGPT consistently delivers highly relevant information, witty responses, and extremely creative ideas on demand, as if it were an exceptionally knowledgeable and intelligent human being?

The secret lies in its intricate inner workings, which enable it to process, analyze, and generate text in a way that closely resembles human speech patterns. In this blog post, we will explore the technology behind ChatGPT, examining the architecture and foundational elements that make it a remarkable and transformative tool.

Undoubtedly, ChatGPT and other Large Language Model (LLM)-based programs involve a high level of complexity, with advanced mathematical concepts like linear algebra, matrix manipulations, and differential calculus powering their algorithms. This blog post is designed to present these concepts in an accessible manner, allowing curious readers without an extensive background in computer science or mathematics to grasp the fundamentals and gain a solid understanding of how these programs operate.

Below is a simplified process flow for ChatGPT. The first section of this blog post will take you through the important building blocks depicted in this flow.

Key Building Blocks: Tokenization, Embeddings, and Neural Networks

To understand the technology that powers ChatGPT, it's essential to first explore several of its key building blocks, which include tokenization, embeddings, and neural networks. These components form the foundation of ChatGPT's language understanding and generation capabilities, enabling it to process and produce human-like text.

Tokenization and Encoding Computers are fundamentally designed to work with numbers, not language or text. In order for computers to interpret human language, text must first be transfered into a numerical format.

Tokenization is the process of converting raw text into a format that can be directly fed into the LLM. This involves breaking down the text into smaller units, called tokens, which can represent words, subwords, or even a few characters. This approach, as opposed to using entire words, allows the model to efficiently handle various linguistic phenomena, such as morphology (how words are formed) and word composition, and to better deal with rare or out-of-vocabulary words.

You can see tokenization at work in the above diagram. Each color in the lower panel represents a different token. I've included a made up word, which shows how the tokenizer handles unknown vocabulary.

Once tokenized, each word can be directly encoded into a numerical representation. There are several schemes for encoding tokens into numbers, the one that ChatGPT uses is called Byte Pair Encoding (BPE). BPE is a method for segmenting text into subwords based on the frequency of their occurrence in the training data. The result of the tokenization and BPE encoding process is a set of numbers, each one representing a particular token. The diagram below is the result of the BPE encoding for the example sentence.

Token Embeddings Once the token is encoded into a number, it can be embedded into a higher-level representation. Token embeddings are a critical aspect of natural language processing. They enable the model to represent words and phrases as vectors (a series of numbers) in a high-dimensional space, capturing the relationships and similarities between them. This vector representation allows the neural network (specifically the transformers in the case of ChatGPT) to process and manipulate text more effectively, as it can identify patterns and connections based on analyzing the embeddings' properties.

The embedding process captures the relationship between words in the English (and other) languages during the training process. The figure below (referenced here) conceptually describes the word embedding process using two parameters: age (y-axis) and gender (x-axis). (Note that in ChatGPT, for example these words would be embedded as numeric tokens, but for clarity, the diagram just displays the word.)

For example, the word "prince" would be embedded on the "male" side of the "gender" (x-axis) and on the young side of "age" (y-axis) and "king" would be in the "male" and "older" quadrant. In this fashion, similar words (prince, nephew) are encoded near to each other, whereas dissimilar words (computer, princess) are encoded in separate regions. In this way, knowledge inherent in the language can be captured.

LLMs such as ChatGPT use token embeddings in exactly this way. However, instead of using two or three dimensions, these LLMs can use literally thousands - GPT-3 uses 12,228 dimentions. Furthermore, the labels on the 12,228 axes (such as gender and age in the above example) are not defined in advance but discovered during the training process. They don't necessarily correspond in a straightforward way to a single category like gender or age - they are completely abstract. Needless to say, humans cannot possibly conceptualize this many dimensions of abstract categories, and there is no way to grapically represent it. However, it is clear that a great deal of "knowledge" about language structure can be embedded with such a scheme and this partially explains ChatGPT's extensive ability to generate coherent text.

Neural Networks At the core of ChatGPT's engine lies a series of deep neural networks, which are responsible for processing the tokenized input and generating output based on the learned patterns from training data. (The training process will be described later in this blog post.)

A neural network is inspired by how neurons function within the human brain. In the brain, synapses form connections between nerve cells, and when a signal passes through the synapse, it activates the neuron on the other side. Similarly, a computer-based neural network consists of artificial neurons, called nodes, connected to each other in a layered architecture (see diagram).

An input signal is presented at the left side of the diagram. In the case of a computer neural network, the input signal is a number. Neurons in the hidden layers are connected in such a way as to combine these signals, and the resulting calculation is presented at the output (on the right side of the diagram).

For simple neural networks (such as one that is trained to recognize cats in images), the calculation presented at the output would perhaps select a word to go along with the image (such as the word "cat"). In the case of more complex algorithms, such as ChatGPT, there are multiple neural networks involved, and the output would be some abstract formulation that is relevant to the input text. We will examine how these multiple neural networks are wired together into an architecture called a "transformer" later in this post.

The key to encoding the information in any such neural network is that each connection between neurons carries a specific weight - which represents the strength of the signal that is transferred. These weighting parameters are discovered during the training process, which will be covered later in this post.

For example, GPT-3 has 175 billion total parameters, which include the weights and biases of connections between the nodes in its neural networks.

Together, these important building blocks - tokenization, word embedding and neural networks - form the foundation of ChatGPT's capabilities. In the following section, we will examine how these building blocks are connected into a higher level architecture called a transformer.

The Transformer Architecture: Structure, Masked Self-Attention, and Positional Encoding

The transformer architecture is indeed a crucial aspect of ChatGPT, as it forms the foundation for the model's language understanding and generation capabilities. Introduced by Vaswani et al. in 2017 in the groundbreaking paper "Attention is All You Need," the transformer architecture has since become the basis for numerous state-of-the-art natural language processing models. In this section, we will explore the overall structure of the transformer architecture and delve into its subcomponents, such as self-attention and positional encoding

Encoder         Decoder

The "Transformer" Architecture

Encoder is the left side and Decoder is the right side

The transformer architecture, as illustrated in the referenced diagram, primarily consists of two main components: the encoder and the decoder. The encoder processes the input text, while the decoder generates the output text. Both the encoder and the decoder comprise multiple layers, where each layer contains a self-attention mechanism (explained in the next section) and a neural network (explained above).

Different types of LLMs utilize various components of the transformer architecture based on their specific needs. For example, machine translation tasks, such as translating from French to English, require both the encoder and decoder sections of a transformer. In contrast, GPT-based architectures, including ChatGPT, primarily rely on the decoder portion of the architecture (the right-hand side of the diagram). In this case, the transformer encoder stack is not used, but ChatGPT instead employs its own token encoding and embedding processes as described in the previous section.

The already-encoded tokens enter the model one at a time from the bottom right of the diagram (confusingly labeled as "Outputs (shifted right)"). As more tokens are processed and accumulate for each sentence, they are reintroduced at the bottom-right of the diagram, enabling the model to maintain the context of the entire input sequence.

Self Attention ChatGPT's self-attention mechanism is designed to capture the importance of each token in the input relative to others, enabling it to maintain the tokens' context and dependencies across a lengthy input sequence. As each token is processed, there are matrices that are built which describe the importance of, and relationship to, other tokens in the input.

The diagram above illustrates the attention mechanism for the sentence (in token form) "the animal didn't cross the street because it was too tired." The attention head is on the word "it" (as seen shaded in grey on the right side of the diagram) and shows that it is paying most self-attention to "the animal". This is self-attention because it is comparing the sentence to itself. The mechanism clearly establishes that there is a strong relationship between the instance of the word "it" and "the animal."

The neural network, which follows each self-attention mechanism, helps the model learn complex, non-linear relationships between the input and output tokens.

Positional Encoding While the self-attention mechanism is powerful for capturing relationships between tokens, it lacks the ability to recognize the order of tokens in the input sequence. Positional encoding is introduced to remedy this shortcoming. It is a technique used to include information about the position of tokens within the sequence into their embeddings. Positional encodings are added to the input embeddings before they are processed by the self-attention mechanism.

The importance of this step can be illustrated by looking at two example sentences using the exact same set of four words. #1: No, I am good. #2: I am no good. Without positional encoding, the model would be unable to distinguish between what is essentially two opposite sentiments.

Decoding Remember that there is no "encoder" section of the architecture (the left side of the architectural block diagram earlier in this post) that is doing the processing in ChatGPT. So, all the self-attention, encoding and neural network processing is part of the (confusingly named) decoder section of the architecture (right side of the diagram) - which obviously does more than "decoding." The actual process of the decoding of embedded information involves selecting the next word (or token) to add to the string of output text that is being presented to the user. This information is added to the input and output token details, and then fed back into the bottom right of the diagram above.

There are several decoding techniques that can be used to create text that is clear, relevant, and meaningful. GPT-based models, like ChatGPT, usually rely on something called top-k sampling (or variations of it).

Top-k sampling is a decoding method that adds an element of randomness to the token selection process. Instead of always picking the most probable token, top-k sampling chooses a token from a group of the top k most likely candidates. The selection is based on the probability distribution of these candidates, meaning that tokens with higher probabilities have a better chance of being chosen. This process allows for more diverse and creative output compared to more predictable methods such as always picking the word that the transformer has deemed most likely.

The Training Process: Pretraining and Fine-tuning ChatGPT

To create a powerful language model like ChatGPT, a thorough and effective training process is essential. The training process involves multiple stages, including pretraining on a large corpus of text and fine-tuning the model on more specific tasks or datasets. The training processes result in the setting of all the parameters in the model including, for example, neural network weights and biases as well as values for word embedding vectors. In this section, we will explore the training process, focusing on how ChatGPT is pretrained and fine-tuned to achieve its impressive language understanding and generation capabilities.

Backprop To fully understand the inner workings of ChatGPT's training process, it's essential to explore a critical building block of large language models: backpropagation (or backprop). Invented by Dr. Geoffrey Hinton and others around 20 years ago, backprop is a groundbreaking concept that has enabled practical applications of neural networks.

During the training process, a neural network is presented with a problem to solve, for which the correct answer is already known. The network processes the input data and generates an output or prediction. This output is then compared to the correct answer, or the desired output, to calculate the difference between them, referred to as the "error" or "loss." The objective is to minimize this error as much as possible, allowing the neural network to make accurate predictions.

Backprop serves as the mechanism for minimizing this error. The technique works by traversing the network in reverse, starting from the output and moving towards the input. It computes how much each component (or neuron) in the network contributes to the overall error and adjusts the connections (or weights) between these neurons accordingly. This process is repeated iteratively, progressively refining the weights to enhance the network's performance.

In essence, backprop is a vital method in the training of neural networks like ChatGPT, optimizing the connections within the network to minimize errors and improve the model's prediction capabilities.

Pretraining During the pretraining stage, ChatGPT uses backprop to determine values for its billions of parameters, including setting all the weights in the neural network. The model is exposed to vast amounts of text from diverse sources, such as websites, books, and articles. This allows the model to learn the structure of the language, grammar, syntax, and common facts by capturing patterns and relationships within the text.

This learning is typically achieved through self-supervised learning, where the model learns to predict the next token in a sentence given the previous tokens, using a process called language modeling. In each training step, a word is masked out so the model has to guess the masked word. For example, the phrase "bacon and ____" is presented, and the model must predict the missing word. The first guess might be far from the actual masked word, but using backprop, the model learns the error and adjusts the weights in the neural network to make better predictions. The backprop process is then repeated many times with different sentences and masked words.

The pretraining process results in a "base model," where, in the case of GPT-3, 175 billion parameters (neural network weights, word embedding locations, etc.) have been established. At this point, the model can understand and generate text but may not be specialized in any specific domain or task. This base model serves as a foundation for further customization through fine-tuning.

Fine-tuning Once the base model has been pretrained, it can be fine-tuned on more specific tasks or datasets to improve its performance in those areas. Fine-tuning involves training the model on a smaller, task-specific dataset using supervised learning, where input-output pairs are provided to guide the model's learning process.

For instance, if we want ChatGPT to perform well in a customer support context, we can fine-tune it on a dataset containing customer support conversations. The model is exposed to examples of correct responses in various support scenarios, and its parameters are adjusted to minimize the difference between the generated output and the desired output.

The net result of the fine tuning step is some tweaks to the 175 billion parameters (in GPT-3's case), so the model is better adapted to a specific area.

Reinforcement Learning from Human Feedback (RLHF)

Reinforcement Learning from Human Feedback (RLHF) is a crucial technique used to improve the performance of ChatGPT by generating higher-quality and safer responses. It combines reinforcement learning with human feedback to optimize the model's behavior more effectively. In this section, we explore the RLHF process and its role in shaping ChatGPT's capabilities.

Reinforcement Learning Reinforcement learning (RL) is a type of machine learning in which an agent learns to make decisions by interacting with an environment. The agent's objective is to learn an optimal strategy, or policy, that allows it to maximize cumulative rewards over time. In RL, the agent receives feedback in the form of rewards or penalties after taking actions in the environment. This feedback helps the agent identify which actions lead to desirable outcomes and adjust its behavior accordingly.

Collecting Human Feedback For ChatGPT, human feedback (the "HF" in RLHF) determines the rewards and/or penalties that the reinforcement learning component uses. Although it is rumored that OpenAI uses thousands of contractors, possibly located in places like Kenya, to provide this feedback, the exact number and location of evaluators remain unconfirmed.

The first step in the RLHF process is to gather human feedback on the model's performance. This typically involves human evaluators rating the quality of the model's responses in various situations. These evaluators follow guidelines to assess aspects such as relevance, coherence, and safety of the generated text. The collected feedback serves as a valuable source of information for guiding the model's learning process.

Training with Proximal Policy Optimization After collecting human feedback, it is used to create a reward function that quantifies the quality of the model's responses. The model is then fine-tuned using an algorithm called Proximal Policy Optimization (PPO).

PPO iteratively refines the model's behavior by further tweaking its parameters based on the reward function. The goal is to maximize the reward, which corresponds to generating higher-quality and safer responses, as indicated by the human feedback. By adjusting the model's parameters using PPO, the model learns to generate text that more closely aligns with the desired behavior defined by the evaluators.

Iterative Improvement The RLHF process is typically performed iteratively, with multiple rounds of human feedback collection and model fine-tuning using PPO. Each iteration helps improve the model's performance, as it continuously learns from the evaluators' assessments and refines its behavior to better align with the desired output characteristics.

The Inference Phase: Putting ChatGPT to Work

After the extensive process of pretraining, fine-tuning, and reinforcement learning from human feedback (RLHF), ChatGPT is ready to be deployed and utilized for various tasks and applications. The inference phase, also known as the model's deployment or serving phase, is when ChatGPT generates output in response to user inputs, leveraging its advanced language understanding and generation capabilities. In this section, we discuss the key aspects of the inference phase and how ChatGPT is put to work.

Input Processing The first step in the inference phase is processing the user input. The input text is tokenized and encoded, following the same tokenization and word embedding procedures used during the model's training process. Positional encodings are also added to the input tokens to preserve their order within the sequence.

Decoding and Generating Output Once the input has been processed, it is fed into the pretrained, fine-tuned, and RLHF-enhanced ChatGPT model. The model uses the transformer architecture to process the input and generate output, employing, for example, top-k sampling, as discussed in the section above.

Post-processing and Output Delivery The generated output tokens are then converted back into human-readable text through a process called detokenization. Any necessary post-processing, such as removing any special tokens, is performed before delivering the final output to the user.

Monitoring and Continuous Improvement During the inference phase, it's essential to monitor the model's performance and gather feedback on its generated responses. This feedback can be used to further improve the model through additional rounds of fine-tuning and RLHF. By continuously iterating on the model and refining its behavior, ChatGPT can be adapted and optimized for an ever-growing range of tasks and applications.

The Future of ChatGPT and AI-driven Language Models

The rapid advancements in natural language processing, as exemplified by ChatGPT, have opened up a world of possibilities for AI-driven language models. As these models continue to evolve, they will increasingly impact various industries, applications, and aspects of our daily lives. In this section, we explore the future of ChatGPT and AI-driven language models, focusing on potential advancements, applications, and challenges.

Advancements in Model Architecture and Training Techniques The development of more sophisticated model architectures, such as the transformer architecture used in ChatGPT, has been instrumental in driving progress in natural language processing. In the future, we can expect further innovations in model architecture and training techniques, enabling even more powerful and efficient language models. These advancements may include novel architectures that better capture the complexities of human language.

Expanding Applications As AI-driven language models become more advanced, they will find applications in a growing range of domains, including customer support, content creation, education, healthcare, and more. The versatility of these models, combined with their ability to understand and generate human-like text, will make them valuable tools for tasks such as conversational AI, text summarization, translation, sentiment analysis, and many others.

Addressing Ethical and Safety Concerns The increasing capabilities of AI-driven language models also raise important ethical and safety concerns, such as the potential for generating harmful or biased content. Ensuring that these models are used responsibly and safely will be a critical aspect of their future development. This may involve refining techniques to better align the models with human values, as well as the development of new methods for mitigating biases and controlling the output of these models.

The future of ChatGPT and AI-driven language models is filled with potential for growth, innovation, and widespread impact across various domains. But, it has challenges. It is important to understand the technology to better use (and not misuse) it.