Lecture
A generative pre-trained transformer ( GPT ) is a type of large language model (LLM), widely used in generative chatbots based on artificial intelligence. GPTs are based on a deep learning architecture called the transformer. They are pre-trained on large datasets of unlabeled content and are able to generate new content.
OpenAI was the first to apply generative pre-training to the transformer architecture, introducing the GPT-1 model in 2018. Since then, the company has released many larger GPT models. The ChatGPT chatbot, released in late 2022 (using GPT-3.5 ), was joined by many competing chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek, and Claude.
GPTs are mainly used to generate text, but can be trained to generate other types of data. For example, GPT-4o can process and generate text, images, and audio. To improve performance on complex tasks, some GPTs, such as OpenAI o3, spend more computation time analyzing the problem before generating a result and are called reasoning models. In 2025, GPT-5 was released with a router that automatically chooses whether to use a faster or a slower reasoning model depending on the task at hand.
The original GPT model
In the 2010s, improved machine learning algorithms, more powerful computers, and an increase in the volume of digitized material led to an AI boom.
Separately worth noting is the concept of generative pre-training (GP), which is a long-established technique in machine learning. GP is a form of self-supervised learning in which a model is first trained on a large unlabeled dataset (the “pre-training” stage) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the “fine-tuning” stage).
The transformer deep learning architecture is the core technology behind GPT (Global Transforming Teaching). Developed by researchers at Google, it was introduced in a paper titled “Attention Is All You Need”, published on June 12, 2017. The transformer architecture solved many of the performance problems associated with older recurrent neural networks (RNNs) for natural language processing (NLP). The use of the attention mechanism in this architecture allows models to process entire sequences of text at once, which enables the training of much larger and more complex models. Since 2017, transformer-based NLP systems have been able to process, analyze, organize, connect, compare, and summarize texts, as well as answer questions arising from textual input.
On June 11, 2018, researchers and engineers at OpenAI published a paper titled “Improving Language Understanding by Generative Pre-Training”, which introduced GPT-1, the first GPT model. It was designed as a large language model based on transformers, which used generative pre-training (GP) on BookCorpus, a diverse text corpus, followed by discriminative fine-tuning to focus on specific language tasks. This semi-supervised approach was regarded as a breakthrough. Previously, the best neural models in natural language processing (NLP) typically used supervised learning on large amounts of manually labeled data — training a large language model using this approach would have been prohibitively expensive and time-consuming.
On February 14, 2019, OpenAI introduced GPT-2, a larger model capable of generating coherent text. Created as a direct scale-up of its predecessor, it had both its parameter count and dataset size increased by a factor of 10. GPT-2 has 1.5 billion parameters and was trained on WebText, a 40-gigabyte dataset containing 8 million web pages. Citing risks of malicious use, OpenAI opted for a “staged release”, initially publishing smaller versions of the model before releasing the full 1.5-billion-parameter model in November.
On February 10, 2020, Microsoft introduced its Turing natural language generation model, which it called “the largest language model ever published, with 17 billion parameters”. The model outperformed all previous language models on a variety of tasks, including text summarization and question answering.
On May 28, 2020, OpenAI introduced GPT-3, a model with 175 billion parameters, trained on a larger dataset than GPT-2. This marked significant progress in few-shot and zero-shot learning capabilities. Given only a few examples, it could perform a variety of tasks for which it had not been specifically trained.
After the release of GPT-3, OpenAI began using reinforcement learning from human feedback (RLHF) to more closely align model behavior with human preferences. This led to the development of InstructGPT, an improved version of GPT-3. OpenAI further refined InstructGPT to create ChatGPT, OpenAI's flagship chatbot product, launched on November 30, 2022. ChatGPT was initially based on GPT-3.5, but later switched to the GPT-4 model, released on March 14, 2023. GPT-4 has also been integrated into parts of several applications, including Microsoft Copilot, GitHub Copilot, Snapchat, Khan Academy, and Duolingo.
ChatGPT's enormous popularity spurred widespread development of competing GPT-based systems by other organizations. EleutherAI released a series of open-weight models, including GPT-J in 2021. Other major technology companies later developed their own GPT models, such as PaLM and Gemini from Google, as well as Llama from Meta AI.
Many subsequent GPT models were trained to be multimodal (able to process or generate multiple types of data). For example, GPT-4o can both process and generate text, images, and audio. In addition, GPT models such as o3 and DeepSeek R1 have been trained using reinforcement learning to generate a multi-step chain of thought before arriving at a final answer, which helps solve complex problems in fields such as mathematics.
On August 7, 2025, OpenAI released GPT-5, which includes a router that automatically chooses whether to use a faster or a slower reasoning model depending on the task.
A foundation model is an AI model trained on large amounts of data, which allows it to be adapted to a wide range of downstream tasks.
The most recent model in OpenAI's GPT-n series is GPT-5.
Other similar models include PaLM from Google, a broad foundation model that has been compared to GPT-3 and has been made available to developers through an API, and GPT-JT from Together, which is reported to be the closest-performing open-source alternative to GPT-3 (and is derived from earlier open-source GPTs ). Meta AI (formerly Facebook ) also has a generative transformer-based foundation large language model known as LLaMA.
Foundation GPTs can also use modalities other than text for input and/or output. GPT-4 is a multimodal LLM capable of processing text and image input (although its output is limited to text). With regard to multimodal output, some models based on generative transformers are used for text-to-image technologies, such as diffusion and parallel decoding. Such models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.
The computational and memory requirements of transformer-based models increase significantly as their size and input sequence length grow. The standard self-attention mechanism has quadratic complexity with respect to input sequence length, as described in “Attention Is All You Need”.
Researchers have proposed a number of efficiency improvements, such as sparse attention mechanisms and memory-efficient architectures, which reduce computational costs while maintaining longer context windows. Models such as BigBird, Reformer, and FlashAttention demonstrate structured attention patterns or optimized computations to improve scalability and efficiency.
This has helped large language models process long input sequences efficiently, with lower memory and computational costs during both training and inference.
A foundation GPT model can be further adapted to create more targeted systems, focused on specific tasks and/or domains. Methods of such adaptation may include additional fine-tuning (beyond that performed for the foundation model), as well as certain forms of prompt engineering.
An important example of this is fine-tuning models to follow instructions, which is, of course, a fairly broad task, but more targeted than the foundation model. In January 2022, OpenAI introduced “InstructGPT” — a series of models that were fine-tuned to follow instructions using a combination of supervised learning and reinforcement learning from human feedback (RLHF) on GPT-3 foundation language models. The advantages of this over the foundation models were higher accuracy, fewer negative/toxic sentiments, and generally better alignment with user needs. OpenAI therefore began using this as the basis for its API service offerings. Other instruction-tuned models were released by other developers, including a fully open-source version.
Another (related) type of task-specific model is chatbots, which carry on human-like dialogue. In November 2022, OpenAI launched ChatGPT — an online chat interface powered by an instruction-tuned language model trained similarly to InstructGPT. They trained this model using RLHF, with human AI trainers providing dialogues in which they played both the user and the AI, and mixed this new dataset of dialogues with the InstructGPT dataset to create a conversational format suitable for a chatbot. Other major chatbots currently include Microsoft's Bing Chat, which uses OpenAI's GPT-4 (as part of a broader close partnership between OpenAI and Microsoft), and Google's competing Gemini chatbot (originally based on the LaMDA family of language models, trained for dialogue, with plans to switch to PaLM ).
Another type of task for which GPTs can be used is the meta-task of generating one's own instructions, for example, devising a series of prompts to “itself” in order to be able to accomplish a more general goal set by a human user. This is known as an AI agent, and specifically a recursive one, since it uses the results of its previous self-instructions to help it form subsequent prompts; the first major example of this was Auto-GPT (which uses OpenAI's GPT models), and others have since been developed.
Chain-of-thought reasoning is a prompting method in which a language model generates intermediate reasoning steps before arriving at a final answer. This approach has been shown to improve performance on tasks requiring multi-step reasoning, such as solving mathematical problems and logical inference.
By providing step-by-step explanations, the model is able to break down complex problems into smaller, manageable parts. This method is often used in combination with the “few-shot” method, in which examples demonstrating the reasoning process are used as input.
GPT systems can be tailored to specific domains or fields of activity. Below are some examples of such models and applications:
Sometimes domain specificity is achieved through software plugins or add-ons. For example, several different companies have developed special plugins that interact directly with OpenAI's ChatGPT interface, and Google Workspace offers add-ons such as “GPT for Sheets and Docs”, which is reported to help leverage spreadsheet functionality in Google Sheets.
Scaling laws describe empirical relationships between the performance of large language models and factors such as model size, dataset size, and computational resources. Empirical studies have shown that performance generally follows approximate power laws as these factors increase.
Larger models trained on more data tend to exhibit lower training loss and better generalization ability. More recent work shows that performance is determined not only by the number of parameters, but also by how model size, data, and computational resources are balanced during training.
These observations have influenced the development of subsequent GPT models, in particular decisions regarding architecture design, dataset composition, and training strategies.
Emergent abilities refer to abilities that appear in large language models only once they reach a certain scale, and are absent in smaller versions of the same models. These abilities are considered “emergent” because they arise as model size, training data, and computational resources increase.
Examples of emergent abilities include multi-step reasoning, in-context learning (the ability to perform tasks based on examples provided in prompts, without additional training), and improved results on complex tests of language knowledge and logical reasoning.
Research shows that these abilities do not scale linearly, but appear once models exceed certain thresholds of size and training scale.
This phenomenon has influenced the development of larger GPT models and has helped improve their effectiveness across a wide range of tasks.
OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, stated in 2023 that “GPT” should be regarded as an OpenAI brand. In April 2023, OpenAI revised its brand usage rules in its terms of service, stating that other companies using its API to run their AI services would no longer be able to include “GPT” in such names or branding. In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notices did not contain explicit legal claims (such as allegations of trademark infringement or cease-and-desist demands ). As of November 2023, OpenAI still prohibits its API licensees from naming their own products using "GPT", but has begun giving its ChatGPT Plus subscribers the ability to create "custom versions of ChatGPT", called GPTs, on the OpenAI website. OpenAI's terms of service state that its subscribers may use "GPT" in the names of these products, although this is "discouraged".
In connection with this, OpenAI applied to the United States Patent and Trademark Office (USPTO) for trademark registration in the United States for the term “GPT” in the field of AI. OpenAI tried to expedite review of its application, but the USPTO denied this request in April 2023. In May 2023, the USPTO responded to the application, acknowledging that “GPT” is both a descriptive and a generic term. As of November 2023, OpenAI continued to pursue its position.
For any kind or extent of trademark protection in the United States, OpenAI would need to prove that the term is genuinely “distinctive” of its specific offerings, beyond being a broader technical term for a type of technology. Some media reports in 2023 suggested that OpenAI might obtain trademark registration indirectly on the strength of its GPT-based chatbot's fame, ChatGPT, for which OpenAI separately sought protection (and which it sought to defend more aggressively). Other reports indicate that registration of the plain term “GPT” appears unlikely, since it is often used as a generic term for AI systems that simply incorporate generative pre-trained transformers. In any case, to whatever extent exclusive rights to this term exist in the United States, others would need to avoid using it for similar products or services in ways that could cause confusion. Should such rights ever become broad enough to affect other established uses in the field, the doctrine of trademark fair use may allow such non-brand-related use to continue.
In the European Union, the European Union Intellectual Property Office registered “GPT” as an OpenAI trademark in spring 2023. However, since spring 2024, the registration has been contested and is undergoing cancellation proceedings.
In Switzerland, the Swiss Federal Institute of Intellectual Property registered “GPT” as an OpenAI trademark in spring 2023.
Evaluation of pre-trained transformer models built using generative algorithms is carried out using a range of benchmark tests and metrics designed to assess their performance on various tasks. Common approaches include accuracy on standard datasets, as well as other characteristics such as robustness, bias, and toxicity.
These models are typically tested on tasks such as natural language understanding, reasoning, question answering, and code generation. They are sometimes combined across multiple tasks to provide a broader assessment of model performance across different domains.
More recent approaches extend these evaluations to include characteristics such as fairness, efficiency, and transparency, in order to obtain a more accurate assessment of these models.
Evaluation remains an active area of research, since existing benchmarks may not accurately reflect the real-world performance or risks associated with large-scale generative models.
Generative pre-trained transformer models have raised a number of ethical and social concerns, particularly regarding bias, misinformation, and environmental impact. Large language models can reproduce and amplify patterns present in their training data, including social biases, which can lead to unfair or misleading results.
These models are also associated with the generation of inaccurate or misleading information, since they are designed to produce fluent text rather than to verify factual accuracy. This affects their use in applications such as automated content generation and the dissemination of information.
Training large-scale models also requires enormous computational resources, leading to increased energy consumption and environmental costs. Concerns about the environmental impact of large AI systems have led to calls for more efficient training methods and greater transparency in reporting resource usage.
Comments