Generative pre-trained transformer

Generative pre-trained transformers (GPT) are a type of large language model (LLM)^[1][2][3] and a prominent framework for generative artificial intelligence.^[4][5] They are artificial neural networks that are used in natural language processing tasks.^[6] GPTs are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content.^[2][3] As of 2023, most LLMs have these characteristics^[7] and are sometimes referred to broadly as GPTs.^[8]

The first GPT was introduced in 2018 by OpenAI.^[9] OpenAI has released very influential GPT foundation models that have been sequentially numbered, to comprise its "GPT-n" series.^[10] Each of these was significantly more capable than the previous, due to increased size (number of trainable parameters) and training. The most recent of these, GPT-4, was released in March 2023.^[11] Such models have been the basis for their more task-specific GPT systems, including models fine-tuned for instruction following—which in turn power the ChatGPT chatbot service.^[1]

The term "GPT" is also used in the names and descriptions of such models developed by others. For example, other GPT foundation models include a series of models created by EleutherAI,^[12] and seven models created by Cerebras in 2023.^[13] Also, companies in different industries have developed task-specific GPTs in their respective fields, such as Salesforce's "EinsteinGPT" (for CRM)^[14] and Bloomberg's "BloombergGPT" (for finance).^[15]