BERT (language model)

Bidirectional Encoder Representations from Transformers (BERT)

Original author(s)	Google AI
Initial release	October 31, 2018
Repository	github.com/google-research/bert
Type	Large language model Transformer (deep learning architecture) Foundation model
License	Apache 2.0
Website	arxiv.org/abs/1810.04805

Bidirectional encoder representations from transformers (BERT) is a language model introduced in October 2018 by researchers at Google.^[1][2] It learns to represent text as a sequence of vectors using self-supervised learning. It uses the encoder-only transformer architecture. BERT dramatically improved the state-of-the-art for large language models. As of 2020^[update], BERT is a ubiquitous baseline in natural language processing (NLP) experiments.^[3]

BERT is trained by masked token prediction and next sentence prediction. As a result of this training process, BERT learns contextual, latent representations of tokens in their context, similar to ELMo and GPT-2.^[4] It found applications for many natural language processing tasks, such as coreference resolution and polysemy resolution.^[5] It is an evolutionary step over ELMo, and spawned the study of "BERTology", which attempts to interpret what is learned by BERT.^[3]

BERT was originally implemented in the English language at two model sizes, BERT_BASE (110 million parameters) and BERT_LARGE (340 million parameters). Both were trained on the Toronto BookCorpus^[6] (800M words) and English Wikipedia (2,500M words).^[1]: 5 The weights were released on GitHub.^[7] On March 11, 2020, 24 smaller models were released, the smallest being BERT_TINY with just 4 million parameters.^[7]