Byte-Pair Encoding (en.wikipedia.org) AI

Byte-pair encoding (BPE) is a text encoding method that iteratively merges the most frequent adjacent byte pairs using a learned lookup table, initially described for data compression. A modified form used in large language model tokenizers builds a fixed vocabulary by repeatedly merging frequent token pairs, aiming for practical training rather than maximum compression. Byte-level BPE extends this by encoding text as UTF-8 bytes, allowing it to represent any UTF-8 text.