Token Byte-Pair Encoder Merge

Token Byte-Pair Encoder Merge

Byte-Pair Encoding (BPE) is a subword tokenization algorithm used by model tokenizers (like Tiktoken). In the training/processing phase, the tokenizer iteratively merges the most frequent adjacent pair of tokens in a sequence into a new, consolidated token.

Tasks

Implement the function merge_bpe_pair(ids, pair, new_id) to:

1. Find all occurrences of the adjacent pair of token IDs (pair[0], pair[1]) in the input sequence ids.
2. Replace each occurrences of the pair with the single integer new_id.
3. Return the modified sequence.

Example

merge_bpe_pair([1, 2, 3, 1, 2, 4], (1, 2), 5)
# Returns: [5, 3, 5, 4]