Tokenization
Why models break text into subwords, how Byte Pair Encoding builds a vocabulary from scratch, and what naive word splitting gets wrong.
–––
Topics / Tokenization
Tokenization
1 post.
Topics / Tokenization
1 post.
Why models break text into subwords, how Byte Pair Encoding builds a vocabulary from scratch, and what naive word splitting gets wrong.