2024-12-26
Extracting SentencePiece tokeniser training settings for vocabulary expansion
How to extract SentencePiece tokeniser training settings for vocabulary expansion.
English | 日本語
2024-06-24
Vocabulary expansion for non-SentencePiece based BPE tokeniser
A note on how to do vocabulary expansion with LLaMA3, OLMo, etc.
2024-06-23
Introduction
Introduction to my blog and the privacy policy.