#preprocessing

Process of splitting text into discrete units (tokens) that language models can process numerically, fundamental to how LLMs understand and generate text.