British Sign Language Glossaries of Curriculum Terms

BSL AI Glossary - Token definition

Translation: A token is a small unit of text, such as a whole word, part of a word, a character, or punctuation, that a language model processes. Instead of handling text all at once, the model breaks it down into tokens and works with these units to analyse or generate language.