Open Access. Powered by Scholars. Published by Universities.®
Ministry of Higher and Secondary Specialized Education of the Republic of Uzbekistan
Articles 1 - 1 of 1
Full-Text Articles in Education
Linguistic Features Tokenization Of Text Corpora Of The Uzbek Language, I.I. Bakaev
Linguistic Features Tokenization Of Text Corpora Of The Uzbek Language, I.I. Bakaev
Bulletin of TUIT: Management and Communication Technologies
The article deals with the topic of tokenization of text corpora of the Uzbek language, which should take into account the linguistic features of the spelling of the language. The types of words are analyzed and a mathematical model of word formation for complex, paired, repeated, and compound words is proposed. And also based on the models developed a finite state machine to demonstrate the spelling rules of the language, and the rules are implemented by regular expressions