a g @sBdZddlmZmZddlmZGdddZGdddeZdS) z Tokenzier classes for ULMFiT ) CollectionList)THAI2FIT_TOKENIZERc@sDeZdZdZedddZeeedddZeedd d Z d S) BaseTokenizerz;Basic class for a tokenizer function. (codes from `fastai`)langcCs ||_dSNrselfrr D/usr/local/lib/python3.9/dist-packages/pythainlp/ulmfit/tokenizer.py__init__szBaseTokenizer.__init__)treturncCs |dS)N )split)r rr r r tokenizerszBaseTokenizer.tokenizer)tokscCsdSrr r rr r r add_special_casesszBaseTokenizer.add_special_casesN) __name__ __module__ __qualname____doc__strr rrrrr r r r r src@s@eZdZdZd edddZeeeedddZd d Z d S) ThaiTokenizerz Wrapper around a frozen newmm tokenizer to make it a :class:`fastai.BaseTokenizer`. (see: https://docs.fast.ai/text.transform#BaseTokenizer) thrcCs ||_dSrrr r r r r szThaiTokenizer.__init__)textrcCs t|S)u This function tokenizes text using *newmm* engine and the dictionary specifically for `ulmfit` related functions (see: `Dictionary file (.txt) `_). :meth: tokenize text using a frozen newmm engine :param str text: text to tokenize :return: tokenized text :rtype: list[str] :Example: Using :func:`pythainlp.ulmfit.ThaiTokenizer.tokenizer` is similar to :func:`pythainlp.tokenize.word_tokenize` using *ulmfit* engine. >>> from pythainlp.ulmfit import ThaiTokenizer >>> from pythainlp.tokenize import word_tokenize >>> >>> text = "อาภรณ์, จินตมยปัญญา ภาวนามยปัญญา" >>> ThaiTokenizer.tokenizer(text) ['อาภรณ์', ',', ' ', 'จิน', 'ตม', 'ย', 'ปัญญา', ' ', 'ภาวนามยปัญญา'] >>> >>> word_tokenize(text, engine='ulmfit') ['อาภรณ์', ',', ' ', 'จิน', 'ตม', 'ย', 'ปัญญา', ' ', 'ภาวนามยปัญญา'] )rZ word_tokenize)rr r r r#szThaiTokenizer.tokenizercCsdSrr rr r r rDszThaiTokenizer.add_special_casesN)r) rrrrrr staticmethodrrrr r r r rs  rN)rtypingrrZpythainlp.tokenizerrrr r r r s