a g @sdZddlZddlmZmZdddddd d d d d dZed eZ e e dddZ e ee dddZe ee dddZdS)a The implementation of tokenizer according to Thai Character Clusters (TCCs) rules proposed by `Theeramunkong et al. 2000. `_ Credits: * TCC: Jakkrit TeCho * Grammar: Wittawat Jitkrittum (`link to the source file `_) * Python code: Korakot Chaovavanich N)ListSetuc[ั]([่-๋]c)? c[ั]([่-๋]c)?k เc็ck เcctาะk เccีtยะk เccีtย(?=[เ-ไก-ฮ]|$)k เc[ิีุู]tย(?=[เ-ไก-ฮ]|$)k เcc็ck เcิc์ck เcิtck เcีtยะ?k เcืtอะk เcื เctา?ะ?k c[ึื]tck c[ะ-ู]tk c[ิุู]์ cรรc์ c็ ct[ะาำ]?k แc็ck แcc์k แctะk แcc็ck แccc์k โctะk [เ-ไ]ctk ก็ อึ หึ ku(cc?[d|ิ]?[์])?cu [ก-ฮ]tu [่-๋]?du อูอุuอ|)textreturnccsn|rt|tsdSt|}d}||krjt||d}|rJ|d}nd}||||V||7}qdS)z TCC generator which generates Thai Character Clusters :param str text: text to be tokenized into character clusters :return: subwords (character clusters) :rtype: Iterator[str] rrN) isinstancestrlen_PAT_TCCmatchspan)r Zlen_textpmnr@/usr/local/lib/python3.9/dist-packages/pythainlp/tokenize/tcc.pytcc>srcCsF|rt|tstSt}d}t|D]}|t|7}||q&|S)z TCC positions :param str text: text to be tokenized into character clusters :return: list of the ending position of subwords :rtype: set[int] r)r rsetrradd)r Zp_setrwrrrtcc_posUs   rcCs tt|S)z Subword segmentation :param str text: text to be tokenized into character clusters :return: list of subwords (character clusters), tokenized from the text :rtype: list[str] )listr)r rrrsegmentis r)__doc__retypingrrreplacesplitZ_RE_TCCcompilejoinrrrintrrrrrrs   "#$ )