a g}@sdZddlZddlmZddlmZddlmZddlm Z e eddd Z d ed Z e e Z eeeed d dZeeedddZdS)a[ Segmenting text into Enhanced Thai Character Clusters (ETCCs) Python implementation by Wannaphong Phatthiyaphaibun This implementation relies on a dictionary of ETCC created from etcc.txt in pythainlp/corpus. Notebook: https://colab.research.google.com/drive/1UTQgxxMRxOr9Jp1B1jcq1frBNvorhtBQ :See Also: Jeeragone Inrut, Patiroop Yuanghirun, Sarayut Paludkong, Supot Nitsuwat, and Para Limmaneepraserth. "Thai word segmentation using combination of forward and backward longest matching techniques." In International Symposium on Communications and Information Technology (ISCIT), pp. 37-40. 2001. N)List)thai_follow_vowels) get_corpus) Tokenizerzetcc.txtlongest)Zengine[uๆฯ])tokensreturncCspt|}d}||krqlt||rb|dkrbt||dkrb||d||7<||=|d8}|d7}q |S)Nr)len_RE_ENDING_CHARsearch)rZ len_tokensirA/usr/local/lib/python3.9/dist-packages/pythainlp/tokenize/etcc.py _cut_subword!s& r)textr cCs |rt|tsgStt|S)aU Segmenting text into ETCCs. Enhanced Thai Character Cluster (ETCC) is a kind of subword unit. The concept was presented in Inrut, Jeeragone, Patiroop Yuanghirun, Sarayut Paludkong, Supot Nitsuwat, and Para Limmaneepraserth. "Thai word segmentation using combination of forward and backward longest matching techniques." In International Symposium on Communications and Information Technology (ISCIT), pp. 37-40. 2001. :param str text: text to be tokenized into character clusters :return: list of clusters, tokenized from the text :return: List[str] ) isinstancestrr _cut_etccZ word_tokenize)rrrrsegment/sr)__doc__retypingrZ pythainlprZpythainlp.corpusrZpythainlp.tokenizerrZ_PAT_ENDING_CHARcompiler rrrrrrrs