a †‹g²ã@sPdZddlZddlmZddlmZmZeedœdd„Zeeedœdd „Z dS) zô Wrapper for PyICU word segmentation. This wrapper module uses :class:`icu.BreakIterator` with Thai as :class:`icu.Local` to locate boundaries between words in the text. :See Also: * `GitHub repository `_ éN)ÚList)Ú BreakIteratorÚLocale)ÚtextÚreturnccs@t tdƒ¡}| |¡| ¡}|D]}|||…V|}q$dS)NÚth)rZcreateWordInstancerZsetTextÚfirst)rZbdÚpÚq©r úB/usr/local/lib/python3.9/dist-packages/pythainlp/tokenize/pyicu.pyÚ _gen_wordss  r cCs,|rt|tƒsgSt dd|¡}tt|ƒƒS)zn :param str text: text to be tokenized into words :return: list of words, tokenized from the text u([^฀-๿ ]+)z \1 )Ú isinstanceÚstrÚreÚsubÚlistr )rr r r Úsegmentsr) Ú__doc__rÚtypingrZicurrrr rr r r r Ús