a g@sdZddlZddlmZddlmZmZmZmZm Z m Z dZ dZ dZ dZGd d d ZGd d d eZGd ddeZGdddeZGdddZdS)z* Command line for PyThaiNLP's tokenizers. N)cli)DEFAULT_SENT_TOKENIZE_ENGINEDEFAULT_SUBWORD_TOKENIZE_ENGINEDEFAULT_WORD_TOKENIZE_ENGINE sent_tokenizesubword_tokenize word_tokenizez@@/~|c@seZdZddZdS) SubAppBasecCstjfitd|}|jdtddd|jdddtd |j|jd |jd d d td |j|jd |jddddd|jddddd|jdd| |}||_ t |j ||j |j |j|jd}t|j||jdS)Nz tokenize text?z input text)typenargshelpz-sz--sep separatorz default: )destrrdefaultz-az--algo algorithmz-wz--keep-whitespacekeep_whitespace store_true)ractionz-nwz--no-whitespace store_falseT)r)Zenginer)argparseArgumentParserrZ make_usage add_argumentstrrr set_defaults parse_argsargs exit_if_emptyr runrprintjoin)selfnameargvparserr resultr*@/usr/local/lib/python3.9/dist-packages/pythainlp/cli/tokenize.py__init__sZ    zSubAppBase.__init__N__name__ __module__ __qualname__r,r*r*r*r+r sr cseZdZfddZZS)WordTokenizationAppcs.d|_t|_t|_t|_tj|i|dSNT) rrrDEFAULT_WORD_TOKEN_SEPARATORrrr"superr,r%r kwargs __class__r*r+r,Ns zWordTokenizationApp.__init__r.r/r0r, __classcell__r*r*r7r+r1Msr1cseZdZfddZZS)SentenceTokenizationAppcs.d|_t|_t|_t|_tj|i|dSr2) rrrDEFAULT_SENT_TOKEN_SEPARATORrrr"r4r,r5r7r*r+r,Ws z SentenceTokenizationApp.__init__r9r*r*r7r+r;Vsr;cseZdZfddZZS)SubwordTokenizationAppcs.d|_t|_t|_t|_tj|i|dSr2) rrrDEFAULT_SUBWORD_TOKEN_SEPARATORrrr"r4r,r5r7r*r+r,`s zSubwordTokenizationApp.__init__r9r*r*r7r+r=_sr=c@seZdZddZdS)AppcCstjdddd}|jdtdd||dd }t|j|t|j}|d d}| d rnt d |n:| d rt d |n$| drt d|nt d|dS)Ntokenizez'Break a text into small units (tokens).uthainlp tokenize [options] "" token_type: subword subword (may not be a linguistic unit) syllable syllable word word sent sentence options: --sep or -s specify custom separator (default is a space) --algo or -a tokenization algorithm (see API doc for more info) --keep-whitespace or -w keep whitespaces in output (default) and should be inside double quotes. Example: thainlp tokenize word -s "|" "ใต้แสงนีออนเปลี่ยวเหงา" --)prog descriptionusage token_typez[subword|word|sent])rrwwordZsuZsubwordsesentzToken type not available: )rrrrrrr!rDlower startswithr1r=r;r#)r%r'r(r rDr*r*r+r,is*        z App.__init__Nr-r*r*r*r+r?hsr?)__doc__rZ pythainlprZpythainlp.tokenizerrrrrrr<r>Z DEFAULT_SYLLABLE_TOKEN_SEPARATORr3r r1r;r=r?r*r*r*r+s  3