a g @s^dZdgZddlZddlZddlmZmZddlZddl Z ddl Tddl Te e jZeedZeZeeeddZe jjeZeed eed Ze ed d d e!edddgZ"e#j$eedge"d%d&j'ddZ(e()ddddd dddddddd Z*d d!d"dd#Z+e,e(e-fe*d d$e+Z.e.j/fied(e0e1e2e3eee0e0fd&d'dZ4dS))z Thai2fit: Thai Wikipeida Language Model for Text Generation Codes are from https://github.com/PyThaiNLP/tutorials/blob/master/source/notebooks/text_generation.ipynb gen_sentenceN)ListUnion)*z texts.csvZ itos_fnamerbth)Ztok_funclangZ pre_rulesZ post_rulesi'F) tokenizer chunksizeZ mark_fieldsi`)ZvocabZ max_vocabZmin_freqtext)cols processorg?@)bsiiTg?g?g{Gz?g333333?) Zemb_szZn_hidZn_layersZ pad_tokenZqrnnZ tie_weightsZout_biasZoutput_pZhidden_pZinput_pZembed_pZweight_pg?gQ?)Z drop_multZclipalphabeta)configZ pretrainedMbP?) start_seqNprob output_strreturncCsB|durttt}tj||d|ddd}|r>d|S|S)u: Text generator using Thai2fit :param str start_seq: word to begin sentence with :param int N: number of words :param bool output_str: output as string :param bool duplicate: allow duplicate words in sentence :return: list words or str words :rtype: List[str], str :Example: :: from pythainlp.generate.thai2fit import gen_sentence gen_sentence() # output: 'แคทรียา อิงลิช (นักแสดง' gen_sentence("แมว") # output: 'แมว คุณหลวง ' Ng?z-*-)Z temperatureZmin_psep)randomchoicelist thwiki_itoslearnZpredictsplitjoin)rrrrZ list_wordr&E/usr/local/lib/python3.9/dist-packages/pythainlp/generate/thai2fit.pyrOs  )NrrT)5__doc____all__rpickletypingrrZpandaspdZfastaiZ fastai.textZpythainlp.ulmfitZ untar_dataZURLsZ IMDB_SAMPLEZimdbZread_csvZdummy_dfZ THWIKI_LSTMZthwikiloadopenr"r Z transformZVocabZ thwiki_vocab TokenizerZ ThaiTokenizerZ pre_rules_thZ post_rules_thttZTokenizeProcessorZNumericalizeProcessorrZTextListZfrom_dfZsplit_by_rand_pctZ label_for_lmZ databunchZdata_lmZ sanity_checkrZtrn_argsZlanguage_model_learnerZAWD_LSTMr#Zload_pretrainedstrintfloatboolrr&r&r&r's|