a gD. @sHdZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlmZmZddlmZddlmZeedd d Zeedd d Ze eefeeeeegefed ddZee edddZee edddZeeeefe ee e eeffeeee eegefe e eefdddZGdddZdS)z Spell checker, using Peter Norvig algorithm. Spelling dictionary can be customized. Default spelling dictionary is based on Thai National Corpus. Based on Peter Norvig's Python code from http://norvig.com/spell-correct.html )Counter)digits) CallableDict ItemsViewIterableListOptionalSetTupleUnion) thai_digits thai_letters)tnc) isthaicharwordreturncCsdS)NTrrrBz_edits1..r cSs$g|]\}}|r||ddqSr Nrr%LRrrrr'Cr(cSs<g|]4\}}t|dkr||d|d|ddqS)r rNr!r*rrrr'Dr(cSs2g|]*\}}|rtD]}|||ddqqSr)rr%r+r,crrrr'Er(cSs&g|]\}}tD]}|||qqSrr.r/rrrr'Fr()ranger"set)rsplitsZdeletesZ transposesZreplacesinsertsrrr_edits1>s r5cCstddt|DS)zO Returns a set of words with an edit distance of 2 from the input word css |]}t|D] }|VqqdSr$)r5)r%e1e2rrr Or(z_edits2..)r2r5rrrr_edits2Ksr9) custom_dictrrrrrcsxt|trt|}t|}t|}t|trHfdd|D}n,t|trlfdd|D}ntd|S)zE Converts a custom dictionary to a list of (str, int) tuples cs(g|] }t|dfdr|dfqS)r r#)r%r)rrrrrr'fsz(_convert_custom_dict..cs g|]}t|r|qSrr;)r%rrrrrrrr'mszVcustom_dict must be either Dict[str, int], Iterable[Tuple[str, int]], or Iterable[str]) isinstancedictlistitemsiternextstrtuple TypeError)r:rrrrr&Z first_memberrr<r_convert_custom_dictRs    rFc @seZdZddddefeeeefeeee eeffeeee e ege fdddZ eeefddd Zeeeed d d Zeed ddZeed ddZeeed ddZeed ddZdS)NorvigSpellCheckerNr-()r:rrrrcCs`|sddtD}|st}t|||||}tt||_|jt7_t|j|_ dS)a_ Initializes Peter Norvig's spell checker object. Spelling dictionary can be customized. By default, spelling dictionary is from `Thai National Corpus `_ Basically, Norvig's spell checker will choose the most likely corrected spelling given a word by searching for candidates of corrected words based on edit distance. Then, it selects the candidate with the highest word occurrence probability. :param str custom_dict: A custom spelling dictionary. This can be: (1) a dictionary (`dict`), with words (`str`) as keys and frequencies (`int`) as values; (2) an iterable (list, tuple, or set) of words (`str`) and frequency (`int`) tuples: `(str, int)`; or (3) an iterable of just words (`str`), without frequencies -- in this case `1` will be assigned to every words. Default is from Thai National Corpus (around 40,000 words). :param int min_freq: Minimum frequency of a word to keep (default = 2) :param int min_len: Minimum length (in characters) of a word to keep (default = 2) :param int max_len: Maximum length (in characters) of a word to keep (default = 40) :param func dict_filter: A function to filter the dictionary. Default filter removes any word with numbers or non-Thai characters. If no filter is required, use None. cSsg|]\}}||fqSrr)r%r&jrrrr'r(z/NorvigSpellChecker.__init__..N) rZ word_freqsrrFrr>_NorvigSpellChecker__WORDSsumvalues _NorvigSpellChecker__WORDS_TOTAL)selfr:rrrrrrr__init__|s+ zNorvigSpellChecker.__init__)rcCs |jS)u3 Returns the spelling dictionary currently used by this spell checker :return: spelling dictionary of this instance :rtype: list[tuple[str, int]] :Example: :: from pythainlp.spell import NorvigSpellChecker dictionary= [("หวาน", 30), ("มะนาว", 2), ("แอบ", 3223)] checker = NorvigSpellChecker(custom_dict=dictionary) checker.dictionary() # output: dict_items([('หวาน', 30), ('มะนาว', 2), ('แอบ', 3223)]) )rJr@rNrrr dictionaryszNorvigSpellChecker.dictionary)wordsrcstfdd|DS)u Returns a list of given words found in the spelling dictionary :param list[str] words: A list of words to check if they exist in the spelling dictionary :return: intersection of the given word list and words in the spelling dictionary :rtype: list[str] :Example: :: from pythainlp.spell import NorvigSpellChecker checker = NorvigSpellChecker() checker.known(["เพยน", "เพล", "เพลง"]) # output: ['เพล', 'เพลง'] checker.known(['ยกไ', 'ไฟล์ม']) # output: [] checker.known([]) # output: [] c3s|]}|jvr|VqdSr$rJ)r%wrPrrr8r(z+NorvigSpellChecker.known..)r?)rNrRrrPrknownszNorvigSpellChecker.knownrcCs|j||jS)ui Returns the probability of an input word, according to the spelling dictionary :param str word: A word to check occurrence probability of :return: word occurrence probability :rtype: float :Example: :: from pythainlp.spell import NorvigSpellChecker checker = NorvigSpellChecker() checker.prob("ครัช") # output: 0.0 checker.prob("รัก") # output: 0.0006959172792052158 checker.prob("น่ารัก") # output: 9.482306849763902e-05 )rJrMrNrrrrprobszNorvigSpellChecker.probcCs |j|S)u  Returns the frequency of an input word, according to the spelling dictionary :param str word: A word to check frequency of :return: frequency of the given word in the spelling dictionary :rtype: int :Example: :: from pythainlp.spell import NorvigSpellChecker checker = NorvigSpellChecker() checker.freq("ปัญญา") # output: 3639 checker.freq("บิญชา") # output: 0 rSrVrrrfreqszNorvigSpellChecker.freqcCsL|s dgS||gp6|t|p6|t|p6|g}|j|jdd|S)u  Returns a list of all correctly-spelled words whose spelling is similar to the given word by edit distance metrics. The returned list of words will be sorted by decreasing order of word frequencies in the word spelling dictionary. First, if the input word is spelled correctly, this method returns a list of exactly one word which is itself. Next, this method looks for a list of all correctly spelled words whose edit distance value is 1 from the input word. If there is no such word, then the search expands to a list of words whose edit distance value is 2. And if that still fails, the list of input words is returned. :param str word: A word to check spelling of :return: list of possibly correct words within 1 or 2 edit distance and sorted by frequency of word occurrence in the spelling dictionary in descending order. :rtype: list[str] :Example: :: from pythainlp.spell import NorvigSpellChecker checker = NorvigSpellChecker() checker.spell("เส้นตรบ") # output: ['เส้นตรง'] checker.spell("ครัช") # output: ['ครับ', 'ครัว', 'รัช', 'ครัม', 'ครัน', # 'วรัช', 'ครัส', 'ปรัช', 'บรัช', 'ครัง', #'คัช', 'คลัช', 'ครัย', 'ครัด'] T)keyreverse)rUr5r9sortrX)rNr candidatesrrrspells%   zNorvigSpellChecker.spellcCsJ|sdSz d|vrt|nt||WSty:Yn0||dS)u Returns the most possible word, using the probability from the spelling dictionary :param str word: A word to correct spelling of :return: the correct spelling of the given word :rtype: str :Example: :: from pythainlp.spell import NorvigSpellChecker checker = NorvigSpellChecker() checker.correct("ปัญชา") # output: 'ปัญหา' checker.correct("บิญชา") # output: 'บัญชา' checker.correct("มิตรภาบ") # output: 'มิตรภาพ' rYrr)floatint ValueErrorr^rVrrrcorrectMs  zNorvigSpellChecker.correct)__name__ __module__ __qualname__rr rrCr`rr r rboolrOrrQrrUr_rWrXr^rbrrrrrG{s(  :2rGN) __doc__ collectionsrstringrtypingrrrrrr r r r Z pythainlpr rZpythainlp.corpusrZpythainlp.utilrrCrfrrr`r#r5r9rFrGrrrrs6  ,         )