:Qgw UddlmZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z ddl mZdZd ed <d Zdd Zed ZeeddZ eeddZeeddZ y)) annotationsN) lru_cache)FinalListTuple)pos_tag) sent_tokenize) word_tokenizez Final[int]CACHE_MAX_SIZEc`tjddtjddy)Naveraged_perceptron_tagger_engT)quiet punkt_tab)nltkdownloadV/var/www/html/answerous/venv/lib/python3.12/site-packages/unstructured/nlp/tokenize.pydownload_nltk_packagesrsMM2$?MM+T*rc.g}tjjD]D}|jds tjj |d}|j |F tj|d||y#ttf$rYywxYw)zEChecks to see if the specified NLTK package exists on the file system nltk_data/)pathsTF) rdatapathendswithosjoinappendfind LookupErrorOSError) package_namepackage_categoryrrs rcheck_for_nltk_packager&sE }}[)77<<k2D T  %&a ~6eD  !s%BBBcVtdd}tdd}|r|s tyy)z;If required NLTK packages are not available, download them.taggersr)r%r$ tokenizersrN)r&r)tagger_availabletokenizer_availables r&_download_nltk_packages_if_not_presentr,%s:."51%K )9 *:r)maxsizec,tt|S)zFA wrapper around the NLTK sentence tokenizer with LRU caching enabled.)r,_sent_tokenizetexts rr r 5+, $ rc,tt|S)zBA wrapper around the NLTK word tokenizer with LRU caching enabled.)r,_word_tokenizer0s rr r <r2rctt|}g}|D]'}t|}|jt |)|S)z>A wrapper around the NLTK POS tagger with LRU caching enabled.)r,r/r4extend_pos_tag)r1 sentencesparts_of_speechsentencetokenss rrrCsO+,t$I-/O1)x/01 r)r$strr%r<returnbool)r1r<r=z List[str])r1r<r=zList[Tuple[str, str]]) __future__rr functoolsrtypingrrrrrr7r r/r r4r __annotations__rr&r,rrrrCs" %% $00  + "  !  ! >" #  >" #  >" # r