:Qg ddlZddlmZmZddlZddlmZmZddlm Z ddl m Z de de fdZ d e fd Zdd e dee de d e fd ZdZy)N)ListOptional) MarianMTModelMarianTokenizer) sent_tokenize)chunk_by_attention_window source_lang target_langcd|d|S)zjConstructs the name of the MarianMT machine translation model based on the source and target language.zHelsinki-NLP/opus-mt--)r r s \/var/www/html/answerous/venv/lib/python3.12/site-packages/unstructured/cleaners/translate.py_get_opus_mt_model_namer s#;-q >> language_codec^t|trt|dk7rtd|dy)NzInvalid language code: z,. Language codes must be two letter strings.) isinstancestrlen ValueError)rs r_validate_language_coders8 mS )S-?1-D%m_4` a  .Ertextreturnc|jdk(r|S||ntj|}|jdrd}t |t |||k(r|St ||} t j|}tj|}t||t}g}|D]} |jt||| dj|S#t$rtd|dwxYw)aTranslates the foreign language text. If the source language is not specified, the function will attempt to detect it using langdetect. Parameters ---------- text: str The text to translate target_lang: str The two letter language code for the target langague. Defaults to "en". source_lang: Optional[str] The two letter language code for the language of the input text. If source_lang is not provided, the function will try to detect it. zhz2Transformers could not find the translation model z>. The requested source/target language combo is not supported.)split_function )strip langdetectdetect startswithrrrfrom_pretrainedrOSErrorrrrappend_translate_textjoin) rr r _source_lang model_name tokenizermodelchunkstranslated_chunkschunks rtranslate_textr0s zz|r '2'> JDUDUVZD[Lt$ K(L)l" ({CJ #33J? --j92$ R_`F#%J  ui!HIJ 88% &&  @ MK K   s ,*CC7c  tj5tjd|jd i||gddd}dddDcgc]}|j |ddc}d S#1swY.xYwcc}w) z8Translates text using the specified model and tokenizer.ignorept max_lengthi)return_tensorspaddingr4NT)max_new_tokensskip_special_tokensrr )warningscatch_warnings simplefiltergeneratedecode)rr,r+ translatedts rr'r'Js  " h'#U^^ t\VYZ  Xb bRSI  Qs  M b     cs3A5B5A>)Nen)r9typingrrr! transformersrrunstructured.nlp.tokenizer unstructured.staging.huggingfacerrrrr0r'r rrrEsb!73F??3?  3 /'/'8C=/'c/']`/'d r