R"hddlZddlZddlmZmZddlmZmZm Z m Z m Z m Z ddl mZ ddlZej 5ej"dedej&vrdej&d<ddlZd e_d e_ddddd lmZej2j5ej6d Zeed e_GddZ y#1swYTxYw#e$rGddZeZY/wxYw)N)ThreadPoolExecutor as_completed)ListTupleUnionOptionalDictLiteral)Pathignore)categoryLITELLM_LOCAL_MODEL_COST_MAPTrueTF)Cachez.storm_local_cachedisk)disk_cache_dirtypeceZdZdZy)LitellmPlaceholderctd)Nz@The LiteLLM package is not installed. Run `pip install litellm`.) ImportError)self_s 8/var/www/html/sandstorm/storm/knowledge_storm/encoder.py __getattr__zLitellmPlaceholder.__getattr__sR N)__name__ __module__ __qualname__rrrrrs rrc eZdZdZ ddeedeedeedeefdZdded efd Z dd e ee efd ed e jfd ZdZ dd e ee efd ed ee jeffdZy)Encodera A wrapper class for the LiteLLM embedding model, designed to handle embedding generation tasks efficiently. It supports parallel processing and local caching of embedding results for improved performance. The Encoder utilizes the LiteLLM library to interact with various embedding models, such as OpenAI and Azure embeddings. Users can specify the desired encoder type and provide relevant API credentials during initialization. Features: - Support for multiple embedding models (e.g., OpenAI, Azure). - Parallel processing for faster embedding generation. - Local disk caching to store and reuse embedding results. - Total token usage tracking for cost monitoring. Note: Refer to the LiteLLM documentation for details on supported embedding models: https://docs.litellm.ai/docs/embedding/supported_embedding N encoder_typeapi_keyapi_base api_versioncd|_i|_d|_|xstjd}|s t d|j dk(r(d|_d|xstjdi|_y|j d k(rXd |_|xstjd |xstjd |xstjd d|_yt d|d)a Initializes the Encoder with the appropriate embedding model. Args: encoder_type (Optional[str]): Type of encoder ('openai', 'azure', etc.). api_key (Optional[str]): API key for the encoder service. api_base (Optional[str]): API base URL for the encoder service. api_version (Optional[str]): API version for the encoder service. NrENCODER_API_TYPEz1ENCODER_API_TYPE environment variable is not set.openaiztext-embedding-3-smallr$OPENAI_API_KEYazurezazure/text-embedding-3-small AZURE_API_KEYAZURE_API_BASEAZURE_API_VERSION)r$r%r&zUnsupported ENCODER_API_TYPE 'z5'. Supported types are 'openai', 'azure', 'together'.)embedding_model_namekargstotal_token_usageosgetenv ValueErrorlower)rr#r$r%r&s r__init__zEncoder.__init__9s %)! !"$Dryy1C'D PQ Q    8 +(@D %#W%K :J0KLDJ    !W ,(FD %"@bii&@$C 2B(C*Lbii8K.LDJ 0>st rresetreturnc0|j}|rd|_|S)z Retrieves the total token usage. Args: reset (bool): If True, resets the total token usage counter after retrieval. Returns: int: The total number of tokens used. r)r1)rr7 token_usages rget_total_token_usagezEncoder.get_total_token_usageas ,, %&D "rtexts max_workersc(|j||S)z Public method to get embeddings for the given texts. Args: texts (Union[str, List[str]]): A single text string or a list of text strings to embed. Returns: np.ndarray: The array of embeddings. r=)_get_text_embeddings)rr<r=s rencodezEncoder.encodeps((K(HHrctjd|j|dd|j}|jdd}|j dij dd}|||fS)NT)modelinputcachingr embeddingusage total_tokensr )litellmrFr/r0dataget)rtextresponserFr:s r_get_single_text_embeddingz"Encoder._get_single_text_embedding|so$$ ++4 IM MM!$[1 ll7B/33NAF Y ++rcttr?|j\}}}|xj|z c_t j |Sg}d}t |5}D cic]} |j|j| | !} } t| D]0} | j\} }}|j| ||f||z }2 ddd|jfd|D cgc]} | d }} |xj|z c_t j |Scc} w#t$r&} td| | t| Yd} ~ d} ~ wwxYw#1swYxYwcc} w)a? Get text embeddings using OpenAI's text-embedding-3-small model. Args: texts (Union[str, List[str]]): A single text string or a list of text strings to embed. max_workers (int): The maximum number of workers for parallel processing. api_key (str): The API key for accessing OpenAI's services. embedding_cache (Optional[Dict[str, np.ndarray]]): A cache to store previously computed embeddings. Returns: Tuple[np.ndarray, int]: The 2D array of embeddings and the total token usage. rr?zAn error occurred for text: Nc,j|dS)Nr)index)xr<s rz.Encoder._get_text_embeddings..sekk!A$&7r)key) isinstancestrrNr1nparrayrsubmitrresultappend Exceptionprintsort)rr<r=rrFtokens embeddingsrHexecutorrLfuturesfutureer[s ` rr@zEncoder._get_text_embeddingssq$ eS !#'#B#B5#I Ay&  " "f , "88I& &  K 8 H" ? ?FLG 'w/ .4mmo+D)V%%tY&?@ F*L    78.89FfQi9 9 ,.xx ##'!88IJK!HH  ":sN!E&$D$ E-D)E, E'$E) E2EEEEE$)NNNN)F))rrr__doc__rrWr6boolintr;rrrXndarrayrArNrr@r rrr"r"$s,'+!%"&%) &sm&#&3- & c] &P 4 C  IE#tCy.1 I IBJJ I,.$S$s)^$.$.$ rzz3  .$rr")!r2numpyrXconcurrent.futuresrrtypingrrrrr r pathlibr warningscatch_warningsfilterwarnings UserWarningenvironrI drop_params telemetrylitellm.caching.cachingrpathjoinhomercacherrr"r rrr{s ?>>#  "";? ) ;9?BJJ5 6"!".WW\\)$))+/CDNfEGMN$N$3""# !"G#s*C ACAC CC C98C9