+#hCdZddlZddlmZmZmZmZmZddlZddl m Z  ddl m Z ddl mZddlmZddlmZmZmZ ddlZGd d ej2Zy#e$r ed wxYw#e$rej.d e YBwxYw)zf Retriever module for Azure AI Search Author: Prajapati Harishkumar Kishorkumar (@HARISHKUMAR1112001) N)AnyCallableListOptionalUniondotdict)AzureKeyCredential) SearchClient)SearchItemPaged) QueryTypeVectorFilterModeVectorizedQueryzvYou need to install azure-search-documents libraryPlease use the command: pip install azure-search-documents==11.6.0b1zd`openai` is not installed. Install it with `pip install openai` to use AzureOpenAI embedding models.)categoryc*eZdZdZddddddddddej dddddfd ed ed ed ed eededee jdeedee de dededede deededee dee dee dee jf(fd Zdeded ed!ede dededede dedede de de d ede f d"Zd#ed$ed%efd&Zd'eeeefdeed(ej.fd)Zd ed*ed ed(eezfd+Zd,ZxZS)-AzureAISearchRMa A retrieval module that utilizes Azure AI Search to retrieve top passages for a given query. Args: search_service_name (str): The name of the Azure AI Search service. search_api_key (str): The API key for accessing the Azure AI Search service. search_index_name (str): The name of the search index in the Azure AI Search service. field_text (str): The name of the field containing text content in the search index. This field will be mapped to the "content" field in the dsp framework. field_vector (Optional[str]): The name of the field containing vector content in the search index. Defaults to None. k (int, optional): The default number of top passages to retrieve. Defaults to 3. azure_openai_client (Optional[openai.AzureOpenAI]): An instance of the AzureOpenAI client. Either openai_client or embedding_func must be provided. Defaults to None. openai_embed_model (Optional[str]): The name of the OpenAI embedding model. Defaults to "text-embedding-ada-002". embedding_func (Optional[Callable]): A function for generating embeddings. Either openai_client or embedding_func must be provided. Defaults to None. semantic_ranker (bool, optional): Whether to use semantic ranking. Defaults to False. filter (str, optional): Additional filter query. Defaults to None. query_language (str, optional): The language of the query. Defaults to "en-Us". query_speller (str, optional): The speller mode. Defaults to "lexicon". use_semantic_captions (bool, optional): Whether to use semantic captions. Defaults to False. query_type (Optional[QueryType], optional): The type of query. Defaults to QueryType.FULL. semantic_configuration_name (str, optional): The name of the semantic configuration. Defaults to None. is_vector_search (Optional[bool]): Whether to enable vector search. Defaults to False. is_hybrid_search (Optional[bool]): Whether to enable hybrid search. Defaults to False. is_fulltext_search (Optional[bool]): Whether to enable fulltext search. Defaults to True. vector_filter_mode (Optional[VectorFilterMode]): The vector filter mode. Defaults to None. Examples: Below is a code snippet that demonstrates how to instantiate and use the AzureAISearchRM class: ```python search_service_name = "your_search_service_name" search_api_key = "your_search_api_key" search_index_name = "your_search_index_name" field_text = "text_content_field" azure_search_retriever = AzureAISearchRM(search_service_name, search_api_key, search_index_name, field_text) ``` Attributes: search_service_name (str): The name of the Azure AI Search service. search_api_key (str): The API key for accessing the Azure AI Search service. search_index_name (str): The name of the search index in the Azure AI Search service. endpoint (str): The endpoint URL for the Azure AI Search service. field_text (str): The name of the field containing text content in the search index. field_vector (Optional[str]): The name of the field containing vector content in the search index. azure_openai_client (Optional[openai.AzureOpenAI]): An instance of the AzureOpenAI client. openai_embed_model (Optional[str]): The name of the OpenAI embedding model. embedding_func (Optional[Callable]): A function for generating embeddings. credential (AzureKeyCredential): The Azure key credential for accessing the service. client (SearchClient): The Azure AI Search client instance. semantic_ranker (bool): Whether to use semantic ranking. filter (str): Additional filter query. query_language (str): The language of the query. query_speller (str): The speller mode. use_semantic_captions (bool): Whether to use semantic captions. query_type (Optional[QueryType]): The type of query. semantic_configuration_name (str): The name of the semantic configuration. is_vector_search (Optional[bool]): Whether to enable vector search. is_hybrid_search (Optional[bool]): Whether to enable hybrid search. is_fulltext_search (Optional[bool]): Whether to enable fulltext search. vector_filter_mode (Optional[VectorFilterMode]): The vector filter mode. Methods: forward(query_or_queries: Union[str, List[str]], k: Optional[int]) -> dspy.Prediction: Search for the top passages corresponding to the given query or queries. azure_search_request( self, key_content: str, client: SearchClient, query: str, top: int, semantic_ranker: bool, filter: str, query_language: str, query_speller: str, use_semantic_captions: bool, query_type: QueryType, semantic_configuration_name: str, is_vector_search: bool, is_hybrid_search: bool, is_fulltext_search: bool, field_vector: str, vector_filter_mode: VectorFilterMode ) -> List[dict]: Perform a search request to the Azure AI Search service. process_azure_result( self, results:SearchItemPaged, content_key:str, content_score: str ) -> List[dict]: Process the results received from the Azure AI Search service and map them to the correct format. get_embeddings( self, query: str, k_nearest_neighbors: int, field_vector: str ) -> List | Any: Returns embeddings for the given query. check_semantic_configuration( self, semantic_configuration_name, query_type ): Checks semantic configuration. Raises: ImportError: If the required Azure AI Search libraries are not installed. Note: This class relies on the 'azure-search-documents' library for interacting with the Azure AI Search service. Ensure that you have the necessary permissions and correct configurations set up in Azure before using this class. Nztext-embedding-ada-002Fzen-UslexiconTsearch_service_namesearch_api_keysearch_index_name field_text field_vectorkazure_openai_clientopenai_embed_modelembedding_funcsemantic_rankerfilterquery_language query_spelleruse_semantic_captions query_typesemantic_configuration_nameis_vector_searchis_hybrid_searchis_fulltext_searchvector_filter_modec||_||_||_d|jd|_||_||_||_||_| |_t|j|_ t|j|j|j|_ | |_ | |_| |_| |_||_||_||_||_||_||_||_t0|e|y)Nzhttps://z.search.windows.net)endpoint index_name credential)r)rrrr*rrrrrr r,r clientrrr r!r"r#r$r%r&r'r(super__init__)selfrrrrrrrrrrrr r!r"r#r$r%r&r'r( __class__s [/var/www/html/sandstorm/venv/lib/python3.12/site-packages/dspy/retrieve/azureaisearch_rm.pyr/zAzureAISearchRM.__init__s.$7 ,!2"4#;#;"<z6AzureAISearchRM.azure_search_request.. s q7Ir3T)keyreverse)get_embeddingscheck_semantic_configurationsearchsortedprocess_azure_result)r0r4r-r5r6rrr r!r"r#r$r%r&r'rr( vector_queryresultssorted_resultss r2azure_search_requestz$AzureAISearchRM.azure_search_requests. ..uc<HL112MzZ -- $!)$0>'90KCX#?^b( !-- $!$0>'9 ( ..uc<HL112MzZ -- %!)#1"/0K$0>'9CX#?^b( !-- %!#1"/$0>'9( 112MzZ -- %!)#1"/0KCX#?^b( !--Es6-R-ISWX22>;P[\r3rG content_key content_scorecg}|D]H}i}|jD] \}}||k(r||d<||k(r||d<|||<"|j|J|S)z~ process received result from Azure AI Search as dictionary array and map content and score to correct format textscore)itemsappend) r0rGrJrKresresulttmpr?values r2rEz$AzureAISearchRM.process_azure_result&su FC$lln % U+%"'CKM)#(CL$CH  % JJsO  r3query_or_queriesreturnc||n |j}t|tr|gn|}|Dcgc]}|s| }}g}|D]}|j|j|j |||j |j|j|j|j|j|j|j|j|j|j |j"}|j%d|D|Scc}w)a  Search with pinecone for self.k top passages for query Args: query_or_queries (Union[str, List[str]]): The query or queries to search for. Returns: dspy.Prediction: An object containing the retrieved passages. c3<K|]}td|diyw) long_textrMNr).0ds r2 z*AzureAISearchRM.forward..ZsO!G[!F)$<=Os)r isinstancestrrIrr-rrr r!r"r#r$r%r&r'rr(extend)r0rUrqueriesqpassagesr5rGs r2forwardzAzureAISearchRM.forward7sADFF(23CS(I#$O_%+1++ PE// $$ ##""**00%%%%''!!''!G$ OOOwO O' P*1,s DDk_nearest_neighborsc>|js|jsJd|jb|sJd|jjj||jj dj }t|||}|gS|j|S)as Returns embeddings for the given query. Args: query (str): The query for which embeddings are to be retrieved. k_nearest_neighbors (int): The number of nearest neighbors to consider. field_vector (str): The field vector to use for embeddings. Returns: list: A list containing the vectorized query. Any: The result of embedding_func if azure_openai_client is not provided. Raises: AssertionError: If neither azure_openai_client nor embedding_func is provided, or if field_vector is not provided. z>Either azure_openai_client or embedding_func must be provided.zfield_vector must be provided.)inputmodelr)vectorrdfields)rr embeddingscreaterdata embeddingr)r0r5rdrrmrFs r2rAzAzureAISearchRM.get_embeddings^s$  $ $(;(; L K L ;  # # / A!A A<((33::dNeNe:fkklmnxx + 6IR^L!> !&&u- -r3cJ|sJd|tjk(sJdy)aW Checks semantic configuration. Args: semantic_configuration_name: The name of the semantic configuration. query_type: The type of the query. Raises: AssertionError: If semantic_configuration_name is not provided or if query_type is not QueryType.SEMANTIC. z-Semantic configuration name must be provided.z&Query type must be QueryType.SEMANTIC.N)r SEMANTIC)r0r$r#s r2rBz,AzureAISearchRM.check_semantic_configurations-+[,[[*Y///Y1YY/r3)__name__ __module__ __qualname____doc__r FULLr^rintopenai AzureOpenAIrboolr PRE_FILTERr/r rIr rErrdspy PredictionrcrrArB __classcell__)r1s@r2rr strt'+<@,D-1 %%&&+*3..+/+0+0-1DH+1 11 1  1 sm 1 1&f&8&891%SM1!*11111 $1 Y'!1"&)#1$#4.%1&#4.'1(%TN)1*%%5%@%@A+1f[[[ [  [  [[[[ $[[&)[[[![ !["-#[zO#^a"%c49n(=%(3-%TXTcTc%N .C .c .QT .Y]`cYc .D Zr3r)rswarningstypingrrrrrrzdsp.utils.utilsr azure.core.credentialsr azure.search.documentsr azure.search.documents._pagingr azure.search.documents.modelsr rr ImportErrorrvwarn ImportWarningRetrieverr<r3r2rs 77 # 93>ZZmZdmmmZ  O HMMnsAA(A%(BB