+#h*ddlZddlmZmZddlZ ddlZddlmZddlmZ ddl Z Gdd ejZy#e $r e dwxYw#e $rejde YBwxYw) N)CallableOptional)register_vector)sqlzcThe 'pgvector' extra is required to use PgVectorRM. Install it with `pip install dspy-ai[pgvector]`z_`openai` is not installed. Install it with `pip install openai` to use OpenAI embedding models.)categoryceZdZdZ ddededeejdeede dedee ed ed e ffd Z d efd Z d ede efdZxZS) PgVectorRMa5 Implements a retriever that (as the name suggests) uses pgvector to retrieve passages, using a raw SQL query and a postgresql connection managed by psycopg2. It needs to register the pgvector extension with the psycopg2 connection Returns a list of dspy.Example objects Args: db_url (str): A PostgreSQL database URL in psycopg2's DSN format pg_table_name (Optional[str]): name of the table containing passages openai_client (openai.OpenAI): OpenAI client to use for computing query embeddings. Either openai_client or embedding_func must be provided. embedding_func (Callable): A function to use for computing query embeddings. Either openai_client or embedding_func must be provided. k (Optional[int]): Default number of top passages to retrieve. Defaults to 20 embedding_field (str = "embedding"): Field containing passage embeddings. Defaults to "embedding" fields (List[str] = ['text']): Fields to retrieve from the table. Defaults to "text" embedding_model (str = "text-embedding-ada-002"): Field containing the OpenAI embedding model to use. Defaults to "text-embedding-ada-002" Examples: Below is a code snippet that shows how to use PgVector as the default retriever ```python import dspy import openai import psycopg2 openai.api_key = os.environ.get("OPENAI_API_KEY", None) openai_client = openai.OpenAI() llm = dspy.OpenAI(model="gpt-3.5-turbo") DATABASE_URL should be in the format postgresql://user:password@host/database db_url=os.getenv("DATABASE_URL") retriever_model = PgVectorRM(conn, openai_client=openai_client, "paragraphs", fields=["text", "document_id"], k=20) dspy.settings.configure(lm=llm, rm=retriever_model) ``` Below is a code snippet that shows how to use PgVector in the forward() function of a module ```python self.retrieve = PgVectorRM(db_url, openai_client=openai_client, "paragraphs", fields=["text", "document_id"], k=20) ``` db_url pg_table_name openai_clientembedding_funckembedding_fieldfieldsembedding_modelinclude_similarityc |s |sJd||_||_tj||_t |j||_|xsdg|_||_||_ | |_ t |1|y)z@ k = 20 is the number of paragraphs to retrieve z8Either openai_client or embedding_func must be provided.text)rN) r r psycopg2connectconnrr rrrrsuper__init__) selfr r r r rrrrr __class__s V/var/www/html/sandstorm/venv/lib/python3.12/site-packages/dspy/retrieve/pgvector_rm.pyrzPgVectorRM.__init__Asj0jj.*,$$V,  "*( .."4 1queryc :|j|}g}tjdj|jDcgc]}tj |c}}|j rmtjdtjdjtj |jz}||z }|||jf}n||jf}tjdj|tj |jtj |j}|j5} | j5} | j||| j} | jD cgc]} | d } } | D];}t!t#| |}|j%t'j(di|= dddddd|Scc}wcc} w#1swYxYw#1swY|SxYw) aSSearch with PgVector for self.k top passages for query using cosine similarity Args: query (str): The query to search for include_similarity (bool): Whether or not to include the similarity for each record Returns: dspy.Prediction: an object containing the retrieved passages. ,z,1 - ({embedding_field} <=> %s) AS similarity)rzOselect {fields} from {table} order by {embedding_field} <=> %s::vector limit %s)rtablerrN)_get_embeddingsrSQLjoinr Identifierrformatrrr rcursorexecutefetchall descriptiondictzipappenddspyExample)rrquery_embeddingretrieved_docsfrsimilarity_fieldargs sql_queryrcurrowsdescripcolumnsrowdatas rforwardzPgVectorRM.forward^s..u5""[[$  NN1 $    " " B&8L8L)M&NO  & &F#_dff=D#TVV,DGG ] &..!3!34NN4+?+?@   YY @$ @# ||~58__E'71:EE@CGS 12D"))$,,*>*>?@  @ @G$ <F  @ @ @s=G:H(1H G?%AH(H?HH HHreturnc|jI|jjj|j|djdj S|j |S)Nfloat)modelinputencoding_formatr)r embeddingscreaterr< embeddingr )rrs rr#zPgVectorRM._get_embeddingssi    )%%0077** '8d1  i  &&u- -r)NNrFNztext-embedding-ada-002F)__name__ __module__ __qualname____doc__strropenaiOpenAIrintlistboolrr=r@r# __classcell__)rs@rr r s*^6:15#.*.#;',$FMM2  %X.   !T#Y'!!%:1S1f.S.T%[.rr )warningstypingrrr/rpgvector.psycopg2rr ImportErrorrMwarn ImportWarningRetriever r"rrrZs{% 1 * D.D. m  *HMMs(**s=AA A*)A*