+#h >ddlZddlmZmZddlZddlZddlmZm Z m Z m Z m Z m Z Gdde ZdZdeedefd Zdeed ed eefd Zd eedeed efdZddZdedeed efdZded efdZded efdZefdeedeegefd eeegeeffdZy)N)AnyCallable)EMF1 DPR_normalizedotdict has_answernormalize_textcFeZdZUdZeeed<fdZdZdZ dZ xZ S)Examplez0A primitive datatype for representing an exampledemosct|dksJt| |r$t|dk(sJ|j|d|jdi|y)Nr)lensuper__init__update)selfargskwargs __class__s W/var/www/html/sandstorm/venv/lib/python3.12/site-packages/dsp/primitives/demonstrate.pyrzExample.__init__sR4yA~~  t9> !> KKQ  fc 6tdiit||}|S)Nr)r dict)rrthe_copys rcopyz Example.copys#65T 5f56rct|}t|jDcic] \}}||vs ||c}}Scc}}w)z=Removes the provided keys from the example and returns a copy)setr items)rkeyskvs rwithoutzExample.withouts84yGA$1GHHGs > > c fd}|jDcgc]}|jdi||}}|j|Scc}w)zWReturns a copy of the example with the demos stage transformed by the provided functioncV |jdS#t$ricYSwxYw)N augmented)r% Exception)examplefns ratzExample.demos_at..at&s1 '{**;77   s  (()r r)r r)rr+r,r*r s ` rdemos_atzExample.demos_at#sK  =AJJG,7 ,GGyyuy%%HsA) __name__ __module__ __qualname____doc__listr__annotations__rrr%r- __classcell__)rs@rr r s%: 9 I &rr cdfd }|S)zUReturns an Augment function that applies the provided transformations to the Examplescg}g}|D]o}tj|}|t||k\rd}D]}|n ||}|d|_|j |Xd|_|j |q|r||zS|S)NTF)dspr rr(append) trainr# return_allrdemosademosr* raw_exampleftransformationss r do_augmentzannotate..do_augment3s +G++g.K 3v;!#3$ %?G*  % "$(! g&(- % k*# +& F? " r)NFr)r?r@s` rannotaterA0s8 rr9r#ctjtjj}|Dcgc]}tj |}}|j ||d|Scc}w)zSample k examples from train.N)randomRandomr7settings branch_idxr shuffle)r9r#rngr*shuffled_trains rsamplerJRsU -- // 0C:?@wckk'*@N@KK "1 AsA&xreturnc |Dcgc]h}tjt|jdg|jgzt|jdg|jgzs|j}}|Scc}w)zORemoves the example x from the train set by comparing the question and history.history)r intersectiongetquestion)r9rKyoutputs rall_butrT[sw   i$ |3 4 i$ |3 4  F MsA-A6passagesanswersc,tfd|DS)z8Returns True if any of the passages contains the answer.c36K|]}t|yw)N)passage_has_answers).0psgrVs r z passage_match..lsES"30Es)any)rUrVs `r passage_matchr^js EHE EErcB|dk\r t||St|||k\S)N?)rr) predictionrVfracs r answer_matchrcos* s{*g&& j' "d **rpassagec xt|Dcgc]}tt|c}t|Scc}w)z0Returns True if the passage contains the answer.)tokenized_answerstext)r rr )rdrVanss rrYrYys2 IPQ#=)<=Q G $ Qs7 inp_examplec:|j|jS)ze Extracts question as a field to vectorize with Vectorizer object. `question` field is used. text_to_vectorize)rrQ)ris r!cast_naive_get_only_question_textrms   k.B.B  CCrc|jjdz|jjz}|j|S)z Extracts question and answer as fields to vectorize with Vectorizer object. `question` and `answer` fields are used. They will be concatenated with the word "Answer" between. z Answer: rk)rQstripanswerr)rirls r"cast_naive_get_question_and_answerrqsK ""${2[5G5G5M5M5OO   .?  @@rcastc  ddlm}Dcgc] }| }}tjj  |j t j}|d |jdtd|j|j|dtdtdttf fd }|Scc}w) a A function that vectorizes train data using `dsm.settings.vectorizer`, then build an ANN/KNN index to search similar questions among `train` samples. Args: train: a bunch of questions to put in index & search later cast: function that contructs text before vectorization. By default, it uses only question. Check `cast_naive_get_question_and_answer` for more details. n_probe: number of closest IVF-clusters to check for neighbours. Doesn't affect bruteforce-based search. knn_args: check `create_faiss_index` function for details on ANN/KNN arguments. Returns: function to search similar Examples from `train` in FAISS-index. r)create_faiss_indexr)emb_dim n_objectsrir#rLc |g}j||\}}|dDcgc]} | }}|Scc}w)Nr)search) rir#inp_example_vector_nearest_samples_idxscur_idx train_sampledrrindexr9 vectorizers rinner_knn_searchzknn..inner_knn_searchsU'k):(;<"',,/A1"E 7KA7NOGwO OPs =r)dsp.utils.ann_utilsrtr7rErastypenpfloat32shaperr9addr intr2) r9rrknn_argsrtcur_elemtrain_casted_to_vectorize all_vectorsrr~rs `` @@rknnrs$7@E FHh F F+.<<+B+BJ67>>rzzJK  !!!$E  >F E KK  IIkg#$w- #!GsC)r`)rCtypingrrnumpyrr7 dsp.utilsrrrrr r r rAr2rrJrTstrboolr^rcrYrmrqrrrrrs  PP#&g#&LD$w-C 4= W g FDIFS FdF +tCyTD7DwD AG A A*K% =% G9g% &%wnd7m+, %r