R"h>ddlZddlZddlZddlZddlmZmZmZm Z m Z m Z ddl m Z ddlmZGddZGdd ZGd d Zy) N)SetDictListOptionalUnionTuple)Encoder) InformationceZdZdZ ddedededeedeedeeed eeed eeefd Zd Z d Z e de fdZ y)ConversationTurna< A class to represent a turn in a conversation. Attributes: role (str): A short phrase of the role of the speaker for the current conversation turn. raw_utterance (str): The response generated by the LM model without polished style and tone. utterance_type (str): The type of utterance (e.g., statement, question). claim_to_make (Optional[str]): The point that this utterance tries to make. Should be empty if the utterance type is questioning. utterance (Optional[str]): The response generated by the model with polished style and tone. Defaults to raw_utterance if not provided. queries (List[str]): The queries used to gather information to have a grounded answer. raw_retrieved_info (List['Information']): A list of Information type that is retrieved. cited_info (Dict[int, 'Information']): A dictionary where the key is the citation index and the value is Information type. role_description (Optional[str]): A few sentences description of the role. Defaults to an empty string if not provided. Nrole raw_utteranceutterance_type claim_to_make utterancequeriesraw_retrieved_info cited_infoc ||n||_||_d|vr|n|jdd|_d|vrdn|jdd|_||ng|_||ng|_||ni|_||_|||_ yd|_ y)N:rr ) rrsplitrrole_descriptionrrrrr) selfrrrrrrrrs :/var/www/html/sandstorm/storm/knowledge_storm/dataclass.py__init__zConversationTurn.__init__s'0&;*tODC1C &)o4::c?1;M")"5w2 "4"@ b )3(>*B,.;.G]Rctjd}ttt|j |j S)Nz \[(\d+)\])recompilelistmapintfindallr)rcitation_patterns rget_all_citation_indexz'ConversationTurn.get_all_citation_index2s3::l3C-55dnnEFGGrc |jDcgc]}|j}}|j|j|j|j |j |j|j|dd Scc}w)N) rrrrrrrrr) rto_dictrrrrrrr)rinfors rr)zConversationTurn.to_dict6sq9=9P9PQdllnQQ!//II $ 5 5||"11!//"4  RsA:conv_turn_dictc |dDcgc]}tj|}}||d|d|dd|d|d|d|d|d  Scc}w) Nrrrrz: rrrr)rrrrrrrr)r from_dict)clsr+r*rs rr-zConversationTurn.from_dictDs5CCW4X ,0K ! !$ '  $[1(9"6*+2n=O.P-QR"9-1)*:;(9   sA)NNNNN)__name__ __module__ __qualname____doc__strrrr rr'r) classmethodrr-rrr r s ((,#''+:>26PPP P  } P C= P$s)$P%T+%67PT+./P.H   t  rr c eZdZdZ ddedeededdeeddeedef d Zd Z d efd Z dd ed efdZ dZ dZ dZdZddedfdZdefdZdedfdZdedfdZdZedZy) KnowledgeNodea5 Class representing a node in the knowledge base. Attributes: name (str): The name of the node. content (list): A list of Information instances. children (list): A list of child KnowledgeNode instances. parent (KnowledgeNode): The parent node of the current node. Nnamecontentparentchildrensynthesize_output!need_regenerate_synthesize_outputc||_| t|n t|_|gn||_||_||_||_y)a- Initializes a KnowledgeNode instance. Args: name (str): The name of the node. content (list, optional): A list of information uuid. Defaults to None. parent (KnowledgeNode, optional): The parent node of the current node. Defaults to None. N)r8setr9r;r:r<r=)rr8r9r:r;r<r=s rrzKnowledgeNode.__init__asD" 181DW#% &.H  !21R.rct|j}|jD]!}|j|j #|S)z Collects all content from the current node and its descendants. Returns: Set[int]: A set containing all content from the current node and its descendants. )r?r9r;updatecollect_all_content)r all_contentchilds rrBz!KnowledgeNode.collect_all_contentysB$,,' ]] >/ *!V+!]]%Ezz_4$ %$}4..??efjfofoeppqr#E  Z(rc|jS)z Returns the parent node of the current node. Returns: KnowledgeNode: The parent node of the current node. )r:rs r get_parentzKnowledgeNode.get_parents{{rc|jS)z Returns the children of the current node. Returns: list: A list of child KnowledgeNode instances. r;rSs r get_childrenzKnowledgeNode.get_childrens}}rcT|jDcgc]}|jc}Scc}w)z3 Returns a list of children names. rGrrDs rget_children_namesz KnowledgeNode.get_children_namess )- 6u 666s%chd|jd|jdt|jdS)z Returns a string representation of the KnowledgeNode instance. Returns: str: String representation of the KnowledgeNode instance. zKnowledgeNode(name=z , content=z , children=))r8r9lenr;rSs r__repr__zKnowledgeNode.__repr__s6%TYYKz$,,{SVW[WdWdSeRffghhrrootcg}|}|rF|j|j||j|jk(rn|j}|rF|dddS)z Get a list of names from the root to this node. Returns: List[str]: A list of node names from the root to this node. N)rOr8r:)rr_path current_nodes rget_path_from_rootz KnowledgeNode.get_path_from_roots]  KK )) *L$5$5$B'..L  DbDzrinformation_indexcf||jvr#d|_|jj|yyNT)r9r=add)rres rinsert_informationz KnowledgeNode.insert_informations. DLL 059D 2 LL  . / 1rreturnc*gfd|S)z Get a list of all descendant nodes. Returns: List[KnowledgeNode]: A list of all descendant nodes. cZ|jD]}j||yNr;rO)noderDcollect_descendents descendentss rrpz>KnowledgeNode.get_all_descendents..collect_descendentss, +""5)#E* +rr5)rrprqs @@rget_all_descendentsz!KnowledgeNode.get_all_descendentss  + D!rcfg}|j}| |j||j}| |S)z Get a list of all predecessor nodes (from current node to root). Returns: List[KnowledgeNode]: A list of all predecessor nodes. )r:rO)r predecessorsrcs rget_all_predecessorsz"KnowledgeNode.get_all_predecessorssA {{ &    -'..L&rc|jt|j|jDcgc]}|j c}|j r|j jnd|j |jdScc}w)z Converts the KnowledgeNode instance to a dictionary representation. Returns: dict: The dictionary representation of the KnowledgeNode. N)r8r9r;r:r<r=)r8r"r9r;r)r:r<r=rYs rr)zKnowledgeNode.to_dictsfIIDLL)6:mmDUD*.++dkk&&4!%!7!7151W1W   EsBc"dfd ||S)a Constructs a KnowledgeNode instance from a dictionary representation. Args: data (dict): The dictionary representation of the KnowledgeNode. Returns: KnowledgeNode: The constructed KnowledgeNode instance. c ||d|d|jk(sJ||d|d|d|jdd|jdd}|dD](}||| }|jj|*|S) Nr:r8r9r<r=T)r8r9r:r;r<r=r;) parent_node)r8getr;rO)r.dataryro child_datarPhelpers rr}z'KnowledgeNode.from_dict..helpers&H~1d8n HXHX6XXX&\Y""&((+>"E26((73 D#:. 1 #CF  $$Z0 1Krrmr5)r.r{r}s @rr-zKnowledgeNode.from_dicts $c4  r)NNNNTrKrm)r/r0r1r2r3rrboolrrBrHrQrTrWrZr^rdr$rirrrur)r4r-r5rrr7r7Vs"&,048+/26SS#S) S 401 S $C= S,0S0 JJ #$7 ix'@ 0C0 T/%:" d?&;  !!rr7c .eZdZdZdedeejjejjfde de fdZ dZ ededeejjejjfde de fd Z d&d eed eej*eeffd ZdZdZdZ d'deefdZdZd(dZ d)deee d eed efdZ d*ded eefdZ d*dede d eefdZ!dZ"dZ#dZ$dZ% d+de&d e'd!e'fd"Z(d#Z)d$Z*d%Z+y ), KnowledgeBasea Represents the dynamic, hierarchical mind map used in Co-STORM to track and organize discourse. The knowledge base serves as a shared conceptual space between the user and the system, allowing for effective collaboration by reducing the user's cognitive load and ensuring that the discourse is easy to follow. The knowledge base is structured as a tree (or mind map) that dynamically organizes collected information and concepts as the conversation progresses. The mind map consists of concepts (nodes) and edges that represent parent-child relationships among topics. Each concept is linked to retrieved information, which is placed under the most appropriate concept based on its associated question and semantic similarity. For more details, please refer to Section 3.2 of Co-STORM paper: https://www.arxiv.org/pdf/2408.15232 Attributes: root (KnowledgeNode): The root node of the hierarchical knowledge base, representing the top-level concept. topicknowledge_base_lmnode_expansion_trigger_countencodercddlm}ddlm}m}ddlm}||_||_|||j|_ |||j||_ |||_ |||_ td |_td t!j"ggd d |_i|_i|_t+j,|_y ) a Initializes a KnowledgeBase instance. Args: topic (str): The topic of the knowledge base expand_node_module (dspy.Module): The module that organize knowledge base in place. The module should accept knowledge base as param. E.g. expand_node_module(self) article_generation_module (dspy.Module): The module that generate report from knowledge base. The module should return string. E.g. report = article_generation_module(self) r )ArticleGenerationModule)InsertInformationModuleExpandNodeModule)KnowledgeBaseSummaryModule)enginer)rinformation_insert_moduler)rr_r8rhashencoded_structurestructure_stringN).collaborative_storm.modules.article_generationr8collaborative_storm.modules.information_insertion_modulerr2collaborative_storm.modules.knowledge_base_summaryrrrrexpand_node_modulearticle_generation_modulegen_summary_moduler7r_rnparray kb_embeddinginfo_uuid_to_info_dictinfo_hash_to_uuid_dict threadingLock_lock) rrrrrrrrrs rrzKnowledgeBase.__init__4s"     ' )@$dll* &#3$&*&D&D)E#  *A$* &#=DU"V#0f#= H!#2$ "  ?A#68#^^% rc|jjDcic]\}}||j}}}|j|jj||j dScc}}w)N)rtreerr)ritemsr)rr_r)rkeyvaluers rr)zKnowledgeBase.to_dictjso373N3N3T3T3V" %/S%C " " ZZII%%'&<&*&A&A   " sA/r{c||d|||}tj|d|_|djDcic]\}}t |t |c}}|_|djDcic]$\}}t |t j|&}}}||_|Scc}}wcc}}w)Nr)rrrrrrr)r7r-r_rr$rr r) r.r{rrrknowledge_baserrrs rr-zKnowledgeBase.from_dictusw-/)E   ,55d6lC##;<BBD1 U Hc%j 1 - ##;<BBD" U Hk++E2 2" " 1G-1 " s B6)B<Nr_rjcX|jddd|}t|}||jdk7rV|jd}|Dcgc]}|j dd}}|j j |}|||d|_|jd |jd fScc}w) NFT)include_indentinclude_full_pathinclude_hash_tagr_r  -> z, rrr)get_node_hierarchy_stringrrrreplacerencode)rr_outline_stringoutline_string_hashoutline_stringsoutlinecleaned_outline_stringsencoded_outlines r&get_knowledge_base_structure_embeddingz4KnowledgeBase.get_knowledge_base_structure_embeddings77 "" 8  #>2 $"3"3F"; ;)7)=)=d)CO=L'29-' #'#ll112IJO+%4$3!D    1 2   0 1  'sB'c*gfd|S)z Traverses the tree downward from the given node. Args: node (KnowledgeNode): The node to start the traversal from. Returns: list: A list of KnowledgeNode instances in the order they were visited. cbj||jD] }| yrm)rOrW)rcrD _traversenodess rrz.KnowledgeBase.traverse_down.._traverses/ LL &%224 !%  !rr5)rrorrs @@r traverse_downzKnowledgeBase.traverse_downs ! $ rcVg}|$|j||j}|$|S)z Traverses the tree upward from the given node. Args: node (KnowledgeNode): The node to start the traversal from. Returns: list: A list of KnowledgeNode instances in the order they were visited. )rOrT)rrors r traverse_upzKnowledgeBase.traverse_ups6 LL ??$D rc>gfd|jS)NcZj||jD] }| yrm)rOr;)rorD_collectrs rrz1KnowledgeBase.collect_all_nodes.._collects) LL   rr_)rrrs @@rcollect_all_nodeszKnowledgeBase.collect_all_nodess   rrycf||jj||S|j||S)a Inserts a new node into the knowledge base under the specified parent node. Args: new_node_name (str): The name of the new node. parent_node_name (str): The name of the parent node. If None, the new node is inserted under the root. duplicate_handling (str): How to handle duplicate nodes. Options are "skip", "none", and "raise error". )rI)r_rQ)r new_node_nameryrIs r insert_nodezKnowledgeBase.insert_nodesL  99&&2D' ((2D) rc|j|k(r|S|jD]}|j||}||cSy)a9 Finds a node by name in the knowledge base. Args: current_node (KnowledgeNode): The node to start the search from. node_name (str): The name of the node to find. Returns: KnowledgeNode: The node with the specified name, or None if not found. N)r8rW find_node)rrc node_namerDresults rrzKnowledgeBase.find_nodesP    ) !..0 E^^E95F!  rc|i}|jdD]}|jd}|dkDs|jdj}|jdvrL|dk(rdn|j |dz }|j |||}|||<t |jD] } | |kDs || = y) aF Creates and inserts nodes into the knowledge base from a string outline. Args: outline_string (str): The outline string where each line starts with '#' denoting the level. duplicate_handling (str): How to handle duplicate nodes. Options are "skip", "none", and "raise error". r#rz# )overviewsummary introductionr N)rryrI)rcountstriplowerrzrr"keys) rrrIlast_node_at_levellineleveltitlerynew_node deeper_levels rinsert_from_outline_stringz(KnowledgeBase.insert_from_outline_strings "((. =DJJsOEqy 4(..0;;=$KK&+qjd6H6L6LUUVY6W ++"' +'9, -5"5)$();)@)@)B$C=L#e+.|<= =r cited_indicesc d}ttD]}||j|D]} jdj| j j| j | j | j} | D]} j | jj | fdfdf dg} A|j5|jjD]} | j| dn| jddj| S) Nc.gfd|S)a Traverses the tree downward from the given node. Args: node (KnowledgeNode): The node to start the traversal from. Returns: list: A list of KnowledgeNode instances in the order they were visited. c||jvrj||jD] }| yrm)r9rOrW)rcrDrindexrs rrz[KnowledgeBase.get_node_hierarchy_string..find_node_contain_index.._traverse1sB+9M9M0MLL.)668%Ee$%rr5)rorrrs `@@rfind_node_contain_indexzHKnowledgeBase.get_node_hierarchy_string..find_node_contain_index%sE % dOLrrcy|vSrgr5)rornodes_to_includes rshould_include_nodezDKnowledgeBase.get_node_hierarchy_string..should_include_nodeGs$++ +rcBy|jD] }|s yy)NFTrV)rorDrrs rshould_omit_child_nodeszHKnowledgeBase.get_node_hierarchy_string..should_omit_child_nodesLs/$ !&u-  !rc g}||}sdnd|dz z}dj|j}s |jn|}rd|zdznd}rdt|jd nd} |vrdnd } |r|j ||||| |r9t|j d kDrsdnd|zx} }|j | d |S|j D]} |j | |dz|S) Nr r rrr z (r\u ⭐rz...)joinrdr8r]r9rOr;extend)cur_rootr to_returnshould_include_current_nodeindent full_path node_infohash_tag content_count special_note child_indentrDrr}rrrinclude_node_content_countpaths_to_highlightr_rrs rr}z7KnowledgeBase.get_node_hierarchy_string..helperTsXI#.A(.K+#1tuqy7I"KK(C(C(C(NO 1BHMM 0@3;,b5ObX--./q1UW %, AS0S /$$!(8*YK |nU/x8x001A5*8denL6&,, ~S-AB &.%6%6GE%,,VE519-EFG rr )rr) r?r_rhrrdrArrrur;r)rrrrrrr_rrcur_nodert predecessorrrDr}rrrrs `````` @@@@@rrz'KnowledgeBase.get_node_hierarchy_stringsN *!U5  $& : 7 5 I:H&**6;;x7R7R7T+UV$((2$++H,H,H,JK#+#@#@#BL'3F (// 0D0DEF$++L9: : ,    B  ". missing_node_handling (str): How to handle missing nodes. Options are "abort", "create", and "raise error". Returns: KnowledgeNode: The target node. rNr c3BK|]}|jk(s|ywrmr).0rDr8s r z2KnowledgeBase.find_node_by_path..sP5UZZ4=OPsabortcreate)rErLTF)rrrz/Insert information error. Unable to find node {z } under {z} )rr_nextr;rQrrNr8) rrbmissing_node_handlingr_ node_namesrc found_noder structurer8s @rfind_node_by_pathzKnowledgeBase.find_node_by_path~s ZZ' $(Ltyyd qrN *DPL$9$9PRVJ!(G3*h6+55d5KH#+L*m; $ > >'+*/)-!?!I $J4&P[\h\m\m[nnrs|r}~ * ) **r informationc|j5|j|||}t|}|jdk(rW|jj |t |jdz}||_||j|<||j|<|^dj|j|j|jjd<|j|jdddy#1swYyxYw)a Inserts information into the knowledge base at the specified path. Args: path (str): The placement path string, connected by " -> " linking the name of nodes. information (Information): The information to insert. missing_node_handling (str): How to handle missing nodes. Options are "abort", "create", and "raise error". Return: uuid of insertion information )rbrr_rar Nr placement) rrr citation_uuidrrzr]rrrdmetari)rrbrrr_ target_nodeinformation_hashinfo_citation_uuids rriz KnowledgeBase.insert_informations "ZZ J)-)?)?1FT*@*K $K0 ((B.%)%@%@%D%D$c$*E*E&F&J&"-? )@R++,<=BM++,>?&KK > > @A++K,E,EFKK..{/H/HI! J J Js C&C<<Dcfd t|j}|jt|j}||k(ryK)z Trims all leaf nodes that do not have any content. Iteratively does it until all leaf nodes have at least one content. c|js |jsy|jDcgc] }|r |c}|_|j xr |j Scc}wrg)r;r9)rorD trim_nodes rrz6KnowledgeBase.trim_empty_leaf_nodes..trim_nodesO==04 VuYuEUUVDM}}$9T\\)9 9Ws AAN)r]get_all_leaf_nodesr_)r before_trim after_trimrs @rtrim_empty_leaf_nodesz#KnowledgeBase.trim_empty_leaf_nodessO  :d5578K dii T4467Jj( rc>gfd|jS)z Helper function to get all leaf nodes. Returns: List[KnowledgeNode]: A list of all leaf nodes in the knowledge base. cr|jsj||jD] }| yrmrn)rorDfind_leaf_nodes leaf_nodess rrz9KnowledgeBase.get_all_leaf_nodes..find_leaf_nodess3==!!$' '& 'rr)rrrs @@rrz KnowledgeBase.get_all_leaf_nodess#  '  "rc4fd|jy)z Merges content of a node with its single child and removes the child node. Iteratively does this from leaf nodes back to the root. c$|jD] }| t|jdk(r^|jd}|jj|j|j|_|jD] }||_ yy)Nr r)r;r]r9rAr:)rorD single_child grandchild merge_nodes rrz:KnowledgeBase.merge_single_child_nodes..merge_nodes "5! "4==!Q&#}}Q/  ##L$8$89 , 5 5 "&---J(,J%- 'rNr)rrs @rmerge_single_child_nodesz&KnowledgeBase.merge_single_child_nodess - 499rc8fdjy)Nc|jD];}dj|jj|jd<=|j D] }| y)Nrr)r9rrdrrr;)ro citation_idxrD_helperrs rr z3KnowledgeBase.update_all_info_path.._helpersb $   KK 7 7 9:++L9>>{K   rr)rr s`@rupdate_all_info_pathz"KnowledgeBase.update_all_info_paths   r conv_turnallow_create_new_nodeinsert_under_rootcz|yt|jj}|r/|D])}|j|jj |+n|j ||||jjDcic]\}}||j}}}|jD]W\}}|jjd|dd|d|_ |jjd|dd|d|_ Y|jD]\} }|jjd|dd|d|_ |jjdd|jjd|dd|d|_ |jjddd|_ycc}}w) N)rbr)rrr#[]z[_z_]z[-1]r) r"rvaluesrir_r8rrrrrr) rr"r#r$info_to_insertr*old_idxold_to_new_citation_idx_mappingnew_idx_s rupdate_from_conv_turnz#KnowledgeBase.update_from_conv_turns   i2299;< & O''TYY^^'N O  * *#*&; + "+!5!5!;!;!=+  T'' '+ '+ !@ E E G  GW"+"5"5"="=G9A"WIR 0#I '0&=&=&E&EG9A"WIR 0'I #  :??A 8JAw"+"5"5"="=WIR AgYa.#I     ' ' 3&/&=&=&E&EWIR AgYa.'I #  # # + +FB 7 8 $ ++ s F7c$|j|Srm)rrSs rget_knowledge_base_summaryz(KnowledgeBase.get_knowledge_base_summary9s&&t,,rc|j|j|j||j|j|jy)a Reorganizes the knowledge base through two main processes: top-down expansion and bottom-up cleaning. The reorganization process ensures that the knowledge base remains well-structured and relevant as new information is added. It consists of the following steps: 1.Top-Down Expansion: Expands nodes that have accumulated significant amounts of information by creating subtopics, ensuring that each concept remains specific and manageable. 2.Bottom-Up Cleaning: Cleans the knowledge base by removing empty leaf nodes (nodes with no supporting information) and merging nodes that have only a single child, simplifying the structure and maintaining clarity. rN)rrrr!rSs r reogranizezKnowledgeBase.reogranize<sQ ""$ %%' t4 ""$ %%' !!#rc&|j|S)Nr2)rrSs r to_reportzKnowledgeBase.to_reportPs--T-BBrrm)NrKr~)FFTFNN)rN)FF),r/r0r1r2r3rdspydspLMHFModelr$r rr)r4rr-rr7rrndarrayrrrrrrrrrrr rirrrr!r rr.r0r3r5r5rrrr#s 4&4&!dhh.>.>!>?4&'* 4&  4&l  !dhh.>.>!>?'*   4/3 ]+ rzz49$ % 4( 04! m,.&=8#(-1(,`$  S * `$}%`$ `$J&(, ((}% (\&(, !J!J!!J }% !JF&$* ',"' '$#'$ $'$ '$R-$(Crr)r6numpyrr rtypingrrrrrrrr interfacer r r7rr5rrr>sE  ::"H H VJ!J!ZnCnCr