R"hRddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZmZmZmZmZddlmZej*ej,dej.eZerdd lmZGd d e ZGd d ZGddZGdde ZGddZ Gdde Z!Gdde Z"Gdde Z#Gdde Z$dZ%Gdde Z&Gdd e Z'Gd!d"e Z(y)#N)ABCabstractmethod) OrderedDict)DictListOptionalUnion TYPE_CHECKING)ArticleTextProcessingz(%(name)s : %(levelname)-8s : %(message)s)levelformat)LoggingWrapperc&eZdZdZdZedZy)InformationTableaO The InformationTable class serves as data class to store the information collected during KnowledgeCuration stage. Create subclass to incorporate more information as needed. For example, in STORM paper https://arxiv.org/pdf/2402.14207.pdf, additional information would be perspective guided dialogue history. cyNselfs :/var/www/html/sandstorm/storm/knowledge_storm/interface.py__init__zInformationTable.__init__! c yrr)kwargss rretrieve_informationz%InformationTable.retrieve_information$ rN)__name__ __module__ __qualname____doc__rrrrrrrrs    rrcLeZdZdZd dZdZdZdZdZdZe d Z d Z y) InformationaClass to represent detailed information. Inherits from Information to include a unique identifier (URL), and extends it with a description, snippets, and title of the storm information. Attributes: description (str): Brief description. snippets (list): List of brief excerpts or snippets. title (str): The title or headline of the information. url (str): The unique URL (serving as UUID) of the information. Nc`||_||_||_||_||ni|_d|_y)a`Initialize the Information object with detailed attributes. Args: url (str): The unique URL serving as the identifier for the information. description (str): Detailed description. snippets (list): List of brief excerpts or snippet. title (str): The title or headline of the information. N) descriptionsnippetstitleurlmeta citation_uuid)rr*r'r(r)r+s rrzInformation.__init__6s7'    ,D" rcht|jtt|jfSr)hashr*tuplesortedr(rs r__hash__zInformation.__hash__Fs,fT]]+,   rct|tsy|j|jk(xrNt|jt|jk(xr!|j |j k(S)NF) isinstancer$r*setr( _meta_str)rothers r__eq__zInformation.__eq__Ns^%- HH ! 6DMM"c%..&99 6 EOO$55 rc t|j|jtt |j |j fdS)N)int _md5_hashr*r/r0r(r5rs rr1zInformation.__hash__Ws> NNDHHeF4==,A&BDNNDTU V   rczd|jjddd|jjddS)z>Generate a string representation of relevant meta information.z Question: questionz , Query: query)r+getrs rr5zInformation._meta_str]s7DIIMM*b9:)DIIMMRY[]D^C_``rct|tttfrt j |d}t jt|jdjS)z'Generate an MD5 hash for a given value.T) sort_keyszutf-8) r3dictlistr/jsondumpshashlibmd5strencode hexdigest)rvalues rr;zInformation._md5_hashasJ edD%0 1JJu5E{{3u:,,W56@@BBrc ||d|d|d|d|jdd}t|jdd |_|S) a~Create a Information object from a dictionary. Usage: info = Information.from_dict(storm_info_dict) Args: info_dict (dict): A dictionary containing keys 'url', 'description', 'snippets', and 'title' corresponding to the object's attributes. Returns: Information: An instance of Information. r*r'r(r)r+N)r*r'r(r)r+r,r&)r@r:r,)cls info_dictinfos r from_dictzInformation.from_dictgs[% !-0z*G$vt,  !!CD rc|j|j|j|j|j|j dS)Nr*r'r(r)r+r,rSrs rto_dictzInformation.to_dict}s:88++ ZZII!//   rr) rr r!r"rr1r7r5r; classmethodrQrTrrrr$r$)s@     aC * rr$c,eZdZdZddefdZddZdZy) ArticleSectionNodez The ArticleSectionNode is the dataclass for handling the section of the article. The content storage, section writing preferences are defined in this node. N section_namec<||_||_g|_d|_y)z section_name: section heading in string format. E.g. Introduction, History, etc. content: content of the section. Up to you for design choice of the data structure. N)rXcontentchildren preference)rrXrZs rrzArticleSectionNode.__init__s! )  rcx|r|jjd|y|jj|y)Nr)r[insertappend)rnew_child_nodeinsert_to_fronts r add_childzArticleSectionNode.add_childs+  MM N 3 MM  0rc:|jj|yr)r[remove)rchilds r remove_childzArticleSectionNode.remove_childs U#rr)F)rr r!r"rIrrbrfrrrrWrWs S1 $rrWceZdZdZdededeefdZedefdZ dZ de efdZ e ed ed efd Zdd Zy )Articlec$t||_yr)rWroot)r topic_names rrzArticle.__init__s&z2 rnodenamereturncx|j|k(r|S|jD]}|j||}|s|cSy)a Return the node of the section given the section name. Args: node: the node as the root to find. name: the name of node as section name Return: reference of the node or None if section name has no match N)rXr[ find_section)rrlrmreresults rrpzArticle.find_sectionsI    $K]] E&&ud3F  rcy)zC Export Article object into string representation. Nrrs r to_stringzArticle.to_stringsrcZdtttfffd |jS)aS Generates a hierarchical tree structure representing the outline of the document. Returns: Dict[str, Dict]: A nested dictionary representing the hierarchical structure of the document's outline. Each key is a section name, and the value is another dictionary representing the child sections, recursively forming the tree structure of the document's outline. If a section has no subsections, its value is an empty dictionary. Example: Assuming a document with a structure like: - Introduction - Background - Objective - Methods - Data Collection - Analysis The method would return: { 'Introduction': { 'Background': {}, 'Objective': {} }, 'Methods': { 'Data Collection': {}, 'Analysis': {} } } rnc`i}|jD]}|||j<|r|SiSr)r[rX)rltreere build_trees rrwz,Article.get_outline_tree..build_trees=D =+5e+<U''( =4 'R 'r)rrIrj)rrws @rget_outline_treezArticle.get_outline_trees(> (S$Y ( $))$$rch|jjDcgc]}|jc}Scc}w)z/ Get first level section names )rjr[rX)ris rget_first_level_section_namesz%Article.get_first_level_section_namess&)- (:(:;1;;;s/rk article_textcy)zH Create an instance of the Article object from a string Nr)rNrkr|s r from_stringzArticle.from_strings rNc| |j}|jDcgc]}|j|s|c}|jdd|j|jdk(r |jsy|Scc}w)Nr>)rjr[prune_empty_nodesrZ)rrlres rrzArticle.prune_empty_nodessk <99D $}} 0F0Fu0ME  a LL DLLB$6 K s A2A2r)rr r!rrWrIrrprrsrxrr{rUr~rrrrrhrhs3&.1 $ %* 3  %%N  ! !'$"5"I"I"K L # ;E). ;% I]209M*-!*-:-  ; ;rr? exclude_urlsrnc(t|tr|n|g}g}fd}tjj j 5}t|j ||}dddD]}|j||S#1swY#xYw)Nc&j|g}g}|D]t}tt|dD]#}tj|d||d|<%t j |}||jd<|j|v|S)N)query_or_queriesrr(r?) rrangelenr remove_citationsr$rQr+r_)qretrieved_data_listlocal_to_returndatarz storm_inforrs r process_queryz)Retriever.retrieve..process_query&s"&''"#<#*# !O+ 3s4 #345A+@*P*PZ(++D$Q' )2248 +, (&&z2 3# "r) max_workers)r3rD concurrentfuturesThreadPoolExecutorrmapextend) rr?rqueries to_returnrexecutorresultsrqs ` ` rretrievezRetriever.retrieve s&eT2%  #"   2 23  A 8<< w?@G A  %F   V $ % A As BBN)r )rr r!r"dspyRetriever:rrr rIrr$rrrrrrs]4==c  GI3S >*:>s) k rrc2eZdZdZdefdZedefdZy)KnowledgeCurationModulez` The interface for knowledge curation stage. Given topic, return collected information. retrieverc||_y)z7 Store args and finish initialization. N)r)rrs rrz KnowledgeCurationModule.__init__Gs #rrncy)z Curate information and knowledge for the given topic Args: topic: topic of interest in natural language. Returns: collected_information: collected information in InformationTable type. Nr)rtopics rresearchz KnowledgeCurationModule.researchMs rN) rr r!r"rrrrrrrrrrBs/#)#   !1    rrc.eZdZdZedededefdZy)OutlineGenerationModulez The interface for outline generation stage. Given topic, collected information from knowledge curation stage, generate outline for the article. rinformation_tablernc y)a~ Generate outline for the article. Required arguments include: topic: the topic of interest information_table: knowledge curation data generated from KnowledgeCurationModule More arguments could be 1. draft outline 2. user provided outline Returns: article_outline of type ArticleOutline Nr)rrrrs rgenerate_outlinez(OutlineGenerationModule.generate_outlineas rN) rr r!r"rrIrrhrrrrrr[s3   -=   rrc 2eZdZdZededededefdZy)ArticleGenerationModule The interface for article generation stage. Given topic, collected information from knowledge curation stage, generated outline from outline generation stage, rrarticle_with_outlinernc y)a$ Generate article. Required arguments include: topic: the topic of interest information_table: knowledge curation data generated from KnowledgeCurationModule article_with_outline: article with specified outline from OutlineGenerationModule Nr)rrrrrs rgenerate_articlez(ArticleGenerationModule.generate_articlezs rN) rr r!r"rrIrrhrrrrrrtsA     ,  &      rrc.eZdZdZedededefdZy)ArticlePolishingModulerr draft_articlernc y)z Polish article. Required arguments include: topic: the topic of interest draft_article: draft article from ArticleGenerationModule. Nr)rrrrs rpolish_articlez%ArticlePolishingModule.polish_articles rN)rr r!r"rrIrhrrrrrrs0  C  g  rrcBtjfd}|S)z2Decorator to log the execution time of a function.ctj}|g|i|}tj}||z }tjjd|dd||jj<|S)N executed in .4f seconds)timeloggerrPr)rargsr start_timerqend_timeexecution_timefuncs rwrapperz#log_execution_time..wrappersnYY[ d,T,V,99;!J. t}}o]>#2FhOP#1 $--  r functoolswraps)rrs` rlog_execution_timers'__T Nrc.eZdZdZdZdZdZdZdZy) LMConfigszAbstract base class for language model configurations of the knowledge curation engine. The language model used for each part should be declared with a suffix '_lm' in the attribute name. cyrrrs rrzLMConfigs.__init__rrc|jD]0}d|vst||tjd|d|d2y)N_lmzLanguage model for z% is not initialized. Please call set_z())__dict__rloggingwarningr attr_names r init_checkzLMConfigs.init_checksK I !gdI&>&F))4YZcYddfg rcg}|jD]T}d|vstt||ds|jt||jgt||_V|S)Nrhistory)rrrrr)rrrs rcollect_and_reset_lm_historyz&LMConfigs.collect_and_reset_lm_historysa 6I !ggdI.F &RwtY7??@35i(0 6 rcXg}|jD]G}d|vstt||ds|jt||j Ii}|D]J}|j D]5\}}||vr|||<||dxx|dz cc<||dxx|dz cc<7L|S)Nrr prompt_tokenscompletion_tokens)rrrr_rr)rrrmodel_name_to_usagerrtokenss rcollect_and_reset_lm_usagez$LMConfigs.collect_and_reset_lm_usages VI !gi(*?'%%gdI&>&R&R&TU  V !# E&+kkm " F%886<' 3' 3OD'ID( 34GHF+MH  #"rc t|jDcic]3}d|vr-tt||dr|t||j5c}Scc}w)Nrr)rrrrrrs rlogz LMConfigs.logsZ"& I%''$ 2JH*U743:::    s8AN) rr r!r"rrrrrrrrrrs   #. rrceZdZdefdZdZdZedee fdZ ede fdZ ede fdZ ede fd Zed Zd Zd Zy )Engine lm_configsc<||_i|_i|_i|_yr)rrlm_costrm_cost)rrs rrzEngine.__init__s$   rcFtjfd}|S)zcDecorator to log the execution time, language model usage, and retrieval model usage of a function.ctj}|i|}tj}||z }|jj<tjjd|ddjj j j<tdr1jjjj<|S)Nrrrr) rrrrPrrrrrrr)rrrrqrrrrs rrz:Engine.log_execution_time_and_lm_rm_usage..wrappersJ4*6*Fyy{H% 2N'5DIIdmm $ KK4==/~c6J(S T*.//*T*T*VDLL 't[)NN==? T]]+Mrr)rrrs`` r"log_execution_time_and_lm_rm_usagez)Engine.log_execution_time_and_lm_rm_usages'     rc t|Dcgc]*}tt||r|jdr|,}}|D],}t||}|j |}t |||.ycc}w)z+Apply decorators to methods that need them.run_N)dircallabler startswithrsetattr)r method_namemethods_to_decorateoriginal_methoddecorated_methods rapply_decoratorszEngine.apply_decoratorss #4y k23 8N8Nv8V   / 9K%dK8O#FFW  D+'7 8 9  s/A1rnc yrrrrs rrun_knowledge_curation_modulez$Engine.run_knowledge_curation_module rrc yrrrkwargs rrun_outline_generation_modulez$Engine.run_outline_generation_modulerrc yrrrs rrun_article_generation_modulez$Engine.run_article_generation_modulerrc yrrrs rrun_article_polishing_modulez#Engine.run_article_polishing_modulerrc yrrrs rrunz Engine.runrrctd|jjD]\}}t|d|ddtd|jjD]:\}}t||jD]\}}td|d|<td|jjD]\}}t|d|y)Nz***** Execution time *****: rrz+***** Token usage of language models: *****z z2***** Number of queries of retrieval models: *****)printrrrr)rkvrrs rsummaryzEngine.summary s *+IIOO% +DAq QCr!C) * + ;<LL&&( 5DAq QCM&'ggi 5" FZL6(34 5 5 BCLL&&( DAq QCr!+  rc.i|_i|_i|_yr)rrrrs rresetz Engine.reset/s   rN)rr r!rrrrrrrrrhrrrr rrrrrrrs9 ( 9 BR9S         w     rrc^eZdZdZddlmZmZdededefdZdZ e d ed e ed d fd Z y)Agenta Interface for STORM and Co-STORM LLM agent This class must be implemented by any subclass of `Agent` to define how the agent generates an utterance. The generated utterance can be influenced by the conversation history, knowledge base, and any additional parameters passed via `kwargs`. The implementation should align with the specific role and perspective of the agent, as defined by the agent's topic, role name, and role description. Args: knowledge_base (KnowledgeBase): The current knowledge base (e.g., mind map in Co-STORM) that contains the accumulated information relevant to the conversation. conversation_history (List[ConversationTurn]): A list of past conversation turns, providing context for generating the next utterance. The agent can refer to this history to maintain continuity and relevance in the conversation. logging_wrapper (LoggingWrapper): A wrapper used for logging important events during the utterance generation process. **kwargs: Additional arguments that can be passed to the method for more specialized utterance generation behavior depending on the agent's specific implementation. Returns: ConversationTurn: A new conversation turn generated by the agent, containing the agent's response, including the role, utterance type, and relevant information from the knowledge base. Notes: - Subclasses of `Agent` should define the exact strategy for generating the utterance, which could involve interacting with a language model, retrieving relevant knowledge, or following specific conversational policies. - The agent's role, perspective, and the knowledge base content will influence how the utterance is formulated. r ) KnowledgeBaseConversationTurnr role_namerole_descriptionc.||_||_||_yr)rrr)rrrrs rrzAgent.__init__Ns " 0rch|jr|jd|jS|jS)Nr )rrrs rget_role_descriptionzAgent.get_role_descriptionSs2  nn%R(=(='>? ?~~rknowledge_baseconversation_historylogging_wrapperrc yrr)rrrrrs rgenerate_utterancezAgent.generate_utteranceXs rN) rr r!r" dataclassrrrIrrrrrrrrrr5sa,;1c1c1S1   % ##34 *   rr))concurrent.futuresrrrrGrErrabcrr collectionsrtypingrrrr r utilsr basicConfigINFO getLoggerrrrrrr$rWrhrrrrrrrrrrrrr)s   ##==( ,,I   8 $/ s $\ \ ~$$4_c_D;;| c 2 c 2 c .  S   7 7 tMSM`+ C+ r