Ig^-ddlmZddlZddlZddlmZmZddlmZm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZGddeZGd d ZGd d Zy) ) annotationsN)BytesIOStringIO)AnyDictIterableListOptionalTuple TypedDictcast)Document)RecursiveCharacterTextSplitterc:eZdZUdZded<ded<ded<ded<y) ElementTypezElement type as typed dict.strurlxpathcontentzDict[str, str]metadataN)__name__ __module__ __qualname____doc____annotations__Z/var/www/html/answerous/venv/lib/python3.12/site-packages/langchain_text_splitters/html.pyrrs% H J LrrcJeZdZdZ d d dZ d dZd dZd dZd dZy)HTMLHeaderTextSplitterzU Splitting HTML files based on specified headers. Requires lxml package. c2||_t||_y)ayCreate a new HTMLHeaderTextSplitter. Args: headers_to_split_on: list of tuples of headers we want to track mapped to (arbitrary) keys for metadata. Allowed header values: h1, h2, h3, h4, h5, h6 e.g. [("h1", "Header 1"), ("h2", "Header 2)]. return_each_element: Return each element w/ associated headers. N)return_each_elementsortedheaders_to_split_on)selfr$r"s r__init__zHTMLHeaderTextSplitter.__init__s$7 #)*=#> rcg}|D]:}|r%|dd|dk(r|ddxxd|dzz cc<*|j|<|Dcgc]}t|d|dc}Scc}w)zCombine elements with common metadata into chunks Args: elements: HTML element content with associated identifying info and metadata rrz  page_contentr)appendr)r%elementsaggregated_chunkselementchunks raggregate_elements_to_chunksz3HTMLHeaderTextSplitter.aggregate_elements_to_chunks.s02 2G!%b)*59LL ""%i0FWY=O4OO0"((1 2+  % "2U:=N O   sA#c vtj|fi|}|jt|jS)zSplit HTML from web URL Args: url: web URL **kwargs: Arbitrary additional keyword arguments. These are usually passed to the fetch url content request. )requestsgetsplit_text_from_filerr)r%rkwargsrs rsplit_text_from_urlz*HTMLHeaderTextSplitter.split_text_from_urlJs1 LL ' '(();<Unable to import lxml, please install with `pip install lxml`.Nzutf-8)encodingz!xsl/html_chunks_with_headers.xslthzhttp://www.w3.org/1999/xhtmlz*//*z*[@class='headers']z*[@class='chunk']z*[@class='xpath']c |jvSN)tag)x header_filters rz=HTMLHeaderTextSplitter.split_text_from_file..s!%%=*@rz*[@class='headers']/*)rrrrrrr))lxmlrB ImportError HTMLParserparsepathlibPath__file__parentXSLT fromstringrr$dictfindallr+rjoinr<filterrIr"r0r)r%filerBeparsertree xslt_path xslt_tree transformresult result_domheaderheader_mappingns_mapr,r.noder/rKs @rr4z+HTMLHeaderTextSplitter.split_text_from_file]s]   "!!7!3{{4(LL*114WW KK * JJy) 4%%c&k2 261I1IJvJ d66756!))&&9 G45#:  gg-4OO''44X> >&eI&6zARS { P  &K   "s5H$% I8I4I 2%II$ H>- H99H>N)F)r$List[Tuple[str, str]]r"bool)r,zList[ElementType]returnList[Document])rrr5rrjrkr<rrjrkr[rrjrk) rrrrr&r0r7r=r4rrrr r sH%*?2?"?" )  8 =9Hrr cheZdZdZ d d dZd dZd dZ d ddZddZddZ dd Z y)HTMLSectionSplitterz` Splitting HTML files based on specified tag and font sizes. Requires lxml package. Nc t||_|Atjtj dz j |_||_ytj|j |_||_y)aCreate a new HTMLSectionSplitter. Args: headers_to_split_on: list of tuples of headers we want to track mapped to (arbitrary) keys for metadata. Allowed header values: h1, h2, h3, h4, h5, h6 e.g. [("h1", "Header 1"), ("h2", "Header 2"]. xslt_path: path to xslt file for document transformation. Uses a default if not passed. Needed for html contents that using different format and layouts. Nzxsl/converting_to_header.xslt) rWr$rQrRrSrTabsoluter_r5)r%r$r_r5s rr&zHTMLSectionSplitter.__init__si $((;#<   X&--0OOhj N  %\\)4==?DN rcgg}}|D]8}|j|j|j|j:|j||}t di|j }|j |S)zSplit documents.) metadatasr)r+r*rcreate_documentsrr5split_documents)r% documentstextsrsdocresults text_splitters rruz#HTMLSectionSplitter.split_documentsswry +C LL)) *   S\\ * +'''C6EE ,,W55rc6|jt|Sr9r:r;s rr=zHTMLSectionSplitter.split_textr>rc|xsigt|z}g}t|D]\}}|j|D]}tj||}|j j D]'} |j | dk(s|d|j | <)i||j }t|j|} |j| |S)z&Create documents from a list of texts.#TITLE#Titler)) len enumerater=copydeepcopyrkeysrr*r+) r%rwrs _metadatasrvir<r/rkeynew_docs rrtz$HTMLSectionSplitter.create_documentss32$U"3   ' *GAt. *==A7 >>..0@C~~c*i7.6w.?s+@:h9%..9"0B0BXV  ) * *rc ddlm}m}||d}t |j j }g}|jdg|z}t|D]\}} | } |dk(rd} d} g} n(| jj} | j} g} | jD]B}|dzt|kr |||dzk(rn$t|ts2| j!|Dd j#| j}|d k7s|j!| || d |S#t$r}td|d}~wwxYw) Nr) BeautifulSoup PageElementzzUnable to import BeautifulSoup/PageElement, please install with `pip install bs4`.z html.parserbodyr}h1 rF)rdrtag_name)bs4rrrNlistr$rfind_allrr<stripname next_elementsr isinstancerr+rY)r%html_docrrr\soupheaderssectionsrrdheader_elementcurrent_headercurrent_header_tagsection_contentr.rs rsplit_html_by_headersz)HTMLSectionSplitter.split_html_by_headerss^  6X}5t//446702--7 23"7+ IAv*0NAv!*%)"(*!/!4!4!:!:!<%3%8%8""$)77 4q53w<'Gwq1u~,Egs+#**73  4 hh/557G"}"0#*$6% 4O    sD## D=, D88D=c:|j|S ddlm}|j }|j t ||}|j |j}|j|}||}t|S#t$r}td|d}~wwxYw)NrrArC) r_rMrBrNrOrPrrUr) r% html_contentrBr\r]r^r`rarbs rconvert_possible_tags_to_headerz3HTMLSectionSplitter.convert_possible_tags_to_headers >> !   " !!#{{8L16:KK/ JJy) 46{ P  sB B BBc |j}|j|}|j|}|Dcgc]>}tt t |d|j t |d|di@c}Scc}w)r@rrrd)r)getvaluerrrr rr$)r%r[ file_contentrsections rr4z(HTMLSectionSplitter.split_text_from_file-s }} ;;LI --l;$   S'),-,,S1D-EF I    sAA=rH)r$rhr_z Optional[str]r5rrjNone)rvzIterable[Document]rjrkrl)rwz List[str]rszOptional[List[dict]]rjrk)rrrjzList[Dict[str, Optional[str]]])rrrjrrm) rrrrr&rur=rtrrr4rrrrorosu$(2!   4 69CG+? $*X& rro) __future__rrrQiorrtypingrrrr r r r r r2langchain_core.documentsr"langchain_text_splitters.characterrrr rorrrrsH"  NNN-M)NNbY Y r