Ig(<ddlmZddlZddlmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZGddeZGd d ZGd d eZGd deZGddZy)) annotationsN)AnyDictListTuple TypedDictUnion)Document)Language)RecursiveCharacterTextSplitterc$eZdZdZdfd ZxZS)MarkdownTextSplitterz=Attempts to split the text along Markdown-formatted headings.c f|jtj}t|dd|i|y)z"Initialize a MarkdownTextSplitter. separatorsN)get_separators_for_languager MARKDOWNsuper__init__)selfkwargsr __class__s ^/var/www/html/answerous/venv/lib/python3.12/site-packages/langchain_text_splitters/markdown.pyrzMarkdownTextSplitter.__init__s.55h6G6GH  9J9&9)rrreturnNone)__name__ __module__ __qualname____doc__r __classcell__)rs@rrr sG::rrc8eZdZdZ d ddZddZd dZy) MarkdownHeaderTextSplitterz4Splitting markdown files based on specified headers.cH||_t|dd|_||_y)a Create a new MarkdownHeaderTextSplitter. Args: headers_to_split_on: Headers we want to track return_each_line: Return each line w/ associated headers strip_headers: Strip split headers from the content of the chunk ct|dS)Nr)len)splits rz5MarkdownHeaderTextSplitter.__init__..*s3uQx=rT)keyreverseN)return_each_linesortedheaders_to_split_on strip_headersrr-r+r.s rrz#MarkdownHeaderTextSplitter.__init__s-!1$* %@$$  +rcg}|D]}|r%|dd|dk(r|ddxxd|dzz cc<*|r||dd|dk7rnt|ddt|dkrN|ddjddddk(r.|js"|ddxxd|dzz cc<|d|dd<|j||Dcgc]}t |d|dc}Scc}w) zCombine lines with common metadata into chunks Args: lines: Line of text / associated header metadata metadatacontentz  r# page_contentr2)r&r'r.appendr )rlinesaggregated_chunkslinechunks raggregate_lines_to_chunksz4MarkdownHeaderTextSplitter.aggregate_lines_to_chunks/s8 -/ /D!%b)*5j9II ""%i0FT)_4LL0!%b)*5j9II)"-j9:SjAQ=RR%b))4::4@DQG3N**""%i0FT)_4LL0484D!"%j1"((.9 /@+  % "2U:=N O   sC!c|jd}g}g}i}g}i}d}d} |D]Q} | j} djttj | } |s@| j dr| jddk(rd}d} n+| j drd}d} n| j | rd}d} |r|j| |jD]3\} } | j | st| t| k(s| t| dk(sB| | jd }|rD|d d |k\r9|j}|d |vr|j|d |r |d d |k\r9|| | t| d jd}|j||d|| <|rA|jdj||jd|j|js|j| nX| r|j| nC|rA|jdj||jd|j|j}T|r#|jdj||d|js|j!|S|Dcgc]}t#|d|dc}Scc}w)zASplit markdown file Args: text: Markdown filer4Fz```Tz~~~ Nr5r1levelname)rBrCdatarD)r3r2r3r2r6)r'stripjoinfilterstr isprintable startswithcountr8r-r&popcopyclearr.r+r=r )rtextr9lines_with_metadatacurrent_contentcurrent_metadata header_stackinitial_metadata in_code_block opening_fencer; stripped_lineseprCcurrent_header_level popped_headerheaderr<s r split_textz%MarkdownHeaderTextSplitter.split_textYs  4 .0%'+-*, +-  U 7D JJLMGGF3??M$JKM  ++E2}7J7J57QUV7V$(M$)M"--e4$(M$)M ++M:$)M$&M&&}5"55< , T ++C0 &#c(2mCH6MQT6T'/2yy~,) ,R 0 9=Q Q-9,<,<,>M -V48HH 0 4 4]65J K) ,R 0 9=Q Q&:$($1#c(*$=$C$C$E. %++F317(.'+22+/99_+E,<,A,A,C (--/--'..}=c< ,f!#**=9$'..'+yy'A(8(=(=(? $))+/446 kU 7n   & & IIo6DTU  $$112EF F1eI&6zARS s=KN)FT)r-zList[Tuple[str, str]]r+boolr.r])r9zList[LineType]rList[Document]rOrHrr^)rrrr rr=r\rrrr#r#s:> "'" +2++ +.( Txrr#c&eZdZUdZded<ded<y)LineTypezLine type as typed dict.zDict[str, str]r2rHr3Nrrrr __annotations__rrrraras" Lrrac0eZdZUdZded<ded<ded<y) HeaderTypezHeader type as typed dict.intrBrHrCrDNrbrrrreres$ J I IrrecteZdZdZdddddddZ d dd Zdd Zdd Zdd ZddZ ddZ ddZ ddZ y )&ExperimentalMarkdownSyntaxTextSplittera An experimental text splitter for handling Markdown syntax. This splitter aims to retain the exact whitespace of the original text while extracting structured metadata, such as headers. It is a re-implementation of the MarkdownHeaderTextSplitter with notable changes to the approach and additional features. Key Features: - Retains the original whitespace and formatting of the Markdown text. - Extracts headers, code blocks, and horizontal rules as metadata. - Splits out code blocks and includes the language in the "Code" metadata key. - Splits text on horizontal rules (`---`) as well. - Defaults to sensible splitting behavior, which can be overridden using the `headers_to_split_on` parameter. Parameters: ---------- headers_to_split_on : List[Tuple[str, str]], optional Headers to split on, defaulting to common Markdown headers if not specified. return_each_line : bool, optional When set to True, returns each line as a separate chunk. Default is False. Usage example: -------------- >>> headers_to_split_on = [ >>> ("#", "Header 1"), >>> ("##", "Header 2"), >>> ] >>> splitter = ExperimentalMarkdownSyntaxTextSplitter( >>> headers_to_split_on=headers_to_split_on >>> ) >>> chunks = splitter.split(text) >>> for chunk in chunks: >>> print(chunk) This class is currently experimental and subject to change based on feedback and further development. zHeader 1zHeader 2zHeader 3zHeader 4zHeader 5zHeader 6)r5z##z###z####z#####z######Ncg|_td|_g|_||_|rt ||_||_y|j|_||_y)Nr?r7) chunksr current_chunkcurrent_header_stackr.dictsplittable_headersDEFAULT_HEADER_KEYSr+r/s rrz/ExperimentalMarkdownSyntaxTextSplitter.__init__s[ ') %26;=!* &*+>&?D #!1'+&>&>D # 0rc |jd}|ra|jd}|j|}|j|}|j |}|ry|j |j s|jxj|z c_t|jd}|jd}|j||n|rj|j |j|||j_|jd|jjd<|j n2|r|j n|jxj|z c_|ra|j |jrb|jD cgc]J} | jjD]+} | r'| j!st#| | j-Lc} } S|jScc} } w)NT)keependsrr@Coder6) splitlinesrL _match_header _match_code _match_horz_complete_chunk_docr.rlr7r&group_resolve_header_stack_resolve_code_chunkr2r+rkisspacer ) rrO raw_linesraw_line header_match code_match horz_match header_depth header_textr<r;s rr\z1ExperimentalMarkdownSyntaxTextSplitter.split_text&sOOTO2  }}Q'H--h7L))(3J))(3J((*))&&33x?3 #<#5#5a#89 *003 **<E((*262J2Ji3""/7A6F6Fq6I""++F3((*((*""//8;/36   "  "[[!..99; dU^^DD  {{ s"AHct|jD]7\}\}}||k(s||f|j|<|jd|dz|_y|jj||fy)Nr@) enumeratermr8)rrridepth_s rr{z|jj|jtd|_y)Nr5r?rj) rlr7r}rmrogetr2rkr8r )r chunk_contentrvalue header_keys rryz:ExperimentalMarkdownSyntaxTextSplitter._complete_chunk_docbs**77 !6!6!8 $ 9 9 @ u!4488uE :?""++J7 @ KK  t11 2%26rcrtjd|}|r|jd|jvr|Sy)Nz^(#{1,6}) (.*)r@)rematchrzro)rr;rs rrvz4ExperimentalMarkdownSyntaxTextSplitter._match_headeros3*D1 U[[^t'>'>>LrcxdDcgc]}tj||}}td|DdScc}w)N)z^```(.*)z^~~~(.*)c3&K|] }|s| ywNr.0rs r zEExperimentalMarkdownSyntaxTextSplitter._match_code..x9u5U9rrnextrr;rulematchess rrwz2ExperimentalMarkdownSyntaxTextSplitter._match_codevs94NOD288D$'OO994@@P7cxdDcgc]}tj||}}td|DdScc}w)N)z ^\*\*\*+\nz^---+\nz^___+\nc3&K|] }|s| ywrrrs rrzEExperimentalMarkdownSyntaxTextSplitter._match_horz..~rrrrs rrxz2ExperimentalMarkdownSyntaxTextSplitter._match_horzzsA-T %)BHHT4   994@@ r)NFT)r-z"Union[List[Tuple[str, str]], None]r+r]r.r]r_)rrfrrHrr)rrHr~z List[str]rrH)rr)r;rHrzUnion[re.Match, None]) rrrr rprr\r{r|ryrvrwrxrrrrhrhsy&R CG!&" 1?11 1")VF 7AArrh) __future__rrtypingrrrrrr langchain_core.documentsr langchain_text_splitters.baser "langchain_text_splitters.characterr rr#rarerhrrrrsZ" ;;-2M:9:||~y[A[Ar