:Qg!dZddlmZddlmZmZmZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZdddddddd d d Zdd ZGd d e Zy)zdImplementation of chunking by title. Main entry point is the `@add_chunking_strategy()` decorator. ) annotations)IterableIteratorOptional)CHUNK_MULTI_PAGE_DEFAULTBoundaryPredicateChunkingOptionsPreChunkCombiner PreChunkeris_on_next_pageis_title)Element) lazypropertyNcombine_text_under_n_charsinclude_orig_elementsmax_charactersmultipage_sectionsnew_after_n_charsoverlap overlap_allc Rtj|||||||}t||S)a Uses title elements to identify sections within the document for chunking. Splits off into a new CompositeElement when a title is detected or if metadata changes, which happens when page numbers or sections change. Cuts off sections once they have exceeded a character length of max_characters. Parameters ---------- elements A list of unstructured elements. Usually the output of a partition function. combine_text_under_n_chars Combines elements (for example a series of titles) until a section reaches a length of n characters. Defaults to `max_characters` which combines chunks whenever space allows. Specifying 0 for this argument suppresses combining of small chunks. Note this value is "capped" at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect. include_orig_elements When `True` (default), add elements from pre-chunk to the `.metadata.orig_elements` field of the chunk(s) formed from that pre-chunk. Among other things, this allows access to original-element metadata that cannot be consolidated and is dropped in the course of chunking. max_characters Chunks elements text and text_as_html (if present) into chunks of length n characters (hard max) multipage_sections If True, sections can span multiple pages. Defaults to True. new_after_n_chars Cuts off new sections once they reach a length of n characters (soft max). Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. overlap_all Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole elements and not subject to text-splitting. Use this with caution as it entails a certain level of "pollution" of otherwise clean semantic chunk boundaries. r)_ByTitleChunkingOptionsnew_chunk_by_title) elementsrrrrrrroptss X/var/www/html/answerous/venv/lib/python3.12/site-packages/unstructured/chunking/title.pychunk_by_titlers>h # & &#=3%-+ ' D 8T **rcttj|||j}|Dcgc]}|j D]}|c}}Scc}}w)z-Implementation of actual "by-title" chunking.)r)r r iter_pre_chunksiter_combined_pre_chunks iter_chunks)rr pre_chunks pre_chunkchunks rrrWsW"""8T2 $. Si9;P;P;R S%E SE SS SsAcZeZdZdZeddZeddZeddZd fd ZxZ S) ra9Adds the by-title-specific chunking options to the base case. `by_title`-specific options: combine_text_under_n_chars A remedy to over-chunking caused by elements mis-identified as Title elements. Every Title element would start a new chunk and this setting mitigates that, at the expense of sometimes violating legitimate semantic boundaries. multipage_sections Indicates that page-boundaries should not be respected while chunking, i.e. elements appearing on two different pages can appear in the same chunk. c0dfd }t|S)a(The semantic-boundary detectors to be applied to break pre-chunks. For the `by_title` strategy these are sections indicated by a title (section-heading), an explicit section metadata item (only present for certain document types), and optionally page boundaries. c3PKtjs tyyw)N)r rr )selfsriter_boundary_predicateszM_ByTitleChunkingOptions.boundary_predicates..iter_boundary_predicatesys#N**%''+s#&)returnzIterator[BoundaryPredicate])tuple)r+r,s` rboundary_predicatesz+_ByTitleChunkingOptions.boundary_predicatesps ( -/00r cX|jjd}| |jS|S)aCombine consecutive text pre-chunks if former is smaller than this and both will fit. - Does not combine table chunks with text chunks even if they would both fit in the chunking window. - Does not combine text chunks if together they would exceed the chunking window. - Defaults to `max_characters` when not specified. - Is reduced to `new_after_n_chars` when it exceeds that value. r)_kwargsgethard_maxr+ arg_values rrz2_ByTitleChunkingOptions.combine_text_under_n_charss-LL$$%AB ) 1t}}@y@r c^|jjd}|tSt|S)z0When False, break pre-chunks on page-boundaries.r)r1r2rboolr4s rrz*_ByTitleChunkingOptions.multipage_sectionss.LL$$%9: +4+<'Q$y/Qr ct||jdkrtd|j|j|jkDr%td|jd|jy)z2Raise ValueError if request option-set is invalid.rz8'combine_text_under_n_chars' argument must be >= 0, got zR'combine_text_under_n_chars' argument must not exceed `max_characters` value, got z > N)super _validater ValueErrorr3)r+ __class__s rr:z!_ByTitleChunkingOptions._validates   * *Q .778:   * *T]] :#>>?s4==/S  ;r )r-ztuple[BoundaryPredicate, ...])r-int)r-r7)r-None) __name__ __module__ __qualname____doc__rr/rrr: __classcell__)r<s@rrrbsR  1 1 A ARR r )rIterable[Element]r Optional[int]rOptional[bool]rrErrFrrErrErrFr- list[Element])rrDrrr-rG)rB __future__rtypingrrrunstructured.chunking.baserrr r r r r unstructured.documents.elementsrunstructured.utilsrrrrr rrNs #//4+ 15,0$()-'+!"&=+=+!.=+* =+ " =+ ' =+%=+=+ =+=+@TGoGr