:Qg,dZddlmZddlZddlZddlZddlZddlZddl m Z m Z m Z m Z ddlmZddlmZmZddlmZddlmZdd lmZed Zdd Zgd gd dZef ddZ d ddZddZddZy)z>Helpers used across multiple partitioners to compute metadata.) annotationsN)AnyCallableIteratorSequence) ParamSpec)ElementElementMetadata)FileType)apply_lang_metadata)get_call_args_applying_defaults_Pctjj|sytjj tjj |}|jdS)a Modification time of file at path `filename`, if it exists. Returns `None` when `filename` is not a path to a file on the local filesystem. Otherwise returns date and time in ISO 8601 string format (YYYY-MM-DDTHH:MM:SS) like "2024-03-05T17:02:53". Nz%Y-%m-%dT%H:%M:%S%z)ospathisfiledtdatetime fromtimestampgetmtimestrftime)filename modify_dates c/var/www/html/answerous/venv/lib/python3.12/site-packages/unstructured/partition/common/metadata.pyget_last_modified_datersL 77>>( #++++BGG,<,rcdfd }|S)aPost-process element-metadata for this document. This decorator adds a post-processing step to a partitioner, primarily to apply metadata that is common to all partitioners. It assumes the following responsibilities: - Hash element-ids. Computes and applies SHA1 hash element.id when `unique_element_ids` argument is False. - Element Hierarchy. Computes and applies `parent_id` metadata based on `category_depth` etc. added by partitioner. - Language metadata. Computes and applies `language` metadata based on a language detection model. - Apply `filetype` (MIME-type) metadata. There are three cases; first one in this order that applies is used: - `metadata_file_type` argument is present in call, use that. - `file_type` decorator argument is populated, use that. - `file_type` decorator argument is omitted or None, don't apply `.metadata.filetype` (assume the partitioner will do that for itself, like `partition_image()`. - Replace `filename` with `metadata_filename` when present. - Replace `last_modified` with `metadata_last_modified` when present. - Apply `url` metadata when present. cHtjdfd }|S)aThe decorator function itself. This function is returned by the `apply_metadata()` function and is the actual decorator. Think of `apply_metadata()` as a factory function that configures this decorator, in particular by setting its `file_type` value. c|i|}tg|i|}t|}|jd}|jdd}tt |||}i}|jdxs }||j |d<|jdxs|jd}|r||d<|jd } | r| |d <|jd } | r| |d <|D]=} | j jr| j jtd i|?|jd d} | dur t|}t|}|S)N languagesdetect_language_per_elementF)r3r@rAmetadata_file_typefiletypemetadata_filenamermetadata_last_modified last_modifiedurlunique_element_ids) r _uniqueify_elements_and_metadatar.r2r mime_typer+attached_to_filenameupdater _assign_hash_idsr<)argskwargsr3 call_argsr@rAmetadata_kwargsrBrrErGr6rH file_typefuncs rwrapperz2apply_metadata..decorator..wrappersT,V,H7NtNvNI8AH" k2I*3--8UW\*] '#%'0KH/1O"+/C!D!Q !-.@.J.J +!}}%89VY]]:=VH.6 +&/]]3K%L "%3I0--&C),&$ L##88  ''(J/(JK  L(1}}5I5'Q !U*+H5-X6HOr)rOz_P.argsrPz _P.kwargsreturn list[Element]) functoolswraps)rTrUrSs` r decoratorz!apply_metadata..decorators*  O  O br)rTCallable[_P, list[Element]]rVr[rI)rSrZs` rapply_metadatar\xs@Zx rc&|Dcgc]}|jj}}tj|Dcgc]\}}t |D]\}}| }}}}t ||D]\}}|j ||Scc}wcc}}}w)a.Converts `.id` of each element from UUID to hash. The hash is based on the `.text` of the element, but also on its page-number and sequence number on that page. This provides for deterministic results even when the document is split into one or more fragments for parallel processing. )r+ page_number itertoolsgroupby enumeratezip id_to_hash) r3e page_numbers_group seq_on_pagepage_seq_numbersr6seq_on_page_counters rrNrNs5==qAJJ**=L=")),7 Au'. K  ),H6F(G0$$./0 O>s BB c,dd}t||S)zEnsure each of `elements` and their metadata are unique instances. This prevents hard-to-diagnose bugs downstream when mutating one element unexpectedly also mutates others because they are the same instance. c3Kt}t}|D]}t||vrtj|}t|j|vr$tj|j|_|j t||j t|j|yw)zLSubstitute deep-copies of any non-unique elements or metadata in `elements`.N)setr/copydeepcopyr+add)r3 seen_elements seen_metadatar6s riter_unique_elementsz>_uniqueify_elements_and_metadata..iter_unique_elementss"%% "%%  G'{m+--0'""#}4#'==1A1A#B   bk *   b!1!12 3M sB)r3rWrVzIterator[Element])r2)r3rss rrJrJ s  $X. //r)rstrrVz str | None)r3zSequence[Element]r4zdict[str, list[str]]rVrW)N)rSzFileType | NonerVzDCallable[[Callable[_P, list[Element]]], Callable[_P, list[Element]]])r3rWrVrW) __doc__ __future__rrnrrrXr_rtypingrrrrtyping_extensionsrunstructured.documents.elementsr r unstructured.file_utils.modelr "unstructured.partition.common.langr unstructured.utilsr rrHIERARCHY_RULE_SETr<r\rNrJrIrrr~sD"  44'D2B>t_ 7  8BT00*>00r"&||I|~*0r