:Qg ddlZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl Z ddl mZmZej dZdededefd Zd edeefd Zd Zd ZdZ ddedede j0dedef dZdZdefdZdefdZd de e j<eefdee de edffdZ!d deeedee de edffdZ"d deeedee de edffdZ#deeedefdZ$dZ%y)!N)Path)ListOptionalUnion)elements_from_jsonelements_to_textzunstructured.evaldocpath output_typereturnc |dk(rtt|}|S|dk(r t|}|Std|d#t$r}tj d||d}~wwxYw)z Convert given input document (path) into cct-ready. The function only support conversion from `json` or `txt` file. jsontxtzZFile type not supported. Expects one of `json` or `txt`, but received z instead.zCould not read the file N)rr_read_text_file ValueErrorloggererror)r r output_cctes W/var/www/html/answerous/venv/lib/python3.12/site-packages/unstructured/metrics/utils.py_prepare_output_cctrs  & )*>> A&A!!A&dirc g}tj|D]s\}}}|D]h}tjj||}|dk(r|j |:|j tjj ||ju|S)z Recursively lists all files in the given directory and its subdirectories. Returns a list of all files found, with each file's path relative to the initial directory. .)oswalkpathrelpathappendjoin)rlistdirdirpath_ filenamesfilename relative_paths r_listdir_recursiver&%s G!#FI! FHGGOOGS9M#x(rww||M8DE  FF Nc4ddddd}|j|S)a Renames aggregated columns in a DataFrame based on a predefined mapping. Parameters: df (pandas.DataFrame): The DataFrame with aggregated columns to rename. Returns: pandas.DataFrame: A new DataFrame with renamed aggregated columns. meanstdevpstdevcount)_mean_stdev_pstdev_count)columns)rename)df rename_maps r_rename_aggregated_columnsr57s$"WU\]J 99Z9 ((r'cLtj|djS)zs Concatenates multiple pandas DataFrame objects along the columns (side-by-side) and resets the index. )axis)pdconcat reset_index)r3s r_format_grouping_outputr<Es 99Ra , , ..r'c t|dk(ry|jj}|Dcgc]*}tt|td||D,c}t j dj fdt|Dt j dtzdt|dz zz|jD]\}}g|D]@}t|trj|d'jt|Bt j dj fd ttDycc}w) zD Displays the evaluation metrics in a formatted text table. rNc3DK|]}tt|ywN)lenstr).0items r z_display..UsCST^Cs  c3LK|]\}}|j|ywr?ljust)rBiheader col_widthss rrDz_display..Ws"X 6 Z]3X!$-r7z.3fc3LK|]}|j|ywr?rG)rBrIrK formatted_rows rrDz_display..as%^q]1%++JqM:^rL)r@r1tolistmaxclickechor enumeratesumiterrows isinstancefloatrrArange)r3headersrJr"rowrCrKrOs @@r_displayr\Ms( 2w!|jj!GSZIOCKC6 CCDJ JJsxxXYwEWXXY JJsS_$sc'lQ.>'??@++-  3  0D$&$$Sz3$$SY/  0  HH^E#mJ\D]^ ^   s/E+ directoryr$r3mode overwritec|dvr td|rt|jdd|jvr|dj t |d<d|jvr#d|jvr|j ddgd|s t||}|jtjj||d |d |d k( y )z Save the metrics report to tsv file. The function allows an option 1) to choose `mode` as `w` (write) or `a` (append) and 2) to `overwrite` the file if filename existed or not. )waz/Mode not supported. Mode must be one of [w, a].T)exist_okr,r$ connector)byinplace Fra)sepr^indexrJN) rrmkdirr1astypeint sort_values_get_non_duplicated_filenameto_csvrrr)r]r$r3r^r_s r_write_to_filerpes :JKK Yt,"**k((-7 RZZK2::$= ; 3TB / 8DII  Y)t$eUY]`U`r'cPtjd|}|rt|dSy)a A function that defines the sorting method for duplicated file names. For example, with filename.ext filename (1).ext filename (2).ext filename (10).ext - this function extracts the integer in the bracket and sort those numbers ascendingly. z(\d+)r)refindallrl)r$numberss r _sorting_keyrv{s+jj8,G72;r'c |jdd\}}dtj|dtj|d}t|Dcgc]}tj||s|c}t }g}|D]D}tj d|} | s|jt| jdF|jd} |D]} | | k(r| dz } n|dzt| zd z|zScc}w) z Checks the duplicity of the file name from the list and run the numerical check of the minimum number needed as extension to not overwrite the exising file. Returns a string of file name in the format of `filename ().ext`. rr7^z(?: \((\d+)\))?\.$)keyz \((\d+)\)z (z).) rsplitrsescapesortedmatchrvsearchrrlgroupsortrA) file_listtarget_filenameoriginal_filename extensionpatternfduplicated_filesrufiler~counternumbers r_uniquity_filers $3#9#9#q#A y299./00A"))IBVAWWXYG)LQrxx7KqLR^_G 0 ,-  NN3u{{1~. /0  LLNG W  qLG   t #c'l 2T 9I EE#Ms D*DcDttj||}|S)zs Helper function to calls the `_uniquity_file` function. Takes in directory and file name to check on. )rrr )rr$s rrnrns bjjox8H Or'scoresroundingcjt|dk(rytj|}|s|St||S)z Find mean from the list. Returns None if no element in the list. Args: rounding (int): optional argument that allows user to define decimal points. Default at 3. rN)r@ statisticsr)round)rrr)s rr-r-s6 6{a ??6 "D  x  r'c|Dcgc]}|| }}t|dkry|stj|Sttj||Scc}w)z Find standard deviation from the list. Returns None if only 0 or 1 element in the list. Args: rounding (int): optional argument that allows user to define decimal points. Default at 3. Nr7)r@rr*rrrscores rr.r.s\"( =5+ AAc|Dcgc]}|| }}t|dkry|stj|Sttj||Scc}w)z Find population standard deviation from the list. Returns None if only 0 or 1 element in the list. Args: rounding (int): optional argument that allows user to define decimal points. Default at 3. Nr7)r@rr+rrs rr/r/s\"( =5+rct|S)z, Returns the row count of the list. )r@)rs rr0r0s v;r'ctjj|std|d t |d5}|j }ddd|S#1swYSxYw#t $r}td|d|d}~wwxYw)zG Reads the contents of a text file and returns it as a string. z The file at z does not exist.ignore)errorsNz+An error occurred when reading the file at z: )rrexistsFileNotFoundErroropenreadOSErrorIOError)rrtextrs rrrs 77>>$ ,tf4D EFFQ $x ( A668D    QCD6A3OPPQs4 A%A A%A"A%"A%% B.A??B)raT))&loggingrrsrpathlibrtypingrrrrRpandasr9unstructured.staging.baserr getLoggerrrArr&r5r<r\ DataFrameboolrprvrrnSeriesrXrlr-r.r/r0rr'rrs (( J   . /33*CDI$ )/ 2Y]!')||;>QU,  F#F83 !% 4;./ !8C= !QVW\^bWbQc !54(5HSM5%PUW[P[J\5$ 6D%) 6Xc] 65QVX\Q\K] 6 4(UQr'