:Qg)BddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z mZmZmZej"dZGdd eZy) N) BaseParser) TextEdgesTable) text_in_bboxget_table_indexcompute_accuracycompute_whitespacecamelotceZdZdZ ddZedZeddZeddZedZ ed Z ed Z d Z d Z d ZdZdZdifdZy)Streama8Stream method of parsing looks for spaces between text to parse the table. If you want to specify columns when specifying multiple table areas, make sure that the length of both lists are equal. Parameters ---------- table_regions : list, optional (default: None) List of page regions that may contain tables of the form x1,y1,x2,y2 where (x1, y1) -> left-top and (x2, y2) -> right-bottom in PDF coordinate space. table_areas : list, optional (default: None) List of table area strings of the form x1,y1,x2,y2 where (x1, y1) -> left-top and (x2, y2) -> right-bottom in PDF coordinate space. columns : list, optional (default: None) List of column x-coordinates strings where the coordinates are comma-separated. split_text : bool, optional (default: False) Split text that spans across multiple cells. flag_size : bool, optional (default: False) Flag text based on font size. Useful to detect super/subscripts. Adds around flagged text. strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. edge_tol : int, optional (default: 50) Tolerance parameter for extending textedges vertically. row_tol : int, optional (default: 2) Tolerance parameter used to combine text vertically, to generate rows. column_tol : int, optional (default: 0) Tolerance parameter used to combine text horizontally, to generate columns. NFc  ||_||_||_|j||_||_||_||_||_| |_ yN) table_regions table_areascolumns_validate_columns split_text flag_size strip_textedge_tolrow_tol column_tol) selfrrrrrrrrrkwargss S/var/www/html/answerous/venv/lib/python3.12/site-packages/camelot/parsers/stream.py__init__zStream.__init__9sQ+&   $"$   $ct|Dcgc]}||D]}|jc}}}t|Dcgc]}||D]}|jc}}}t|Dcgc]}||D]}|jc}}}t|Dcgc]}||D]}|j c}}}||||f}|Scc}}wcc}}wcc}}wcc}}w)aReturns bounding box for the text present on a page. Parameters ---------- t_bbox : dict Dict with two keys 'horizontal' and 'vertical' with lists of LTTextLineHorizontals and LTTextLineVerticals respectively. Returns ------- text_bbox : tuple Tuple (x0, y0, x1, y1) in pdf coordinate space. )minx0y0maxx1y1)t_bbox directiontxminyminxmaxymax text_bboxs r _text_bboxzStream._text_bboxQs &LY&:KLQADDLDLM&LY&:KLQADDLDLM&LY&:KLQADDLDLM&LY&:KLQADDLDLM4t,  MLLLsC C )C C cd}g}g}|D]}|jjs"tj||j|s+|j t |dg}|j}|j ||j t |dt|dkDr|jd}|S)acGroups PDFMiner text objects into rows vertically within a tolerance. Parameters ---------- text : list List of PDFMiner text objects. row_tol : int, optional (default: 2) Returns ------- rows : list Two-dimensional list of text objects grouped into rows. ratolc|jSrr"r)s rz$Stream._group_rows..s 144rkeyc|jSrr4r5s rr6z$Stream._group_rows..s qttrr) get_textstripnpiscloser#appendsortedlenpop)textrrow_yrowstempr)__s r _group_rowszStream._group_rowshs" Azz|!!#zz%G<KKt @ADDDE A  F4^45 t9q=!B rc4g}|D]}|s|j||d}|dk\rg|d|dkstj|d|d|r,t|d|d}t |d|d}||f|d<w|j||dks|d|dkretj|d|dt |r|j|t|d|d}t |d|d}||f|d<|j||S)ahMerges column boundaries horizontally if they overlap or lie within a tolerance. Parameters ---------- l : list List of column x-coordinate tuples. column_tol : int, optional (default: 0) Returns ------- merged : list List of merged column x-coordinate tuples. rrr1)r>r<r=r$r!abs)lrmergedhigherlower upper_bound lower_bounds r_merge_columnszStream._merge_columnss-" .F f%r ?ayE!H, q 58*1'*%(F1I&> &)%(F1I&> &1;%?r  f-!^ayE!H,::fQiqJP"MM&1*-eAhq *BK*-eAhq *BK*5{)CF2J f-/ .0 rc |Dcgc]Q}t|dkDr?t|Dcgc]}|j|jzdz  c}t|z ndS}}}t dt|Dcgc]}||||dz zdz }}|j d||j |t dt|dz Dcgc]}||||dzf}}|Scc}wcc}}wcc}wcc}w)aQMakes row coordinates continuous. Parameters ---------- rows_grouped : list Two-dimensional list of text objects grouped into rows. text_y_max : int text_y_min : int Returns ------- rows : list List of continuous row y-coordinate tuples. rrr)r@sumr#r&rangeinsertr>) rows_grouped text_y_max text_y_minrr)row_midsirDs r _join_rowszStream._join_rowss&" =@FQJCA.q!$$+". /#a& 8A M  >C1c(m=TU!xA.!3UU Az" J05aTQ0GH1a$q1u+&HH / VIs"C*#C%C*6C0 C5%C*cd|rtj||}|Dcgc] }t|}}|Dcgc]9}t|t|k(s|D]}|j|j f;}}}|j tjt||Scc}wcc}}w)aAdds columns to existing list by taking into account the text that lies outside the current column x-coordinates. Parameters ---------- cols : list List of column x-coordinate tuples. text : list List of PDFMiner text objects. ytol : int Returns ------- cols : list Updated list of column x-coordinate tuples. r) rrGr@r$r"r%extendrQr?)colsrBrrYelementsr)new_colss r _add_columnszStream._add_columnss& %%dG%? @ .sB'B,!B,cDt|}tdt|Dcgc]}||d||dz dzdz }}|jd||j |tdt|dz Dcgc]}||||dzf}}|Scc}wcc}w)a8Makes column coordinates continuous. Parameters ---------- cols : list List of column x-coordinate tuples. text_x_min : int text_y_max : int Returns ------- cols : list Updated list of column x-coordinate tuples. rrr)r?rTr@rUr>)r` text_x_min text_x_maxr[s r _join_columnszStream._join_columnss"d|;@CI;NOaad1q5k!n,1OO Az" J05aTQ0GH1a$q1u+&HH PIs BBc|jD|j7t|jt|jk7r tdyyy)Nz1Length of table_areas and columns should be equal)rrr@ ValueError)rs rrzStream._validate_columnssM    'DLL,D4##$DLL(99 !WXX:-E 'rcB|jdt|j}|j||j }|j j ||j||}t|sdd|j|jfdi}|S)a1A general implementation of the table detection algorithm described by Anssi Nurminen's master's thesis. Link: https://dspace.cc.tut.fi/dpub/bitstream/handle/123456789/21520/Nurminen.pdf?sequence=3 Assumes that tables are situated relatively far apart vertically. c4|j |jfSrr#r"xs rr6z2Stream._nurminen_table_detection..sqtteQTT]rr7)rrN) sortrrgenerate get_relevant textedgesr_get_table_areasr@ pdf_width pdf_height)r textlinesrrrelevant_textedges table_bboxs r_nurminen_table_detectionz Stream._nurminen_table_detection s 23t}}5 9%&335 01..y:LM :aA4HJrcVg|_|j|j}|jg}|jD]o}|j d\}}}}t |}t |}t |}t |}t ||||f|j}|j|q|j|}||_ yi}|jD]M} | j d\}}}}t |}t |}t |}t |}d|||||f<O||_ y)N,) rrrhorizontal_textrsplitfloatrr_ryrx) rhor_textregionr%r&x2y2 region_textrxareas r_generate_table_bboxzStream._generate_table_bbox&s3    #++H!!-"001F%+\\#%6NBBrBrBrBrB".BB/?AUAU"VKOOK0177AJ%J(( 4!%CBB2Y2Y2Y2Y/3 BB+,  4%rc i}t||j|d<t||j|d<|djd|djd||_|j |j\}}}}|j |jd|j}|j|||} |D cgc] } t| } } |j|j|dk7r|j|jd} | D cgc] } t| } } | jd || j|td t| d z Dcgc]}| || |d zf} }| | fSt| s ||fg} | | fSt!t#| | j$}|d k(r]t't)d | } t| r!t!t#| | j$}nt+j,d |d z|D cgc]0} t| |k(s| D]}|j.|j0f2} } }|j3t5| |j6 } g}td t| D]r}| |d z d }| |d }|j9|jDcgc]6}|j|D]"}|j.|kDr|j0|kr|$8c}}t|jDcgc]B}|j|D].}|j.| dd kDs|j0| d d kr|0D}}}|j9||j;| ||j} |j=| ||} | | fScc} wcc} wcc}wcc}} wcc}}wcc}}w)N horizontalverticalc4|j |jfSrrlrms rr6z3Stream._generate_columns_and_rows..Hs!$$ rr7c4|j|j fSr)r"r#rms rr6z3Stream._generate_columns_and_rows..IsqttaddUmrr^r{rrc |dk7SNrrms rr6z3Stream._generate_columns_and_rows..is Q!VrzNo tables found in table area )rrI)rr| vertical_textror'r/rGrr\r@rr}r~rUr>rTr$setcountlistfilterwarningswarnr"r%rQr?rr_rcrg)r table_idxtkr'rerXrfrWrVrDrYrar`cr[ncolsr) inner_textleftrightr( outer_texts r_generate_columns_and_rowsz!Stream._generate_columns_and_rowsBs+B0D0DE|)"d.@.@Az|!!&=!>z$;< 9=9U6 J J'' L(A4<<'X |ZD$01qCF11 << # Y(?2(E << *005D&*+E!H+D+ KK: & KK #49!SY]4KLqT!Wd1q5k*LDLZTzSx=#Z01PTzMCMx~~>A: $F+;X$FGH8} #CMx~~ F (FyST}o&VW*6%&#a&E/STNOQTT144L **6$D GAJE%%.2[[ )%)[[%;!" ttd{qtte| &*[[!![[3ttd2hqk)QTTDGAJ->  !!*-((z4<<H))$ JGTzq2,M&s+ N+N0.N5#N:7!N:;O AOc t||}|j}g}dD]}|j|D]p}t||||j|j |j \} } | dddk7s>|j| | D]\} } } | |j| | _ rtd|gg}|j}tj||_|jj|_t!|}d|_||_||_|dz|_t+t,j.j1|j2j5dd |_g}|j9|j:Dcgc]0}|j<|j>|j@|jBf2c}|j9|jDDcgc]0}|j<|j>|j@|jBf2c}||_#d|_$d|_%|jL|_'|Scc}wcc}w) N)rr)rrrr)rIrIdstreamrzpage-r)(r set_all_edgesr'r rrrr>cellsrBr datapd DataFramedfshaper flavoraccuracy whitespaceorderintospathbasenamerootnamereplacepager_r|r"r#r%r&r_text_image _segmentsrr _textedges)rrr`rDrtable pos_errorsr(r)indiceserrorr_idxc_idxrBrrrrs r_generate_tablezStream._generate_tablesdD!##% 4 >I[[+ >!0#"nn# "2A;(*%%e,.5>*ud9= E*516> > >$c:%6$78zz<<%hhnn '-  !%!m ))$--8@@"MN  8L8LM1qttQTT144.MN 8J8JK1qttQTT144.KL  >> NKs 5H<5Ic\|j||tjj|j}|st j d||js@|jrtj|dgStjd|gS|jg}tt|jjddD]E\}}|j!||\}} |j#||| } || _|j'| G|S)Nz Processing z8 is image-based, camelot only works on text-based pages.zNo tables found on c |dSrrrms rr6z'Stream.extract_tables..s 1rT)r8reverse)_generate_layoutrrrrloggerinfor|imagesrrr enumerater?rxkeysrr_bboxr>) rfilenamesuppress_stdout layout_kwargs base_filename_tablesrrr`rDrs rextract_tableszStream.extract_tabless h 6((7  KK+m_5 6##{{ $o&)) I 3M?CDI !!#& 4??'')~t L  "MIr88BGJD$((D$?EEK NN5 !  "r) NNNFFr2rr)r)r)__name__ __module__ __qualname____doc__r staticmethodr/rGrQr\rcrgrryrrrrrrrrrs$P%0,""H))V46.Y 2%8FP+Z8=Brr)rloggingrnumpyr<pandasrbasercorerrutilsrr r r getLoggerrrrrrrsD #WW   9 %BZBr