(#hH8dZddlZddlZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z mZmZmZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZm Z dd l!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4ejje6Z7ddhZ8de"de&de9fdZ:dededede&de e e9e;ff dZddddddddddddddddddddddde%dfded ee9d7ee9d!e=d"e=d#e=d$e=d%e=d&e9d8e=d'ee9d(e=d)e=d*e=d+e=d,e=d-eee9efd.e=d/e=d0ee;d1ee e9d2ee e9d9ee9d4eed5edee&dee9f6d:Z?y);z4 Extraction configuration and processing functions. N)copydeepcopy)AnyDictOptionalSetTupleUnion)_ElementElementXPath strip_tags) HtmlElement)baseline)content_fingerprintduplicate_test)compare_extraction)build_html_output convert_tagsprune_unwanted_nodes tree_cleaning)extract_commentsextract_content)Documentextract_metadata)DEFAULT_CONFIG Extractor use_config) LANGID_FLAGcheck_html_langlanguage_filter load_htmlnormalize_unicode)build_json_outputcontrol_xml_outputxmltotxtxmltocsv)REMOVE_COMMENTS_XPATHmarkdowntxtdocumentoptionsreturnc d|jvr|jjdD]o}|jdk7st |dk(s"|j r/|j r<|j}|O|jdk7s_|j|qt||}t)|S|jdk(r!t||j}t)|S|jdk(r!t||j}t)|S|jdk(r!t||j}t)|S|jr:d }d D]-}t||s||d t!t||d z }/|d z }nd }|t#|j|j}|j$3|d t#|j$|jj'}t)|S)zMConvert XML tree to chosen format, clean the result and output it as a stringxml*graphicrcodecsvjsonhtmlz--- ) titleauthorurlhostname descriptionsitenamedate categoriestags fingerprintidlicensez:  )formatbodyitertaglentexttail getparentremover&r( formattingr% with_metadatargetattrstrr' commentsbodystripr$)r,r-elementparent returnstringheaderattrs M/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/core.pydetermine_returnstringrZ,s }}))#. +G y(LA%   **,%&***>MM'* +*(G< J \ **G 5 '*<*<= D \ **A 6 !(73H3HI > \ **; 6 !(73H3HI 8 \ **3  F  J8T*bWXt-D)E(FbIIF J g FF (8==':L:L"M!NO  ,*^2hx7L7LgN`N`.a-bciikL \ ** cleaned_treecleaned_tree_backup tree_backupct||\}}}|jst|t|||||\}}}||jkr=|j dk(s.t t|\}}}tjd||||fS)z?Execute the standard cascade of extractors used by Trafilatura. precisionz+non-clean extracted length: %s (extraction)) rfastrrmin_extracted_sizefocusrLOGGERdebug)r\r]r^r-postbody temp_textlen_texts rYtrafilatura_sequenceries%4L'$J!Hi <<(:  [ !     ) %)X',,,W]]k5Q(0+1F(G%)X BHM Y ((r[FTpython filecontentr9ra no_fallbackfavor_precision favor_recallinclude_comments output_formattarget_languageinclude_tablesinclude_imagesinclude_formatting include_links deduplicatedate_extraction_paramsrOonly_with_metadata max_tree_size url_blacklistauthor_blacklistas_dict prune_xpathconfigc  |r|}tjdt|rtjdt|r td|rt |t s?t d)id|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d|d|d|} t |}|tjd|t|jrP|jsts>t||jdur&tjd|jt|jrt||j |j"|j|j$}|j |j&vr&tj)d|j t|j*rT|j,r|j.r |j s0tjd|jtt1}|6t |t2r|g}t5||Dcgc] }t7|c}}t9t;||}t;|}t=|||j xs |j }|j>rtA||\}}}}ntCddd}}}|jDdk(rt5|tF}tI||||\} }!}"|jJrtM| |jJkDr+tjOdtM| tQ| d tM| |jJkDr0tjOd!tM| |jt|j>r/||jRkr tjOd"|j|"|jTkr7||jVkr(tjOd#|"||jt|jXr4t[| |d$ur&tjOd%|jt|jrEt]|!||j|\}#}|#d$ur&tjOd&|jt|j`d(k(r[tc| |jd|_3|j>r"tc||jd|_||_4|jf|_5n|!|c|_5|_4| |_6|s|S|joScc}w#t^tf$r#tj)d'|jYywxYw)*alInternal function for text extraction returning bare Python variables. Args: filecontent: HTML code as string. url: URL of the webpage. fast: Use faster heuristics and skip backup extraction. no_fallback: Will be deprecated, use "fast" instead. favor_precision: prefer less text but correct extraction. favor_recall: prefer more text even when unsure. include_comments: Extract comments along with the main text. output_format: Define an output format, Python being the default and the interest of this internal function. Other values: "csv", "html", "json", "markdown", "txt", "xml", and "xmltei". target_language: Define a language to discard invalid documents (ISO 639-1 format). include_tables: Take into account information within the HTML element. include_images: Take images into account (experimental). include_formatting: Keep structural elements related to formatting (present in XML format, converted to markdown otherwise). include_links: Keep links along with their targets (experimental). deduplicate: Remove duplicate segments and documents. date_extraction_params: Provide extraction parameters to htmldate as dict(). with_metadata: Extract metadata fields and add them to the output. only_with_metadata: Only keep documents featuring all essential metadata (date, title, url). url_blacklist: Provide a blacklist of URLs as set() to filter out documents. author_blacklist: Provide a blacklist of Author Names as set() to filter out authors. as_dict: Will be deprecated, use the .as_dict() method of the document class. prune_xpath: Provide an XPath expression to prune the tree before extraction. can be str or list of str. config: Directly provide a configparser configuration. options: Directly provide a whole extractor configuration. Returns: A Python dict() containing all the extracted information or None. Raises: ValueError: Extraction problem. H"no_fallback" will be deprecated in a future version, use "fast" insteadzR"as_dict" will be deprecated, use the .as_dict() method on bare_extraction results:max_tree_size is deprecated, use settings.cfg file insteadr~rprar`recallcommentsrNlinksimagestablesdeduplangr9rOrxr{rz date_paramsNzempty HTML tree: %sFzwrong HTML meta language: %szblacklisted URL: %szno metadata: %srFrDrzoutput tree too long: %shiz'output tree too long: %s, discarding %sznot enough comments: %sz+text and comments not long enough: %s %s %sTz!discarding duplicate document: %szwrong language: %szdiscarding data: %srj)8warningswarnPendingDeprecationWarning ValueError isinstancerr#rderrorrrar r!sourcerOrr9rr{rzwarningrxr=r7rrQrr rrrrrr rcr)riryrIrermin_extracted_comm_sizemin_output_sizemin_output_comm_sizerrr" TypeErrorrEr'rNrJrRraw_textrFr|)$rkr9rarlrmrnrorprqrrrsrtrurvrwrOrxryrzr{r|r}r~r-treer,xr\r]rR temp_comments len_commentsrfrgrhis_not_target_langs$ rYbare_extractionrsD V %  ` % UVV *Wi8  '  &    &  *   " "  !  (  2 .! "(# $/% *p% < LL. 4  <STL(< -tW) %)X  8}w444 7XG8T*8}w444 =MNN !    w/N/N N LL2GNN C w.. .w;;; LL=     ==^Hg>$F LL>X %K      (#6HNN#c)C0A0A,BB$H "(G 44r[)@__doc__loggingrrrtypingrrrrr r lxml.etreer r r r lxml.htmlrr deduplicationrrexternalrhtmlprocessingrrrrmain_extractorrrmetadatarrsettingsrrrutilsr r!r"r#r$r0r%r&r'r(xpathsr) getLogger__name__rdrrQrZintriboolrrrr[rYrs599;;!>( >0;;KJ)   8 $5! 6+X6+ 6+c6+r))$)) )  8S#  )>!!!%) $7;$#'(,+/!% #'1d;d; #d; d; d;  d;  d;d;d;c]d;d;d;d;d;d;%T#s(^4d; !d;"#d;$C=%d;&CH%'d;(s3x()d;*+d;,#-d;. /d;0i 1d;2eHd38n,-.3d;R#!! %) $7;$#'(,+/"&!% #'555 #5}5  5  5  55555c]55555 !5"%T#s(^4#5$%5&'5(C=)5*CH%+5,s3x(-5.3-/50#152 354i 556c]75r[