(#hD dZ ddlZdZddlZddlZ ddlZdZddlm Z ddl m Z ddl m Z mZmZmZmZmZddlmZ ddlZdZ ddlZdZ ddlZdZ dd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'ejPe)Z*ddhZ+ejXdejZZ.ejXdejZZ/ejXdZ0e$dddddZ1ejXdejdejfzZ4ejXdZ5ejXdZ6hdZ7ddhZ8dZ9ejXdZ:ejXdejvZd!e=de?fd"Z@d#e=deeAfd$ZBdee=eAfdeAfd%ZCd&eAde?fd'ZDd(eAd&eAdeAfd)ZEd*eAdee#fd+ZFd*e dee#fd,ZGe d-.d/eAdeAfd0ZHd1eAdeAfd2ZIdTd1eAd3ed4deAfd5ZJe d6.dUd7eAd8e?d9e?deeAfd:ZKdUd;eAd8e?d9e?deeAfd<ZLd=e!de!fd>ZMe d6.d1eAdeAfd?ZNd@e!de?fdAZOdBeeAde?fdCZPdDe dEeQde fdFZRdGeQdHe de?fdIZSdVd=e#dJeAdKe?de?fdLZTdMeAdNeAdeeAfdOZUdMeAdNeAdJeAdPe dee?e ff dQZVd@e!de?fdRZWd1eeAde?fdSZXy#e$rdZYwxYw#e$rdZYwxYw#e$rdZYywxYw#e$rdZYwxYw#e$rdZYwxYw#e$rdZYwxYw)Wzj Module bundling functions related to HTML and text processing, content filtering and language detection. NTF) lru_cache)islice)AnyListLiteralOptionalTupleUnion normalize)detect) from_bytes)_Element) HtmlElement HTMLParser fromstring) HTTPResponseutf-8utf_8z^< ?! ?DOCTYPE.+?/ ?>z(z(|<[^>]*>)) collect_idsdefault_doctypeencodingremove_comments remove_pisz(?])\n)flagsz^https?://|/+$z3[^\s]+\.(avif|bmp|gif|hei[cf]|jpe?g|png|webp)(\b|$)>phitdrefcellheaditemquotecodepre)zhttp-equiv="content-language"zproperty="og:locale"z ([a-z]{2})z\W*(Drucken|E-?Mail|Facebook|Flipboard|Google|Instagram|Linkedin|Mail|PDF|Pinterest|Pocket|Print|QQ|Reddit|Twitter|WeChat|WeiBo|Whatsapp|Xing|Mehr zum Thema:?|More on this.{,8}$)$ filecontentreturnc4t|ts|Str|dddk(r tj|Str|dddk(r tj|Str tj|Str tj|S|S#t $rt jdY~wxYw#tj$rt jdYwxYw#tj$rYwxYw#tj$rY|SwxYw)z Don't trust response headers and try to decompress a binary string with a cascade of installed packages. Use magic numbers when available. Nszinvalid GZ files(/zinvalid ZSTD file) isinstancebytesHAS_GZIPgzip decompress ExceptionLOGGERwarningHAS_ZSTD zstandard ZstdError HAS_BROTLIbrotlierrorHAS_ZLIBzlib)r&s N/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/utils.pyhandle_compressed_filer<^s k5 )KO6 .??;/ /KO':: 0'' 4 4 $$[1 1 ??;/ / - . NN, - . "" 0 NN. / 0 ||    zz     sGBB9"C'>DB65B69(C$#C$'C=<C=DDdatacF |jdy#t$rYywxYw)zLSimple heuristic to determine if a bytestring uses standard unicode encodingzUTF-8FT)decodeUnicodeDecodeError)r=s r;isutf8rAs, G  s    bytesobjectct|rdgSg}t/t|d}||j|jt |dkr t |}n!t |dd|ddzxs t |}t |dkDr)|j |Dcgc]}|jc}|Dcgc] }|tvs |c}Scc}wcc}w)z="Read all input or first chunk and return a list of encodingsrNri'iixr) rAcchardet_detectappendlowerlenrextendrUNICODE_ALIASES)rBguessescchardet_guessdetection_resultsrgs r;detect_encodingrOskyG"(5jA  % NN>//1 2 ;%&{3&{5D'9Kq >? ;!!?":A ;;? ;sC< C Cct|tr|Sd}t|}t|D]} |j |}n|xst|ddS#t t f$rtjd|d}YSwxYw)zCheck if the bytestring could be GZip and eventually decompress it, guess bytestring encoding and try to decode to Unicode string. Resort to destructive conversion otherwise.Nzwrong encoding detected: %srreplace)rerrors) r+strr<rOr? LookupErrorr@r1r2)r&htmltextguessed_encodings r; decode_filerWs+s#H)5K+K8 "))*:;H   Ks;KK/0  NN8:J KH sA'A?>A? beginningc d|vS)zOAssess if the object is proper HTML (awith a corresponding tag or declaration).html)rXs r;is_dubious_htmlr\s  "" htmlstringc8d|vr3|jd\}}}tjd|ddz|z}tt |j D]=\}}d|vr,|j drtjd|d}|S|d kDs<|S|S) z>Repair faulty HTML strings to make then palatable for libxml2.doctype )countzz\1>) partition DOCTYPE_TAGsub enumerateiter splitlinesendswith FAULTY_HTML)r^rX firstline_restilines r;repair_faulty_htmlrssI'11$7 1d __R!_zsanitize..6s&eaq.'I&esu␤rb)rrfilterrkrQAttributeError)rrrs ` r;sanitizer/sct^T::yy&eSWSbSbSd&efgoopxz|}} sAA A A r|cR|jD]}|j}| |jnd}|jtvxs|tv}|jtvxs |tvxs|}|j D]S}d|vs|j |r"|j ddd|jvs9|j j|U|jrt|j|||_ |jst|j|||_ |S)z?Trims spaces, removes control characters and normalizes unicoderb:rcr) rj getparenttagSPACING_PROTECTEDFORMATTING_PROTECTEDattribsplitnsmappoprrtail)r|elemparent parent_tagrr attributes r; sanitize_treer;s L!#)#5VZZ2 %66Y*HY:Y%99qZK_=_qcq /Ii{{9-a1H1KSWS]S]1]KKOOI. / 99 NNKDI 99 NNKDI%L& Kr]c dj|jjS#ttf$rYywxYw)z/Remove unnecessary spaces within a text string.rrb)rrstriprrrs r;rrSs<xx '--// I &s,/AAelementcdD]!}|j|d}t|s!y|jjD]$\}}|j dst|s$yy)z*Check if an element is a valid img element)data-srcsrcrbTrF)get is_image_fileritems startswith)rattrrvalues r;is_image_elementr]si#kk$#   #>>//1 KD%z*}U/C  r]imagesrcc`|t|dkDryttj|S)zCheck if the observed string corresponds to a valid image extension. Use a length threshold and apply a regex on the content.i F)rGboolIMAGE_EXTENSIONsearch)rs r;rrks.3x=4/ &&x0 11r]iterablenc#Kt|}tt||x}r|tt||x}ryyw)zChunk data into smaller pieces.N)rjtupler)rriteratorbatchs r; make_chunksrssFH~H!,- -% - !,- -% -s >AAmy_lenoptionsc||jkr!tjd|jy||jkDr"tjd||jyy)z=Check if the document length is within acceptable boundaries.ztoo small/incorrect for URL %sFztoo large: length %s for URL %sT) min_file_sizer1r8url max_file_size)rrs r;is_acceptable_lengthr{sO %%% 5w{{C %%% 6 L r]target_languagestrictcPtD]G}|jd|d}|stfd|Drytj d|y|r>|j d}|r+tfd|Drytj d ytj d y) zrCheck HTML meta-elements for language information and split the result in case there are several languages.z .//meta[@z ][@content]c3K|]9}tj|jddjv;yw)contentrbN RE_HTML_LANGrrrFrrrs r;rz"check_html_lang..s6l^b?l&8&8)R9P9V9V9X&YYl?ATz%s lang attr failedFz //html[@lang]c3K|]9}tj|jddjv;yw)langrbNrrs r;rz"check_html_lang..s6i[_?l&8&8&"9M9S9S9U&VVirzHTML lang failedzNo relevant lang elements found)TARGET_LANG_ATTRSfindallanyr1debugxpath)r|rrrelemss ` r;check_html_langrs" yk:; lfkll LL. 5  ?+ ichii LL+ , LL23 r] temp_text temp_commentsctdurFt|t|kDrtj|ntj|\}}|Stj dd}|S)zARun external component (if installed) for language identificationTz3Language detector not installed, skipping detectionN) LANGID_FLAGrG py3langidclassifyr1r2)rrresultros r;language_classifierrsdd9~M 22   y )##M2  M LM Mr]docmetac|[t|||_|j>|j|k7r/tjd|j|jd|fSd|fS)zFFilter text based on language detection and store relevant informationzwrong language: %s %sTF)rlanguager1r2r)rrrrs r;language_filterrsc".y-H    'G,<,<,O NN2G4D4Dgkk R= '>r]c|j |jn |j}| xsC|jxs1tt t j |jS)zFilter out unwanted text)rrrrr RE_FILTERmatchrk)rtesttexts r; textfilterrsN&||3w||H< a8++- aS(J]J]J_5`1aar]c>t|xr|j S)zJDetermine if a string is only composed of spaces and/or control characters)rrrs r;text_chars_testrs < 0 000r])r)FF)F)Y__doc__r.r- ImportErrorloggingrer:r9 functoolsr itertoolsrtypingrrrrr r unicodedatar r7r6r4r3rrcchardetr rDcharset_normalizerr lxml.etreer lxml.htmlrrrurllib3.responser getLogger__name__r1rIcompileIrgrmHTML_STRIP_TAGSr{UNICODE MULTILINErURL_BLACKLIST_REGEXrrrrr IGNORECASErr,r<rrArSrOrWr\rsr~rrrrrrrrrrintrrrrrrrr[r]r;rs H H ==!JH K 2*99)   8 $G$bjj0"$$7 bjj+RTT2 "**45UEGeivz{ -RZZ 5LM bjj!23"**STPUONrzz-(  BJJ[ ]] , ""%"J4<<49<2LU5#:.L3L2#s#t# 3 3 3  +)>*#*(;"7*Z 5@s@s@@ >c>c> *c*8T0U*be*  4#tT^fgj^k( 3  d W_`cWd X0 4 h 4 2HSM2d2###st+TVZ0 3 s x} s3WZ_deiknen_o"bbTb1HSM1d1u HHJ HK OsiJJ#J1J? K KJ J #J.-J.1J<;J<?K  K  KKK&%K&