(#h% dZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZdd lmZd d lmZd d lmZej2d ZeeddZedd'dededede fdZ!d(dededeefdZ"d)dedede#fdZ$GddZ%dedefdZ&d\Z'Z(Z)Z*Gd d!Z+e+eZ,d"eddfd#Z-d$ed%ede fd&Z.y)*z>Code parts dedicated to duplicate removal and text similarity.N)SequenceMatcher) lru_cache)blake2b)add)RLock)AnyDictListOptionalUnion)_Element)LRU_SIZE)trimz\.[^/?#]{2,63}$ bit_countc6t|jdS)N1)bincount)xs V/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/deduplication.pyrsSV\\#5Fimaxsize reference new_string thresholdreturnctjd|}tjd|}td||j|k\S)zIReturn the similarity ratio between two short strings, here domain names.N)STRIP_EXTENSIONsubrratio)rrrs ris_similar_domainr%sE ##B 2I $$R4J 4J 7 = = ?9 LLr inputstringlengthcNg}|jD]C}|jtj}|j s3|j |Eg}t dddD]5}|Dcgc]}t||kDs|}}t||dz k\s3|cS|Scc}w)zbSplit input into list of tokens and adjust length threshold to make sure there is enough data.)splitstripstring punctuationisalnumappendrangelen)r&r'tokenstokensampleits r sample_tokensr9#sF""$! F../ ==? MM% !F 1b" #2s1vz!22 v;&1* $M M3s /B"B"cdjt|j}t|j |j S)z=Create a bag of words and generate a hash for a given string.  digest_size)joinr9r-rencodedigest)r&r' teststrings rgenerate_bow_hashrB3s=- 45;;=J :$$&F ; B B DDrc eZdZdZddgZ ddededeeddfdZdedefd Z e d d ede efd Z dedefdZ defdZdedeefdZdeeeefdeefdZdedefdZdedefdZy)SimhashzAImplement a basic Charikar hashing approach of string similarity.hashr'Nr& existing_hashrcd||_|j|xs|j||_y)z&Store length and existing or new hash.N)r'validate create_hashrE)selfr&r'rFs r__init__zSimhash.__init__>s+ MM-0QD4D4D[4Q rc|tjt|jdj dS)z&Return a numerical hash of the string.r<big)int from_bytesrr?r@)rJr&s r_hashz Simhash._hashHs1~~ K&&(a 8 ? ? A5  ri@rr5ct|jDcgc]}|j|d|zzrdndc}Scc}w)z2Create vector to add to the existing string vectorrr*)r2r'rQ)rJr5r7s r_vector_to_addzSimhash._vector_to_addZs<DICUVaTZZ&!q&1r9VVVs"=c dg|jzt||jD]+}ttt|j |-t fdt|jDS)zCalculates a Charikar simhash. References used: https://github.com/vilda/shash/ https://github.com/sean-public/python-hashes/blob/master/hashes/simhash.py Optimized for Python by @adbar. rc3:K|]}|dk\s d|zyw)rrN).0r7vectors r z&Simhash.create_hash..jsHaa16Hs  )r'r9listmaprrSsumr2)rJr&r5rXs @rrIzSimhash.create_hash_sk t{{""; < HE#c64+>+>u+EFGF HH5#5HHHrc2t|jddS)z3Convert the numerical hash to a hexadecimal string.r+N)hexrErJs rto_hexzSimhash.to_hexls499~ab!!r inputhashcF t|dS#ttf$rYywxYw)z2Convert the hexadecimal hash to a numerical value.N)rO TypeError ValueErrorrJras r _hash_to_intzSimhash._hash_to_intps+ y"% %:&  s   ct|tr dtt|cxkrdkr|St|trA|j r dt|cxkrdkr t|S|j |Sy)z9Validate the input hash and return it, or None otherwise.N) isinstancerOr3strisdigitrgrfs rrHzSimhash.validatewsy i %"C N0C*Ir*I +J i %  "rS^'Ar'A9~%(B$$Y/ /r other_hashcFt|j|jz S)zJReturn distance between two hashes of equal length using the XOR operator.)BIN_COUNT_FUNCrErJrns rhamming_distancezSimhash.hamming_distancesdii*//9::rcX|j|j|z |jz S)zjCalculate how similar this hash is from another simhash. Returns a float from 0.0 to 1.0. )r'rrrqs r similarityzSimhash.similaritys' d33J??4;;NNr)r!@N)__name__ __module__ __qualname____doc__ __slots__rlrOr rKrQrr rSrIr`rgr rHrrrfloatrtrVrrrDrD:sG"I'+ RRR } R  R   $uWCWDIWW Is Is I""chsm (5c?";  ;3;3;OSOUOrrDcontentc4t|jS)zACalculate a simhash hex value for meaningful bits of the content.)rDr`)r|s rcontent_fingerprintr~s 7  " " $$r)rrr+c^eZdZdZd deddfdZdedefdZdedefd Zde d eddfd Z dd Z y)LRUCachea Pure-Python Least Recently Used (LRU) cache using a circular doubly linked list Adapted from CPython functools.py lru_cache decorator implementation https://github.com/python/cpython/blob/3.9/Lib/functools.py#L524 First adapted by https://github.com/vbarbaresi rrNct|_||_i|_g|_|j|jddg|jddd|_y)NF)rlockrcacherootfull)rJrs rrKzLRUCache.__init__sEG  +- !  499dD9 !  rlinkc|\}}}}||c|t<|t<|jt}|x|t<|jt<||t<|j|t<|S)N)NEXTPREVr)rJr link_prev link_next_keyresultlasts r _move_linkzLRUCache._move_linksb-1* 9dF+4i( $4yy'++T TYYt_T YYT  rkeyc|j5|jj|}|r|j|cdddS dddy#1swYyxYw)zgTests if the key that is asked for is in the cache and retrieve its value from the linked list.Nr*)rrgetr)rJrrs rrz LRUCache.getsSYY -::>>#&Dt, - - -  -s .AAvaluec|j5|jj|}|r)|j|||j|t<n|j r|j }||c|t<|t<|t|_|j t}dx|j t<|j t<|j|=||j|<nu|j t}||j ||g}|x|t<x|j t<|j|<t|j|jk\|_dddy#1swYyxYw)z Stores a given key in the cache.N) rrrrRESULTrrKEYrrr3r)rJrrroldrootoldkeyrs rputz LRUCache.puts*YY @::>>#&D%*/ 3'99"iiG471GCL'&/!( DI!YYs^F9==DIIcNTYYv%6 6*'.DJJsO 99T?D $))S%8DEIIDJI44::c?!$DJJ4<< ?DIA @ @ @s E E!!E*c|j5|jj|j|jddg|jddd|_dddy#1swYyxYw)zDelete all cache content.NF)rrclearrrr_s rrzLRUCache.clearsS YY  JJ    IItyy$=DIIaLDI   s A AA())rN) rvrwrxryrOrKrrrrlrrrVrrrrsd  d ssss#@s#@3#@4#@JrrrAcrtj|}|dk7r|dznd}tj||y)zImplement LRU cache.r*rN)LRU_TESTrr)rAcachevalrs r put_in_cachers/||J'H$NHqLE LLU#relementoptionsctdj|j}t||jkDr>t j |}||jkDrt j||dzyt|y)z(Check for duplicate text with LRU cache.r;rTF) rr>itertextr3min_duplcheck_sizerrmax_repetitionsrr)rrrArs rduplicate_testrslchhw//123J :333<< + g-- - LLX\ 2 r)g?)ru))/ryrer.difflibr functoolsrhashlibroperatorr threadingrtypingrr r r r lxml.etreer settingsrutilsrcompiler"getattrrOrprlr{boolr%r9bytesrBrDr~rrrrrrrrrVrrrs@@  #33"**/0k+FG 4MM#M%MRVMM s C c  E3EEUEPOPOf%%% %dCPPf H %$S$T$ H s t r