#h2NdZddlZddlZddlZddlZddlZ ddlZdZ ddl Z dZ ddl m Z m Z ddlmZmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZmZmZdd lm Z d d l!m"Z"d dl#m$Z$d dl%m&Z&m'Z'd dl(m)Z)d dl*m+Z+m,Z,m-Z-ej\e/Z0GddZ1e1Z2GddeZ3GddZ4GddZ5GddZ6de7de6fdZ8y#e$rdZYwxYw#e$rdZ YwxYw)zd Defines a URL store which holds URLs along with relevant information and entails crawling helpers. NTF) defaultdictdeque)datetime timedelta)Enum) itemgetter)Lock)Any DefaultDictDequeDictListOptionalTupleUnion)RobotFileParser) normalize_url) filter_links) lang_filter validate_url) clear_caches) get_base_urlget_host_and_path is_known_linkc`eZdZdZdZd deddfdZededefdZ dedefd Z de defd Z y) CompressorzYUse system information on available compression modules and define corresponding methods.) compressor decompressor compressionreturnNc |rtrtjn#|rtrtjn |j |_|rtrtj|_y|rtrtj|_y|j |_yN) HAS_BZ2bz2compressHAS_ZLIBzlib _identicalr decompressr)selfr s M/var/www/html/sandstorm/venv/lib/python3.12/site-packages/courlan/urlstore.py__init__zCompressor.__init__:sjw LL"-( w NN %0H CG// datac|S)zReturn unchanged data.)r/s r,r)zCompressor._identicalFs  r.cN|jtj|dS)z9Pickle the data and compress it if a method is available.)protocol)rpickledumpsr+r/s r,r&zCompressor.compressKsv||D1=>>r.cJtj|j|S)zADecompress the data if a method is available and load the object.)r5loadsrr7s r,r*zCompressor.decompressOs||D--d344r.T) __name__ __module__ __qualname____doc__ __slots__boolr- staticmethodr r)r&bytesr*r1r.r,rr6se_.I  D  D  ?S?S?5u55r.rceZdZdZdZdZdZy)Statez0Record state information about a domain or host.rN)r;r<r=r>OPEN ALL_VISITEDBUSTEDr1r.r,rDrDWs6 DK Fr.rDc<eZdZdZdZej fdeddfdZy) DomainEntryz7Class to record host-related information and URL paths.)countrulesstate timestamptotaltuplesrNr!Nchd|_d|_||_d|_d|_t |_y)Nr)rLrMrNrOrPrrQ)r+rNs r,r-zDomainEntry.__init__bs. 04 ! (, +07 r.)r;r<r=r>r?rDrGr-r1r.r,rKrK^s$=KI&+jj3e3T3r.rKc4eZdZdZdZdededdfdZdefdZy) UrlPathTuplezBClass storing information for URL paths relative to a domain/host.)urlpathvisitedrUrVr!Nc>|jd|_||_y)Nutf-8)encoderUrV)r+rUrVs r,r-zUrlPathTuple.__init__os%nnW5 $ r.c8|jjdS)zGet the URL path as string.rX)rUdecoder+s r,pathzUrlPathTuple.pathss||""7++r.) r;r<r=r>r?strr@r-r]r1r.r,rTrTks0H&I%%d%t%,c,r.rTc ZeZdZdZdZ dBdedeedededed df d Z dCd e ed ed e ee e ffd Z ded e e fdZdDdZ dEdedee e deedee e d df dZ dFde edeed e eeeffdZ dGdee edee ed ed dfdZ dHdedededeeded df dZd e ed dfd!ZdDd"Zd e efd#Zd e efd$Zded efd%Zd efd&Zded e efd'Zded e efd(Zde ed e efd)Z de ed e eeeffd*Z!ded efd+Z"ded efd,Z#dIded-ed eefd.Z$ dJd/e%d0ed e efd1Z& dKd0ed/ed e efd2Z'd3ed4ee(d dfd5Z)d3ed ee(fd6Z*dLd3ed7e%d e%fd8Z+d e efd9Z,d efd:Z-d;e%d efd<Z.d e efd=Z/dDd>Z0dDd?Z1d@ed dfdAZ2y)MUrlStorezNDefines a class to store domain-classified URLs and perform checks against it.) compresseddonelanguagestricttrailing_slashurldict_lockNrarcrdtrailingverboser!c|_d_|_|_|_t t _t_ dtdtddffd }|ritjjdsItjtj|tjtj |yyy)NFnumframer!ctjdtjj t j dy)Nz.dump_unvisited_urlss7 LLNDLL!   % % ' HHQKr.win)rarbrcrdrerrKrfr rgr rrplatform startswithsignalSIGINTSIGTERM)r+rarcrdrhrirts` r,r-zUrlStore.__init__s!+ '/ " $,6A+6N 6  S    3<<2259 MM&--)< = MM&..*= >:7r.r/rVcltt}tj|D]} t |\}}|durt j d|t|jJt||j|j|jdurt j d|tt||j|j|j}t|\}}||jt|||S#t tf$rt j#d|Y wxYw)NFzInvalid URL: %szWrong language: %s)rdrcrezDiscarding URL: %s)rrdictfromkeysrrnro ValueErrorrcrrdrerrappendrT TypeErrorwarning) r+r/rV inputdicturlvalidation_result parsed_urlhostinforUs r, _buffer_urlszUrlStore._buffer_urlss!z%UrlStore._set_done..s V1EJJ!6 Vs')T)rballrfvaluesrgr\s r, _set_donezUrlStore._set_donesSyyS V @S@S@U VV !   ! !Wy ! !s AAto_rightrOto_leftcx|jdrd|ddz}||jvrS|}nP|jdr?d|ddz}||jvr)|j||j|<|j|=||jvrZ|j|jtjury|j |}|Dchc]}|j c}nt}t||jfd|D||jfd|D|j5|jr(tj||j|_n||j|_t!||j|_|||j|_t'd |Dr#tj(|j|_n5tj*|j|_|j,rd |_dddycc}w#1swYyxYw) Nzhttp://httpszhttps://httpr3c3XK|]!}t|jr|#ywr#rr]rtknowns r,rz'UrlStore._store_urls..s Ra=53QR**c3XK|]!}t|jr|#ywr#rrs r,rz'UrlStore._store_urls..s U!mAFFHe6TAUrc34K|]}|jywr#)rV)rus r,rz'UrlStore._store_urls..s+199+F)rwrfrNrDrIrr]rsetextend extendleftrgrarr&rQrprPrOrrHrGrb) r+rrrOr candidateurlsrrs @r, _store_urlszUrlStore._store_urlss   Y '&*,IDLL("   z *+IDLL('+||I'> V$LL+ T\\ !||F#))U\\9??6*D'+,!QVVX,E7DEE   KKR8R R   OOUwU U ZZ &.8.A.A$.G V$+.2 V$+),TDLL &$1: V$.+d++-2->-> V$*-2ZZ V$*99 %DI & &- & &s H+d}i}tj|}t|D]g}t|\}}||k7r9|}|j |D cic]} | j | j }} ||vsT|dk(s |dk(s_||se||=it|Scc} w)NrrE)r|r}sortedrrr]rVlist) r+rr last_domain known_pathsremaining_urlsrrrUrs r, _search_urlszUrlStore._search_urlss&* 13 t,.) (C 1# 6 Hg;&& <@OOHD  B Bs /BB c|j5tt|_dddt t j }tjd|y#1swY>xYw)zRe-initialize the URL store.Nz UrlStore reset, %s objects in GC) rgrrKrfrrrrnro)r+rks r,resetzUrlStore.resetMsI ZZ 4&{3DL 4jjl 7=  4 4s AA(cHt|jjS)z#Return all known domains as a list.)rrfkeysr\s r,get_known_domainszUrlStore.get_known_domainsWsDLL%%'((r.c|jjDcgc]%\}}|jtjk(s$|'c}}Scc}}w)ziFind all domains for which there are unvisited URLs and potentially adjust done meta-information.)rfrrNrDrG)r+rrs r,get_unvisited_domainszUrlStore.get_unvisited_domains[s8#ll002Ldaagg6KLLLs %A A ct||jvr*|j|jtjk7Sy)z9Tell if all known URLs for the website have been visited.F)rfrNrDrGrs r,is_exhausted_domainzUrlStore.is_exhausted_domain`s/ T\\ !<<'--; ;r.c4t|jS)zFReturn the number of websites for which there are still URLs to visit.)rprr\s r,unvisited_websites_numberz"UrlStore.unvisited_websites_numbergs4--/00r.cl|j|Dcgc]}||jzc}Scc}w)zLGet all already known URLs for the given domain (ex. "https://example.org").)rr]r+rrs r,find_known_urlszUrlStore.find_known_urlsms*+/??6+BCa!CCCs1c|j|s=|j|Dcgc]"}|jr||jz$c}SgScc}w)z,Get all unvisited URLs for the given domain.)rrrVr]rs r,find_unvisited_urlszUrlStore.find_unvisited_urlsqsF''//3v/FX!aiiFQVVX%X X Ys AAc(|j|dS)z:Take a list of URLs and return the currently unknown ones.rrrr+rs r,filter_unknown_urlszUrlStore.filter_unknown_urlsw  a 00r.c(|j|dS)z as_visitedc"|j|s|j|}|D]}|jr|rdd|_|j5|j|xj dz c_ddd|j ||tj||jzcS|j5tj|j|_ ddd|jy#1swYxYw#1swY&xYw)zSRetrieve a single URL and consider it to be visited (with corresponding timestamp).TrNrO)rrrVrgrfrLrrnowr]rDrHrNr)r+rr url_tuplesrs r,get_urlzUrlStore.get_urls''/0J! /{{!&* !ZZ< LL066!;6<((x||~(V!CHHJ.. /ZZ ;).):):DLL & ; <<  ; ;s #C9=#D9D D time_limitmax_urlscg}|jjD]\}}|jtjk7r$|j r3t j|j z j|kDsc|j|}|w|j|t||k\sn|j|S)zaGet a list of immediately downloadable URLs according to the given time limit per domain.) rfrrNrDrGrOrr total_secondsrrrpr)r+rrrwebsiteentryrs r,get_download_urlszUrlStore.get_download_urlss"ll002 NGU{{ejj(OOLLNU__4CCE Rll7+?KK$4yH,   r.c|j}|sgS|t|zxsd}g}|D]b}|j|}g}|D]} t||k\st|t|z|k\rnl| jr:|j | j d| _|j 5|j|xjdz c_dddtj} |j|j} | r| | z j|kDrd} n!|t| | z jdz } |D]} |j | || zf| |z } | td| |z z}|j|||e|j!t#|t%dS#1swY}xYw) zcGet up to the specified number of URLs along with a suitable backoff schedule (in seconds).rTNgz.2frr)key)rrprrVrr]rgrfrLrrrOrfloatrrrrr)r+rr potential per_domaintargetsrrurlpathsrroriginal_timestamp schedule_secsrU total_diffs r,establish_download_schedulez$UrlStore.establish_download_schedules ..0 IY/41 +-" GF0J"$H! 8MZ/G s8}4A{{OOCHHJ/"&CK8 V,22a7288 8,,.C!%f!5!?!? &,,;;= J # *U00??A#F.! $ , v/?@A+  ,yMJ,FGGJ   VZ:  FE" GH g:a=11/88s 0#F77G rrMcn|jrtj|}||j|_y)z)Store crawling rules for a given website.N)rarr&rfrM)r+rrMs r, store_ruleszUrlStore.store_ruless* ??''.E&+ W#r.c||jvrQ|jr,tj|j|jS|j|jSy)z7Return the stored crawling rules for the given website.N)rfrarr*rM)r+rs r,rzUrlStore.get_rulessM dll "!,,T\\'-B-H-HII<<(.. .r.defaultcvd}|j|} |jd}|xs|S#t$rYwxYw)zBReturn the delay as extracted from robots.txt, or a given default.N*)r crawl_delayAttributeError)r+rrdelayrMs r,get_crawl_delayzUrlStore.get_crawl_delaysKw' %%c*E   s , 88cp|jjDcgc]}|jc}Scc}w)z2Return all download counts for the hosts in store.)rfrrL)r+rs r,get_all_countszUrlStore.get_all_countss'!%!4!4!67A777s3cVtd|jjDS)z!Find number of all URLs in store.c34K|]}|jywr#)rPrs r,rz,UrlStore.total_url_number..s:q177:r)sumrfrr\s r,total_url_numberzUrlStore.total_url_number s :DLL$7$7$9:::r. thresholdc\tfd|jjDS)z^Find out if the download limit (in seconds) has been reached for one of the websites in store.c3<K|]}|jk\ywr#)rL)rrrs r,rz6UrlStore.download_threshold_reached..sGA177i'Gs)anyrfr)r+rs `r,download_threshold_reachedz#UrlStore.download_threshold_reacheds!G1D1D1FGGGr.clg}|jD]"}|j|j|$|S)z Return a list of all known URLs.)rfrr)r+rrs r, dump_urlszUrlStore.dump_urlss6ll 6F KK,,V4 5 6 r.c||jD]-}tdj|j|d/y)z"Print all unvisited URLs in store. TflushN)rfprintjoinrrs r,rqzUrlStore.print_unvisited_urlss5ll KF $))D44V<=T J Kr.c|jD]c}tdj|j|Dcgc],}||j dt |j .c}deycc}w)z5Print all URLs in store (URL + TAB + visited or not).r TrN)rfrrrr]r^rVrs r, print_urlszUrlStore.print_urls srll F  "&!8"(1668*Bs199~.>?   s1A4filenamecv|`t|d5}tj||dddy#1swYyxYw)zWrite the URL store to disk.wbN)rgopenr5dump)r+routputs r,writezUrlStore.write/s6 J (D ! &V KKf % & & &s/8)FNFTF)F)r!N)NNNr#)NNF)FNTr:)g$@i')d )r3)3r;r<r=r>r?r@rr^r-rr r rTrrrrrintrr rrrrrrrrrrrrrrrrrrrrrrrr r rrrqrr$r1r.r,r`r`xsTI!"& ??3-? ?  ?  ? ?<05 I (,  S% -- . D|)<!37(,15 4&4&5./4&H% 4& % -. 4&  4&n8<$I$'/}$ eCHo $4%)*. :tCy!:T#Y': :  :(" === = sm =  = =.EtCyETE>)49)MtCyM #$131 DcDd3iD#$s) 1S 1d3i11$s)1U38_8M1;C;D;HCHDHctx}.! c 0682222/222 c22l,3,x/H,T, /)B  s  U  5  8S 8;#;HEHdH49K &c&d&r.r`rr!ct|d5}tj|}dddt_|S#1swYxYw)zLoad a URL store from disk.rbN)r!r5loadr rg)rr# url_stores r, load_storer,6s@ h (KK' (fIO ((s <A)9r>rloggingr5rxrrr%r$ ImportErrorr(r' collectionsrrrrenumroperatorr threadingr typingr r r r rrrrurllib.robotparserrcleanrcorerfiltersrrmetarurlutilsrrr getLoggerr;rnrrrDrKrTr`r^r,r1r.r,r;s  GH +(   / .DD   8 $55<\ D 3 3 , ,{&{&| OGHs"CC'C$#C$'C10C1