(#hd/dZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z mZmZmZmZmZ ddlZddlmZmZd d lmZmZd d lmZmZmZd d lm Z d d l!m"Z"m#Z#m$Z$ejJe&Z'e ddZ(dZ)dZ*dZ+GddZ,de-de-de ee-ee-ffdZ.de-de ee-ee-ee-ffdZ/de-de-dee fdZ0de-dee fdZ1de-dee-de2fd Z3d!ee-de2fd"Z4 d5de-d#e,d$ee-ddfd%Z5d&eed#e,ddfd'Z6 d6d(e-d)ee-d*ee d!eee-d+eee-d,ee-de,fd-Z7 d7d#e,d.e2de,fd/Z8e*e+ddde ddfde-d0e9d1e9d!eee-d2eee-d)ee-d3ed*ee d,ee-de ee-ee-ffd4Z:y#e$rYSwxYw)8zC Functions dedicated to website navigation and crawling/spidering. N) ConfigParser)sleep)ListOptionalTuple)RobotFileParser)UrlStore extract_linksfix_relative_urls get_base_urlis_navigation_pageis_not_crawlable)XPathtostring)baselineprune_unwanted_nodes)Responsefetch_response fetch_url)DEFAULT_CONFIG) LANGID_FLAG decode_file load_htmlF) compressedstrictz /robots.txt ic eZdZdZgdZ ddedeedeedeeddf d Zdedefd Z dedefd Z d e ddfd Z dee ede efdZdedefdZy)CrawlParametersz6Store necessary information to manage a focused crawl.) startbaselangrulesrefi known_numis_on prune_xpathNr r"r#r(returnc||_|j||_|j||_||_|xst |j|_d|_d|_ d|_ ||_ y)NrT) r _get_base_urlr!_get_referencer$r" get_rulesr#r%r&r'r()selfr r"r#r(s O/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/spider.py__init__zCrawlParameters.__init__0sh  ++E2 ++E2#' 050M4999M  *5c<t|}|std||S)z#Set reference domain for the crawl.zcannot start crawl: )r ValueError)r.r r!s r/r+zCrawlParameters._get_base_urlAs' '3E7;< < r1cX|jddk\r|jdddS|S)zDetermine the reference URL./rr)countrsplit)r.r s r/r,zCrawlParameters._get_referenceHs-*/++c*:a*?u||C#A&JUJr1 url_storect|j|j|_t |j |j|_y)z*Adjust crawl data based on URL store info.N)boolfind_unvisited_urlsr!r'lenfind_known_urlsr&)r.r9s r/update_metadatazCrawlParameters.update_metadataLs8)77 BC Y66tyyABr1todocr|sgS|Dcgc]#}||jk7s|j|vs"|%c}Scc}w)z.Prepare the todo list, excluding invalid URLs.)r r$)r.r@us r/ filter_listzCrawlParameters.filter_listQs2IEa1 ?txx1}EEEs 444linkc|j xs|jjd|xr|j|vxr t| S)z9Run checks: robots.txt rules, URL type and crawl breadth.*)r# can_fetchr$r)r.rDs r/ is_valid_linkzCrawlParameters.is_valid_linkWsI^ >tzz33C> +D  +$T** r1NNN)__name__ __module__ __qualname____doc__ __slots__strrrr0r+r,r r?rrCr;rHr1r/rr,s<cI #+/%) 66sm6( 6 c] 6  6"33KCKCKCCdC Fc 3FS F  # $ r1r htmlstringhomepager)cd|vrd|vr||fSt|}|||fS|jd}|r|dnd}|rd|vrtjd|||fS|j ddj j jd d}|jd st|}t||}t|}|tjd |y tjd |||fS)z:Check if there could be a redirection by meta-refresh tag.z "refresh"z "REFRESH"z@.//meta[@http-equiv="refresh" or @http-equiv="REFRESH"]/@contentr;zno redirect found: %srzurl=httpzfailed redirect: %s)NNzsuccessful redirect: %s) rxpathlogginginfosplitstriplowerreplace startswithr r rwarning)rQrR html_treeresultsresulturl2base_url newhtmlstrings r/refresh_detectionrf`s*$J)F8##*%I8##ooJG#WQZF S& ,h78## << Q  % % ' - - / 7 7 CD ??6 "% 40dOM-t4 LL*D1 $ r1cHt|d}|r |jsy|j|dfvr,tjd|j|j}t |j}t ||\}}|ytjd|||t|fS)zBCheck if the homepage is redirected and return appropriate values.FdecoderIr5zfollowed homepage redirect: %szfetching homepage OK: %s) rdataurlrXrYrrfdebugr )rRresponserQnew_htmlstring new_homepages r/probe_alternative_homepagerpshu5H 8==||Hc?* 5x||D<<X]]+J$5Z#J NL MM,l; <l)C CCr1 robots_urlrjct}|j| |j|j|S#t$r }t j d|Yd}~yd}~wwxYw)zEParse a robots.txt file with the standard library urllib.robotparser.zcannot read robots.txt: %sN)rset_urlparse splitlines ExceptionLOGGERerror)rqrjr#excs r/ parse_robotsrzs[  E MM* DOO%& L  137s> A'A""A'rdcJ|tz}t|}|r t||SdS)z?Attempt to fetch and parse robots.txt file for a given website.N)ROBOTS_TXT_URLrrz)rdrqrjs r/r-r-s*N*J Z D-1< D );t;r1languagec|r=|r;tr5t|\}}}tj|\}}t ||k(Sy)zRun a baseline extraction and use a language detector to check if the content matches the target language. Return True if language checks are bypassed.T)rr py3langidclassifyr;)rQr}_textrbs r/is_target_languagersBh;j) 4&&t, Fh&'' r1r@c&td|DS)z6Probe if there are still navigation URLs in the queue.c32K|]}t|yw)N)r ).0rks r/ z&is_still_navigation..s73!#&7s)any)r@s r/is_still_navigationrs 7$7 77r1paramsrkc ht||jsy|r|jt|jtr|jg|_t |}|Ft ||jDcgc] }t|c}}t|j}gg}}t||xs |jd|jddD]B}|j|st|r|j|2|j|Dtj!||ycc}w)zExamine the HTML code and process the retrieved internal links. Extract and filter new internal links after an optional language check. Store the links in todo-list while prioritizing the navigation ones.NFT) pagecontentrk external_boolr}with_navr)urls appendleft)rr"r( isinstancerOrrrrrir r!rHr append URL_STOREadd_urls)rQrrktreexlinkslinks_priorityrDs r/ process_linksrs j&++ 6f((4 f((# ."("4"4!5F $  'ASAS.TAuQx.TUD!$..0J>E  6;;  ##D)  d #  ! !$ ' LL  En='/Us3D/ rmc| |jsytj|jgdt t |j||j y)z2Convert urllib3 response object and extract links.NT)visited)rjrrrkrrr!)rmrs r/process_responsersG x}}  ~t4+hmm,ffkkBr1r r"r#knownr(ct||||}tj|xsgdtj|j|tj |j |j |s1tj|jgdt|d}|S|jt|S)zInitialize crawl by setting variables, copying values to the URL store and retrieving the initial page if the crawl starts.T)rr)rF)initial) rrrrC store_rulesr!r#r crawl_pager?)r r"r#r@rr(rs r/ init_crawlrsUD% =FEKR6 F..t45 &++v||4 >FD1 M y) Mr1rctj|j}|s6d|_t tj |j|_|S|xjdz c_|r:t|\}}}|rB|r@|r>tj|gt|||nt|d}t|||jt|S)z6Examine a webpage, extract navigation links and links.Fr)rkrh)rget_urlr!r'r=r>r&r%rprrrrr?)rrrkrQrR new_base_urlrms r/rrs   FKK (C  y88EF  HHMH-G-L* Hl (|   z * *f# 6!#e46* 9% Mr1 max_seen_urlsmax_known_urls known_linksconfigc Nt||||||} tj| j|j dd} | j r_| j |krP| j|krAt| } t| | j r| j |kr| j|krAttjtj| j}ttjtj| j}||fS)aBasic crawler targeting pages of interest within a website. Args: homepage: URL of the page to first page to fetch, preferably the homepage of a website. max_seen_urls: maximum number of pages to visit, stop iterations at this number or at the exhaustion of pages on the website, whichever comes first. max_known_urls: stop if the total number of pages "known" exceeds this number. todo: provide a previously generated list of pages to visit / crawl frontier. known_links: provide a list of previously known pages. lang: try to target links according to language heuristics. config: use a different configuration (configparser format). rules: provide politeness rules (urllib.robotparser.RobotFileParser() format). prune_xpath: remove unwanted elements from the HTML pages using XPath. Returns: List of pages to visit, deque format, possibly empty if there are no further pages to visit. Set of known links. DEFAULT SLEEP_TIME)default)rrget_crawl_delayr!getfloatr'r%r&rrlistdictfromkeysr<r>) rRrrr@rr"rr#r(r sleep_times r/focused_crawlerr2s:$t[+ NF** V__Y E+J  M1f6F6F6WF# j  M1f6F6F6W  i;;FKKHI JDt}}Y%>%>v{{%KLMK  r1)rT)NNNNN)F);rMrX configparserrtimertypingrrrurllib.robotparserrcourlanr r r r r rr ImportError lxml.etreerrcorerr downloadsrrrsettingsrutilsrrr getLoggerrJrwrr| MAX_SEEN_URLSMAX_KNOWN_URLSrrOrfrprzr-r;rrrrrrintrrPr1r/rs %((. '0::$66   8 $ e 4  1 1 h##"# 8C=(3- '(#LDD 8C=(3-# 67D2 S  0I << 9<3(3-D8d3i8D8#>#> #> ##> #>L Cx  C  C  C '+ $!%!%   3- O $ 49   DI   # :  B'( $'+)'+!%.... 49  . $s)$ . 3- . . O $.#. 49d3i  .u  sFF  F