(#hq& d Z ddlZddlZddlmZ ddlmZ ddlmZm Z m
Z
m Z m Z ddl
mZmZmZmZmZmZ ddlmZ dd lmZmZ dd
lmZmZ ej8 e Z ej> d Z ej> d ejB Z" ej> d
Z# ej> d Z$ ej> d Z% ej> d Z& ej> d Z' ej> d Z( ej> d Z)g dZ* G d d Z+dddefde,de e, de-d e.d e/d e e, f d Z0de,d e e, d e-fd!Z1d"e,d e e, fd#Z2d$e e, d"e,d e e, fd%Z3y)&z#
Deriving link info from sitemaps.
N)islice)sleep)CallableListSetOptionalPattern) clean_urlextract_domain filter_urlsfix_relative_urls get_hostinfo lang_filter )is_similar_domain) fetch_url is_live_page) MAX_LINKSMAX_SITEMAPS_SEENz.(?:)?zzhref=["\'](.+?)["\']zg(?:blogger|blogpost|ghost|hubspot|livejournal|medium|typepad|squarespace|tumblr|weebly|wix|wordpress)\.z ^.{0,5}<\?xml| !1!128b
d&&'r* linkc L || j k( ryt | j | }t || j xs d}|rt || j syt
|d }|t j d| y| j sMt j | s8t | j | s"t j d| j | yt j | r | j j# | y| j$ j# | y)z^Examine a link and determine if it's valid and if it leads to
a sitemap or a web page.Nr% T)fastzcouldn't extract domain: %sz-link discarded, diverging domain names: %s %s)r r
r r
r r r r, errorr WHITELISTED_PLATFORMSsearchr r warningDETECT_SITEMAP_LINKr appendr! )r' r1 newdomains r( handle_linkzSitemapObject.handle_linkX s 4## # 5t / /06B;tT-=-=> "4d3 LL6 =
)00;%dkk9= NN?i
% %d + $ $T * II T "r* regexindexhandlerc fdt |j | j t D D ]
} || t j dt
| j t
| j | j y)zJExtract links from the content using pre-defined regex, index and handler.c 3 ( K | ] }| y w)N ).0mr= s r( z.SitemapObject.extract_links..~ s
AeH
s z%%s sitemaps and %s links found for %sN)
r finditerr r r, r- lenr r! r )r' r< r= r> matchs ` r(
extract_linkszSitemapObject.extract_linksz sl
$U^^DLL%A9 M
E
EN 3 !! " N
r* c d j vryt j d j dt j dt
ddf fd } j
t d| y) z7Extract links corresponding to a given target language.z hreflang=Nzhreflang=[\"'](z.*?|x-default)[\"']attrsr# c j | r-t j | }|rj |d yyy)z!Examine language code attributes.r N)r6 HREFLANG_REGEXr; )rJ
lang_match
lang_regexr' s r( handle_lang_linkzASitemapObject.extract_sitemap_langlinks..handle_lang_link s@ ' + 2 25 9
$$Z]3 (r* r )r recompiler DOTALLstrrH XHTML_REGEX)r' rO rN s ` @r( extract_sitemap_langlinksz'SitemapObject.extract_sitemap_langlinks sb dll * ZZ t / / 00C Dbii
4C 4D 4
;+;r s 9 9 - . 2
8 $
RZZI
J
bjj+RYY7 34"
n => bjj!?@ rzz./ bjj ' BJJz* ~% ~%F "&) C C# C C C
C
#Y
CLc Xc] t *7# 7$s) 7x} s tCy r*