#hG .dZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z m Z mZmZddlmZej$dZej$dZej$d Zej$d ej,Z d!d ed ed e e de eefffdZ d"d edeeed ed eefdZd ed efdZd ed efdZd ed e eeffdZd ed e eeeffdZded ed efdZ deedeed eefdZ!d#d ededed efdZ"dedeed efd Z#y)$zD Functions related to URL manipulation and extraction of URL parts. N)unescape)AnyListOptionalSetTupleUnion)urljoinurlsplit urlunsplit SplitResult)get_tldz{(?:(?:f|ht)tp)s?://(?:[^/?#]{,63}\.)?([^/?#.]{4,63}\.[^/?#]{2,63}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[0-9a-f:]{16,})(?:/|$)z (?<=\D):\d+z ^www[0-9]*\.z(?:feed(?:burner|proxy))urlfastreturnNNc^|rt|tsy|r\tj|}|rEtj d|dj dd}|j dd}|r||fSt|dd }|y|jtj d|jfS) z0Cached function to extract top-level domain infor@.rT) as_object fail_silently) isinstancestr DOMAIN_REGEXmatchSTRIP_PORT_REGEXsubsplitrdomainCLEAN_FLD_REGEXfld)rr domain_match full_domain clean_matchtldinfos M/var/www/html/sandstorm/venv/lib/python3.12/site-packages/courlan/urlutils.py get_tldinfor*s jc* #))#. *..r<?3H3H3Mb3QRK%++C03K"K//cT>G >>?..r7;;? ?? blacklistcV| t}t||\}}|r ||vr||vr|SdS)z;Extract domain name information using top-level domain infoNr)setr*)rr,rr"r&s r)extract_domainr01sIE %c5FK 62{)7S r+ct|trtt|}|St|tr|}|St dt |)z3Parse a string or use urllib.parse object directly.zwrong input type:)rrr rr TypeErrortype)r parsed_urls r)_parser5AsO#shsm,  C %  +T#Y77r+crt|}|jr|jdz}nd}||jzS)ziStrip URL of some of its parts to get base URL. Accepts strings and urllib.parse ParseResult objects.z://r)r5schemenetloc)rr4r7s r) get_base_urlr9Ls<J""U* J%% %%r+ct|}t|}tdd|j|j|j g}|dk(rd}|r|st d|||fS)zvDecompose URL in two parts: protocol + host/domain and path. Accepts strings and urllib.parse ParseResult objects.r/zincomplete URL: )r5r9r pathqueryfragment ValueError)rr4hostnamepathvals r)get_host_and_pathrBWspJJ'H R*"2"2J4G4GHG"} 7+C5122 W r+c:t|d}t|}||fS)zXConvenience function returning domain and host info (protocol + host/domain) from a URL.Tr.)r0r9)r domainnamebase_urls r) get_hostinforFgs#$/JC H x r+baseurlc|jdr|St|j}t|}|j|dfvr)|jr|St |j dSt ||S)z8Prepend protocol and host information to relative links.{rhttp)r7) startswithr r8r7r _replacer )rGr base_netloc split_urls r)fix_relative_urlsrOnss ~~c 7#**K I R00   J),,F,;<< 7C  r+ link_list urlfilterc|tt|S|Dcgc] }||vs| }}|s%|Dcgc]}tj|s|}}tt|Scc}wcc}w)zDReturn a list of links corresponding to the given substring pattern.)sortedr/FEED_WHITELIST_REGEXsearch)rPrQl filtered_lists r) filter_urlsrX~soc)n%% )<1Y!^Q$ID9L#)#D9OV,.. S=r+link known_linksc||vry|ddk(r|jdn|dz}||vry|jdrH|jdrd|ddznd|ddz}|ddk(r|jdn|dz}||vs||vryy ) zDCompare the link and its possible variants to the existing URL base.Trr;rJhttpsNF)rstriprK)r`ra slash_test protocol_tests r) is_known_linkris {&*"X_S!$*J[  v!%!9FT"1X wab?Q  R C'   %$  K ':+D r+)F)NF)T)$__doc__rehtmlrtypingrrrrrr urllib.parser r r r tldrcompilerrr#IrTrboolr*r0r5r9rBrFrOrXr_rir+r)rts 99CCrzz 2::n-"**_-!rzz"=rttD!@ @@ 5 eCHo -.@.BG   !#c(+ :>  c]   &c&c& 3 5c?   c eHSM3$67  !s ! ! ! &49 &# &49 &SS#c(tr+