(#h% d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
ddl m Z m
Z
mZmZmZmZ ddlmZ dd lmZ dd
lmZ dd lmZ ej4 e Z h d Z ej< d
Z ej< d Z ej< d Z! ej< d Z" ej< d Z# ej< d Z$ G d d Z%de&de'fdZ(de e& de%de e& fdZ)de&de%de e& fdZ*de&de%de e& fd Z+d e&de%de e& fd Z,de%d e
e& de e& fd Z- d(d!e&d"e
e& d#e'd$e.de e& f
d%Z/d&e&d"e
e& de e& fd'Z0y))z>
Examining feeds and extracting links for further processing.
N)islice)sleep)ListOptional) check_url clean_url filter_urlsfix_relative_urls get_hostinfo is_valid_url )is_similar_domain) fetch_url) MAX_LINKS) load_html> text/rdftext/rsstext/xml text/atom
text/plain text/rdf+xml text/rss+xml
text/atom+xmlapplication/rdfapplication/rssapplication/xmlapplication/atomapplication/jsonapplication/rdf+xmlapplication/rss+xmlapplication/atom+xmlapplication/feed+jsonapplication/x-atom+xmlapplication/x.atom+xmlz<(feed|rss|\?xml)z(?:\s*)(?:)?(?:\s*)z \bcomments\bzn\.(?:atom|rdf|rss|xml)$|\b(?:atom|rss)\b|\?type=100$|feeds/posts/default/?$|\?feed=(?:atom|rdf|rss|rss2)|feed$c
D e Zd ZdZg dZ d dededededee d df d
Zy) FeedParametersz.Store necessary information to proceed a feed.basedomainextlangrefNbaseurlr) referenceexternal target_langreturnc J || _ || _ || _ || _ || _ y )Nr' )selfr- r) r. r/ r0 s N/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/feeds.py__init__zFeedParameters.__init__M s' ! ! !#. ! )FN) __name__
__module__ __qualname____doc__ __slots__strboolr r5 r6 r4 r& r& I sU 48I %)
" " " "
" c]
"
"r6 r& feed_stringr1 c L t j | ry| dd }d|v xs d|v S )z$Check if the string could be a feed.TNd zt |j |d s%t j d|j |d |j |d d|v sd|v s|j | |S )zGExamine links to determine if they are valid and
lead to a web page)languagefeedr
z'Rejected, diverging domain names: %s %sr
feedburner feedproxy) sortedsetr
r( r r+ r* r r) LOGGERwarningappend)rF rG output_linksitemlinkcheckeds r4 handle_link_listrV d s Ls8}% & d3D6;;7 JJ$&)&--D=v}}gVWj ##GAJ/
T
![D%8 %#&& r6 c ` t | s| j d rh t j | j dg D cg c]& }|j d xs |j d ( }}|D cg c] }|| c}S t j d|j g S d| v rVd t t j | t D D cg c]" }d |vr d
|vrt j | d $ c}S d | v rQt t j | t" j$ t D cg c] }|d j' c}S g S c c}w c c}w # t j
j $ r$ t j d|j Y g S w xY wc c}w c c}w )
z ).0ms r4 z find_links.. s !s zatom+xmlz
rel="self"r
z)rE
startswithjsonloadsgetdecoderJSONDecodeErrorrO debugr) r
LINK_ATTRSfinditerr LINK_HREFsearch
LINK_ELEMENTSreDOTALLstrip)r? rG rS
candidatescrT r^ s r4
find_linksrq s [ ) ! !# &
G !%
; 7 ; ;GR H HHUO5txx~5
$.?a ??
LL4fmm D ; $Z%8%8%Ey Q
%,d*B
T "1 %
; M 2 2; JIV
aDJJL
I= @<<//
G 6
F
G
sA 'E" +E0E" 6E >E E" 'F&9F+
E" "=F#"F#c | s"t j d|j g S t | j | }t || D cg c]' }||j k7 r|j d dkD r|) }}|r+t j dt | t | |S t j d|j |S c c}w )z7Extract and refine links from Atom, RSS and JSON feeds.zEmpty feed: %s/ z!Links found: %s of which %s validzInvalid feed for %s) rO rf r) rq rn rV r, countlen)r? rG
feed_linksrT rR s r4
extract_linksrx s %v}}5 K - - /8J %Z8 6:: $**S/A"5
L /Z #lBS
*FMM: s
,C
htmlstringc N t | }|"t j d|j g S |j d D cg c]P }|j d t v s%t j |j dd r|j dd R }}|sS|j d D cg c]9 }t j |j dd r|j dd ; }}g }t j | D ]i }t |j | }t | }|s'||j k7 s7t | sCt j | rY|j! | k t j dt# | t# | |S c c}w c c}w )zxParse the HTML and try to extract feed URLs from the home page.
Adapted from http://www.aaronsw.com/2002/feedfinder/zInvalid HTML/Feed page: %sz //link[@rel="alternate"][@href]typehref z
//a[@href]z%Feed URLs found: %s of which %s valid)r rO rf r( xpathrc
FEED_TYPESLINK_VALIDATION_RErj dictfromkeysr
r r, r BLACKLISTrQ rv )ry rG treerT feed_urls output_urlss r4 determine_feedr sv Z D | 16;;?
JJ @A 88F z ) $ $TXXfb%9 :
I
<0
!((&")=>
HHVR
K
i( % d3
" T" $$T* t $ % LL/Y [AQ A
s
AF />F" urlfilterc | j rlt d| j d| j d }|rDt || }t || }t
j
dt | | j |S g S )z2Alternative way to gather feed links: Google News.z*https://news.google.com/rss/search?q=site:z&hl=z&scoring=n&num=100z!%s Google news links found for %s)r+ r r) rx r rO rf rv )rG r
downloadedrw s r4 probe_gnewsr ss
{{ 8 tFKK=Xj k
&z6:J$Z;J LL3S_fmm
Ir6 rZ r0 r/
sleep_timec ~ t | \ }}|t j d| g S t ||| || }d}t | }|t || } | sVt
|| D ]+ }
t |
} | s| j t | | - t | t | dz kD r| }| r.t | | } t j dt | | | S t j d| nAt j d| | j d |k7 rt | t || S t || S )a Try to find feed URLs.
Args:
url: Webpage or feed URL as string.
Triggers URL-based filter if the webpage isn't a homepage.
target_lang: Define a language to filter URLs based on heuristics
(two-letter string, ISO 639-1 format).
external: Similar hosts only or external URLs
(boolean, defaults to False).
sleep_time: Wait between requests on the same website.
Returns:
The extracted links as a list (sorted list of unique links).
NzInvalid URL: %srt z%s feed links found for %sz No usable feed links found: %sz Could not download web page: %srs )r rO rP r& r rx r extendrv r rf errorrn r try_homepager ) rZ r0 r/ r r) r- rG r r rw rJ r? s r4 find_feed_urlsr s/ * #3'OFG
~(#.
GVS(K
HFI3 J ":v6
&z6:
J 'o %%mK&HI
J
3x#g,** $Z;J LL5s: O 5s; 6< 99S>W $ * 5 5 vy ))r6 r- c F t j d| t | | S )zhShift into reverse and try the homepage instead of the particular feed
page that was given as input.z&Probing homepage for feeds instead: %s)rO rf r )r- r0 s r4 r r 4 s LL97C '; //r6 )NFg @)1r: ra loggingrl itertoolsr timer typingr r courlanr r r r
r r
deduplicationr downloadsr settingsr utilsr getLoggerr7 rO r compilerB rg ri rk r r r& r< r= rE rV rq rx r r floatr r r> r6 r4 r s ! - 8 $
, rzz./
RZZ.
/
BJJ '
A
BJJ ' RZZ
" "&73 74 7tCy . T#Y 6%C % %DI %Ps N tCy 0)s )N )tCy )X
8C=
T#Y
$ "& 6* 6*# 6* 6* 6*
#Y 6*r0# 0HSM 0d3i 0r6