(#h^K r d Z ddlZddlZddlmZ ddlmZ ddlmZm Z m
Z
m Z ddl m
Z
ddlmZmZ dd lmZmZ ej( e Z ej. d
Zd ed efd
Zh dZddhZ h dZ h dZ h dZ h dZ ej. dejB ej. dejB ej. dejB ej. dejB ej. dejB ej. dejB dZ"d d hZ#d d hZ$d ed e%fd!Z& G d" d# Z' G d$ d% Z( ej. d&ejB ej. d'ejB d(Z) ej. d)ejB Z*d*ed e+fd+Z,i fd ed,ed e+fd-Z-y).a Minimalistic fork of readability-lxml code
This is a python port of a ruby port of arc90's readability project
http://lab.arc90.com/experiments/readability/
Given a html document, it pulls out the main body text and cleans it up.
Ruby port by starrhorne and iterationlabs
Python port by gfxmonk
For list of contributors see
https://github.com/timbertson/python-readability
https://github.com/buriy/python-readability
License of forked code: Apache-2.0.
N)sqrt)
attrgetter)AnyDictOptionalSet)tostring) HtmlElementfragment_fromstring ) load_htmltrimz\.( |$)stringreturnc & t | t d S )Nxml)encodingmethod)r str)r s Y/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/readability_lxml.py _tostringr & s FS 77 >
apdloluldivimgpretable
blockquoter article> tdr r" > ddr dtlir r formasideaddress>
h1h2h3h4h5h6thnavfooterheader> r r r' r embedinputzcombx|comment|community|disqus|extra|foot|header|menu|remark|rss|shoutbox|sidebar|sponsor|ad-break|agegate|pagination|pager|popup|tweet|twitterz#and|article|body|column|main|shadowzKarticle|body|content|entry|hentry|main|page|pagination|post|text|blog|storyzbutton|combx|comment|com-|contact|figure|foot|footer|footnote|form|input|masthead|media|meta|outbrain|promo|related|scroll|shoutbox|sidebar|sponsor|shopping|tags|tool|widgetz.<(?:a|blockquote|dl|div|img|ol|p|pre|table|ul)z+https?:\/\/(?:www\.)?(?:youtube|vimeo)\.com)unlikelyCandidatesReokMaybeItsACandidateRe
positiveRe
negativeRedivToPElementsRevideoRebodyhtmlr r elemc F t t | j S )z7Return the length of the element with all its contents.)lenr text_content)r? s r text_lengthrC W s tD%%'( ))r c , e Zd ZdZddgZdededdfdZy) Candidatez,Defines a class to score candidate elements.scorer? r Nc || _ || _ y N)rF r? )selfrF r? s r __init__zCandidate.__init__a s !
!% r )__name__
__module__ __qualname____doc__ __slots__floatr
rJ r r rE rE \ s) 2&!I&e &; &4 &r rE c e Zd ZdZg dZddedededdfdZdefd Z d
e
ee f d e defd Z d
e
ee f de
e fd
ZdedefdZde
ee f fdZdedefdZdede fdZddZddZded
e
ee f defdZy)Documentz,Class to build a etree document out of html.docmin_text_length retry_lengthrU rV rW r Nc . || _ || _ || _ y)a Generate the document
:param doc: string of the html content.
:param min_text_length: Set to a higher value for more precise detection of longer texts.
:param retry_length: Set to a lower value for better detection of very small texts.
The Document class is not re-enterable.
It is designed to create a new Document() for each HTML file to process it.
API method:
.summary() -- cleaned up content
NrT )rI rU rV rW s r rJ zDocument.__init__k s . (r c @ | j j dd D ] }|j d} |r| j | j | j }| j
| }|r| j || }n\|du rd}t j d ut j d | j j d }||n | j }| j || }t |xs d }|r|| j k rd}|S ) z
Given a HTML file, extracts the text of the article.
Warning: It mutates internal DOM representation of the HTML document,
so it is better to call other API methods before this one.
scriptstyleTFz5Ended up stripping too much - going for a safer parsez=Ruthless and lenient parsing did not work. Returning raw htmlr= )rU iter drop_treeremove_unlikely_candidates&transform_misused_divs_into_paragraphsscore_paragraphsselect_best_candidate get_articleLOGGERdebugfindsanitizerA rW ) rI r? ruthless
candidatesbest_candidater# r= cleaned_articlearticle_lengths r summaryzDocument.summary| s HHMM(G4 D NN //1 7 7 9 ..0J ! 7 7
CN **:~Ft# $HLLO S xx}}V,"&"2$ "mmGZ @O !6B 7NNT->-> > " "r ri rj c t d|j dz }t d }|j j }| t | n |j g}|D ] }d}||j k( s||v r|| j |k\ rd}nf|j dk( rW| j | } |j xs d}
t |
} | dkD r| d k s | dk r | d
k( rt j |
rd}|s|j | |S ) N
皙?zFTr r\ P g ?r )
maxrF r r? getparentlisttagget_link_densitytextrA DOT_SPACEsearchappend) rI ri rj sibling_score_thresholdoutputparentsiblingssiblingrz link_density node_content node_lengths r rc zDocument.get_article s #&b.*>*>*D"E $X. $$..0#)#54 |sy t |j t d d }t j t
j r?|d d D ]7 }t j d|j j |j 9 t t | S )NrF T)keyreverse z Top 5: %s %s)
sortedvaluesr rd isEnabledForloggingDEBUGre r? ru rF nextr] )rI ri sorted_candidates candidates r rb z Document.select_best_candidate s " Z%8$
w}} - .r 2
R ^Y^^-?-? Q
RD*+,,r r? c l t | xs d}t d |j d D }||z S )Nr c 3 2 K | ] }t | y wrH )rC ).0links r z,Document.get_link_density.. s M+d+Ms z.//a)rC sumfindall)rI r? total_length link_lengths r rv zDocument.get_link_density s5 "4(-A M V8LMM \))r c i }| j j ddd D ] }|j }||j }t |j }t | }|| j k rZ||fD ] }|||vs | j | ||< dt |j d z t |dz d z }|| xj |z
c_
||| xj |dz z
c_
|j D ], \ }} | xj d| j | z
z c_
. |S ) Nr r r$ r ,d )
rU r] rs r rB rA rV
score_nodesplitminrF itemsrv )
rI ri r? parent_nodegrand_parent_node elem_text
elem_text_lennoderF r s
r ra zDocument.score_paragraphs sY
HHMM#ud3 AD ..*K" + 5 5 7 T . . 01I NM t 3 33$&78
=#J(>'+t' > >O ? r c d}t d |j d |j d f D ]= }t d j | r|dz }t d j | s9|dz
}? |S )Nr classidr: r9 )filtergetREGEXESry )rI r? weight attributes r class_weightzDocument.class_weight sr txx'8$((4.&IJ I|$++I6" |$++I6"
r c | j | }t |j }|j }|t v r|dz
}n)|t
v r|dz
}n|t v r|dz }n
|t v r|dz }t || S )Nr r ) r r ru lower
DIV_SCORES BLOCK_SCORESBAD_ELEM_SCORESSTRUCTURE_SCORESrE )rI r? rF ru names r r zDocument.score_node s| !!$'$((myy{ : QJE
\
! QJE
_
$ QJE
%
% QJE%%r c
| j j d D ] }dj t d |j d |j d f }t | dk rM|j t vs`t d j | syt d j | r|j y )Nz.//* r r r r7 r8 ) rU r joinr r rA ru
FRAME_TAGSr ry r^ )rI r? attrss r r_ z#Document.remove_unlikely_candidates s HH$$V,
!DHHVD488G+r T)r r br)rU r r ry r mapr rt ru rw stripr insertr enumeratetailr^ )rI r? p_elemposchilds r r` z/Document.transform_misused_divs_into_paragraphs) s0 HH$$X. D -.55ItDz23 HH$$X. &DyyTYY__ .,V4)-D& TY Av& $Yt_d C
&
U::%**"2"2"4 0 8F.3jj$+FKKKa099$OO%
&
&r r c X |j dddddd D ]; }| j | dk s| j | dkD s,|j = |j d d
D ] }|j |j d D ]M }d |j v r-t
d
j
|j d rd|_ >|j O t }t |j d D ] }||v r | j | }||v r|| j nd}||z dk r3t j d|j || |j j|j j! d dk sd}t" D ci c] } | t% |j' d| " }
} |
dxx dz cc< |
dxx t% |j' d z cc< t) | } | j | } |j+ }
|
|
|v r||
j nd}|
d r |
d d|
d d z z kD r d |
d d }nx|
d |
d kD r|j t, vrd }nW|
d |
d d z kD rd!}nE| | j. k r|
d dk( rd"| d#}n&| | j. k r|
d d$kD rd"| d%}n|d&k r| d'kD r
d(| d)d*| }n|d&k\ r| d+kD r
d(| d)d*| }n|
d, dk( r| d-k s|
d, dkD rd.}n| sd/}g }|j1 D ]" }t) | }|s|j3 | n t% | dz }|j1 d0 D ]1 }t) | }|s|j3 | t% | |k\ s1 n |r6t5 | d1kD r(d2}|j7 |j d3d4d5d6 nd2}|s|j t j d7||j |xs d8 || _ t; | j8 S c c} w )9Nr+ r, r- r. r/ r0 r g Q ?r( textareaiframesrcr< VIDEOz6//table|//ul|//div|//aside|//header|//footer|//sectionz+Removed %s with score %6.3f and weight %-3sr ro Tz.//r' r r6 z.//input[@type="hidden"]r r r g?ztoo many images ()zmore
s than
sr z less than 3x
s than sztoo short content length z without a single imager z and too many imagesr rp ztoo many links z.3fz for its weight g ?r5 K z<