"#h81ddlmZddlmZmZmZddlZddlZddlZddlZddl m Z m Z m Z m Z ddlddlmZmZdZej&d fd Zej&fd Zej&fd Zd Zedk(reyy))absolute_import)divisionprint_functionunicode_literalsN)escapePY3URLErrorurllib)*) get_stoplist get_stoplistsc dtjjtjjtjdt t ttttttd zS)NaUsage: %(progname)s -s STOPLIST [OPTIONS] [HTML_FILE] Convert HTML to plain text and remove boilerplate. -o OUTPUT_FILE if not specified, output is written to stdout --encoding=... default character encoding to be used if not specified in the HTML meta tags (default: %(default_encoding)s) --enc-force force specified encoding, ignore HTML meta tags --enc-errors=... errors handling for character encoding conversion: strict: fail on error ignore: ignore characters which can't be converted replace: replace characters which can't be converted with U+FFFD unicode replacement characters (default: %(default_enc_errors)s) --format=... output format; possible values: default: one paragraph per line, each preceded with

or (headings) boilerplate: same as default, except for boilerplate paragraphs are included, too, preceded with detailed: one paragraph per line, each preceded with

tag containing detailed information about classification as attributes krdwrd: KrdWrd compatible format --no-headings disable special handling of headings --list-stoplists print a list of inbuilt stoplists and exit -V, --version print version information and exit -h, --help display this help and exit If no HTML_FILE specified, input is read from stdin. STOPLIST must be one of the following: - one of the inbuilt stoplists; see: %(progname)s --list-stoplists - path to a file with the most frequent words for given language, one per line, in UTF-8 encoding - None - this activates a language-independent mode Advanced options: --length-low=INT (default %(length_low)i) --length-high=INT (default %(length_high)i) --stopwords-low=FLOAT (default %(stopwords_low)f) --stopwords-high=FLOAT (default %(stopwords_high)f) --max-link-density=FLOAT (default %(max_link_density)f) --max-heading-distance=INT (default %(max_heading_distance)i) r) progname length_low length_high stopwords_lowstopwords_highmax_link_densitymax_heading_distancedefault_encodingdefault_enc_errors) ospathbasenamesysargvLENGTH_LOW_DEFAULTLENGTH_HIGH_DEFAULTSTOPWORDS_LOW_DEFAULTSTOPWORDS_HIGH_DEFAULTMAX_LINK_DENSITY_DEFAULTMAX_HEADING_DISTANCE_DEFAULTDEFAULT_ENCODINGDEFAULT_ENC_ERRORSM/var/www/html/sandstorm/venv/lib/python3.12/site-packages/justext/__main__.pyusager)s[, Z  !1!1#((1+!>?$&*,08(, Y6 6r'Tc |D]P}|jdk(r|jrd}nd}n|r'd}td|dt|jd| Ry ) z Outputs the paragraphs as: text of the first paragraph text of the second paragraph ... where is

, or which indicates standard paragraph, heading or boilerplate respecitvely. goodhpb FquotefileN) class_typeheadingprintrtext) paragraphsfpno_boilerplate paragraphtags r(output_defaultr=Js[ O   6 )    C 3y~~U CD2N Or'c |D]_}d|j|jt|j|jt |j dfz}t||ay)z Same as output_default, but only

tags are used and the following attributes are added: class, cfclass and heading. z6

%sFr0r2N)r4cf_classintr5xpathrr7r6)r8r9r;outputs r(output_detailedrCasd   I      !! " OO 9>> / M   f2r'c|D]U}|jdvr|jrd}nd}nd}|jD]"}td||j fz|$Wy)a Outputs the paragraphs in a KrdWrd compatible format: classfirst text node classsecond text node ... where class is 1, 2 or 3 which means boilerplate, undecided or good respectively. Headings are output as undecided. )r+neargoodrz%i %sr2N)r4r5 text_nodesr6strip)r8r9r;cls text_nodes r( output_krdwrdrLqsn @   #7 7  C"-- @I (c9??#455B ? @ @r'cddl}ddlm} |jtjdddgd\}}tjdd}tj}tr!|tjj}n|tj}d}d } d } t } t"} t$} t&}t(}t*}d}t,}d }t.} D]\}}|d vr(t ttjd|d vrUt t0j2j5tjdd |dtjd|dk(rAt dj7t9t;tjd|dk(r tj<|dd}|dk(r|jCdk(r tE}t0j2jG|rQ tj<|dd}tE|Dcgc]}|jIc}}|jK|t;vr tO|}tQjRd|r1tAd|ddj7t9t;tAd|z|dk(r |}djU| |d!k(rd"}|d#k(r2|jCd$vr|jC}?tAd%|z|d&k(r|d'vr|} ZtAd(|z|d)k(rd"} q|d*k(r tY|} |d.k(r tY|} |d/k(r t]|} |d1k(r t]|}|d2k(r t]|}|d3k(s tY|}|r|}| tAd4|sd}d} rB tQjRd5|drt_j`|d}nt=|dd}|je}|tjur|jKt||| | | |||| ||| }| d k(r tg||y| d6k(rtg||d 7y| d8k(r ti||y| d9k(r tk||ytmd:| z#|j $r^}t |tjt ttjtjdYd}~d}~wwxYw#t>$r}tAd|d|d}~wwxYwcc}w#t>$r}tAd|d|d}~wtL$r}tAd|zd}~wwxYw#tV$rtAd |zwxYw#tZ$rtAd+|d,|d-wxYw#tZ$rtAd+|d,|d-wxYw#tZ$rtAd+|d,|d0wxYw#tZ$rtAd+|d,|d0wxYw#tZ$rtAd+|d,|d0wxYw#tZ$rtAd+|d,|d-wxYw#t>tbf$r}tAd|dd|d}~wwxYw#tn$rm}t t0j2j5tjdd;|tjtjdYd}~yd}~wwxYw)z--list-stoplists z-owz Can't open z for writing: z-snonerz for reading: zLUnicode decoding error when reading the stoplist (probably not in UTF-8): %sz^\w*$zUnknown stoplist: z Available stoplists: zFile not found: %sz --encodingzUknown character encoding: %sz --enc-forceTz --enc-errors)strictignorereplacezInvalid --enc-errors value: %sz--format)rS boilerplatedetailedkrdwrdzUknown output format: %sz --no-headingsz --length-lowzInvalid value for z: 'z'. Integer expected.z --length-highz--stopwords-lowz'. Float expected.z--stopwords-highz--max-link-densityz--max-heading-distancezNo stoplist specified.z [^:/]+://r\)r:r]r^zUnknown format: %sz: )8getoptjustextrNrr GetoptErrorr6stderrr)exitcodecslookupstdinr stdoutbufferrrr r!r"r#r$r%rrrjoinsortedropenIOErrorJustextInvalidOptionslowersetisfilerIcloseUnicodeDecodeErrorr rematchencode LookupErrorr@ ValueErrorfloatr urlopenr readr=rCrLAssertionError JustextError)r_VERSIONoptsargse stream_writerfp_infp_outstoplistformat no_headingsrrrrrrencodingrforce_default_encoding enc_errorsoa fp_stoplistl html_textr8s r(mainrs. ]]388AB<<78 dMM&)"-M IIE szz001szz*H FK#J%K)M+N/7H'"#JQd RDAq$$eg ''GG$$SXXa[17<= ((dii} 789 dB#[[C8Fd779&"uHww~~a( P*0++af*EK'*{+K!AGGI+K'LH'--/mo-#/?88Ha0#8$%tyy 1H'I!K#LL #88Lq8P"QQl"U'($II./m#)-&n$779 ??!"J/0PST0TUUjHHF/0JQ0NOOo%" n$R!$QJo%R"%a&K''P$)!HM((P%*1XN**P',Qx$..R+.q6(Cd RL "'H  '(@A ANM  D88L$q'2"NN473E a#.E JJL  ! KKMY*k >+;=Q #3ZA  Y  :v . } $ :ve D z ! J / x  *f -!!5!>? ?M    acjj! egCJJ'  PB/;>Z YZ Z/ZZZ \A#[<<\__main__) __future__rrrrrdrrsr_compatrr r r coreutilsr rr)rgr=rCrLr__name__r&r'r(rsw'AA 22.7t#&**TO.$'::  "%@.tn zFr'