(#he(dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZddlmZmZej.j0d k7r/eej.d rej.j5d ej6j0d k7r/eej6d rej6j5d d e de fdZde de fdZde de fdZddZde ddfdZ e!dk(reyy)z. Implementing a basic command-line interface. N)version)python_version)Any) cli_crawler cli_discoveryexaminefile_processing_pipelineload_blacklistload_input_dictprobe_homepageurl_processing_pipeline write_result)PARALLEL_CORESSUPPORTED_FMT_CLIzUTF-8 reconfigurezutf-8)encodingparserreturnc |jdd}|j}|jdd}|jdd}|j}|jdd}|jd d }|j}|jd d d t|jddt|jdddt|jddtt |jdddt|jddd|jddd t|jd!d"t|jd#d$d|jd%d&d'd(d)*|jd+d,d'd(d)*|jd-d.d'd(d)*|jd/d0d'd(d)*|jd1d2d'd(d)*|jd3d4d|jd5d6d7t8|jd9d:d;d|jdd?d|jd@dAd|jdBdCdD|jdEdFdD|jdGdHd|jdIdJd|jdKdLt|jdMdNd|jdOdPt|jdQdRd|jdSdTd|jdUdVt dWX|jdYdZd|jd[d\d|jd]d^d|jd_d`d|jdadbd|jdcddd|jdedfd|jdgdhdidjdkl|jdmdndodptdqdrts|S)tz,Add argument groups and arguments to parser.Inputz%URLs, files or directories to processOutputz+Determines if and how files will be written NavigationzLink discovery and web crawling Extractionz-Customization of text and metadata processingFormatzSelection of the output formatz-iz --input-filez'name of input file for batch processing)helptypez --input-dirz5read files from a specified directory (relative path)z-uz--URLzcustom URL downloadz --parallelzAspecify a number of cores/threads for downloads and/or processing)rrdefaultz-bz --blacklistz:file containing unwanted URLs to discard during processingz--listz/display a list of URLs without downloading them store_true)ractionz-oz --output-dirz6write results in a specified directory (relative path)z --backup-dirz9preserve a copy of downloaded files in a backup directoryz --keep-dirsz-keep input directory structure and file namesz--feedz.look for feeds and/or pass a feed URL as input?TF)rnargsconstrz --sitemapzBlook for sitemaps for the given website and/or enter a sitemap URLz--crawlzJcrawl a fixed number of pages within a website starting from the given URLz --explorez=explore the given websites (combination of sitemap and crawl)z--probez?probe for extractable content (works best with target language)z --archivedz=try to fetch URLs from the Internet Archive if downloads failz --url-filterzLonly process/output URLs containing these patterns (space-separated strings)+)rr"rz-fz--fastz!fast (without fallback detection)z --formattingz,include text formatting (bold, italic, etc.)z--linksz5include links along with their targets (experimental)z--imagesz.include image sources in output (experimental)z --no-commentszdon't output any comments store_falsez --no-tableszdon't output any table elementsz--only-with-metadataz4only output those documents with title, URL and datez--with-metadataz&extract and add metadata to the outputz--target-languagez*select a target language (ISO 639-1 codes)z --deduplicatez+filter out duplicate documents and sectionsz --config-filezAoverride standard extraction parameters with a custom config filez --precisionz;favor extraction precision (less noise, possibly less text)z--recallz8favor extraction recall (more text, possibly more noise)z--output-formatzdetermine output formattxt)rchoicesrz--csvzshorthand for CSV outputz--htmlzshorthand for HTML outputz--jsonzshorthand for JSON outputz --markdownzshorthand for MD outputz--xmlzshorthand for XML outputz--xmlteizshorthand for XML TEI outputz--validate-teizvalidate XML TEI outputz-vz --verbosecountrz&increase logging verbosity (-v or -vv))r rrz --versionz!show version information and exitrz Trafilatura trafilaturaz - Python )rr r) add_argument_groupadd_mutually_exclusive_group add_argumentstrintrrrr) rgroup1 group1_exgroup2group3 group3_exgroup4group5 group5_exs L/var/www/html/sandstorm/venv/lib/python3.12/site-packages/trafilatura/cli.pyadd_argsr8sE & &w0W XF335I  & &x1^ _F  & &|5V WF335I  & &|5d eF  & &x1Q RF335I 4F "=T "42 "  ` .: mY " N+- nU " X "  L+-8M!u>;a!u>9i!u>;\!u>9^!u>  \+- k!- h@+- K+-  T+-  M+- 8,.  >,. .S+- )E+- +I " J+- ` "  Z+-  W+- ,6 1 %'77+-88+-88+-<6+-77+-:;+- (6+- k'1E  0w}56jAQ@RS  Margscvtjd}t|}t|j S)z(Define parser for command-line argumentsz&Command-line interface for Trafilatura) description)argparseArgumentParserr8map_args parse_args)r:rs r7r@r@s1  $ $1Y ZF f F F%%' ((r9c@dD]}t||s||_|S|S)z2Map existing options to format and output choices.)csvhtmljsonmarkdownxmlxmltei)getattr output_format)r:otypes r7r?r?s5F 4 !&D   K  Kr9cRttjdd}t|y)z Run as a command-line utility. rN)r@sysargv process_args)r:s r7mainrOs chhqrl #Dr9cxd}|jdk(r4tjtjtj nB|jdk\r3tjtjtj |jrt|j|_|js|js |jr t|}n|jr t|n|jr t!|n|j"r t%|nz|j&s |j(rt+|}t-||}nJt/tj0j2j5||j(}t7|||dk7rtj8|yy)z8Perform the actual processing according to the argumentsrr)streamlevel)urlN)verboselogging basicConfigrLstdoutWARNINGDEBUG blacklistr explorefeedsitemaprcrawlrprober input_dirr input_fileURLr rr stdinbufferreadrexit)r: exit_code url_storeresults r7rNrNs(I ||q3::W__E  3::W]]C ~~'7  ||tyyDLL!$'  D t  & DHH#D) +D)< ))..0$DHHEVT"A~ r9__main__)rN)"__doc__r=rVrLimportlib.metadatarplatformrtypingr cli_utilsrrr r r r r rrsettingsrrrXrhasattrrstderrr8r@r?rOrN__name__r9r7rvs &#???8::'!gcjj-&HJJG,::'!gcjj-&HJJG,PSPSPf)S)S)33 *s*t*Z zFr9