*#hlzddlZddlZddlmZddlmZddlmZmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZd d lmZd d lmZd dlm Z m!Z!m"Z"d dl#m$Z$d dl%m&Z&d dl'm(Z(m)Z)d dl*m+Z+m,Z,e-e$j\Z/e&j`e1Z2Gdde-Z3Gdde4Z5dZ6e$j\ddge$jngde$jpgdiZ9dZ:ejvejxdkrddgZ=ndd gZ=e$j\e$jne$jpgZ>e>Dcic]0}|e9|Dcgc]}e=D]}|j|e:!c}}2c}}}Z@e$j\d"giZAe6gZBe@eAgZCejvejxdkrgd#ZDnd$d%gZDd&ZEgd'ZFd(e-d)eGfd*ZHd+ee e e-fd)e e-ee e-d,fffd-ZId.e-d(e-d)eGfd/ZJd.e-d(e-d)eGfd0ZK dAd1e e-ge e-fd2e-d)e e-e e-ffd3ZLd1e e-ge e-fd)e e-fd4ZM dBd(e-d2e-d5e e e-d6e ed)e e-f d7ZNdCd2e-d6e ed)e e-e e-ffd8ZO dCd2e-d6e ed)e e-fd9ZP dCd:e-d6e ed)ee-fd;ZQ dDdd,e e-ZSGd?d@e e-eSfZTycc}}wcc}}}w)EN)partial) has_magic)PathPurePath)CallableDictListOptionalSetTupleUnion)get_fs_token_paths)HTTPFileSystem) HfFileSystem)version) thread_map)config)DownloadConfig)!_prepare_path_and_storage_options xbasenamexjoin)Split)logging) is_local_pathis_relative_path)glob_pattern_to_regexstring_to_dictc eZdZy)UrlN__name__ __module__ __qualname__P/var/www/html/sandstorm/venv/lib/python3.12/site-packages/datasets/data_files.pyr r r&r c eZdZy)EmptyDatasetErrorNr!r%r&r'r*r*"r(r&r*zFdata/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*traintraining) validationvaliddevval)testtestingeval evaluationz-._ 0-9z2023.9.0z{keyword}[{sep}/]**z**[{sep}/]{keyword}[{sep}/]**z**/*[{sep}/]{keyword}[{sep}/]**)keywordsepz**)z metadata.csv**/metadata.csvzmetadata.jsonl**/metadata.jsonlr7r8z*[])z README.mdz config.jsonzdataset_info.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.jsonpatternreturnc4tfdtDS)Nc3&K|]}|v ywNr%).0wilcard_characterr9s r' z%contains_wildcards..^sY0A G+Ys)anyWILDCARD_CHARACTERS)r9s`r'contains_wildcardsrC]s YEXY YYr&patterns DataFilesListc t|tr@|jDcic]$\}}t|t|tr|n|g&c}}St|tr t |giSt|trt d|Dr|D]W}t|tr8t|dk(r*d|vr&t|jdttfrKtd||Dcgc]}|d }}tt|t|k7rtd||Dcic]-}t|dt|dtr|dn|dg/c}St |iStt |Scc}}wcc}wcc}w)a/ Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list of patterns c3<K|]}t|tywr=) isinstancedict)r>r9s r'r@z$sanitize_patterns..osAWz'4(Assplitpathz]Expected each split to have a 'path' key which can be a string or a list of strings, but got z*Some splits are duplicated in data_files: ) rHrIitemsstrlistSANITIZED_DEFAULT_SPLITrAlenget ValueErrorsetsanitize_patterns)rDkeyvaluer9splitss r'rUrUas(D!ZbZhZhZjkJCQVC:eT#:%Gkk Hc "'(44 Hd # AA A# w-G )7*"7;;v#6d D$wxxAB 7??7gg&?F?3v;3v;. #MfX!VWW (GG$%*WV_VZ:[wvbijpbqarr  ,X6 6 h003l@s)E=, F,2Fmatched_rel_pathc<t|jjDcgc]}|jds|}}t|jjDcgc]}|jds|}}t |t |k7Scc}wcc}w)u When a path matches a pattern, we additionnally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False __)rparentparts startswithrQ)rYr9partdata_dirs_to_ignore_in_pathdata_dirs_to_ignore_in_patterns r'"_is_inside_unrequested_special_dirrbs85==M4N4U4U4[4["uD_c_n_nos_t4"u"u7?7H7O7O7U7U%otY]YhYhimYnd%o"%o * +s3Q/R RR#v%osBB B7BcTt|jDcgc]&}|jdst|dhk(r%|(}}t|jDcgc]&}|jdst|dhk(r%|(}}t |t |k7Scc}wcc}w)u: When a path matches a pattern, we additionnally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False .)rr]r^rTrQ)rYr9r_hidden_directories_in_pathhidden_directories_in_patterns r'?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirrgsl""2399"T__S=QZ]^bZchkglZl"""'*00%DOOC4HQTUYQZ_b^cQc%!% ) *c2O.P PP "%s!B B B B%3B%B%pattern_resolver base_pathc tD]}|jdd} ||}t|dkDs-|Dchc],}t t |t t |d.}}tDcgc]}||vst|c}t|ttz z}|Dcic]}||j|gc}cStD]e} g} | jD]6\}} | D],} ||}t|dkDs| j|68| sQ| Dcic]}|| | c}cStdd|#t$rYRwxYwcc}wcc}wcc}w#t$rYwxYwcc}w)a+ Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rrK)rKCouldn't resolve pattern  with resolver )ALL_SPLIT_PATTERNSreplaceFileNotFoundErrorrQrrrDEFAULT_SPLITSrNsortedrTformatALL_DEFAULT_PATTERNSrMappend) rhri split_patternr9 data_filesprXrK sorted_splits patterns_dictnon_empty_splitsrDs r'_get_data_files_patternsr|s,[ '' 37 )'2J z?Q $ y|-B9]C[-\]^ef F 6DWEuPVSZWZ`^,,[MMZZ5EM00u0=>>Z Z[ . O ,224 OE8# !1'!:Jz?Q&$++E2   =MNEE=//N N O 7yP`Oab cc3!    X[) Os@E 1E3 E"= E"-E'5E,) E; EE, E8 7E8 cg}tD]*} ||}t|dkDr|j|,|r|Stdd|#t$rYMwxYw)zM Get the supported metadata patterns from a directory or repository. rrlrm)METADATA_PATTERNSrQrurp)rhnon_empty_patternsr9metadata_filess r'_get_metadata_files_patternsrs~$ -g6N>"Q&"))'2 !! 7yP`Oab cc !   s'A  AAallowed_extensionsdownload_configc t|r t||}nAt|r4tjj |dtj z}nd}t||\}}t||\}}}|jddjddxs |j}|jddjdd}ttt|hz } t|jt r |jn|jd} | dk7r| dznd} |j#|d j%D cgc]\} } | d dk(rt| | vrt'tjj)| |tjj)||s`t+tjj)| |tjj)||s| j-| r| n| | z}} } |D cgc]3} t/fd t| jddd Dr| 5}} t1|t1|kr>t3t|t|z }t4j7d|d|n|}|s$d|d}|dt3z }t9||Scc} } wcc} w)a Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. glob.glob, Path.glob, Path.match or fnmatch do not support ** with a prefix/suffix other than a forward slash /. For instance, this means **.json is the same as *.json. On the contrary, the fsspec glob has no limits regarding the ** prefix/suffix, resulting in **.json being equivalent to **/*.json. More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicilty mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rrstorage_optionsz::z://fileT)detailtypeNc3,K|] }d|zv yw)rdNr%)r>suffixrs r'r@z"resolve_pattern..osg&3<#55gsrdrz Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rrrosrL splitdriver6rrrK root_markerrTFILES_TO_IGNORErrHprotocolrNglobrMrbrelpathrgr^rArQrOloggerinforp)r9rirrrfs_ fs_base_path fs_patternfiles_to_ignorerprotocol_prefixfilepathr matched_pathsoutinvalid_matched_files error_msgs ` r'resolve_patternr!sb  7+ w GG&&w/2RVV;  @ZijG_!'?KHB1??4(+11%8<NLt$Q'--e4R8J/*i.@-AAO(c:r{{ AH*2f*XX M %* gIhDWD]D]^aDbcdceDfgg   s8c-( ($(]);c#h)F$G ! KK27);hi~hA  &wiq1  ) 9$?Q:R9ST TI ** J=  s CK'8K$c|tt||} t||S#t$rt d|ddwxYw)uh Get the default pattern from a directory testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. Some examples of supported patterns: Input: my_dataset_repository/ ├── README.md └── dataset.csv Output: {"train": ["**"]} Input: my_dataset_repository/ ├── README.md ├── train.csv └── test.csv my_dataset_repository/ ├── README.md └── data/ ├── train.csv └── test.csv my_dataset_repository/ ├── README.md ├── train_0.csv ├── train_1.csv ├── train_2.csv ├── train_3.csv ├── test_0.csv └── test_1.csv Output: {'train': ['train[-._ 0-9/]**', '**/*[-._ 0-9/]train[-._ 0-9/]**', 'training[-._ 0-9/]**', '**/*[-._ 0-9/]training[-._ 0-9/]**'], 'test': ['test[-._ 0-9/]**', '**/*[-._ 0-9/]test[-._ 0-9/]**', 'testing[-._ 0-9/]**', '**/*[-._ 0-9/]testing[-._ 0-9/]**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train/ │ ├── shard_0.csv │ ├── shard_1.csv │ ├── shard_2.csv │ └── shard_3.csv └── test/ ├── shard_0.csv └── shard_1.csv Output: {'train': ['train[-._ 0-9/]**', '**/*[-._ 0-9/]train[-._ 0-9/]**', 'training[-._ 0-9/]**', '**/*[-._ 0-9/]training[-._ 0-9/]**'], 'test': ['test[-._ 0-9/]**', '**/*[-._ 0-9/]test[-._ 0-9/]**', 'testing[-._ 0-9/]**', '**/*[-._ 0-9/]testing[-._ 0-9/]**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train-00000-of-00003.csv ├── train-00001-of-00003.csv ├── train-00002-of-00003.csv ├── test-00000-of-00001.csv ├── random-00000-of-00003.csv ├── random-00001-of-00003.csv └── random-00002-of-00003.csv Output: {'train': ['data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'test': ['data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'random': ['data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']} In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. rir)riThe directory at z doesn't contain any data filesN)rrr|rpr*rirresolvers r'get_data_patternsrsPh)_]Hj'IFF j"3I;>] ^_eiijs !;cxtt||} t|S#t$rtd|ddwxYw)zE Get the supported metadata patterns from a local directory. rrz" doesn't contain any metadata fileN)rrrrprs r'get_metadata_patternsrsM)_]Hm+H55 m"3I;>` abhllms 9 data_filecrt||\}}t||\}}}t|tr)|j |}|j |j fSt|tr|jtjrttj|j}d|ttjdzdjdddz}|j |}|j |j fS|j|}dD]}||vst||fcSy ) Nrr)endpointtokenzhf://rz /resolve/@)ETagetagmtimer%)rrrHr resolve_pathrepo_idrevisionrr^r HF_ENDPOINTrrQrorrN) rrrrr resolved_pathhffsrrVs r'_get_single_origin_metadatars!"C9^m!nI!)_MHB1"l# 2 %%}'='=>> B 'I,@,@ASAS,TV%7%7?T?TUiF,>,>(?!(C(EFNN{\_abcc )))4 %%}'='=>> 779 D(% $;S N$ $% r&rwc ttt|||tjdt |dkxstj S)NrzResolving data files) max_workers tqdm_classdescdisable)rrrrtqdmrQis_progress_bar_enabled)rwrrs r'_get_origin_metadatarsI +_M<< #J2%NW-L-L-N)N  r&cjeZdZdZdeedeeeffd ZdZe ddeede jjde ed e eed e ed df d Ze ddeede ed e eed e ed df d Ze ddeede ed e eed e ed df dZdeed dfdZxZS)rEa List of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover DataFilesList has an additional attribute ``origin_metadata``. It can store: - the last modified time of local files - ETag of remote files - commit sha of a dataset repository Thanks to this additional attribute, it is possible to hash the list and get a different hash if and only if at least one file changed. This is useful for caching Dataset objects that are obtained from a list of data files. rworigin_metadatac2t||||_yr=)super__init__r)selfrwr __class__s r'rzDataFilesList.__init__ s $.r&cPtg|||j|jzSr=)rEr)rothers r'__add__zDataFilesList.__add__$s(_t_e_d.B.BUEZEZ.Z[[r&rD dataset_inforirrr:cd|jd|jd|xsdjd}|j||||S)Nzhf://datasets/r/rrirr)idsharstrip from_patterns)clsrDrrirrs r' from_hf_repozDataFilesList.from_hf_repo's]%\__$5Q|7G7G6H)/WYIZ[bbcfg    >Pbq!  r&c||n%tjj}|j||||SNr)rresolveas_posixr)rrDrirrs r'from_local_or_remotez"DataFilesList.from_local_or_remote5sE"+!6IDFNNPbq!  r&c ||n%tjj}g}|D]!} |jt ||||#t||}|||S#t $rt |sYQwxYw)Nrr)rrrextendrrprr)rrDrirrrwr9rs r'rzDataFilesList.from_patternsBs"+!6IDFNNexts r'r@z2DataFilesList.filter_extensions..]s<#4#: list of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover each list is a DataFilesList. It is possible to hash the dictionary and get a different hash if and only if at least one file changed. For more info, see ``DataFilesList``. This is useful for caching Dataset objects that are obtained from a list of data files. Changing the order of the keys of this dictionary also doesn't change its hash. NrDrirrr:c|}|jD]3\}}t|tstj||||n|||<5|Sr)rMrHrErrrDrirrrrVpatterns_for_keys r'rz"DataFilesDict.from_local_or_remoteusje%-^^%5  !C!""2MB 22$''9$3 3& H  r&rc|}|jD]4\}}t|tstj|||||n|||<6|S)N)rrirr)rMrHrEr) rrDrrirrrrVrs r'rzDataFilesDict.from_hf_reposme%-^^%5  !C!""2MB**$!-''9$3 +& H  r&c|}|jD]3\}}t|tstj||||n|||<5|Sr)rMrHrErrs r'rzDataFilesDict.from_patternssje%-^^%5  !C!""2MB ++$''9$3 ,& H  r&rc~t|}|jD]\}}|j|||<|Sr=)rrMr)rrrrVdata_files_lists r'rzDataFilesDict.filter_extensionssDd4jl$(JJL E C&88DCH E r&r)r"r#r$rrrrNr r rEr rrrrrrrrr%r&r'rres $(2648 sE$s)]":;;<C=%T#Y/  ".1   * $(2648 sE$s)]":;;<&,,88C=  %T#Y/  ".1  .$(2648 sE$s)]":;;<C=%T#Y/  ".1   *DI/r&r)r)NNr=)@N)Urr functoolsrrrpathlibrrtypingrrr r r r r rfsspecrfsspec.implementations.httprr packagingrtqdm.contrib.concurrentrrrdownloadr#download.streaming_download_managerrrrrXrutilsrutils.file_utilsrrutils.py_utilsrrrNTRAINrP get_loggerr"rr rpr*SPLIT_PATTERN_SHARDED VALIDATIONTESTSPLIT_KEYWORDSNON_WORDS_CHARSFSSPEC_VERSIONparse#KEYWORDS_IN_PATH_NAME_BASE_PATTERNSrqrs#DEFAULT_PATTERNS_SPLIT_IN_PATH_NAMEDEFAULT_PATTERNS_ALLrnrtr~rBrboolrCrUrbrgr|rrrrrrrEr)rKr5r9s000r'rs= "DDD%6(.$dd=Aekk*   H % #  ) a KK':& ; JJ9  =7==44+@Ba*b'+@Bc*d'++u//< ''   %e,  :   wO< < '# KK$,,' =7==44 ZZZ#1dD#o 6#14U4PS9VeKeEf@f;g#1LSSsStSB;QVY;Qdg;Qlp;Q~DF'dud3i/0'd=@'d #tCy.'dTd8SE49J6 !J0-J60J6