*#hhdZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlZddlmZmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&e#e'Z(eGddZ)eGddZ*Gdde+Z,Gdde+Z-eGddZ.eGddZ/Gdde e0e/fZ1eGddZ2y) a DatasetInfo and MetricInfo record information we know about a dataset and a metric. This includes things that we know about the dataset statically, i.e.: - description - canonical location - does it have validation and tests splits - size - etc. This also includes the things that can and should be computed once we've processed the dataset as well: - number of examples (in each split) - etc. N) dataclass)Path)ClassVarDictListOptionalUnion) DatasetCardDatasetCardData)config)FeaturesValue)is_remote_filesystem) SplitDict) TaskTemplatetask_template_from_dict)Version) get_logger)asdict unique_valuesc*eZdZUdZeed<dZeed<y)SupervisedKeysDatainputoutputN)__name__ __module__ __qualname__rstr__annotations__rJ/var/www/html/sandstorm/venv/lib/python3.12/site-packages/datasets/info.pyrr9sE3OFCr#rc*eZdZUdZeed<dZeed<y)DownloadChecksumsEntryDatarkeyvalueN)rrrr'r r!r(r"r#r$r&r&?sCME3Or#r&ceZdZdZy)MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nrrr__doc__r"r#r$r*r*EsEr#r*ceZdZdZy)NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr+r"r#r$r.r.Is9r#r.cVeZdZUdZeeed<dZeeed<dZ e deddfdZ y)PostProcessedInfoNfeaturesresources_checksumsc|j@t|jts%tj|j|_yyyN)r1 isinstancer from_dict)selfs r$ __post_init__zPostProcessedInfo.__post_init__Rs8 == $Z x-P$..t}}=DM.Q $r#post_processed_info_dictreturnc tj|Dchc]}|j}}|di|jDcic] \}}||vs ||c}}Scc}wcc}}wNr" dataclassesfieldsnameitems)clsr9f field_nameskvs r$r6zPostProcessedInfo.from_dictWs`'2'9'9#'>?!qvv? ?]'?'E'E'G\tq!1P[K[ad\]]@\A A!A!) rrrr1rrr!r2dictr8 classmethodr6r"r#r$r0r0MsL#'Hhx '*.$.> ^^:M^^r#r0ceZdZUdZej eZeed<ej eZ eed<ej eZ eed<ej eZ eed<dZ e eed<dZe eed <dZe eed <dZe eeed <dZe eed <dZe eed <dZe eed<dZe eeefed<dZe eed<dZe eed<dZe e ed<dZ!e e ed<dZ"e e ed<dZ#e e ed<gdZ$e%eeed<dZ& d)de efdZ'd*dZ(dZ)e*dedfdZ+e* d+dede ed dfd!Z,e*d"ed dfd#Z-d,d-d$Z.d.d%Z/d efd&Z0e*d'ed dfd(Z1y)/ DatasetInfoa Information about a dataset. `DatasetInfo` documents datasets, including its name, version, and features. See the constructor arguments and properties for a full list. Not all fields are known on construction and may be updated later. Attributes: description (`str`): A description of the dataset. citation (`str`): A BibTeX citation of the dataset. homepage (`str`): A URL to the official homepage for the dataset. license (`str`): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. features ([`Features`], *optional*): The features used to specify the dataset's column types. post_processed (`PostProcessedInfo`, *optional*): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index. supervised_keys (`SupervisedKeysData`, *optional*): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). builder_name (`str`, *optional*): The name of the `GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name. config_name (`str`, *optional*): The name of the configuration derived from [`BuilderConfig`]. version (`str` or [`Version`], *optional*): The version of the dataset. splits (`dict`, *optional*): The mapping between split name and metadata. download_checksums (`dict`, *optional*): The mapping between the URL to download the dataset's checksums and corresponding metadata. download_size (`int`, *optional*): The size of the files to download to generate the dataset, in bytes. post_processing_size (`int`, *optional*): Size of the dataset in bytes after post-processing, if any. dataset_size (`int`, *optional*): The combined size in bytes of the Arrow tables for all splits. size_in_bytes (`int`, *optional*): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files). task_templates (`List[TaskTemplate]`, *optional*): The task templates to prepare the dataset for during training and evaluation. Each template casts the dataset's [`Features`] to standardized column names and types as detailed in `datasets.tasks`. **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`]. default_factory descriptioncitationhomepagelicenseNr1post_processedsupervised_keystask_templates builder_name dataset_name config_nameversionsplitsdownload_checksums download_sizepost_processing_size dataset_size size_in_bytes)rWr[r]r1rY_INCLUDED_INFO_IN_YAMLc2|j>t|jts$tj|j|_|j>t|jt s$t j|j|_|j st|j tsYt|j trt|j |_n$tj|j |_|j>t|jts$tj|j|_ |jlt|jtsRt|jttfrt|j|_ ntdi|j|_ |j t|j ttfrM|j Dcgc]}t|t"r|n t%|!}}|Dcgc]}|| c}|_nNt|j t"r|j g|_n!t%|j }||gng|_|j `t|j |_|j9|j Dcgc]}|j'|jc}|_yyycc}wcc}wcc}wr<)r1r5rr6rRr0rXrr rYrfrom_split_dictrSrtuplelistrTrralign_with_features)r7template templatess r$r8zDatasetInfo.__post_init__sa == $Z x-P$..t}}=DM    *:d>Q>QSd3e"3"="=d>Q>Q"RD  << #Jt||W,M$,,,&t||4 &00> ;; ":dkk9+M#33DKK@DK    +Jt?S?SUg4h$.. >'94;O;O'P$'9'QD'+':':&;#243F3FG4<4Hxjb#    *"&t':':";D }}(QUQdQd'DLH00?'#) +'c's$L :LL"Lstorage_optionscT|dk7r&tjdt|j}t j ||}|d}t | }|rtjjntj}|j||tjd5}|j||ddd|jrB|j||tj d5}|j#|dddyy#1swYXxYw#1swYyxYw)ahWrite `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`. Args: dataset_info_dir (`str`): Destination directory. pretty_print (`bool`, defaults to `False`): If `True`, the JSON will be pretty-printed with the indent level of 4. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("rotten_tomatoes", split="validation") >>> ds.info.write_to_directory("/path/to/directory/") ``` deprecated'fs' was deprecated in favor of 'storage_options' in version 2.9.0 and will be removed in 3.0.0. You can remove this warning by passing 'storage_options=fs.storage_options' instead.rgrwb) pretty_printN)warningswarn FutureWarningrgfsspecget_fs_token_pathsrospathjoin posixpathopenr DATASET_INFO_FILENAME _dump_inforQLICENSE_FILENAME _dump_license) r7dataset_info_dirrmfsrgfs_token_pathsis_local path_joinrCs r$write_to_directoryzDatasetInfo.write_to_directorysB   MMg  !00O223CUde(6q(9+B//$,BGGLL).. WWY/1M1MNPT U :YZ OOALO 9 : <<#3V5L5LMtT &XY""1% & &  : : & &s'D6DDD'c|jtjt||rdndj dy)zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsrencode)r7filerms r$ryzDatasetInfo._dump_info s- 4::fTl 1$OVVW^_`r#cX|j|jjdy)zTDump license in `file` file-like object open in bytes mode (to support remote files)rN)rrQr)r7rs r$r{zDatasetInfo._dump_licenses 4<<&&w/0r# dataset_infosc |Dcgc]}||j}}djtd|Dj}djtd|Dj}djtd|Dj}djtd|Dj}d}d}d} |D cgc]} | j| j} } t | dkDr(t t| dj| dd} n"t | rt t| d} | r| nd} |||||||| Scc}wcc} w) Nz c34K|]}|jywr4)rN.0infos r$ z)DatasetInfo.from_merge..s/[T0@0@/[c34K|]}|jywr4)rOrs r$rz)DatasetInfo.from_merge..,UtT]],Urc34K|]}|jywr4)rPrs r$rz)DatasetInfo.from_merge..rrc34K|]}|jywr4)rQrs r$rz)DatasetInfo.from_merge..s+STDLL+Srr r)rNrOrPrQr1rSrT) copyrurstriprTlenrcset intersection) rBr dset_inforNrOrPrQr1rSrTrall_task_templatess r$ from_mergezDatasetInfo.from_mergesp;HbiILa)b bkk-/[]/["[\bbd ;;},U},UUV\\^;;},U},UUV\\^++m+S]+SSTZZ\?LodtObObOnd11oo ! "Q &!"I#&8&;"<"I"IK]^_^`Ka"bcN # $!#&8&;"<=N+9t#+)  %cpsE6E6'E;9E;r|r:c|dk7r&tjdt|j}t j ||}|d}t jd||s tdt| }|rtjjntj}|j||tj dd 5}t#j$|}d d d |j'S#1swYxYw) aCreate [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_dir (`str`): The directory containing the metadata file. This should be the root directory of a specific dataset version. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import DatasetInfo >>> ds_info = DatasetInfo.from_directory("/path/to/directory/") ``` rirjrkrzLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rrencodingN)rnrorprgrqrrloggerr ValueErrorrrsrtrurvrwr rxrloadr6) rBr|r}rgr~rrrCdataset_info_dicts r$from_directoryzDatasetInfo.from_directory0sJ   MMg  !00O223CUde(6q(9 01A0BCDde e+B//$,BGGLL).. WWY/1M1MNPS^eW f -jk $ !  -}}.// - -s C==Drc tj|Dchc]}|j}}|di|jDcic] \}}||vs ||c}}Scc}wcc}}wr<r=)rBrrCrDrErFs r$r6zDatasetInfo.from_dictks_'2'9'9#'>?!qvv? ?V'8'>'>'@Utq!ADTadUVV@UrGc |j}|jdi|jjDcic]\}}||s|tj|!c}}ycc}}wr<)__dict__updaterArdeepcopy)r7other_dataset_info ignore_none self_dictrErFs r$rzDatasetInfo.updatepseMM   /77==?AqM4==## s$A#c |jdi|jjDcic]\}}|tj|c}}Scc}}wr<) __class__rrArr)r7rErFs r$rzDatasetInfo.copyzsCt~~XATATAV WADMM!$4!4 WXX Ws Aci}t|}|D]b}||jvst||}t|dr|j ||<>t|dr|j ||<^|||<d|S)N _to_yaml_list_to_yaml_string)rr_getattrhasattrrr)r7 yaml_dictrr'r(s r$ _to_yaml_dictzDatasetInfo._to_yaml_dict}s "4L$ +Cd111c*5/2%*%8%8%:IcNU$56%*%:%:%r?r@rA)rBrrCrDrErFs r$_from_yaml_dictzDatasetInfo._from_yaml_dictsMM), == $ 0$,$<$?!qvv? ?Ny'8Mtq!AC)FriNF)riN)T)rrK)r:rK)2rrrr,r>fieldr rNr!rOrPrQr1rrrRr0rSrrTrrrUrVrWrXr rrYrHrZr[intr\r]r^r_rr8rryr{rIrrr6rrrrr"r#r$rKrK]s>,^){((=K=%K%%c:Hc:%K%%c:Hc:$;$$S9GS9#'Hhx '26NH./648OX01837NHT,/07#'L(3-&"&L(3-&!%K#%-1GXeCL) *1!FHTN!)--#'M8C='*.(3-."&L(3-&#'M8C='3HT#Y/'Thl3&V^_cVd3&ja1 tM':  :W[80"80FNtn80 8080tW$W=WW Y t OOOOr#rKcReZdZdd dZed dZededdfdZdeddfdZy) DatasetInfosDictr:Nc >i}tjj|tj}tjj|d}|s|j |}|j |tjj|r_t|dd5}|jD cic]\}} |t| } }} tj| ||rdnddddtjj|r"tj|} | j} n d} t!} |rJ|j#| | tdt%| zdzn| } | j't)|yycc} }w#1swYxYw) N README.mdwrrrrz--- z --- )rsrtrur DATASETDICT_INFOS_FILENAMErrexistsrwrArrdumpr rdatar to_dataset_card_datar saver) r7dataset_infos_dir overwritermtotal_dataset_infosdataset_infos_pathdataset_readme_pathrCrWrdataset_infos_dict dataset_carddataset_card_datas r$rz#DatasetInfosDict.write_to_directoryst WW\\*;V=^=^_ ggll+>, -(#@ UAQdQjQjQl&7M{IK !22&"& ,a\tT  U 77>>- .&++,?@L , 1 1 L / 1    4 45F GMYMa Gc*;&<>"'',,'8+F G + 0 06G1H;1V W \ \ !22112CDD 77>>"'',,'8&:[:[\ ]bggll#4f6W6WXcjk op?Ciil>P>P>R:K):$[%:%:;L%MM  5L   s)F E; #F;FF rc t|jdttfrt|dtr=||dDcic](}|jddtj |*c}Stj |d}|djdd|_||j |iS|Scc}w)NrrWdefault)r5rrcrHrKrrW)rBrdataset_info_yaml_dictrs r$rz'DatasetInfosDict.from_dataset_card_datas '++N;dD\ J+N;TB 7H6W 3/22=)LkNiNi2O +::;L^;\] +<^+L+P+PQ^`i+j (L44lCDD5Ls-Cc|rcd|vr.t|dtr|djdd|di}n0d|vr*t|dtr|dDcic]}|d| }}ni}i||j Dcic]\}}||j c}}}|j D] \}}||d< t |dk(rMtt|j|d<|djdd}|dk7r d|i|d|d<yyg|d<t|j D]1\}}|jddd|i|}|dj|3yycc}wcc}}w)NrrWrr ) r5rHrrcrArrnextitervaluespopsortedappend) r7rdataset_metadata_infosconfig_metadatarWrrdset_info_yaml_dictrs r$rz%DatasetInfosDict.to_dataset_card_datas !22zBSTbBcei7j%n599-SUfguUv*& #44DUVdDegk9l,=^+L*'$M2OC*&* *,&#(#X\XbXbXde>Tk9; 7 7 99e#  5H4M4M4O A0 05@#M2 A&'1,48>Q>X>X>Z9[4\!.1/?CCMSWX )+&{9+N;9%n5,57!.1;ABUB[B[B];^U7K!7*..}dC.;[-cLb-c*%n5<<=ST UC *fs  E(<E-)FF)r:N)r:r) rrrrrIrr rrr"r#r$rrsT98&K]$&Uo&U$&Ur#rceZdZUdZeed<eed<eed<ejeZ eed<ejeZ eed<ejeZ eed<eje Z eeed <eje Zeeed <d Zeed <d Zeeed<d Zeeed<d Zeeed<d Zeeed<dZddZeddZededdfdZy ) MetricInfoaInformation about a metric. `MetricInfo` documents a metric, including its name, version, and features. See the constructor arguments and properties for a full list. Note: Not all fields are known on construction and may be updated later. rNrOr1rLinputs_descriptionrPrQ codebase_urlsreference_urlsF streamableNformat metric_namerW experiment_idc|jX|jjD]:\}}t|trt d|d|j jyy)NzSWhen using 'numpy' format, all features should be a `datasets.Value` feature. Here z is an instance of )rr1rAr5rrrr)r7r'r(s r$r8zMetricInfo.__post_init__sj ;; ""mm113  U!%/$ #u$78P8P7QS  #r#cttjj|tj dd5}t jt|||rdndddd|jr_ttjj|tjdd5}|j|jdddyy#1swYuxYw#1swYyxYw)aWrite `MetricInfo` as JSON to `metric_info_dir`. Also save the license separately in LICENCE. If `pretty_print` is True, the JSON will be pretty-printed with the indent level of 4. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.info.write_to_directory("/path/to/directory/") ``` rrrrNr) rwrsrtrur METRIC_INFO_FILENAMErrrrQrzr)r7metric_info_dirrmrCs r$rzMetricInfo.write_to_directory's"'',,0K0KLc\c d Khi IIfTlA>> from datasets import MetricInfo >>> metric_info = MetricInfo.from_directory("/path/to/directory/") ``` zLoading Metric info from zCCalling MetricInfo.from_directory() with undefined metric_info_dir.rrN) rrrrwrsrtrur rrrr6)rBrrCmetric_info_dicts r$rzMetricInfo.from_directory;s~  //@ABbc c "'',,0K0KLW^ _ ,cd#yy|  ,}}-.. , ,s BBrc tj|Dchc]}|j}}|di|jDcic] \}}||vs ||c}}Scc}wcc}}wr<r=)rBrrCrDrErFs r$r6zMetricInfo.from_dictRs_'2'9'9#'>?!qvv? ?U'7'='='?Ttq!1 CSadTUU@TrGr)r:r)rrrr,r r!rr>rrrPrQrcrrrrboolrrrrWrr8rrIrrHr6r"r#r$rrsM/k//DD%K%%c:Hc:$;$$S9GS90{00FM49F 1 1 1$ GNDIGJ FHSM "&K#%!%K#%#'M8C='&(//,VV,VVr#r)3r,rr>rrsrvrnrpathlibrtypingrrrrr rqhuggingface_hubr r rr r1rr filesystemsrrYrtasksrrutilsr utils.loggingrutils.py_utilsrrrrrr& Exceptionr*r.r0rKr rrr"r#r$rs   !88 8%-8%1 H       FIF:):  ^ ^  ^ uOuO uOp jUtC,-jUZ QVQV QVr#