Ig9 LdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddlm Z m Z m Z m Z mZmZmZddlZddlZddlmZddlmZmZmZerddlmZdd lmZmZmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.e.j^e0Z1e*e e2jfZ4d d d dZ5ejldk\r ejnZ7n ejnZ7GddedZ8GddedZ9GddedZ:GddedZ;GddedZ<Gdde8e9e:e;ed Z?e)e>je>_@e>jj8e>jjjd!d"d#$e>j_yy)%z8 Processing saving/loading class for common processors. N)Path)AnyDictListOptionalTuple TypedDictUnion)custom_object_save)ChannelDimensionis_valid_imageis_vision_available)PILImageResampling)PaddingStrategyPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategy) CHAT_TEMPLATE_NAMEPROCESSOR_NAMEPushToHubMixin TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_funcdirect_transformers_import download_urlis_offline_mode is_remote_urlloggingrFeatureExtractionMixinImageProcessingMixin) AutoTokenizerAutoFeatureExtractorAutoImageProcessor) ceZdZUdZeeeeeeeefe d<eeeeeeefe d<eeeeeeeefe d<ee e d<ee e e fe d<ee e e fe d<eee d<eee d <ee e d <eee d <ee e d <ee e d <ee e d<ee e d<ee e d<ee e d<ee e d<ee e d<y) TextKwargsa Keyword arguments for text processing. For extended documentation, check out tokenization_utils_base methods and docstrings associated. Attributes: add_special_tokens (`bool`, *optional*) Whether or not to add special tokens when encoding the sequences. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*) Activates and controls padding. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*): Activates and controls truncation. max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. stride (`int`, *optional*): If set, the overflowing tokens will contain some tokens from the end of the truncated sequence. is_split_into_words (`bool`, *optional*): Whether or not the input is already pre-tokenized. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. return_overflowing_tokens (`bool`, *optional*): Whether or not to return overflowing token sequences. return_special_tokens_mask (`bool`, *optional*): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*): Whether or not to return `(char_start, char_end)` for each token. return_length (`bool`, *optional*): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*): Whether or not to print more information and warnings. padding_side (`str`, *optional*): The side on which padding will be applied. text_pair text_targettext_pair_targetadd_special_tokenspadding truncation max_lengthstrideis_split_into_wordspad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbose padding_sideN)__name__ __module__ __qualname____doc__rr rrr__annotations__boolstrrrintZ/var/www/html/answerous/venv/lib/python3.12/site-packages/transformers/processing_utils.pyr+r+Qs#Ji):DOTRcMddeffy"3T)_dK\F]]^^uY0A4 ?TXYjTk%klmm & 4o- ..dC!3344  SM!$'  %#D>)#D>)'~- (.$TN*D>! d^3-rGr+F)totalcTeZdZUdZeeed<eeee fed<ee ed<eeee fed<ee de fed<eeed<ee ed <eeed <ee e e e fed <ee e e e fed <eeed <eeee fed<eeed<ee ed<ee ee fed<y) ImagesKwargsaW Keyword arguments for image processing. For extended documentation, check the appropriate ImageProcessor class methods and docstrings. Attributes: do_resize (`bool`, *optional*): Whether to resize the image. size (`Dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. size_divisor (`int`, *optional*): The size by which to make sure both the height and width can be divided. crop_size (`Dict[str, int]`, *optional*): Desired output size when applying center-cropping. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `List[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. pad_size (`Dict[str, int]`, *optional*): The size `{"height": int, "width" int}` to pad the images to. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. do_resizesize size_divisor crop_sizerresample do_rescalerescale_factor do_normalize image_mean image_stddo_padpad_sizedo_center_crop data_formatinput_data_formatN)r>r?r@rArrCrBrrDrEr floatrr rFrGrHrKrKs#J~ 4S> ""3-S#X''u136788UO#4. ud5k1233eT%[0122 TNtCH~&&TN"*++c+;&; <==rGrKceZdZUdZeeed<eeee fed<ee ed<eded<eeed<ee ed<eeed <ee e e e fed <ee e e e fed <eeed <eeed <ee ed<ee ee fed<y) VideosKwargsa Keyword arguments for video processing. Attributes: do_resize (`bool`): Whether to resize the image. size (`Dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. size_divisor (`int`, *optional*): The size by which to make sure both the height and width can be divided. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `List[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. rLrMrNrrPrQrRrSrTrUrVrXrYrZN)r>r?r@rArrCrBrrDrEr[r rr rFrGrHr]r]s@~ 4S> ""3-+,,UO#4. ud5k1233eT%[0122 TNTN"*++c+;&; <==rGr]ceZdZUdZeeed<eedee edeee fed<eee e e fed<eeed<ee ed<eeed<ee ed <y ) AudioKwargsa Keyword arguments for audio processing. Attributes: sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_attention_mask (`bool`, *optional*): Whether or not [`~ASTFeatureExtractor.__call__`] should return `attention_mask`. sampling_ratez np.ndarray raw_speechr0r2r1r5r7N) r>r?r@rArrErBr rr[rCrDrrFrGrHr_r_s:C= |T%[$|:LdSWX]S^N__`aa eD#67 88   %#D>)rGr_c(eZdZUeeeefed<y) CommonKwargsreturn_tensorsN)r>r?r@rr rDrrBrFrGrHrcrcsU3 ?344rGrcceZdZUdZiej Zeed<iej Zeed<ie j Z e ed<ie j Z e ed<ie j Ze ed<y)ProcessingKwargsa' Base class for kwargs passing to processors. A model should have its own `ModelProcessorKwargs` class that inherits from `ProcessingKwargs` to provide: 1) Additional typed keys and that this model requires to process inputs. 2) Default values for existing keys under a `_defaults` attribute. New keys have to be defined as follows to ensure type hinting is done correctly. ```python # adding a new image kwarg for this model class ModelImagesKwargs(ImagesKwargs, total=False): new_image_kwarg: Optional[bool] class ModelProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: ModelImagesKwargs _defaults = { "images_kwargs: { "new_image_kwarg": False, } "text_kwargs": { "padding": "max_length", }, } ``` For Python 3.8 compatibility, when inheriting from this class and overriding one of the kwargs, you need to manually update the __annotations__ dictionary. This can be done as follows: ```python class CustomProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: CustomImagesKwargs CustomProcessorKwargs.__annotations__["images_kwargs"] = CustomImagesKwargs # python 3.8 compatibility ```python common_kwargs text_kwargs images_kwargs videos_kwargs audio_kwargsN)r>r?r@rArcrBrgr+rhrKrir]rjr_rkrFrGrHrfrfs#J#  & &#M<  $ $K#  & &#M<#  & &#M<!  % %!L+rGrfcbeZdZUdZddgZdgZgZeee d<dZ dZ dZ gZ eee d<dZd eeeffd Zd efd Zd eeej*ffd ZdZd&defdZedeeej*fd eeeefeeefffdZedeeeffdZ d'dedeed eeeffdZ e d(deeej*fdeeeej*fdededeeeefdef dZ!ed)dZ"edZ#e$d Z%e&d!Z'd"Z( d*d#eeeeefdeed$ed efd%Z)y)+ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. feature_extractor tokenizer chat_templateoptional_call_argsN valid_kwargsc L|jD]}t|||j|d!|D]}||jvst d|dt ||jD]\}}||vrt d|d|||<t |t |jk7rJtdt |jddj|jdt |d|jD]\}}t||d }tj||}t|trtd |D}ntt|}t||s(t d t!|j"d |d |dt|||y)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: , z. Got z arguments instead._classc3BK|]}|tt|ywNgetattrtransformers_module.0ns rH z*ProcessorMixin.__init__..~s$j\]\iW-@!%D$jsz Received a z for argument z, but a z was expected.)optional_attributessetattrpop attributes TypeErrorziplen ValueErrorjoinitemsrzAUTO_TO_BASE_CLASS_MAPPINGget isinstancetupler{typer>) selfargskwargsoptional_attributekeyargattribute_name class_name proper_classs rH__init__zProcessorMixin.__init__cs#'":": T  D,fjj9KT.R S T GC$//)">se1 EFF G$'tT__#= - C'"CNCSST UVV),~&  - v;#doo. .*3t+?*@ TYYW[WfWfMgLhhnt9+02  $*<<> / NC .)9'@AJ377 JOJ*e,$$jj$jj &':JG c<0!$s)"4"4!5^NCSS[\f[gguv D.# . /rGreturncjtj|j}tj|j }|j }|Dcgc]}||jjvs|}}|dgz }|jDcic] \}}||vs ||}}}|jj|d<d|vr|d=d|vr|d=d|vr|d=d|vr|d=|jDcic]1\}}t|ts|jjdk(s||3}}}|Scc}wcc}}wcc}}w)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this processor instance. auto_mapprocessor_classroimage_processorrnrpBeamSearchDecoderCTC) copydeepcopy__dict__inspect signaturer parameters __class__rrr>rr)routputsig attrs_to_savexkvs rHto_dictzProcessorMixin.to_dictsEt}}- . $1XqQdnn>W>W5WX X*% #)<<>H41aQ-5G!Q$HH$(NN$;$; ! & {#  &() & (*+ f $'   1q.1Q[[5I5IMc5c qD   1YI sD$-D$ D)D))6D/cX|j}tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. Tindent sort_keys )rjsondumps)r dictionarys rHto_json_stringzProcessorMixin.to_json_strings'\\^ zz*Q$?$FFrGjson_file_pathct|dd5}|j|jdddy#1swYyxYw)z Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this processor instance's parameters will be saved. wutf-8encodingN)openwriter)rrwriters rH to_json_filezProcessorMixin.to_json_files<.# 8 0F LL,,. / 0 0 0s 8Ac |jDcgc]}d|dtt||}}dj|}|jj d|d|j Scc}w)Nz- z: rz: z )rreprrzrrr>r)rnameattributes_reprs rH__repr__zProcessorMixin.__repr__svPTP_P_`RvRWT4-@(A'BC``))O4..))*#o->d4CVCVCXBYZZas"A0 push_to_hubc |jdd}| This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_id)config_set_processor_classrrprrrrrrzchat template saved in rzprocessor saved in )rr))rwarningswarn FutureWarningrrosmakedirssplitpathsep _create_repo_get_files_timestamps _auto_classrrzrr init_kwargsappendr hasattrrrr>save_pretrainedrrrrrprrrrloggerinfosetkeysr_upload_modified_files)rsave_directoryrrrrrfiles_timestampsrattrsaconfigs attributeoutput_processor_fileoutput_chat_template_fileprocessor_dictchat_template_json_stringrs rHrzProcessorMixin.save_pretraineds-0 $4d;  % MME zz'4(4 l-F7O NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I     'IMY~WT>2YEYafg\]A7N)O UVVgGg NN4 t^G D"oo 6Nn5Iy"89..t~~/F/FG  % %n 5  6    '"&// :#D.9 i)@A!--j9 :!# ^^ L$&GGLLAS$T!    ) OT-?-?@VZ[^bb &/wG 86 67 8 KK12K1LM N ~""$ %*;)< <   3 4 KK-.C-DE F   ' ' -jj) (  ~""$ %*;)< <I%&&kZg> 8 8s6M%MMM(pretrained_model_name_or_pathc |jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jd d} |jd d } |jd d} |jd d} d| d} | | | d<tr|stjdd}t |}t j j|}t j j|rDt j j|t}t j j|d}t j j|r|}d}d}nVt|r|}t|}d}n;t}t} t||||||||| | | d }t||||||||| | | d }d}|Dt#|dd5}|j%}dddt'j(d}||d<|i|fS t#|dd5}|j%}dddt'j(}|rtjd|ntjdd |d|vr|dtj-d!|s,d"|vrt/|d"||d"<d#|vrt1|d#||d#<||fS#t$rt $rtd|d|dtdwxYw#1swYxYw#1swYxYw#t&j*$rtd|dwxYw)$a From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a processor of type [`~processing_utils.ProcessingMixin`] using `from_args_and_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Returns: `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the processor object. cache_dirNforce_downloadFresume_downloadproxiesrlocal_files_onlyrevision subfolder_from_pipeline _from_auto processor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueTzchat_template.json) rrrrrr user_agentrr%_raise_exceptions_for_missing_entrieszCan't load processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerrrrpz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at zChat templates should be in a 'chat_template.json' file but found key='chat_template' in the processor's config. Make sure to move your template to its own file.rcustom_pipelines)rr rrrDrrisdirrrisfiler!rrrEnvironmentError ExceptionrreadrloadsJSONDecodeError warning_oncerr)clsrrrrrrrrrr from_pipelinerris_localprocessor_filechat_template_fileresolved_processor_fileresolved_chat_template_filerpreadertextrs rHget_processor_dictz!ProcessorMixin.get_processor_dict2s$JJ{D1 $4e< **%6=**Y- 7D)!::&8%@::j$/JJ{B/  #3T:  **\59#.?S  $+8J' (  %5 KKE F# (+,I(J%77==!>? 77==6 7WW\\*GXN!#.KMa!b  77>>7 8&C #*. 'H 8 9:N&23P&Q #*. '+N!3 - *51"'#1#$3%5)%':? +'$/:1&'#1#$3%5)%':? /+8 & 213I %V{{} % JJt,_=M&3F? # # *v:  -sWE %{{} %!ZZ-N  KK56M5NO P KK5n5E_UlTmn o n ,1P1\   ^  ^+-G":.0M.z*"^35W"#568U612v%%y$  &01N0OP99V8WX//=.>eE  % % % %## "45L4MMhi  s<?.K>?L-<M L:M>,L*-L7:M?M#M)rc |j}|jdd}|jdd}d|vr|d=d|vr|d=|j||j}||i|}| t |d|t |j D]+}t||st |||j|-|j|tjd||r||fS|S) a Instantiates a type of [`~processing_utils.ProcessingMixin`] from a Python dictionary of parameters. Args: processor_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~processing_utils.ProcessingMixin.to_dict`] method. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the processor object. Returns: [`~processing_utils.ProcessingMixin`]: The processor object instantiated from those parameters. return_unused_kwargsFrpNrr)processor_configrrz Processor ) rrvalidate_init_kwargsrrrrrrupdaterr) r rrrrrp unused_kwargsrrs rHfrom_args_and_dictz!ProcessorMixin.from_args_and_dicts"(,,.%zz*@%H ?D9   .01  'z*00._b_o_o0p 00  $ I >v{{}% 9Cy#& 3 38 9  m$ j ,- f$ $ rGModelProcessorKwargstokenizer_init_kwargsc @iiiiid}iiiiidt}D]}|jj|ij|<|j|jj D]@}||vst |j|rt|j|n||}|||<B|jt|t|z } |D]}|j|jj D]n}||vr0||j|d} | dk7r/|| vr+td|d|d||vr|j|d} nd} | dk7sV| |||<|j|ptfd|DrT|jD]@\}} |vs | jD]#\} } | |vs | ||| <|j| %Bn|D]}||vs|||d|<|D]}||j|d|S)a Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance. The order of operations is as follows: 1) kwargs passed as before have highest priority to preserve BC. ```python high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"} processor(..., **high_priority_kwargs) ``` 2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API. ```python processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}}) ``` 3) kwargs passed during instantiation of a modality processor have fourth priority. ```python tokenizer = tokenizer_class(..., {"padding": "max_length"}) image_processor = image_processor_class(...) processor(tokenizer, image_processor) # will pass max_length unless overriden by kwargs at call ``` 4) defaults kwargs specified at processor level have lowest priority. ```python class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False): _defaults = { "text_kwargs": { "padding": "max_length", "max_length": 64, }, } ``` Args: ModelProcessorKwargs (`ProcessingKwargs`): Typed dictionary of kwargs specifically required by the model passed. tokenizer_init_kwargs (`Dict`, *optional*): Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults. Returns: output_kwargs (`Dict`): Dictionary of per-modality kwargs to be passed to each modality-specific processor. )rhrirkrjrg __empty__zKeyword argument z+ was passed two times: in a dictionary for z and as a **kwarg.c3&K|]}|v ywrxrF)r}rdefault_kwargss rHrz/ProcessorMixin._merge_kwargs..bs7sn$7srg)r _defaultsrrrBrrrorzrrraddanyr)rrrr output_kwargs used_keysmodality modality_keyvaluenon_modality_kwargs kwarg_valuesubdictsubkeysubvaluerr"s @rH _merge_kwargszProcessorMixin._merge_kwargss^    E ' CH';'E'E'I'I(TV'W'\'\'^N8 $ 4 D DX N ^ ^ c c e C #88#4>><@  =2<@ >CN8,\: C C ^,"&kC ,>>% 0H 4 D DX N ^ ^ c c e 0 v%"("2"6"6|["QK"k1lFY6Y(/ ~>33;*FM(3F;%MM&12 2 Fi':@+M/237 F & KH ( # * *=+I J KrGrrrrrc ,||d<||d<||d<||d<|jdd}|)tjdt| t d|}|||d <|j |fi|} |j |fi|\} }|j| | fi|S) a[ Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rrrrrNrrr)rrrrr_get_arguments_from_pretrainedrr) r rrrrrrrrrrs rHfrom_pretrainedzProcessorMixin.from_pretrainedusN({#1 %5!"%z$4d;  % MME   l#E  #F7O1s112OZSYZ!7!7!78U!`Y_!`%s%%dNEfEErGct|ts |j}ddlmcm}t ||st|d||_y)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) rrDr>transformers.models.automodelsautorrr)r  auto_class auto_modules rHregister_for_auto_classz&ProcessorMixin.register_for_auto_classsC *c*#,,J66{J/ |+FGH H$rGc <g}|jD]}t||d}t|tr7td|D}|j dd}|r |d|d}n|d}ntt |}|j |j|fi||S)Nrvc3DK|]}|tt|ndywrxryr|s rHrz@ProcessorMixin._get_arguments_from_pretrained..s#o_`1=(;Q ?VZ Zos use_fastTr r)rrzrrrr{rr3) r rrrrrclassesr=attribute_classs rHr2z-ProcessorMixin._get_arguments_from_pretraineds!nn bN (8&?@J*e,odnoo!::j$7 6&-ajO&-ajO")*=z"J KK7778U`Y_` a b rGcNt||jd}t|ddS)Nrmodel_input_names)rzr)rfirst_attributes rHrAz ProcessorMixin.model_input_namess'!$(:;(;TBBrGc|j}i}t|t|z }|r=dj|}tj d|d|Dcic]}||| }}|Scc}w)NruzISome kwargs in processor config are unused and will not have any effect: z. )rrrrwarning)rrrkwargs_from_configr unused_keysunused_key_strrs rHrz#ProcessorMixin.validate_init_kwargss-224 ,-L0AA !YY{3N NN[\j[kkmn >IIQ 0 33IMIJs A+c xt|rtjdt|t|jkDrJt dt|jddj |jdt|dt ||jDcic]\}}|| c}}Scc}}w)a Matches optional positional arguments to their corresponding names in `optional_call_args` in the processor class in the order they are passed to the processor call. Note that this should only be used in the `__call__` method of the processors with special arguments. Special arguments are arguments that aren't `text`, `images`, `audio`, nor `videos` but also aren't passed to the tokenizer, image processor, etc. Examples of such processors are: - `CLIPSegProcessor` - `LayoutLMv2Processor` - `OwlViTProcessor` Also note that passing by position to the processor call is now deprecated and will be disallowed in future versions. We only have this for backward compatibility. Example: Suppose that the processor class has `optional_call_args = ["arg_name_1", "arg_name_2"]`. And we define the call method as: ```python def __call__( self, text: str, images: Optional[ImageInput] = None, *arg, audio=None, videos=None, ) ``` Then, if we call the processor as: ```python images = [...] processor("What is common in these images?", images, arg_value_1, arg_value_2) ``` Then, this method will return: ```python { "arg_name_1": arg_value_1, "arg_name_2": arg_value_2, } ``` which we could then pass as kwargs to `self._merge_kwargs` zPassing positional arguments to the processor call is now deprecated and will be disallowed in v4.47. Please pass all arguments as keyword arguments.zExpected *at most* zK optional positional arguments in processor callwhich will be matched with  z+ in the order they are passed.However, got z positional arguments instead.Please pass all arguments as keyword arguments instead (e.g. `processor(arg_name_1=..., arg_name_2=...))`.)rrrrqrrr)rr arg_valuearg_names rH'prepare_and_validate_optional_call_argsz6ProcessorMixin.prepare_and_validate_optional_call_argssX t9 MMB  t9s4223 3%c$*A*A&B%CD..1hht7N7N.O-PQ #D {+}}  @C4I`I`?ab(; 8)#bbbs% B6 conversationtokenizec |$|j |j}n td|jj|f||d|S)a Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input conversations to turn them into a single tokenizable string. Args: conversation (`List[Dict, str, str]`): The conversation to format. chat_template (`Optional[str]`, *optional*): The Jinja template to use for formatting the conversation. If not provided, the tokenizer's chat template is used. tokenize (`bool`, *optional*, defaults to `False`): Whether to tokenize the output or not. **kwargs: Additional keyword arguments zNo chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.)rprN)rprroapply_chat_template)rrMrprNrs rHrPz"ProcessorMixin.apply_chat_template-sd.  !!- $ 2 2  m 2t~~11  (5 LR  rG)Frx)NFFNmain) AutoProcessor)NF)*r>r?r@rArrrqrrDrBfeature_extractor_classtokenizer_classrrrrrrrrr rPathLikerrrCr classmethodrrrrfrr0r3r:r2propertyrA staticmethodrrLrPrFrGrHrmrmTs&{3J*+$&S &"OK L$s) $/L&c3h&P G G 05bkk1A+B 0[ d'4d'LV&,1#r{{2B,CV& tCH~tCH~- .V&V&p+d38n++`15z.z (~z c4i zx8<$!&,0=F',S"++-='>=FE#r{{"234=F =F  =F c4i() =F=F=F~%%2"CC  8cz(, " Dc3h01"  }"  " " rGrmc dtfd  fdfdd}||}|}||}|}|r|r||fS||s||s|r|rtjd||fStd)a For backward compatibility: reverse the order of `images` and `text` inputs if they are swapped. This method should only be called for processors where `images` and `text` have been swapped for uniformization purposes. Note that this method assumes that two `None` inputs are valid inputs. If this is not the case, it should be handled in the processor's `__call__` method before calling this method. rcHt|txr|jdS)Nhttp)rrD startswith)vals rHis_urlz1_validate_images_text_input_order..is_urlZs#s#>v(>>rGc~t|ttfr|D] }|r yyt|s |syy)NFT)rlistrr)imgsimg$_is_valid_images_input_for_processorr^s rHrczO_validate_images_text_input_order.._is_valid_images_input_for_processor]sF dT5M * !;C@  ! !&&,rGct|tryt|ttfrt |dk(ry|D] }|cSy)NTrF)rrDr`rr)tt_s"_is_valid_text_input_for_processors rHrgzM_validate_images_text_input_order.._is_valid_text_input_for_processorhsI a  D%= )1v{ ?9#>> ?rGc||xs|duSrxrF)input validators rH _is_validz4_validate_images_text_input_order.._is_validus05D=0rGzYou may have used the wrong order for inputs. `images` should be passed before `text`. The `images` and `text` inputs will be swapped. This behavior will be deprecated in transformers v4.47.zGInvalid input type. Check that `images` and/or `text` are valid inputs.)rCrr r) imagesrrkimages_is_validimages_is_text text_is_validtext_is_imagesrcrgr^s @@@rH!_validate_images_text_input_orderrqRs?t?  1 (LMO7?Nd$FGM9$?N=t| >t|Tbgu v V| ^ __rGrrRzprocessor files)object object_class object_files)BrArrrrsystypingrpathlibrrrrrrr r numpynptyping_extensionsdynamic_module_utilsr image_utilsr rrrtokenization_utils_baserrrrrutilsrrrrrrrrrrr r!r" get_loggerr>r__file__parentr{r version_infoUnpackr+rKr]r_rcrfrmrqrformatrFrGrHrs  EEE4NN/"   H %1h1F1FG/40 w ]]F  % %F7 %7 t4>9E4>n->9E->`$*)5$*N59E54z<{L`e4n{  ^{  |7`t'~'A'AB%%1)7)C)C)K)K)R)RGX*S*N&2rG