o  hv@s,dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl Z ddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#eraeraddl$Z$e"%e&Z'e dZ(Gd d d eZ)Gd d d eZ*ee*j+e*_+e*j+jdure*j+jj,d ddde*j+_dSdS)zH Feature extraction saving/loading class for common feature extractors. N)UserDict) TYPE_CHECKINGAnyOptionalUnion)custom_object_save)FEATURE_EXTRACTOR_NAMEPushToHubMixin TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_func download_urlis_flax_available is_jax_tensoris_numpy_arrayis_offline_mode is_remote_urlis_tf_availableis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsSequenceFeatureExtractorcseZdZdZddeeeefdedee fffdd Z dedeefd d Z defd d Z d dZ ddZddZddZddZddeeee ffddZddeeee ffddZdddZZS) BatchFeatureaU Holds the output of the [`~SequenceFeatureExtractor.pad`] and feature extractor specific `__call__` methods. This class is derived from a python dictionary and can be used as a dictionary. Args: data (`dict`, *optional*): Dictionary of lists/arrays/tensors returned by the __call__/pad methods ('input_values', 'attention_mask', etc.). tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. Ndata tensor_typecst||j|ddS)N)r)super__init__convert_to_tensors)selfrr __class__y/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/feature_extraction_utils.pyr!Ls zBatchFeature.__init__itemreturncCst|tr |j|Std)z If the key is a string, returns the value of the dict associated to `key` ('input_values', 'attention_mask', etc.). zRIndexing with integers is not available when using Python based feature extractors) isinstancestrrKeyErrorr#r(r&r&r' __getitem__Ps  zBatchFeature.__getitem__cCs z|j|WStytwN)rr,AttributeErrorr-r&r&r' __getattr__Zs   zBatchFeature.__getattr__cCs d|jiSNrrr#r&r&r' __getstate__` zBatchFeature.__getstate__cCsd|vr |d|_dSdSr2r3)r#stater&r&r' __setstate__cszBatchFeature.__setstate__cC |jSr/)rkeysr4r&r&r'r:hr6zBatchFeature.keyscCr9r/)rvaluesr4r&r&r'r;lr6zBatchFeature.valuescCr9r/)ritemsr4r&r&r'r<pr6zBatchFeature.itemscs|durdSt|tst|}|tjkr)tstdddl}|j|j}|fS|tjkrFt s5tdddl fddj}|fS|tj krat sRtdddl m}|jt}|fSd fdd t}|fS) NNNzSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.cst|ttfr;t|dkr;t|dtjrt|}n t|dttfr;t|ddkr;t|ddtjr;t|}t|tjrF|S|S)Nr) r*listtuplelennpndarrayarray from_numpytensor)value)torchr&r' as_tensors     z5BatchFeature._get_is_as_tensor_fns..as_tensorzEUnable to convert output to JAX tensors format, JAX is not installed.csnt|ttfr0t|dtttjfr0dd|D}tt|dkr0|dur0dd|Dtd}tj||dS)NrcSsg|]}t|qSr&)r@.0valr&r&r' szIBatchFeature._get_is_as_tensor_fns..as_tensor..rcSsg|]}t|qSr&)rAasarrayrIr&r&r'rLs)dtype) r*r>r?rArBr@setobjectrM)rFrN value_lens)rHr&r'rHs $r/)r*r TENSORFLOWr ImportError tensorflowconstant is_tensorPYTORCHrrGJAXr jax.numpynumpyrCrr)r#rtfrVjnpr&)rHrGr'_get_is_as_tensor_fnsss<  (     z"BatchFeature._get_is_as_tensor_fnscCsj|dur|S||\}}|D]!\}}z||s"||}|||<Wq|dkr/tdtd|S)a5 Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. Noverflowing_valueszKUnable to create tensor returning overflowing values of different lengths. zUnable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.)r]r< ValueError)r#rrVrHkeyrFrEr&r&r'r"s  zBatchFeature.convert_to_tensorsc Ost|dgddl}i}|d}|dd}|durEt|dkrE|d}t|r*nt|ts8t|s8t|tr;|}n t dt|d| D]2\}} t| |j rc| | rc| j |i|||<qIt| |j rw|durw| j ||d ||<qI| ||<qI||_|S) a Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in different `dtypes` and sending the `BatchFeature` to a different `device`. Args: args (`Tuple`): Will be passed to the `to(...)` function of the tensors. kwargs (`Dict`, *optional*): Will be passed to the `to(...)` function of the tensors. To enable asynchronous data transfer, set the `non_blocking` flag in `kwargs` (defaults to `False`). Returns: [`BatchFeature`]: The same instance after modification. rGrNdevice non_blockingFz*Attempting to cast a BatchFeature to type z. This is not supported.)rarb)rrGgetr@rr*r+rintr_r<Tensoris_floating_pointtor) r#argskwargsrGnew_datararbargkvr&r&r'rgs(    zBatchFeature.tor=r/)r)r)__name__ __module__ __qualname____doc__rdictr+rrr r!r.r1r5r8r:r;r<r]r"rg __classcell__r&r&r$r'r=s. ;rc@sfeZdZdZdZddZdefddZe     d+d e ee j fd e e ee j fd e d e de e ee fdef ddZd,de ee j fde fddZed e ee j fdeeeefeeefffddZedeeefdefddZdeeeffddZede ee j fdefdd Zdefd!d"Zd#e ee j ffd$d%Zd&d'Zed-d)d*ZdS).FeatureExtractionMixinz This is a feature extraction mixin used to provide saving/loading functionality for sequential and image feature extractors. Nc Ksl|dd|_|D](\}}zt|||Wq ty3}ztd|d|d||d}~wwdS)z'Set elements of `kwargs` as attributes.processor_classNz Can't set z with value z for )pop_processor_classr<setattrr0loggererror)r#rir`rFerrr&r&r'r!szFeatureExtractionMixin.__init__rucCs ||_dS)z%Sets processor class as an attribute.N)rw)r#rur&r&r'_set_processor_classs z+FeatureExtractionMixin._set_processor_classFmainpretrained_model_name_or_path cache_dirforce_downloadlocal_files_onlytokenrevisionc Ks||d<||d<||d<||d<|dd}|dur*tdt|dur(td|}|dur2||d <|j|fi|\} }|j| fi|S) a Instantiate a type of [`~feature_extraction_utils.FeatureExtractionMixin`] from a feature extractor, *e.g.* a derived class of [`SequenceFeatureExtractor`]. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a feature extractor file saved using the [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model feature extractor should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the feature extractor files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final feature extractor object. If `True`, then this functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of `kwargs` which has not been used to update `feature_extractor` and is otherwise ignored. kwargs (`Dict[str, Any]`, *optional*): The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]. Examples: ```python # We can't instantiate directly the base class *FeatureExtractionMixin* nor *SequenceFeatureExtractor* so let's show the examples on a # derived class: *Wav2Vec2FeatureExtractor* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h" ) # Download feature_extraction_config from huggingface.co and cache. feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "./test/saved_model/" ) # E.g. feature_extractor (or model) was saved using *save_pretrained('./test/saved_model/')* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("./test/saved_model/preprocessor_config.json") feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False ) assert feature_extractor.return_attention_mask is False feature_extractor, unused_kwargs = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False, return_unused_kwargs=True ) assert feature_extractor.return_attention_mask is False assert unused_kwargs == {"foo": False} ```rrrruse_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r)rvwarningswarn FutureWarningr_get_feature_extractor_dict from_dict) clsr~rrrrrrirfeature_extractor_dictr&r&r'from_pretraineds&X z&FeatureExtractionMixin.from_pretrainedsave_directory push_to_hubc Ks|dd}|dur tdt|dddurtd||d<tj|r.t d|dtj |dd |rX|d d}|d | tjj d }|j |fi|}||}|jdurdt|||d tj|t}||td||r|j|||||dd|gS)az Save a feature_extractor object to the directory `save_directory`, so that it can be re-loaded using the [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] class method. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file will be saved (will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. rNrrrzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_id)configzFeature extractor saved in )rr)rvrrrrcr_ospathisfileAssertionErrormakedirssplitsep _create_repo_get_files_timestamps _auto_classrjoinr to_json_fileryinfo_upload_modified_files) r#rrrirrrfiles_timestampsoutput_feature_extractor_filer&r&r'save_pretrainedsB      z&FeatureExtractionMixin.save_pretrainedr)c Ksp|dd}|dd}|dd}|dd}|dd}|dd}|d d} |d d} |d d} | durJtd t|durHtd | }|dd} |dd} d| d}| durc| |d<tro| sotdd} t|}t j |}t j |rt j |t }t j |r|}d}n;t|r|}t|}n0t }zt||||||| |||| d }Wntytytd|d|dt dwz!t|dd }|}Wdn1swYt|}Wntjytd|dw|r td|n td|d||s4d |vr&t|d ||d <d!|vr4t|d!||d!<||fS)"a6 From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] using `from_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. Returns: `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the feature extractor object. rNrFresume_downloadproxies subfolderrrrrrr_from_pipeline _from_autofeature extractor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueT) rrrrrrr user_agentrz"Can't load feature extractor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z fileutf-8encodingz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at auto_mapcustom_pipelines)rvrrrr_rryrr+rrisdirrr rrrrOSError ExceptionopenreadjsonloadsJSONDecodeErrorr r )rr~rirrrrrrrrr from_pipelinerris_localfeature_extractor_fileresolved_feature_extractor_filereadertextrr&r&r'rs                         z1FeatureExtractionMixin.get_feature_extractor_dictrcKs|dd}g}|D]\}}||vr|||<||q |D]}||dq |di|}td||r>||fS|S)a  Instantiates a type of [`~feature_extraction_utils.FeatureExtractionMixin`] from a Python dictionary of parameters. Args: feature_extractor_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the feature extractor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~feature_extraction_utils.FeatureExtractionMixin.to_dict`] method. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the feature extractor object. Returns: [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature extractor object instantiated from those parameters. return_unused_kwargsFNzFeature extractor r&)rvr<appendryr)rrrir to_remover`rFfeature_extractorr&r&r'r6s  z FeatureExtractionMixin.from_dictcCs8t|j}|jj|d<d|vr|d=d|vr|d=|S)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. feature_extractor_type mel_filterswindow)copydeepcopy__dict__r%rn)r#outputr&r&r'to_dict[s  zFeatureExtractionMixin.to_dict json_filecCsLt|dd }|}Wdn1swYt|}|di|S)a Instantiates a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] from the path to a JSON file of parameters. Args: json_file (`str` or `os.PathLike`): Path to the JSON file containing the parameters. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature_extractor object instantiated from that JSON file. rrNr&)rrrr)rrrrrr&r&r'from_json_filehs   z%FeatureExtractionMixin.from_json_filecCsb|}|D]\}}t|tjr|||<q|dd}|dur'||d<tj|ddddS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. rwNruT)indent sort_keys ) rr<r*rArBtolistrvrdumps)r# dictionaryr`rFrwr&r&r'to_json_string{s   z%FeatureExtractionMixin.to_json_stringjson_file_pathcCsBt|ddd}||WddS1swYdS)z Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this feature_extractor instance's parameters will be saved. wrrN)rwriter)r#rwriterr&r&r'rs"z#FeatureExtractionMixin.to_json_filecCs|jjd|S)N )r%rnrr4r&r&r'__repr__szFeatureExtractionMixin.__repr__AutoFeatureExtractorcCsDt|ts|j}ddlmm}t||st|d||_dS)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoFeatureExtractor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoFeatureExtractor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r*r+rntransformers.models.automodelsautohasattrr_r)r auto_class auto_moduler&r&r'register_for_auto_classs   z.FeatureExtractionMixin.register_for_auto_class)NFFNr})F)r)rnrorprqrr!r+r| classmethodrrPathLikerboolrrr?rrrrPreTrainedFeatureExtractorrrrrrrrr&r&r&r'rtsR   o=  t$  rtrrzfeature extractor file)rP object_class object_files)-rqrrrr collectionsrtypingrrrrrZrAdynamic_module_utilsrutilsr r r r r rrrrrrrrrrrrrrrG get_loggerrnryrrrtrformatr&r&r&r's6   T @ @