o  hc @sdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mZddlmZmZmZmZmZmZmZddlZddlZddlZddlmZdd lmZm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDddlEmFZFmGZGe@rddlHmIZIddlJmKZLerddlmMZMeBNeOZPdejQvrdejQd<n ejQddkrePRdz ddlSZTddlSmUZVWn#eWeXfyddlTZTddlTmUZVeeTjYjZdkre[dYnweNZ\eeej]eej^ee_ej]fee_ej^fej]ej^fZ`ddZaGdddZbddZcGd d!d!ZdGd"d#d#ZeGd$d%d%ZfGd&d'd'ZgGd(d)d)ZhGd*d+d+edZiGd,d-d-Zjd.d/Zkd0d1Zld2d3ZmdYd4d5Znd6e4fdZd9d:Zod[dd?Zq d[d@dAZrd\dBdCZsd\dDdEZtd\dFdGZudHdIZvGdJdKdKeTjwebe)e9ZxGdLdMdMeTjyjzZ{GdNdOdOeTjyjzZ|GdPdQdQeTjyjzZ}d]d^dWdXZ~dS)_zTF general model utils.) annotationsN)Mapping)Path) TYPE_CHECKINGAnyCallableDictListOptionalUnion)parse)DataCollatorWithPaddingDefaultDataCollator)get_tf_activation)PretrainedConfig)custom_object_save)GenerationConfigTFGenerationMixin)convert_batch_encoding expand_1dload_attributes_from_hdf5_groupsave_attributes_to_hdf5_group shape_list)SAFE_WEIGHTS_INDEX_NAMESAFE_WEIGHTS_NAMETF2_WEIGHTS_INDEX_NAMETF2_WEIGHTS_NAMETF_WEIGHTS_NAMEWEIGHTS_INDEX_NAME WEIGHTS_NAME ModelOutputPushToHubMixin cached_file download_url find_labelshas_fileis_offline_mode is_remote_urlis_safetensors_availableis_tf_symbolic_tensorloggingrequires_backendsworking_or_temp_dir)convert_file_size_to_intget_checkpoint_shard_files) safe_open) save_file)PreTrainedTokenizerBaseTF_USE_LEGACY_KERAS1zTransformers is only compatible with Keras 2, but you have explicitly set `TF_USE_LEGACY_KERAS` to `0`. This may result in unexpected behaviour or errors if Keras 3 objects are passed to Transformers models.)backendzYour currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.cCs0|jjdkr|Sttd|jj}tj||dS)Nr axis)shaperanklistrangetf reduce_mean)y_truey_predreduction_axesrBr/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py dummy_lossvs rDc@seZdZdZd d ddZd S) TFModelUtilsMixinzC A few utilities for `keras.Model`, to be used as a mixin. Fonly_trainableboolreturnintcCs$|rttdd|jDS|S)a9 Get the number of (optionally, trainable) parameters in the model. Args: only_trainable (`bool`, *optional*, defaults to `False`): Whether or not to return only the number of trainable parameters Returns: `int`: The number of parameters. css |] }t|jVqdSN)npprodr9as_list.0wrBrBrC sz3TFModelUtilsMixin.num_parameters..)rIsumtrainable_variables count_params)selfrFrBrBrCnum_parameterss z TFModelUtilsMixin.num_parametersN)F)rFrGrHrI)__name__ __module__ __qualname____doc__rVrBrBrBrCrE~srEcsjtdddurtdtfdd}|_tds)tdtjdr8fd d }|_d _tt j d rHt j S) a Decorate a Keras Layer class to support Keras serialization. This is done by: 1. Adding a `transformers_config` dict to the Keras config dictionary in `get_config` (called by Keras at serialization time. 2. Wrapping `__init__` to accept that `transformers_config` dict (passed by Keras at deserialization time) and convert it to a config object for the actual layer initializer. 3. Registering the class as a custom object in Keras (if the Tensorflow version supports this), so that it does not need to be supplied in `custom_objects` in the call to `keras.models.load_model`. Args: cls (a `keras.layers.Layers subclass`): Typically a `TF.MainLayer` class in this project, in general must accept a `config` argument to its initializer. Returns: The same class object, with modifications for Keras deserialization. config_classNz2Must set `config_class` to use @keras_serializablecs|r t|dtr |dn|dd}t|tr*|}||g|Ri|n(t|trNt|dkrA|g|Ri|n||g|Ri|ntd||_||_dS)Nrconfigz?Must pass either `config` (PretrainedConfig) or `config` (dict)) isinstancerpopdict from_dictlen ValueError_config_kwargs)rUargskwargsr\)r[ initializerrBrC wrapped_inits&     z(keras_serializable..wrapped_init get_configz=Only use @keras_serializable on keras.layers.Layer subclasses _is_defaultcs,t|}|j|d<||j|S)Nr\)superrircto_dictupdaterd)rUcfgclsrBrCris z&keras_serializable..get_configTregister_keras_serializable) __init__getattrAttributeError functoolswrapshasattr TypeErrorri_keras_serializablekerasutilsrq)rprhrirB)rpr[rgrCkeras_serializables      r|c@eZdZdZddZdS)TFCausalLanguageModelingLossz Loss function suitable for causal language modeling (CLM), that is, the task of guessing the next token. Any label of -100 will be ignored (along with the corresponding logits) in the loss computation. c Cstjjdtjjjd}|jjr8tt |dd}t t |dt |df|}t t |d|}|||S|tj ||}tj|dk|jd}||}t|t|} t | dS) NT from_logits reductionrr6dtyper )rzlossesSparseCategoricalCrossentropy ReductionNONEr\tf_legacy_lossr= not_equalreshape boolean_maskrnnrelucastr reduce_sum rUlabelslogitsloss_fn active_lossreduced_logits unmasked_loss loss_mask masked_lossreduced_masked_lossrBrBrChf_compute_losss   z,TFCausalLanguageModelingLoss.hf_compute_lossNrWrXrYrZrrBrBrBrCr~ r~c@r})TFQuestionAnsweringLossz8 Loss function suitable for question answering. cCsFtjjdtjjjd}||d|d}||d|d}||dS)NTrstart_positionr end_positionr g@rzrrrr)rUrrr start_lossend_lossrBrBrCrs z'TFQuestionAnsweringLoss.hf_compute_lossNrrBrBrBrCr rc@r})TFTokenClassificationLossz Loss function suitable for token classification. Any label of -100 will be ignored (along with the corresponding logits) in the loss computation. c Cstjjdtjjjd}trtj|dkrt d|j j r]tj|dkr6t dt |ddk}nt |ddk}t t |dt|df|}t t |d|}|||S|tj||}tj|dk|jd }||}t|t|} t | d S) NTrrzSUsing `-1` to mask the loss for the token is deprecated. Please use `-100` instead.rrr6rrr)rzrrrrr=executing_eagerlymath reduce_anyprintr\rrrrrrrrrrrBrBrCr s"     z)TFTokenClassificationLoss.hf_compute_lossNrrBrBrBrCrrrc@r})TFSequenceClassificationLossz= Loss function suitable for sequence classification. cCsj|jjdks |jddkr%tjjtjjjd}|jjdkr$tj|dd}n tjj dtjjjd}|||S)Nr )rrr7Tr) r9r:rzrMeanSquaredErrorrrr= expand_dimsrrUrrrrBrBrCr.s   z,TFSequenceClassificationLoss.hf_compute_lossNrrBrBrBrCr)rrc@r})TFMultipleChoiceLossz1Loss function suitable for multiple choice tasks.cCs tjjdtjjjd}|||S)NTrrrrBrBrCr?s z$TFMultipleChoiceLoss.hf_compute_lossNrrBrBrBrCr<s rc@seZdZdZdS)TFMaskedLanguageModelingLossz Loss function suitable for masked language modeling (MLM), that is, the task of guessing the masked tokens. Any label of -100 will be ignored (along with the corresponding logits) in the loss computation. N)rWrXrYrZrBrBrBrCrDsrc@r})TFNextSentencePredictionLossz Loss function suitable for next sentence prediction (NSP), that is, the task of guessing the next sentence. Any label of -100 will be ignored (along with the corresponding logits) in the loss computation. c Cstjjdtjjjd}|jjr2tt |dd}t t |d|}t t |d|}|||S|tj ||d}tj |dk|jd}||} | S)NTrrr)rr6)r?r@r)rzrrrrr\rr=rrrrrrr) rUrrrnext_sentence_active_lossnext_sentence_reduced_logitsnext_sentence_labelunmasked_ns_loss ns_loss_maskmasked_ns_lossrBrBrCr[s z,TFNextSentencePredictionLoss.hf_compute_lossNrrBrBrBrCrPrrcKsi}d|vr|ddur|dn|j|d<|ddur|dn|j|d<|ddur.|dn|j|d<d|vrI|ddurA|dnt|dd|d<|S)a Process the input booleans of each model. Args: config ([`PretrainedConfig`]): The config of the running model. **kwargs: The boolean parameters Returns: A dictionary with the proper values for each boolean output_attentionsNoutput_hidden_states return_dict use_cache)rrrrs)r\rffinal_booleansrBrBrCbooleans_processingrs rcs,ttfdd}|_|S)a Decorator that processes the inputs to a Keras layer, passing them to the layer as keyword arguments. This enables downstream use of the inputs by their variable name, even if they arrive packed as a dictionary in the first input (common case in Keras). Args: func (`callable`): The callable function of the TensorFlow model. Returns: A callable that wraps the original `func` with the behavior described above. csfdd|Dfdd|D}|di|ttjjdd|d|jjvr6d}n|j}t |fi|}|fi|S)Ncs$i|]\}}|tjvr||qSrB)r_ parametersrOkeyval)original_signaturerBrC s$zHunpack_inputs..run_call_with_unpacked_inputs..csi|] \}}|vr||qSrBrBr kwargs_callrBrCrrr EncoderDecoder) itemsrmr_zip__code__ co_varnames __class__rWr\input_processing)rUrerffn_args_and_kwargsr\unpacked_inputsfuncrrrCrun_call_with_unpacked_inputss  z4unpack_inputs..run_call_with_unpacked_inputs)inspect signaturerurv __signature__)rrrBrrC unpack_inputss rc Kstt|j}t|dd}|ddt|}|d}||d}i}tj tt t t ttt jf} d|dvrItdt|dd|d<d |dvr^td t|dd |d <d |dvrxd |vrxtd t|dd |d <nd |dvrd |vr|dd |d <|r|di|d<nt|ddkrtdt|dd|d|D]&\} } t| | st| s| dur| || <qtdt| d| d| dt|t tfr/t|D]E\} } t| r | jdd}||vr| ||<q| ||| <qt| | s| dur| ||| <qtdt| d| d|| dnt|trd|vrGtdt|d|d<d |vrYtd t|d |d <t|D]<\} } t| | sn| durt| || <q_| |vrd|vrtd| d|dq_tdt| d| d| dn t|s|dur|||<ntdt|d| d|d|D]}|t|vr|dkr||||j||<qd|vr |ddurt|dr|djdd}|d||<n|d|d<|d=d|vr|d=i}|D]:\}}t|tj r4|j tj!kr4t"|tj#||<qt|t jrL|j t j!krL|$t j#||<q|||<q|}~|durodd|D}|%t&dd|i||S)a Process the input of each TensorFlow model including the booleans. In case of a list of symbolic inputs, each input has to be named accordingly to the parameters name, i.e. `input_ids = keras.Input(shape=(128,), dtype='int32', name="input_ids")` otherwise the order of the tensors will not be guaranteed during the training. Args: func (`callable`): The callable function of the TensorFlow model. config ([`PretrainedConfig`]): The config of the running model. **kwargs: The inputs of the model. Returns: Two lists, one for the missing layers, and another one for the unexpected layers. rfNrUrinputsrzeThe `inputs` argument is deprecated and will be removed in a future version, use `input_ids` instead. input_idsdecoder_cached_stateszzThe `decoder_cached_states` argument is deprecated and will be removed in a future version, use `past_key_values` instead.past_key_valuespastziThe `past` argument is deprecated and will be removed in a future version, use `past_key_values` instead.zAThe following keyword arguments are not supported by this model: .z Data of type z is not allowed only z is accepted for :rezThe parameter z( does not belongs to the parameter list z and will be ignored.cSsi|] \}}|dvr||qS))rrrrrB)rOkvrBrBrCr\s z$input_processing..r\rB)'r_rrrrGr^r;keysr=TensorrIr!tuplerKndarraywarningswarn FutureWarningrarbrr] is_tensortype enumerater*namesplitrloggerwarningdefaultrint64rint32astypermr)rr\rfr has_kwargsparameter_namesmain_input_name main_inputoutput allowed_typesrriinput tensor_namer cast_outputrr boolean_dictrBrBrCrs                      rcCsn|dur||r|t|d}|dr|dd}d|vr5t|ddkr5d|ddd}|S)N/r model.) startswithrarjoin)r_prefixrBrBrCstrip_model_name_and_prefixls  r10GB weights_namestrc Cs2t|}g}g}d}d}|D]'}|j|jj}|||kr(||g}d}||||7}||7}q||t|dkrI||didfSi} i} t|D]:\} } |dd| dddt|dd} | dd| dddt|dd} | | | <| D] }|j}| | |<qqQd |i}|| d }| |fS) a? Splits a model state dictionary in sub-checkpoints so that the final size of each sub-checkpoint does not exceed a given size. The sub-checkpoints are determined by iterating through the `state_dict` in the order of its keys, so there is no optimization made to make each sub-checkpoint as close as possible to the maximum size passed. For example, if the limit is 10GB and we have weights of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB], [6+2+2GB] and not [6+2+2GB], [6+2GB], [6GB]. If one of the model's weight is bigger that `max_shard_size`, it will end up in its own sub-checkpoint which will have a size greater than `max_shard_size`. Args: weights (`Dict[str, tf.RessourceVariable]`): The list of tf.RessourceVariable of a model to save. max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): The maximum size of each sub-checkpoint. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). rr Nz.h5-05dz-of- .safetensors total_size)metadata weight_map) r.numpysizerappendrarreplacer)weightsmax_shard_sizersharded_state_dicts current_blockcurrent_block_sizer item weight_sizer shardsidxshard shard_fileweight weight_namer indexrBrBrCtf_shard_checkpointvs@      &  r FcCst}t}t}t}i} t|jD]?\} } | j} |dur0| |r0| t|d} | d} d| vsIt| ddksId| ddd} | | | | | <q|D]!} t || | ||d\}}}| || || |t qU||}|rt|dkst|dkrd|jj}t|dkrdd d |D}|d |d 7}t|dkrdd d |D}|d |d 7}t||||fS)a This is the same as `load_tf_weights` but for a sharded checkpoint. Detect missing and unexpected layers and load the TF weights from the shard file accordingly to their names and shapes. This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being loaded in the model. Args: model (`keras.models.Model`): The model in which to load the checkpoint. shard_files (`str` or `os.PathLike`): A list containing the sharded checkpoint names. ignore_mismatched_sizes`bool`, *optional`, defaults to `True`): Whether or not to ignore the mismatch between the sizes strict (`bool`, *optional*, defaults to `True`): Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint. Returns: Three lists, one for the missing layers, another one for the unexpected layers, and a last one for the mismatched layers. Nrrr ignore_mismatched_sizesrr#Error(s) in loading state_dict for ,cSg|]}d|dqS"rBrOrrBrBrC z+load_tf_sharded_weights.. Missing key(s): rcSr%r&rBr(rBrBrCr)r*)setrrrrralstriprradd load_tf_shardrmgccollectrrW RuntimeError)model shard_filesr"strictrunexpected_keys saved_keysmismatched_keys model_keysmodel_layer_maprr layer_namersaved_weight_names_setunexpected_keys_setmismatched_keys_set missing_keys error_messagestr_missing_keysstr_unexpected_keysrBrBrCload_tf_sharded_weightssJ          rCc Cst}i}t}t}zt|d~} tt| d} g} | D]j} | | } t| || <|| | |vr:|| q|j|| }|| }|durt ||j krz t |t |}Wn$t y~}z|rx|| |j t |fWYd}~q|d}~ww|}| ||fqWdn1swYt| |||fWSty}z?z#t|}|drtdt d|d|1swYWntt fytd|d |d wWYd}~dSd}~ww) a Loads a shard from a sharded checkpoint file. Can be either H5 or Safetensors. Handles missing keys and unexpected keys. Args: model (`keras.models.Model`): Model in which the weights are loaded model_layer_map (`Dict`): A dictionary mapping the layer name to the index of the layer in the model. resolved_archive_file (`str`): Path to the checkpoint file from which the weights will be loaded ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether to ignore the mismatched keys Returns: `keras.models.Model`: Three lists, one for the layers that were found and successfully restored (from the shard file), one for the mismatched layers, and another one for the unexpected layers. r layer_namesNversionYou seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.zUnable to locate the file z_ which is necessary to load this pretrained model. Make sure you have saved the model properly.z4Unable to load weights from TF checkpoint file for 'z' at 'z'. If you tried to load a TF model from a sharded checkpoint, you should try converting the model by loading it in pytorch and saving it locally. A convertion script should be released soon.)r,h5pyFilerrKasarrayr.rK int_shaper9rrbrbatch_set_value ExceptionopenreadrOSErrorUnicodeDecodeError)r3r:resolved_archive_filer"rr< saved_weightsr8r6sharded_checkpoint_filesaved_h5_model_layers_nameweight_value_tuplesr;h5_layer_objectsymbolic_weightsaved_weight_valuearrayefrBrBrCr/st     )    r/cCst}g}t}|D]"}t||||d\} } } |t| || || tq tj|} |r{t| dks@t|dkr{d|jj } t| dkr_d dd| D}| d|d7} t|dkrwd d d|D}| d|d7} t | | ||fS) a This is the same as `load_tf_weights_from_safetensors` but for a sharded TF-format safetensors checkpoint. Detect missing and unexpected layers and load the TF weights from the shard file accordingly to their names and shapes. This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being loaded in the model. Args: model (`keras.models.Model`): The model in which to load the checkpoint. shard_files (`str` or `os.PathLike`): A list containing the sharded checkpoint names. ignore_mismatched_sizes`bool`, *optional`, defaults to `True`): Whether or not to ignore the mismatch between the sizes strict (`bool`, *optional*, defaults to `True`): Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint. Returns: Three lists, one for the missing layers, another one for the unexpected layers, and a last one for the mismatched layers. r!rr#r$cSr%r&rBr(rBrBrCr)r*z.r+rcSr%r&rBr(rBrBrCr)r*) r, load_tf_weights_from_safetensorsrrmr0r1 intersectionrarrWrr2)r3r4r"r5rr6all_missing_keysr8rmissing_layersunexpected_layersmismatched_layersr?r@rArBrBrBrC(load_tf_sharded_weights_from_safetensorsWs2        rdcCs$|drt}nt}|||||dS)a Detect missing and unexpected layers and load the TF weights from the shard file accordingly to their names and shapes. Args: model (`keras.models.Model`): The model to load the weights into. resolved_archive_file (`str`): The location of the H5 file. ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether or not to ignore weights with shapes that don't match between the checkpoint of the model. Returns: Three lists, one for the missing layers, another one for the unexpected layers, and a last one for the mismatched layers. r r!)endswithr^load_tf_weights_from_h5)r3rSr"r load_functionrBrBrCload_tf_weightss rhc Csg}t|d}tt|d}tdd|jD|}t|dd|jD}t} t} g} |jD]} | j|vr|| j} | j| j}i}t| dD]&}d | ddd}|durf|d|}t | |||<| |qN|D]}|durt| d}d |j dd||j d|dd}n d |j ddd}||d}|dur|d r|dd d }||d}| ||durt||jkr z t |t|}Wn&ty }z|r|||jt|fWYd}~qw|d}~ww|}| ||fqwq4Wdn 1s!wYt| |t| | |t| | |||fS) NrDrEcSh|]}|jqSrBrrOlayerrBrBrC z*load_tf_weights_from_h5..cSrirBrjrkrBrBrCrmrn weight_namesrr z embeddings:0izweight:0)rHrIr,rr;layersrtrainable_weightsnon_trainable_weightsrrrKrJr.ragetrerKrLr9rrbrrMextend)r3rSr"rrcrUrVrarbr<symbolic_weights_namesrWrlrXsymbolic_weightsrTrrrY delimetersymbolic_weight_namerZr[r\rBrBrCrfsn             W rfc s.t|dd}g}fdd|jD}t|}tt|t|}tt|t|} |jD]R} t| jd} | |vr|| } t | | j kr|z t | t | } Wn&t t jjfy{} z|ru|| | j t | fWYd} ~ q0| d} ~ wwt| | q0Wdn1swY|| |fS)Nr= frameworkcsg|] }t|jdqS)r)rrrNr{rBrCr)sz4load_tf_weights_from_safetensors..r{)r0rr;rr,rr get_tensorrKrLr9r=rrberrorsInvalidArgumentErrorr set_value)r3rSr"rsafetensors_archivercroloaded_weight_namesrarbrr weight_valuer\rBr{rCr^ s2      r^cCst|\}}||}tj|drItj|td|gddggdd}t||}tt|dgd}tj|td|gddggdd}||fSt |tddgt||g}tt|dgd}||fS)aV This function aims to reduce the embeddings in case new_num_tokens < old_num_tokens or to pad with -1 in case new_num_tokens > old_num_tokens. A mask is also computed in order to know which weight in the embeddings should be kept or not. Example: - if new_num_tokens=5 and old_num_tokens=4 and old_embeddings=[w1,w2,w3,w4] - mask=[True,True,True,True,False] and current_weights=[w1,w2,w3,w4,-1] - if new_num_tokens=4 and old_num_tokens=5 and old_embeddings=[w1,w2,w3,w4,w5] - mask=[True,True,True,True] and current_weights=[w1,w2,w3,w4] rrconstant_valuesr TF) rr=rgreaterpadvalueconvert_to_tensorminfillslice)old_embeddingsnew_num_tokensold_num_tokensold_embedding_dim size_diffcurrent_weightsnum_tokens_to_copymaskrBrBrCinit_copy_embeddings-s"  "   rc seZdZdZdZdZdZdZdZdZ dZ dZ dZ e ddd Zd d Ze dd dZdddZfddZddZeejjfddZeejjfddZeejjfddZeejjfddZeejjfddZeejjfdd Zed!d"Zed#d$Z dd*d+Z!d,d-Z"e#j$d.d/Z%e dd1d2Z&d3d4Z'edd6d7Z(dd9d:Z)d;d<Z* = >     >dddLdMZ+ N O     dŇfdPdQ Z,fdRdSZ-dTdUZ.dVdWZ/dXdYZ0        dddddeZ1dfdgZ2ddidjZ3dkdlZ4ddmdnZ5ddpdqZ6ddsdtZ7dudvZ8ddwdxZ9 ddd|d}Z:ddddZ;ddZddZ?dddZ@ddZAddddZBdddZCddZD      ddddZEeddddddddddddZF      ddddZGedddZHZIS)TFPreTrainedModela Base class for all TF models. [`TFPreTrainedModel`] takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to: - resize the input embeddings, - prune heads in the self-attention heads. Class attributes (overridden by derived classes): - **config_class** ([`PretrainedConfig`]) -- A subclass of [`PretrainedConfig`] to use as configuration class for this model architecture. - **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. - **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP models, `pixel_values` for vision models and `input_values` for speech models). NrFrHDict[str, tf.Tensor]cCsi}|jD].\}}dd|jD}|jddurd|d<tj||jd||<|dkr5t||||<q|jjrbdt |j j vrbd|vrb|j d kr^tjdd |jjftjdd |d<|Std |S) z| Dummy inputs to build the network. Returns: `Dict[str, tf.Tensor]`: The dummy inputs. cSsg|] }|dur |ndqS)Nr6rB)rOdimrBrBrCr)z2TFPreTrainedModel.dummy_inputs..rNr )r9rtoken_type_idsencoder_hidden_statesrr6r9rrzModel has cross-attention but we couldn't infer the shape for the encoder hidden states. Please manually override dummy_inputs!)input_signaturerr9r=onesr zeros_liker\add_cross_attentionrrcallrr hidden_sizefloat32NotImplementedError)rUdummiesrspec dummy_shaperBrBrC dummy_inputsxs(  zTFPreTrainedModel.dummy_inputscCs>t|j|jddWddS1swYdS)N) input_shape)r= name_scoperbuildrUrBrBrCbuild_in_name_scopes"z%TFPreTrainedModel.build_in_name_scopercCdS)zC :str: Identifies that this is a TensorFlow model. r=rBrrBrBrCrzszTFPreTrainedModel.frameworkcCsdSrJrBrUrrBrBrCrszTFPreTrainedModel.buildcsptj|i|t|tstd|jjd|jjd||_|j|_| r-t |nd|_ | |jdS)NzParameter config in `zt(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = z(.from_pretrained(PRETRAINED_MODEL_NAME)`)rkrrr]rrxrrWr\ name_or_path can_generaterfrom_model_configgeneration_config_set_save_specr)rUr\rrfrrBrCrrs  zTFPreTrainedModel.__init__cCs |jSrJ)r\rlrrBrBrCris zTFPreTrainedModel.get_configc$t|i|\}}tj|i|SrJ)rrkfitrUrerfrrBrCrzTFPreTrainedModel.fitcrrJ)rrktrain_on_batchrrrBrCrrz TFPreTrainedModel.train_on_batchcrrJ)rrk test_on_batchrrrBrCrrzTFPreTrainedModel.test_on_batchcrrJ)rrkpredict_on_batchrrrBrCrrz"TFPreTrainedModel.predict_on_batchcrrJ)rrkpredictrrrBrCrrzTFPreTrainedModel.predictcrrJ)rrkevaluaterrrBrCrrzTFPreTrainedModel.evaluatecKs6t|tr|j|fi|S||jj|fi|SrJ)r]r _from_configr[r`rpr\rfrBrBrC from_configs zTFPreTrainedModel.from_configcKs||fi|S)zZ All context managers that the model should be initialized under go here. rBrrBrBrCrszTFPreTrainedModel._from_config head_masktf.Tensor | Nonenum_hidden_layersrI tf.TensorcCs&|dur |||}|Sdg|}|S)a$ Prepare the head mask if needed. Args: head_mask (`tf.Tensor` with shape `[num_heads]` or `[num_hidden_layers x num_heads]`, *optional*): The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard). num_hidden_layers (`int`): The number of hidden layers in the model. Returns: `tf.Tensor` with shape `[num_hidden_layers x batch x num_heads x seq_length x seq_length]` or list with `[None]` for each layer. N)_convert_head_mask_to_5drUrrrBrBrC get_head_masks   zTFPreTrainedModel.get_head_maskcCs|jjdkr|ddddddf}tj||dd}n|jjdkr-|dddddddf}|jjdks [num_hidden_layers x batch x num_heads x seq_length x seq_length]r Nr)repeatsr8r6zhead_mask.dim != 5, instead )r9r:r=repeatrrrrrBrBrCrs  z*TFPreTrainedModel._convert_head_mask_to_5dcCs||}||S)a6 Args: Method used for serving the model. Does not have a specific signature, but will be specialized as concrete functions when saving with `save_pretrained`. inputs (`Dict[str, tf.Tensor]`): The input of the saved model as a dictionary of tensors. )rserving_output)rUrrrBrBrCservings zTFPreTrainedModel.servingDict[str, tf.TensorSpec]cCstt|jj}i}d|vr2|jjdrd}nd}dD]}||vr1tj dg|tj |d||<qd|vrgd }t |j d rE|j j }n|j }t |d rS|j|d <ntd t |drf|j|d<|d<nt |dru|j|d<|d<ntdtj |tjdd|d<d|vrtd|S)z This property should return a dict mapping input names to tf.TensorSpec objects, representing the expected shape and dtype for model inputs. It is used for both serving and for generating dummy inputs. rForMultipleChoicer6)rattention_maskrdecoder_input_idsdecoder_attention_maskNrj pixel_values)NNNN vision_config num_channelsr zhCould not infer number of channels from config, please override input_signature to specify input shapes. image_size input_sizezgCould not infer input image shape from config, please override input_signature to specify input shapes.input_featuresz4Audio models need a manually defined input_signature)r;rrrrrrWrer= TensorSpecrrwr\rrrrrr)rU model_inputssig text_dims input_namepixel_values_shaperrBrBrCrs>      z!TFPreTrainedModel.input_signaturec Cst|ts|S|D]l}|drt|jddsd||<n7|dr-t|jdds-d||<n&|dkr=t|jdds=d||<n|d krSt|jddrOt|jd dsSd||<t||ttfruz t||||<Wq t tj j fytYq wq |S) zz Prepare the output of the saved model. Can be overridden if specific serving modifications are required. hidden_statesrFN attentionsrrrcross_attentionsr) r]r!rersr\rr;r=rrbr}r~)rUrrrBrBrCr=s.      z TFPreTrainedModel.serving_outputrGcCs$dt|jvrdt|jvrdSdS)z Returns whether this model can generate sequences with `.generate()`. Returns: `bool`: Whether this model can generate sequences with `.generate()`. GenerationMixinFT)rprepare_inputs_for_generationgeneraterorBrBrCrUs zTFPreTrainedModel.can_generatekeras.layers.LayercCs"t||j|}||ur|St)z Returns the model's input embeddings layer. Returns: `tf.Variable`: The embeddings layer mapping vocabulary to hidden states. )rsbase_model_prefixget_input_embeddingsr)rU main_layerrBrBrCrcsz&TFPreTrainedModel.get_input_embeddingscCstj|s t|tj|d}||||jd}tj|d}t|d}t ||WddS1s=wYdS)Nz weights.h5)epochoptimizer_statezextra_data.picklewb) ospathisdirmkdirr save_weights optimizer get_weightsrOpickledump)rUcheckpoint_dirr weights_path extra_dataextra_data_pathr]rBrBrC_save_checkpointqs    "z"TFPreTrainedModel._save_checkpointTdataset'datasets.Dataset' batch_sizeshuffle tokenizer#Optional['PreTrainedTokenizerBase'] collate_fnOptional[Callable]collate_fn_argsOptional[Dict[str, Any]]drop_remainderOptional[bool]prefetchc  slt|dgddl} |dur|durtdd}nt|dd}|dur$i}t|| js.tdtt |j j t |j dtt |jj vrV|j|d||d \} } nfd d |jD} || }|j|d||d \} } t| } fd d | D}fdd | D}t|dkr|dn|}t|dkr|dn|}|dur|}|j||||||||d}|S)a Wraps a HuggingFace [`~datasets.Dataset`] as a `tf.data.Dataset` with collation and batching. This method is designed to create a "ready-to-use" dataset that can be passed directly to Keras methods like `fit()` without further modification. The method will drop columns from the dataset if they don't match input names for the model. If you want to specify the column names to return rather than using the names that match this model, we recommend using `Dataset.to_tf_dataset()` instead. Args: dataset (`Any`): A [~`datasets.Dataset`] to be wrapped as a `tf.data.Dataset`. batch_size (`int`, *optional*, defaults to 8): The size of batches to return. shuffle (`bool`, defaults to `True`): Whether to return samples from the dataset in random order. Usually `True` for training datasets and `False` for validation/test datasets. tokenizer ([`PreTrainedTokenizerBase`], *optional*): A `PreTrainedTokenizer` that will be used to pad samples to create batches. Has no effect if a specific `collate_fn` is passed instead. collate_fn (`Callable`, *optional*): A function that collates samples from the dataset into a single batch. Defaults to `DefaultDataCollator` if no `tokenizer` is supplied or `DataCollatorWithPadding` if a `tokenizer` is passed. collate_fn_args (`Dict[str, Any]`, *optional*): A dict of arguments to pass to the `collate_fn` alongside the list of samples. drop_remainder (`bool`, *optional*): Whether to drop the final batch, if the batch_size does not evenly divide the dataset length. Defaults to the same setting as `shuffle`. prefetch (`bool`, defaults to `True`): Whether to add prefetching to the end of the `tf.data` pipeline. This is almost always beneficial for performance, but can be disabled in edge cases. Returns: `Dataset`: A `tf.data.Dataset` which is ready to pass to the Keras API. datasetsrNrK)return_tensors)rr z.Dataset argument should be a datasets.Dataset!cols_to_retain)rrrr cs g|] }|vr|dvr|qS)) label_idslabelrB)rOfeature)rrBrCr)s z8TFPreTrainedModel.prepare_tf_dataset..)rrrcs g|] }|vr|vr|qSrBrBrOcolr model_labelsrBrCr) csg|]}|vr|qSrBrBr)rrBrCr)r*r )columns label_colsrrrrrr)r,r rrr]Datasetrxr;rrrrr%r_get_output_signaturerfeaturesremove_columnsra to_tf_dataset)rUrrrrrrrrr output_signature_unwanted_columnsoutput_columns feature_colsr tf_datasetrBrrCprepare_tf_dataset~sZ .        z$TFPreTrainedModel.prepare_tf_datasetrmspropauto_with_warningc s|dvr tdd}|dkrt}d|_nd|_tttjj j } d| vr;t j d |||||||d|d St j d |||||||d|d S) z This is a thin wrapper that sets the model's loss output head as the loss if the user does not specify a loss function themselves. )r# passthroughaNo loss specified in compile() - the model's internal loss computation will be used as the loss. Don't panic - this is a common way to train TensorFlow models in Transformers! To disable this behaviour please pass a loss argument, or explicitly pass `loss=None` if you do not want your model to compute a loss. You can also specify `loss='auto'` to get the internal loss without printing this info string.autoTFsteps_per_execution)rlossmetrics loss_weightsweighted_metrics run_eagerlyr&)rr'r(r)r*r+ experimental_steps_per_executionNrB) rinforD_using_dummy_lossr;rrrzModelcompilerrrk) rUrr'r(r)r*r+r&rf parent_argsrrBrCr0sB   zTFPreTrainedModel.compilecs:ttjdrtj|i|Stdt|j|i|S)N compute_lossa0The old compute_loss method is deprecated as it conflicts with the Keras compute_loss method added in TF 2.8. If you want the original HF compute_loss, please call hf_compute_loss() instead. From TF versions >= 2.8, or Transformers versions >= 5, calling compute_loss() will get the Keras method instead.) rwrzr/rkr2rrrrrrrBrCr2s zTFPreTrainedModel.compute_losscCsntt|jj}|jdur|jSd|vrdddSd|vr#dddSd |vr,dd d Sd |vr5d ddSiS)Nstart_positions start_logits end_logits)r3 end_positionssentence_order_labelprediction_logits sop_logits)rr7rseq_relationship_logits)rr mc_labelsr mc_logits)rr;)r;rrrr_label_to_output_map)rU arg_namesrBrBrC get_label_to_output_name_mapping+s     z2TFPreTrainedModel.get_label_to_output_name_mappingcstt|jj}t|j|ddD}|j s+t t j t dkr+t |}tj|\}}t|tr=|}ttrF|j rdurtdkrttt jrtt|t jre|d|i}tt}||vrs||<n9ttrt|t jr|d|i}D]#\}}||vr||vr|||<q||d|vr||vr||||<qdurŇfdd|Ds|j stdttrՇfd dDt } |j rd |vr||d d d } n||d d } |j r|j| j| j||jd} nd} ttrAtdkrAtd| vr&| td} nt| ddkr6| d} n| d} \} nSttrSfdd| D} nAtt s_ttrt| ddkrs| !dd} n| !} | dt} nt| ddkr| d} n| d} | dur|j| ||jd} Wdn 1swY|j"j#| |j$| d|j%&| |i} |j'D]}|(}t|tr| )|q|| |j*<q| S) A modification of Keras's default `train_step` that correctly handles matching outputs to labels for our models and supports directly training on the loss output head. In addition, it ensures input keys are copied to the labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure that they are available to the model during the forward pass. cSi|]\}}||qSrBrBrrBrBrCrFz0TFPreTrainedModel.train_step..2.11.0Nr rci|] \}}|vr||qSrBrBr label_kwargsrBrCrgrRCould not find label column(s) in input dict and no separate labels were provided!ci|] \}}|||qSrBrsrlabel_to_outputrBrCrmr return_lossT)trainingrLrMregularization_lossesr'crDrBrBryrBrCrr)tape)+r;rrrrr%rr?rr.r r= __version__rrzr{unpack_x_y_sample_weightr]r_copyrarnextiterrsrb GradientTape compiled_lossr'rrpopitemrto_tuplerminimizerScompiled_metrics update_stater(resultrmr)rUdatar>output_to_labelx sample_weight label_kwargrrrSr@r'rreturn_metricsmetricr`rBrFrKrRrC train_step:s                   )  zTFPreTrainedModel.train_stepcstt|jj}t|j|ddD}|j s+t t j t dkr+t |}tj|\}}t|tr=|}ttrF|j rdurtt|jj}tdkr}tt jr}t|t jrn|d|i}tt}||vr|||<n9ttrt|t jr|d|i}D]#\}}||vr||vr|||<q||d|vr||vr||||<qdur·fdd|Ds|j stdttrއfd dD|j rd |vr||d d d } n||d d} |j r|j| j| j||jd} nd} ttrFtdkrFtd| vr+| td} nt| ddkr;| d} n| d} \} nSttrXfdd| D} nAttsdttrt| ddkrx| dd} n| } | dt} nt| ddkr| d} n| d} | dur|j| ||jd} |j!"| |i} |j#D]} | $}t|tr| %|q|| | j&<q| S)r@cSrArBrBrrBrBrCrrBz/TFPreTrainedModel.test_step..rCNr rcrDrBrBrrErBrCrrrGcrHrBrIrrJrBrCrrrLTF)rLrMrNrOr'crDrBrBrrQrBrCrr)'r;rrrrr%rr?rr.r r=rTrrzr{rUr]r_rVrarrWrXrsrbrZr'rrr[rr\r^r_r(r`rmr)rUrar>rbrcrdrerrr@r'rrfrgr`rBrhrC test_steps                    zTFPreTrainedModel.test_step model_namelanguage Optional[str]licensetagsfinetuned_fromtasks dataset_tagsOptional[Union[str, List[str]]] dataset_argsc  Cszddlm} | j||j|||||||| | d } | } ttj|dd}| | WddS1s6wYdS)at Creates a draft of a model card using the information available to the `Trainer`. Args: output_dir (`str` or `os.PathLike`): The folder in which to create the model card. model_name (`str`, *optional*): The name of the model. language (`str`, *optional*): The language of the model (if applicable) license (`str`, *optional*): The license of the model. Will default to the license of the pretrained model used, if the original model given to the `Trainer` comes from a repo on the Hub. tags (`str` or `List[str]`, *optional*): Some tags to be included in the metadata of the model card. finetuned_from (`str`, *optional*): The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo of the original model given to the `Trainer` (if it comes from the Hub). tasks (`str` or `List[str]`, *optional*): One or several task identifiers, to be included in the metadata of the model card. dataset_tags (`str` or `List[str]`, *optional*): One or several dataset tags, to be included in the metadata of the model card. dataset (`str` or `List[str]`, *optional*): One or several dataset identifiers, to be included in the metadata of the model card. dataset_args (`str` or `List[str]`, *optional*): One or several dataset arguments, to be included in the metadata of the model card. r )TrainingSummary) keras_historyrlrnrorkrprqrrrrtz README.mdrPN) modelcardru from_kerashistory to_model_cardrOrrrwrite)rU output_dirrkrlrnrorprqrrrrtrutraining_summary model_cardr]rBrBrCcreate_model_cards$ )  "z#TFPreTrainedModel.create_model_cardcCs^t||j}|durtdz||WdSty.td|||YdSw)z Set model's input embeddings Args: value (`tf.Variable`): The new weights mapping hidden states to vocabulary. Nz>The model does not implements the base_model_prefix attribute.Building the model)rsrrset_input_embeddingsrtrr-r)rUrrrBrBrCrJs   z&TFPreTrainedModel.set_input_embeddingsUnion[None, keras.layers.Layer]cCsR|dur'|}z|WSty&td||YSwdS)z Returns the model's output embeddings Returns: `tf.Variable`: The new weights mapping vocabulary to hidden states. Nr) get_lm_headget_output_embeddingsrtrr-rrUlm_headrBrBrCr^s    z'TFPreTrainedModel.get_output_embeddingscCsZ|dur+|}z||WdSty*td|||YdSwdS)z Set model's output embeddings Args: value (`tf.Variable`): The new weights mapping hidden states to vocabulary. Nr)rset_output_embeddingsrtrr-rrUrrrBrBrCrrs   z'TFPreTrainedModel.set_output_embeddingscCstdt|S)z Get the layer that handles a bias attribute in case the model has an LM head with weights tied to the embeddings Return: `keras.layers.Layer`: The layer that handles the bias, None if not an LM model. zVThe method get_output_layer_with_bias is deprecated. Please use `get_lm_head` instead.)rrrrrrBrBrCget_output_layer_with_biassz,TFPreTrainedModel.get_output_layer_with_biasUnion[None, str]cCstdtdS)z Get the concatenated _prefix name of the bias from the model name to the parent layer Return: `str`: The _prefix name of the bias. zMThe method get_prefix_bias_name is deprecated. Please use `get_bias` instead.N)rrrrrBrBrCget_prefix_bias_names z&TFPreTrainedModel.get_prefix_bias_name#Union[None, Dict[str, tf.Variable]]cCsF|dur!|}z|WSty ||YSwdS)z Dict of bias attached to an LM head. The key represents the name of the bias attribute. Return: `tf.Variable`: The weights representing the bias, None if not an LM model. N)rget_biasrtrrrBrBrCrs    zTFPreTrainedModel.get_biascCsP|dur&|}z||WdSty%|||YdSwdS)z Set all the bias in the LM head. Args: value (`Dict[tf.Variable]`): All the new bias attached to an LM head. N)rset_biasrtrrrBrBrCrs  zTFPreTrainedModel.set_biascCr)z The LM Head layer. This method must be overwritten by all the models that have a lm head. Return: `keras.layers.Layer`: The LM head layer if the model has one, None if not. NrBrrBrBrCrszTFPreTrainedModel.get_lm_headr Optional[int]*Union[keras.layers.Embedding, tf.Variable]cCsTt|tjjr||S|dus||jjkr||S| |}||j_|S)a Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method. Arguments: new_num_tokens (`int`, *optional*): The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens without doing anything. Return: `tf.Variable` or `keras.layers.Embedding`: Pointer to the input tokens of the model. N) r]rrzrp Embedding_v2_resized_token_embeddingsr\ vocab_size_get_word_embedding_weight_resize_token_embeddingsrUr model_embedsrBrBrCresize_token_embeddingss  z)TFPreTrainedModel.resize_token_embeddingskeras.layers.EmbeddingcCs2|dus ||jjkr|S||}||j_|S)aJ Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Arguments: new_num_tokens (`int`, *optional*): The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens without doing anything. Return: `keras.layers.Embedding`: Pointer to the input tokens of the model. N)r\rr_v2_resize_token_embeddingsrrBrBrCrs   z.TFPreTrainedModel._v2_resized_token_embeddingscCs|t|tjr|St|dd}|dur|St|dd}|dur |S|t|dd}|dur0|St|dd}|dur<|SdS)Nrdecoder)r]r=rrsr)r3embedding_layerembedsrBrBrCrs     z,TFPreTrainedModel._get_word_embedding_weightcCs||}|||}|dur"|}|||}|||dur:||}|||}||| ||SrJ) rr_get_resized_embeddingsr_get_resized_lm_head_biasrr_get_resized_lm_head_decoderrr)rUrrnew_embeddingsold_lm_head_biasnew_lm_head_biasold_lm_head_decodernew_lm_head_decoderrBrBrCrs        z*TFPreTrainedModel._resize_token_embeddingsc Cs|}|||}|||dur$|}|||}||||k}|durF|sF||}|||}| ||SrJ) r_v2_get_resized_embeddingsrr_v2_get_resized_lm_head_biasrrrrr) rUrrrrr tied_weightsrrrBrBrCr2s       z-TFPreTrainedModel._v2_resize_token_embeddingscCs~i}|D]\}}t|dkrdt|dfnt|\}}||}|dur*|gn||g} tj|drx|dur>d|ggnddgd|gg} tj|t| dd} t ||} |dur_| gnd| g} t t| d}tj|t| dd}n#|durdgnddg}t |t|t| } t t| d}|j | dd|j d dd }t|| |}|||||<q|S) a Build a resized bias from the old ones. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_lm_head_bias (`tf.Variable`): Old lm head bias to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the linear matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns None Return: `tf.Variable`: Pointer to the resized bias. r NrrrTFzerosrr9rg trainabler)rr=r:rrrrrrrrr add_weightrrwhereassign)rUrrrattrr first_dimrr final_shape padding_shape current_biasr mask_shape bias_mask slice_fromnew_bias init_biasrBrBrCrGs6*"   z+TFPreTrainedModel._get_resized_lm_head_biasrDict[str, tf.Variable]c Csi}|D]L\}}t|dkrdt|dfnt|\}}||}||kr2|dd|f} n|dur;d|ggnddgd|gg} t|t| } | ||<q|S)a\ Build a resized bias from the old ones. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_lm_head_bias (`Dict[str, tf.Variable]`): Old lm head bias to be resized. new_num_tokens (`int`): New number of tokens in the linear matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. Return: `tf.Tensor`: Values for the resized bias. r Nr.)rr=r:rrrr) rUrrrrrrrrrrrBrBrCr|s*" z.TFPreTrainedModel._v2_get_resized_lm_head_biasc Cs|}t|||k}|dur@|s@t|d}t||\}}|j||fdd|jddd}t ||| }| ||S)a Build a resized decoder from the old ones. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_lm_head_decoder (`tf.Variable`): Old lm head decoder to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the linear matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns None Return: `tf.Variable`: Pointer to the resized decoder or None if the output embeddings are different from the input ones. Nr rTrrr) r=rrrrrrrrrrr) rUrrris_input_output_equalsr decoder_maskcurrent_decoder init_decoderrBrBrCrs    z.TFPreTrainedModel._get_resized_lm_head_decoder tf.Variablec Cspt|d}t|jdd}t||\}}|j|jdd||gt|tj d}t ||| }| ||S)as Build a resized Embedding weights from a provided token Embedding weights. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_embeddings (`tf.Variable`): Old embeddings to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens `tf.Variable` module of the model without doing anything. Return: `tf.Variable`: Pointer to the resized Embedding Module or the old Embedding Module if `new_num_tokens` is `None` r initializer_range{Gz?rr)rr9rgr) rrsr\rrrrget_initializerr=rrrr) rUrrr init_rangeembeddings_maskcurrent_embeddingsrinit_embeddingsrBrBrCrs  z)TFPreTrainedModel._get_resized_embeddingsrcCsd}gd}|D]}t|j|rt|j|}qtjj||jtjj|d|j j ddd}|t dgg|j |krC|j d|}nt j|j |j |j dgdd}|j ||S) a Build a resized Embedding layer from a provided Embedding layer. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. Args: old_embeddings (`keras.layers.Embedding`): Old embeddings to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the embedding matrix. Return: `keras.layers.Embedding`: Resized Embedding layer. r)rinitializer_factorinit_stdstddevNi) input_dim output_dimembeddings_initializerrrr7)rwr\rsrzrprr initializersTruncatedNormal embeddingsrr=constantrconcatr)rUrrr'potential_initialization_variable_namesvar_namerrrBrBrCrs(     z,TFPreTrainedModel._v2_get_resized_embeddingscCst)a Prunes heads of the base model. Arguments: heads_to_prune (`Dict[int, List[int]]`): Dictionary with keys being selected layer indices (`int`) and associated values being the list of heads to prune in said layer (list of `int`). For instance {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on layer 1 and heads 2 and 3 on layer 2. )r)rUheads_to_prunerBrBrC prune_heads s zTFPreTrainedModel.prune_headsr 5GBrUnion[int, str] create_prsafe_serializationtokenOptional[Union[str, bool]]c % Ks| dd} | durtdt| durtd| } | dur"| | d<tj|r3t d|ddStj |dd |r]| d d} | d | tjj d } |j | fi| } ||}|rt|jd ddur|t|jjts|t|jj dd|j_|dur|j|j}tdd|jDrdd|jD}|j|}||d}n|}tj|dt|}|j|d|dtd||jjddg|j_|j durt!|||jd|j"||#r|j$"||rt%nt&}tj||}t'|j(||d\}}t)|D]-}tj||}|*dd*dd}|+|r5tj|r5||,vr5t-|q |dur_|rQdd|j(D}t.||d d!id"n|/|td#|n|rdt0nt1}tj||}t2|d$d%d&}t3j4|ddd'd(}|5|Wdn 1swYtd)|d*t6|d+|d|D]\}}|rd,d|D} t.| tj||d d!id"qt7j8tj||d$d-\}g}!t9|d.d/d0D]C}"d1|"j:vst6|"j: d2dkr|"j:}#n d2|"j: d2dd}#|j;|#|"<j=|"<j>d3}$|"<|$dd<|!?|#@d4qtA|d5|!Wdn 1s3wYq|rI|jB|| || | d6dSdS)7a< Save a model and its configuration file to a directory, so that it can be re-loaded using the [`~TFPreTrainedModel.from_pretrained`] class method. Arguments: save_directory (`str`): Directory to which to save. Will be created if it doesn't exist. saved_model (`bool`, *optional*, defaults to `False`): If the model has to be saved in saved model format as well or not. version (`int`, *optional*, defaults to 1): The version of the saved model. A saved model needs to be versioned in order to be properly loaded by TensorFlow Serving as detailed in the official documentation https://www.tensorflow.org/tfx/serving/serving_basic push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). signatures (`dict` or `tf.function`, *optional*): Model's signature used for serving. This will be passed to the `signatures` argument of model.save(). max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard which will be bigger than `max_shard_size`. create_pr (`bool`, *optional*, defaults to `False`): Whether or not to create a PR with the uploaded files or directly commit. safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save the model using `safetensors` or the traditional TensorFlow way (that uses `h5`). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.rzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_idr torch_dtyperr css|] }|jtjkVqdSrJ)rr=r)rOrrBrBrCrQz sz4TFPreTrainedModel.save_pretrained..cSs:i|]\}}|tj|j|jtjkrtjn|j|jdqS)r)r=rr9rrrr)rOrrrBrBrCr{ s z5TFPreTrainedModel.save_pretrained..)serving_default int64_serving saved_modelF)include_optimizer signatureszSaved model created in r6)r\)rz.binrr cSi|] }t|j|qSrBrrrrNrBrBrCr rformatr=)r zModel weights saved in rPzutf-8)encoding)indent sort_keys z:The model is bigger than the maximum size per checkpoint (z) and is going to be split in z^ checkpoint shards. You can find where each parameters has been saved in the index located at cSrrBrrNrBrBrCr r)modecSs|jSrJrj)rcrBrBrC sz3TFPreTrainedModel.save_pretrained..)rrrrutf8rE)rr)Cr^rrrrbrrisfilererrormakedirsrsep _create_repo_get_files_timestampsrsr\r]rrrget_concrete_functionranyvaluesrrsaver-rrW architectures _auto_classrsave_pretrainedrrrrr rlistdirrrrremovesafe_save_filerrrrOjsondumpsr{rarHrIsortedrcreate_datasetrr9rrencoder_upload_modified_files)%rUsave_directoryrrF push_to_hubrrrrrrfrrrfiles_timestampsr int64_specrsaved_model_dirroutput_model_filerrfilename full_filenameweights_no_suffix state_dictsave_index_file index_filecontentrrshard_state_dictrprlr; param_dsetrBrBrCr" s 5                " z!TFPreTrainedModel.save_pretrainedmain)r\ cache_dirr"force_downloadlocal_files_onlyrrevisionuse_safetensorspretrained_model_name_or_path!Optional[Union[str, os.PathLike]]r\3Optional[Union[PretrainedConfig, str, os.PathLike]]rr"rrrrc3s | dd} | dd} | dd}| dd}| dd}| dd}| d d}| d d}| d d}| d d}| d d}| dd}| dd}| dd}|durhtdt|durftd|}|durqtddd|d}|dur||d<tr|stdd}| durt sd} t |t s|dur|n|}|j j |f|d|| |||||||d | \}}n| }|durt|dd}d}|durt|}tj|}|r| rtjtj|trtj|t}n| r tjtj|tr tj|t}d}n| dur$tjtj|tr$tj|t}n| dur@tjtj|tr@tj|t}d}nptjtj|trUtj|t}n[tjtj|trltj|t}d}nD| r}tdtdtd|dtjtj|tstjtj|trtdtdtd|d tdtd!tdtd|d" tj|r|}d}ntj|d#r|d#}d}nt|r|}t|} n| rt}n | durt}nt}z|||| |||||dd|d$ }!t||fi|!} | dur|tkrt}t|tfi|!} | dur4|tkr4t|tfi|!} | dur4d}| durN|tkrNt|tfi|!} | durNd}| dur|||||d%}"t|tfi|"rhd}n#t|tfi|"r|t|d&td t|d&td!tdt Wn#tyt!ytd'|d(|d)td!tdt w|rtd*||} | "tjj#d+}ntd*|d,| nd} |rt$|| |||| |||||d- \} }#d}$|tkr(t%| d.d/ }%|%&}&Wdn 1swY|&dus|&'d0d1vr t(d2| d3|&'d0d4k}$n>|tkrft%| d5d.d/ }%|%&}&Wdn 1sEwY|&dusW|&'d0d1vr_t(d2| d3|&'d0d4k}$||_)|j*r|'d6dur|d7|'d6|d <||g| Ri|}'|durt+|'dr|'j,}| rd8d9l-m.}(|(|'| d|||d:S|durt/j0j12| |'3Wdn 1swYn|'3|$r|sd8d;l-m4})t%| d.d/}*|)|'|*dd||||d<WdS1swYn|$rd8d=l-m5}+|+|'| dd||||d<Sz>|rI| D]},tj|,d>|,fq|tkr=t6|'| ||d?\}-}.}/nt7|'| ||d?\}-}.}/n t8|'| ||d?\}-}.}/WnAt(y}0z4zt9| }%|%:;d@rpt(dAt|01swwYWntdur|j>D] fdEdD|.D}.qt?|.d5krtdF|dG|'j@jAdH|.dI|'j@jAdJ|'j@jAdK n tdL|'j@jAdMt?|-d5kr tdN|'j@jAdO|dP|-dQnt?|/d5kr$tdR|'j@jAdS|dT|'j@jAdUt?|/d5krFdVdWdD|/D}1tdX|'j@jAdO|dY|1dQ|'BrrztCj |f||| |||||||dZ | |'_DWnt(yqtd[Ynw|r|-|.|/d\}2|'|2fS|'S)]aS! Instantiate a pretrained TF 2.0 model from a pre-trained model configuration. The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task. The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those weights are discarded. Parameters: pretrained_model_name_or_path (`str`, *optional*): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~TFPreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *PyTorch state_dict save file* (e.g, `./pt_model/pytorch_model.bin`). In this case, `from_pt` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards. - `None` if you are both providing the configuration and state dictionary (resp. with keyword arguments `config` and `state_dict`). model_args (sequence of positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. config (`Union[PretrainedConfig, str]`, *optional*): Can be either: - an instance of a class derived from [`PretrainedConfig`], - a string valid as input to [`~PretrainedConfig.from_pretrained`]. Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~TFPreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch state_dict save file (see docstring of `pretrained_model_name_or_path` argument). ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). cache_dir (`str`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies: (`Dict[str, str], `optional`): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (e.g., not try downloading the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. tf_to_pt_weight_rename (`Callable`, *optional*): A function that is called to transform the names of weights during the PyTorch to TensorFlow crossloading process. This is not necessary for most models, but is useful to allow composite models to be crossloaded correctly. use_safetensors (`bool`, *optional*, defaults to `None`): Whether or not to use `safetensors` checkpoints. Defaults to `None`. If not specified and `safetensors` is not installed, it will be set to `False`. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Examples: ```python >>> from transformers import BertConfig, TFBertModel >>> # Download model and configuration from huggingface.co and cache. >>> model = TFBertModel.from_pretrained("google-bert/bert-base-uncased") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model = TFBertModel.from_pretrained("./test/saved_model/") >>> # Update configuration during loading. >>> model = TFBertModel.from_pretrained("google-bert/bert-base-uncased", output_attentions=True) >>> assert model.config.output_attentions == True >>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable). >>> config = BertConfig.from_json_file("./pt_model/my_pt_model_config.json") >>> model = TFBertModel.from_pretrained("./pt_model/my_pytorch_model.bin", from_pt=True, config=config) ```from_ptFresume_downloadNproxiesoutput_loading_infortrust_remote_codemirrorload_weight_prefix_from_pipeline _from_auto subfolderr _commit_hashtf_to_pt_weight_renameadapter_kwargsrrTzgThe argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.r3 tensorflow) file_typerzfrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=True) rreturn_unused_kwargsrr!r"rrrr(r'r*zError no file named z or z found in directory zu. Please make sure that the model has been saved with `safe_serialization=True` or do not set `use_safetensors=True`.zc but there is a file for PyTorch weights. Use `from_pt=True` to load this model from those weights.z, rz.index) rrr"r!rr user_agentrr) _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entriesr*)rr"rrrz& does not appear to have a file named zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named zloading weights file rz from cache at ) rrr"r!rrr2rr*r=ryr)ptr=flaxmlxz"The safetensors archive passed at zf does not contain the valid metadata. Make sure you save your model with the `save_pretrained` method.r5rrrr )$load_pytorch_checkpoint_in_tf2_model)allow_missing_keysr#rr+)$load_pytorch_state_dict_in_tf2_model) tf_inputsr9r#rr"r+)-load_sharded_pytorch_safetensors_in_tf2_modelzError retrieving files r!rFrGz}Unable to load weights from h5 file. If you tried to load a TF 2.0 model from a PyTorch checkpoint, please set from_pt=True. c g|] }t|dur|qSrJresearchr(patrBrCr) rz5TFPreTrainedModel.from_pretrained..cr=rJr>r(rArBrCr) rz)Some layers from the model checkpoint at z! were not used when initializing z: z, - This IS expected if you are initializing z from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing z from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).z8All model checkpoint layers were used when initializing z. zSome layers of z3 were not initialized from the model checkpoint at z and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the layers of z/ were initialized from the model checkpoint at zf. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.rc Ss*g|]\}}}d|d|d|dqS)z- z: found shape z in the checkpoint and z in the model instantiatedrB)rOrshape1shape2rBrBrCr) szSome weights of z= and are newly initialized because the shapes did not match: ) rrr!r"rrrr)r(r'zZGeneration config file not found, using a generation config created from the model config.)r?r6r8)Er^rrrrbrrr'r-r)r]rr[from_pretrainedrsrrrrrrr rrrrrEnvironmentErrorr(r$r#r&rrNrrr/r0r rsrQr_requires_load_weight_prefixrwr+modeling_tf_pytorch_utilsr8r=compatv1variable_scoperr:r<rdrCrhrOrPrrR_keys_to_ignore_on_load_missing"_keys_to_ignore_on_load_unexpectedrarrWrrr)3rprr\rr"rrrrr model_argsrfr r!r"r#rr$rr& from_pipeliner/r) commit_hashr+r2 config_path model_kwargs is_shardedis_local archive_filerrScached_file_kwargshas_file_kwargssharded_metadatasafetensors_from_ptr]safetensors_metadatar3r8r:rr<filer?r6r8r\mismatched_warning loading_inforBrArCrE s                                            $                 z!TFPreTrainedModel.from_pretrainedrr use_temp_dirrprivateOptional[Union[int, str]]Optional[Union[bool, str]]rc  KsV|durtdt|durtd|}d| vr"td| d}| dd} | dd} tj|r@|} |tjj d}n|d d} |j |||| | d }|dur\tj|  }t | |d ?} | | }|j | |d t|d rt|dr| t|jd} | | |jdi| |j| |||||dWddS1swYdS)u Upload the model files to the 🤗 Model Hub while synchronizing a local clone of the repo in `repo_path_or_name`. Parameters: repo_id (`str`): The name of the repository you want to push your model to. It should contain your organization name when pushing to a given organization. use_temp_dir (`bool`, *optional*): Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default to `True` if there is no directory named like `repo_id`, `False` otherwise. commit_message (`str`, *optional*): Message to commit while pushing. Will default to `"Upload model"`. private (`bool`, *optional*): Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists. token (`bool` or `str`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). Will default to `True` if `repo_url` is not specified. max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). create_pr (`bool`, *optional*, defaults to `False`): Whether or not to create a PR with the uploaded files or directly commit. Examples: ```python from transformers import TFAutoModel model = TFAutoModel.from_pretrained("google-bert/bert-base-cased") # Push the model to your namespace with the name "my-finetuned-bert". model.push_to_hub("my-finetuned-bert") # Push the model to an organization with the name "my-finetuned-bert". model.push_to_hub("huggingface/my-finetuned-bert") ``` Nrrrepo_path_or_namezpThe `repo_path_or_name` argument is deprecated and will be removed in v5 of Transformers. Use `repo_id` instead.repo_url organizationrr)r_rrcrd) working_dirr^)rryr)r|rk)rrrrB)rrrrbr^rrrrrrr-rrrwrrrmrr)rUrr^rr_rrrrbase_model_card_argsrcrdrework_dirr rBrBrCr - sX4       "zTFPreTrainedModel.push_to_hub TFAutoModelcCsDt|ts|j}ddlmm}t||st|d||_dS)a Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"TFAutoModel"`): The auto class to register this new model with. rNz is not a valid auto class.) r]rrWtransformers.models.automodelsr%rwrbr)rp auto_class auto_modulerBrBrCregister_for_auto_class s   z)TFPreTrainedModel.register_for_auto_class)rHr)rHrrJ)rrrrIrHr)rHr)rHrG)rHr)rTNNNNT)rrrrIrrGrrrrrrrrrrG)r"r#NNNNN)NNNNNNNN)rkrrlrmrnrmrormrprmrqrmrrrsrrsrtrs)rHr)rHr)rHr)rrrHr)rrrHr)rrrrIrHr)rHr)rrrrIrHr)Fr FNrFFN)rrrrGrrGrr)rrr\rrrr"rGrrGrrGrrrrrr)NNNrNNF)rrr^rrrmr_rrr`rrarrarrGrHr)rh)JrWrXrYrZr[rrrr.r=rLrMrGpropertyrrrzrrrrirurvrzr/rrrrrr classmethodrrrrr=functionrrrrrrr!r0r2r?rirjrrrrrrrrrrrrrrrrrrrrrrEr rm __classcell__rBrBrrCrUs               .   k 6ll <     ! 5" & #/ 8Z mrcs2eZdZdZd fdd ZddZddZZS) TFConv1DaO 1D-convolutional layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2). Basically works like a linear layer but the weights are transposed. Args: nf (`int`): The number of output features. nx (`int`): The number of input features. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation to use to initialize the weights. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the `__init__` of `keras.layers.Layer`. rc s(tjdi|||_||_||_dS)NrB)rkrrnfnxr)rUrsrtrrfrrBrCrr s zTFConv1D.__init__cCsP|jrdSd|_|jd|j|jgt|jd|_|jdd|jgtd|_ dS)NTrr9rgbiasr ) builtrrtrsrrrr=zeros_initializerrvrrBrBrCr s zTFConv1D.buildcCsRt|dd\}}t|d|jg}t||j|j}t||||jg}|S)Nr6r)rr=rrtmatmulrrvrs)rUrcbzslrBrBrCr s z TFConv1D.callr)rWrXrYrZrrrrrqrBrBrrCrr s  rrcsXeZdZdZddfdd Zfd d Zfd d ZddddZddZddZ Z S)TFSharedEmbeddingsa Construct shared token embeddings. The weights of the embedding layer is usually shared with the weights of the linear decoder when doing language modeling. Args: vocab_size (`int`): The size of the vocabulary, e.g., the number of unique tokens. hidden_size (`int`): The size of the embedding vectors. initializer_range (`float`, *optional*): The standard deviation to use when initializing the weights. If no value is provided, it will default to \\(1/\sqrt{hidden\_size}\\). kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the `__init__` of `keras.layers.Layer`. NrrIrrOptional[float]c sDtjdi|||_||_|dur|dn||_tdtdS)Ngz^`TFSharedEmbeddings` is scheduled for deletion in v4.32, use `keras.layers.Embedding` instead.rB)rkrrrrrrrDeprecationWarning)rUrrrrfrrBrCrr szTFSharedEmbeddings.__init__cs0|jd|j|jgt|jd|_t|dS)z Build shared token embedding layer Shared weights logic adapted from https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e32192a944688925ef78659/official/transformer/v2/embedding_layer.py#L24 rruN)rrrrrrrkrrrrBrCr szTFSharedEmbeddings.buildcs8|j|j|jd}t}tt|t|S)N)rrr)rrrrkrir_r;r)rUr\ base_configrrBrCri s  zTFSharedEmbeddings.get_config embeddingrrrrrHcCs4|dkr ||S|dkr||Std|d)am Get token embeddings of inputs or decode final hidden state. Args: inputs (`tf.Tensor`): In embedding mode, should be an int64 tensor with shape `[batch_size, length]`. In linear mode, should be a float tensor with shape `[batch_size, length, hidden_size]`. mode (`str`, defaults to `"embedding"`): A valid value is either `"embedding"` or `"linear"`, the first one indicates that the layer should be used as an embedding layer, the second one that the layer should be used as a linear decoder. Returns: `tf.Tensor`: In embedding mode, the output is a float32 embedding tensor, with shape `[batch_size, length, embedding_size]`. In linear mode, the output is a float32 with shape `[batch_size, length, vocab_size]`. Raises: ValueError: if `mode` is not valid. Shared weights logic is adapted from [here](https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e32192a944688925ef78659/official/transformer/v2/embedding_layer.py#L24). rlinearzmode z is not valid.) _embedding_linearrb)rUrrrBrBrCr s   zTFSharedEmbeddings.callcCst|j|S)z)Applies embedding based on inputs tensor.)r=gatherr)rUrrBrBrCr3 szTFSharedEmbeddings._embeddingcCsHt|dd}t|d|jg}tj||jdd}t|||jgS)z Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [..., hidden_size] Returns: float32 tensor with shape [..., vocab_size]. NrT) transpose_b)rr=rrryrr)rUr first_dimsrcrrBrBrCr7 s zTFSharedEmbeddings._linearrJ)rrIrrIrr~)r)rrrrrHr) rWrXrYrZrrrrirrrrqrBrBrrCr} s  r}cs6eZdZdZddfdd Zdd d Zd dZZS)TFSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`PretrainedConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation to use to initialize the weights. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the `__init__` of `keras.layers.Layer`. rr\rrfloatc stjd i|t|dr|jnd|_|jdkrtt|do"|j|_|jrGt|dr8|jr8|jdkr8|j}n|j }t j j |t |dd|_d|_t|d d}|dur\d |_t||_t|d oe|jdk|_|jrrt j |j|_t|d o{|jdk|_|jrt j |j|_|j |_ dS)Nsummary_use_projlastattnsummary_proj_to_labelsrsummary)kernel_initializerrFsummary_activationTsummary_first_dropoutsummary_last_dropoutrB)rkrrrw summary_typerr has_summaryr num_labelsrrzrpDenserrhas_activationrsr activationrhas_first_dropoutDropout first_dropoutrhas_last_dropout last_dropout)rUr\rrf num_classesactivation_stringrrBrCrrf s0     zTFSequenceSummary.__init__NFcCst|tttfs |}n-t|ttfr-|d}t|dkr |dnd}t|dks,Jdn |d}|dd}|jdkrF|dddf}nj|jd krT|dddf}n\|jd kratj|dd }nO|jdkrt |}|dur|t |dd |d d}t |}t|t|dkrtj |dd }tj ||t|dd }tj |t|dd }n|jdkrt|jr|j||d}|jr||}|jr||}|jr|j||d}|S)Nrr r6zToo many inputs.r cls_indexrrfirstmeanr7) batch_dimsrrN)r]r_rr;rarsrr=r>rrrrsqueezerrrrrrrrr)rUrrrMrr hidden_shape cls_shaperBrBrCr sL          zTFSequenceSummary.callcCsb|jrdSd|_t|dddur/td|j|jWddS1s(wYdSdS)NTr)rwrsr=rrrrrrBrBrCr s "zTFSequenceSummary.buildr|)r\rrr)NF)rWrXrYrZrrrrrqrBrBrrCrH s  #3rrrrrH"keras.initializers.TruncatedNormalcCstjj|dS)a Creates a `keras.initializers.TruncatedNormal` with the given range. Args: initializer_range (*float*, defaults to 0.02): Standard deviation of the initializer range. Returns: `keras.initializers.TruncatedNormal`: The truncated normal initializer. r)rzrr)rrBrBrCr s rrJ)rr)FFN)FNr|)rrrHr)rZ __future__rrur0rrrrr?rcollections.abcrpathlibrtypingrrrrr r r rHrrKr-r=packaging.versionr rrractivations_tfrconfiguration_utilsrdynamic_module_utilsr generationrrtf_utilsrrrrrr{rrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r- utils.hubr.r/ safetensorsr0safetensors.tensorflowr1rr2 get_loggerrWrenvironrtf_kerasrzr5rKModuleNotFoundError ImportErrorrTmajorrb tf_loggerrrrTFModelInputTyperDrEr|r~rrrrrrrrrrr rCr/rdrhrfr^rr/rrpLayerrrr}rrrBrBrBrCs   $    X           A * "!, . E CZ 7  d (p+h}