o  hn @s>ddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlmZddlZddlmZddlZddlmZmZddlmZmZddlmZm Z ddl!m"Z"d d l#m$Z$d d l%m&Z&d d l'm(Z(m)Z)d d l*m+Z+d dl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mm?Z?m@Z@d dlAmBZBeBrddlCmDZDddlEmFZGddlEmHZIe If one of the model's weight is bigger that `max_shard_size`, it will end up in its own sub-checkpoint which will have a size greater than `max_shard_size`. Args: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): The maximum size of each sub-checkpoint. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). r/seprNz.msgpack-05dz-of- total_size)metadata weight_map) r(rsizedtypeitemsizeappendlenr enumeratereplacekeys)paramsmax_shard_sizesharded_state_dicts current_blockcurrent_block_sizerDweightsitem weight_sizerFshardsidxshard shard_file weight_namerEindexr3r3r4flax_shard_checkpointUs:       &   r]c@seZdZdZdZdZdZdZeZ dde j dfde d e jd ed ed e jd ef ddZdIdejjd ededefddZddZeddZedefddZede fddZ ede jfddZ!ede"eeffdd Z#ede$fd!d"Z%edefd#d$Z&e#j'de"eeffd%d Z#dIde"eefd e jd&e(de(fd'd(Z)dIde"eefd&e(fd)d*Z*dIde"eefd&e(fd+d,Z+dIde"eefd&e(fd-d.Z,ed/d0Z-ed1d2Z.edefd3d4Z/ee j fddd5d5d5dd6d7d8e"ee0j1fd e jde2e"e ee0j1fd9e2e"ee0j1fd:ed;edefd?d@Z3  5 A  5dJdBe"ee0j1fd=e2e"eefdCefdDdEZ4edKdGdHZ5dS)LFlaxPreTrainedModela$ Base class for all models. [`FlaxPreTrainedModel`] takes care of storing the configuration of the models and handles methods for loading, downloading and saving models. Class attributes (overridden by derived classes): - **config_class** ([`PretrainedConfig`]) -- A subclass of [`PretrainedConfig`] to use as configuration class for this model architecture. - **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. - **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP models, `pixel_values` for vision models and `input_values` for speech models). N input_ids)rrrTconfigmodule input_shapeseedrH_do_initc Cs|durtd|durtd||_||_t||_||_||_|r*t |nd|_ ||_ |rB| |j|}t dd|}nt|j |d} t | |j}td|jjd||_ttt||_|rp||_dSdS)Nzconfig cannot be Nonezmodule cannot be NonecSs|SNr3)rOr3r3r4sz.FlaxPreTrainedModel.__init__..)rczVModel weights are not initialized as `_do_init` is set to `False`. Make sure to call `z2.init_weights` manually to initialize the weights.) ValueError_config_modulerkeyrHrc can_generaterfrom_model_configgeneration_config_is_initialized init_weightsr. eval_shaperloggerinfo __class____name___params_shape_treesetrr rN_required_paramsrO) selfrarbrcrdrHre random_paramsparams_shape_treeinit_fnr3r3r4__init__s4   zFlaxPreTrainedModel.__init__rngrOreturncCtd|)Nz&init method has to be implemented for NotImplementedError)ryr~rcrOr3r3r4rpz FlaxPreTrainedModel.init_weightscCr)Nz8gradient checkpointing method has to be implemented for rryr3r3r4enable_gradient_checkpointingrz1FlaxPreTrainedModel.enable_gradient_checkpointingcKs||fi|S)zZ All context managers that the model should be initialized under go here. r3)clsrakwargsr3r3r4 _from_configsz FlaxPreTrainedModel._from_configcCsdS)z= :str: Identifies that this is a Flax model. flaxr3rr3r3r4 frameworkszFlaxPreTrainedModel.frameworkcC|jSrf)rirr3r3r4razFlaxPreTrainedModel.configcCrrf)rjrr3r3r4rbrzFlaxPreTrainedModel.modulecCs|jstd|jS)Nz`params` cannot be accessed from model when the model is created with `_do_init=False`. You must call `init_weights` manually and store the params outside of the model and pass it explicitly where needed.)rorh_paramsrr3r3r4rOs zFlaxPreTrainedModel.paramscCrrf)rxrr3r3r4required_paramsrz#FlaxPreTrainedModel.required_paramscCrrf)rvrr3r3r4r{ rz%FlaxPreTrainedModel.params_shape_treecCs`|jstdt|trt|}tt|}t|j |dkr+td|j |||_ dS)Nz}`params` cannot be set from model when the model is created with `_do_init=False`. You store the params outside of the model.rzVSome parameters are missing. Make sure that `params` include the following parameters ) rorh isinstancer r rwrrNrKrr)ryrO param_keysr3r3r4rOs  maskc srfdd}|durtj||St|}tj|\}}t|t|D]\}} |r4||| || <q&t|S)zk Helper method to cast floating-point values of given parameter `PyTree` to given `dtype`. cs*t|tjrt|jtjr|}|Srf)rjnpndarray issubdtyperHfloatingastype)paramrHr3r4conditional_cast's z?FlaxPreTrainedModel._cast_floating_to..conditional_castN) r. tree_utiltree_mapr tree_flattenzipsortedrNr) ryrOrHrr flat_params flat_mask_maskedrkr3rr4_cast_floating_to!s z%FlaxPreTrainedModel._cast_floating_tocC||tj|S)a Cast the floating-point `params` to `jax.numpy.bfloat16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on TPU to explicitly convert the model parameters to bfloat16 precision to do full half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans, `True` for params you want to cast, and should be `False` for those you want to skip. Examples: ```python >>> from transformers import FlaxBertModel >>> # load model >>> model = FlaxBertModel.from_pretrained("google-bert/bert-base-cased") >>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision >>> model.params = model.to_bf16(model.params) >>> # If you want don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model = FlaxBertModel.from_pretrained("google-bert/bert-base-cased") >>> flat_params = traverse_util.flatten_dict(model.params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> model.params = model.to_bf16(model.params, mask) ```)rrbfloat16ryrOrr3r3r4to_bf168%zFlaxPreTrainedModel.to_bf16cCr)ay Cast the floating-point `params` to `jax.numpy.float32`. This method can be used to explicitly convert the model parameters to fp32 precision. This returns a new `params` tree and does not cast the `params` in place. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans, `True` for params you want to cast, and should be `False` for those you want to skip Examples: ```python >>> from transformers import FlaxBertModel >>> # Download model and configuration from huggingface.co >>> model = FlaxBertModel.from_pretrained("google-bert/bert-base-cased") >>> # By default, the model params will be in fp32, to illustrate the use of this method, >>> # we'll first cast to fp16 and back to fp32 >>> model.params = model.to_f16(model.params) >>> # now cast back to fp32 >>> model.params = model.to_fp32(model.params) ```)rrfloat32rr3r3r4to_fp32_szFlaxPreTrainedModel.to_fp32cCr)a Cast the floating-point `params` to `jax.numpy.float16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on GPU to explicitly convert the model parameters to float16 precision to do full half-precision training or to save weights in float16 for inference in order to save memory and improve speed. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans, `True` for params you want to cast, and should be `False` for those you want to skip Examples: ```python >>> from transformers import FlaxBertModel >>> # load model >>> model = FlaxBertModel.from_pretrained("google-bert/bert-base-cased") >>> # By default, the model params will be in fp32, to cast these to float16 >>> model.params = model.to_fp16(model.params) >>> # If you want don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model = FlaxBertModel.from_pretrained("google-bert/bert-base-cased") >>> flat_params = traverse_util.flatten_dict(model.params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> model.params = model.to_fp16(model.params, mask) ```)rrfloat16rr3r3r4to_fp16zrzFlaxPreTrainedModel.to_fp16c Csz3|drt|}t|dd}W|St|d}t||}WdW|S1s,wYW|Sttjj fyy}z6zt|}| drPt dt |1sVwYWnt t fymtd|dwWYd}~|Sd}~ww) N .safetensors.r@rbversionYou seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.Unable to convert to Flax deserializable object. )endswithsafe_load_fileropenr readrmsgpack exceptions ExtraData startswithOSErrorrhUnicodeDecodeErrorEnvironmentError)rresolved_archive_filestatestate_fefr3r3r4load_flax_weightss6       z%FlaxPreTrainedModel.load_flax_weightsc Csi}|D]r}zt|d}t||}Wdn1swYWnAttjjfyU}z!t|}|dr@tdt |1sFwYWYd}~nd}~wt t fyet d|dwt |dd}| |~tqt|ddS) ab This is the same as [`flax.serialization.from_bytes`] (https:lax.readthedocs.io/en/latest/_modules/flax/serialization.html#from_bytes) but for a sharded checkpoint. This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being loaded in the model. Args: shard_files (`List[str]`: The list of shard files to load. Returns: `Dict`: A nested dictionary of the model parameters, in the expected format for flax models : `{'model': {'params': {'...'}}}`. rNrrrrr?r@)rr rrrrrrrrhrrrupdategccollectr)r shard_filesstate_sharded_dictrZrrrrr3r3r4load_flax_sharded_weightss0       z-FlaxPreTrainedModel.load_flax_sharded_weightscCs$dt|jvrdt|jvrdSdS)z Returns whether this model can generate sequences with `.generate()`. Returns: `bool`: Whether this model can generate sequences with `.generate()`. GenerationMixinFT)strprepare_inputs_for_generationgenerate)rr3r3r4rlsz FlaxPreTrainedModel.can_generateFmain)ra cache_dirignore_mismatched_sizesforce_downloadlocal_files_onlytokenrevisionpretrained_model_name_or_pathrrrrrrc0s | dd} | dd} | dd}| dd}| dd}| dd}| d d}| d d }| d d }| dd}| dd}|durVtdt|durTtd|}|d ur_tddd|d}|durm||d<try|sytdd }t |t s|dur|n|}|j j |f|d || |||| ||||d | \}}n| }|durt|dd}||d<d}|durt|}tj|}tj|rtjtj||trtj||t}ntjtj||trtj||t}d }ntrtjtj|trtj|t}nn| r+tjtj||tr+tj||t}nT| rGtjtj||trGtj||t}d }n8trdtjtj|trdtj|t}d }tdtjtj||tr|tdtd|dtdtdtd|dtjtj||r|}d }nt|r|}t|}n| rt}nt}z|||| |||| |dd|d }t ||fi|}|dur|tkrt |tfi|}|durd }|dur| rt |tfi|}|durd }|dur t}t |tfi|}|dur\| ||||d!} t!|tfi| r'd }tdt!|tfi| r;t|d"tdt!|tfi| rOt|d"td#t|d"tdtdWn!tyft"y~td$|d%|d&tdtd w|rtd'||}|#tjj$d(}ntd'|d)|nd}|rt%|||||| |||| ||d* \}}d}!|tkrt&|dd+ }"|"'}#Wdn 1swY|#dus|#(d,d-vrt)d.|d/|#(d,d0k}!||g| Rd |i|}$| s |!rt*|$||}%n#|r|+|}%n|,|}%|r+t-j./t0j1|%}%n t-j./d1d2|%}%d3|%vr|j2t3|$j4d4vr^|j2|%d4vr^|%d4|j2|%d4<|%d3|j2|%d3<|j2t3|$j4d4vr|j2|%d4vr|j2|%d4i|j2|%d3id5}%n(|j2t3|$j4vr|j2|%vr|%|j2}%|j2t3|$j4vr|j2|%vr|j2|%i}%t5|%}%t5t6|r|$j7n|$j4}&|$j8t9|%:}'t9|%:|$j8}(|( D]})d6|)d(vr|(;|)q|'r|std7|d8|'d9|'|_|+|%|+j=|&|+j=f|&|+|%|+<qtd:|+d;|%|+j=d<|&|+j=d=q|'rN|rN|'D] },|&|,|%|,<qD|(D]})|%|)=qPt?|(d>krztd?|d@|$j@jAdA|(dB|$j@jAdC|$j@jAdD n tdE|$j@jAdFt?|'d>krtdG|$j@jAdH|dI|'dJnt?|*d>krtdK|$j@jAdL|dM|$j@jAdNt?|*d>krdOdPdQ|*D}-tdG|$j@jAdH|dR|-dJt-j./dSd2|%fdTdQD}.fdUdQD}/t?|.d>kr tdV|$j@jAdW|dX|.dYt?|/d>kr%tdV|$j@jAdZ|dX|/dY|$BrQztCj |f||| |||| |||d[ | |$_DWnt)yPtd\Ynw|r[tE|%|$_7|$S|$tE|%fS)]aw Instantiate a pretrained flax model from a pre-trained model configuration. The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task. The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those weights are discarded. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~FlaxPreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *pt index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_pt` should be set to `True`. dtype (`jax.numpy.dtype`, *optional*, defaults to `jax.numpy.float32`): The data type of the computation. Can be one of `jax.numpy.float32`, `jax.numpy.float16` (on GPUs) and `jax.numpy.bfloat16` (on TPUs). This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified all the computation will be performed with the given `dtype`. **Note that this only specifies the dtype of the computation and does not influence the dtype of model parameters.** If you wish to change the dtype of the model parameters, see [`~FlaxPreTrainedModel.to_fp16`] and [`~FlaxPreTrainedModel.to_bf16`]. model_args (sequence of positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. config (`Union[PretrainedConfig, str, os.PathLike]`, *optional*): Can be either: - an instance of a class derived from [`PretrainedConfig`], - a string or path valid as input to [`~PretrainedConfig.from_pretrained`]. Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Examples: ```python >>> from transformers import BertConfig, FlaxBertModel >>> # Download model and configuration from huggingface.co and cache. >>> model = FlaxBertModel.from_pretrained("google-bert/bert-base-cased") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model = FlaxBertModel.from_pretrained("./test/saved_model/") >>> # Loading from a PyTorch checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable). >>> config = BertConfig.from_json_file("./pt_model/config.json") >>> model = FlaxBertModel.from_pretrained("./pt_model/pytorch_model.bin", from_pt=True, config=config) ```from_ptFresume_downloadNproxiesuse_auth_tokentrust_remote_code_from_pipeline _from_autoreT subfolderr_ _commit_hashadapter_kwargsrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.zgThe argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.modelr) file_typerfrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=True) rreturn_unused_kwargsrrrrrrrrrrrHzASupport for sharded checkpoints using safetensors is coming soon!zError no file named z found in directory zc but there is a file for PyTorch weights. Use `from_pt=True` to load this model from those weights.z or r) rrrrrr user_agentrr _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entriesr)rrrrrz& does not appear to have a file named zk but there is a sharded file for PyTorch weights. Use `from_pt=True` to load this model from those weights.zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named zloading weights file z from cache at ) rrrrrrrrrr)rformat)pttfrz"The safetensors archive passed at zf does not contain the valid metadata. Make sure you save your model with the `save_pretrained` method.rcSst|tjdddS)Ncpu)backendr)r. device_put local_devicesr1r3r3r4rgsz5FlaxPreTrainedModel.from_pretrained.. batch_statsrO)rOrnum_batches_trackedzThe checkpoint z is missing required keys: zI. Make sure to call model.init_weights to initialize the missing weights.z)Trying to load the pretrained weight for z failed: checkpoint has shape z, which is incompatible with the model shape zd. Using `ignore_mismatched_sizes=True` if you really want to load this checkpoint inside this model.rz(Some weights of the model checkpoint at z! were not used when initializing z: z, - This IS expected if you are initializing z from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing z from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).z9All model checkpoint weights were used when initializing z. zSome weights of z3 were not initialized from the model checkpoint at z and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z/ were initialized from the model checkpoint at zf. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training. c Ss*g|]\}}}d|d|d|dqS)z- z: found shape z in the checkpoint and z in the model instantiatedr3).0rkshape1shape2r3r3r4 sz7FlaxPreTrainedModel.from_pretrained..z= and are newly initialized because the shapes did not match: cSrrfrr1r3r3r4rgscg|] }|tjkr|qSr3)rrrk param_dtypesr3r4rcrr3)rrrrr3r4rrzSome of the weights of zD were initialized in float16 precision from the model checkpoint at z: z You should probably UPCAST the model weights to float32 if this was not intended. See [`~FlaxPreTrainedModel.to_fp32`] for further information on how to do this.zE were initialized in bfloat16 precision from the model checkpoint at ) rrrrrrrrrrzZGeneration config file not found, using a generation config created from the model config.)Fpopwarningswarn FutureWarningrhrrwarningr$rsrr config_classfrom_pretrainedcopygetattrrospathisdirisfilejoinrrr*rrrrrrr%r"r r# ExceptionsplitrAr)r+rEgetrrrrr.rrrarraybase_model_prefixdictr{rr rOrrwrNremove _missing_keysshaperJrKrtrurlrrnr)0rrrHrarrrrrr model_argsrrrrrr from_pipelinerrer commit_hashrr config_path model_kwargs is_shardedis_local archive_filefilenamercached_file_kwargshas_file_kwargssafetensors_from_ptrsafetensors_metadatarr random_state missing_keysunexpected_keysunexpected_keymismatched_keysrk missing_keymismatched_warning fp16_params bf16_paramsr3rr4rs                                                            z#FlaxPreTrainedModel.from_pretrainedr>save_directorysafe_serializationc KsX|dd}|durtdt|durtd|}|dur"||d<tj|r3t d|ddStj |dd |r]|d d} |d | tjj d } |j | fi|} ||} tj|}|jjd dg|j_|jdur{t|||jd|j||r|j||rtnt} tj|| } t|dur|n|j|\}}t|D])}tj||}| dddd}| |rtj|r||!vrt"|q|dur|r|dur|n|j}t#|dd}t$|| ddidnt%| d}|dur|n|j}t&|}|'|Wdn 1swYnttj|t(}t%|ddd}t)j*|dddd}|'|Wdn 1sHwYt+d|dt,|d |d|-D]0\}}t%tj||dd!}t.|d"d}t&|}|'|Wdn 1swYqbt+d#| |r|j/|| | | |d$dSdS)%a  Save a model and its configuration file to a directory, so that it can be re-loaded using the `[`~FlaxPreTrainedModel.from_pretrained`]` class method Arguments: save_directory (`str` or `os.PathLike`): Directory to which to save. Will be created if it doesn't exist. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`): The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard which will be bigger than `max_shard_size`. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save the model using `safetensors` or through msgpack. rNrrrzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_idr)raz.binr_rrr@rr)rEwbwzutf-8)encoding)indent sort_keysrz:The model is bigger than the maximum size per checkpoint (z) and is going to be split in z^ checkpoint shards. You can find where each parameters has been saved in the index located at )moder?zModel weights saved in )r0r)0rrrrrhr r r rrerrormakedirsrrA _create_repo_get_files_timestampsabspathrtrura architectures _auto_classrsave_pretrainedrlrnrrr r]rOlistdirrMrrNrrsafe_save_filerr writerjsondumpsrsrKitemsr_upload_modified_files)ryr-rO push_to_hubrPrr.rrr0r1files_timestamps weights_nameoutput_model_filerWr\r full_filenameweights_no_suffix flat_dictr model_bytessave_index_filecontentrZrY shard_bytesr3r3r4rA0s (                   z#FlaxPreTrainedModel.save_pretrained FlaxAutoModelcCsDt|ts|j}ddlmm}t||st|d||_dS)a Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"FlaxAutoModel"`): The auto class to register this new model with. rNz is not a valid auto class.) rrrutransformers.models.automodelsautohasattrrhr@)r auto_class auto_moduler3r3r4register_for_auto_classs   z+FlaxPreTrainedModel.register_for_auto_classrf)NFr>NF)rT)6ru __module__ __qualname____doc__rrmain_input_namer@rwrrrrr/ModulerintrHboolr}r.randomrr rrpr classmethodrpropertyrrrarbr rOrrr{setterrrrrrrrrlr PathLikerrrAr[r3r3r3r4r^s 3  &''  .     A   r^rrTzmodel checkpoint)object object_class object_filescCs(t|j|_d|j_t||j|_dSrf)r!__call__r^r) model_class docstringr3r3r4overwrite_call_docstrings rncCs.t|j|_t||||j||d|j|_dS)N) checkpoint output_typer model_clsrreal_checkpoint)r!rkrru)rlrorprrrrrr3r3r4append_call_sample_docstrings  rscCs$t|j|_t||d|j|_dS)N)rpr)r!rkr')rlrprr3r3r4 append_replace_return_docstringss  rt)r>)NNN)[rrEr r functoolsrpicklertypingrrrrrr flax.linenlinenr/r. jax.numpynumpyrmsgpack.exceptionsrflax.core.frozen_dictr r flax.serializationr r flax.traverse_utilrr jax.randomrconfiguration_utilsrdynamic_module_utilsr generationrrmodeling_flax_pytorch_utilsrutilsrrrrrrrrrr r!r"r#r$r%r&r' utils.hubr(r)utils.import_utilsr* safetensorsr+safetensors.flaxr,rr-rC get_loggerrurrr5r7r8r:r=ACT2FNr]r^rIr^rrnrsrtr3r3r3r4sp         L         A H