o  hM@sdZddlZddlZddlZddlZddlmZmZddlm Z m Z ddl m Z m Z mZmZmZmZmZddlmZddlmZdd lmZmZmZmZmZmZmZmZm Z e r^dd l!m"Z"e #e$Z%d Z&iZ'iZ(iZ)gZ*erdd l+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5d dl6m7Z7m8Z8e1e'd<e5e'd<e4e0e3e-e.e/dZ(e2e,dZ)e9e(:e9e':ddgZ*GdddeZ;GdddeZe Gddde=Z?e Gdd d Z@dS)!z-Generation configuration class and utilities.N)ABCabstractmethod) dataclass is_dataclass) TYPE_CHECKINGAnyCallableDictListOptionalUnion) __version__)PretrainedConfig) GENERATION_CONFIG_NAME ExplicitEnumPushToHubMixin cached_file download_urlextract_commit_hash is_remote_urlis_torch_availablelogging)PreTrainedModel)_from_model_config _commit_hash_original_object_hashtransformers_version) HQQQuantizedCache HybridCacheHybridChunkedCache MambaCacheOffloadedStaticCacheQuantizedCacheConfigQuantoQuantizedCacheSlidingWindowCache StaticCacheStaticCacheConfig)#SynthIDTextWatermarkLogitsProcessorWatermarkLogitsProcessor quantizedstatic)r,offloaded_staticsliding_windowhybridhybrid_chunkedmamba)quantoHQQ offloadeddynamicc@s4eZdZdZdZdZdZdZdZdZ dZ d Z d Z d S) GenerationModezg Possible generation modes, downstream of the [`~generation.GenerationMixin.generate`] method. contrastive_search greedy_searchsampleassisted_generationdola_generation beam_search beam_sampleconstrained_beam_searchgroup_beam_searchN) __name__ __module__ __qualname____doc__CONTRASTIVE_SEARCH GREEDY_SEARCHSAMPLEASSISTED_GENERATIONDOLA_GENERATION BEAM_SEARCH BEAM_SAMPLECONSTRAINED_BEAM_SEARCHGROUP_BEAM_SEARCHrMrM/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/generation/configuration_utils.pyr6Rsr6c@seZdZdZdZddZddZddZd d Zdde eejfde e eejfdefddZe   d?de eejfde e eejfde e eejfdedede e eefdeddfd d!Zed"e eejffd#d$Zed%eeefddfd&d'Zd(eeefdd fd)d*Zdeeeffd+d,Zdeeeffd-d.Zd@d0ed1edefd2d3ZdAd4e eejfd0efd5d6Zed7eddfd8d9Zd:d;Z d S)BGenerationConfiga[Y Class that holds a configuration for a generation task. A `generate` call supports the following generation methods for text-decoder, text-to-text, speech-to-text, and vision-to-text models: - *greedy decoding* if `num_beams=1` and `do_sample=False` - *contrastive search* if `penalty_alpha>0.` and `top_k>1` - *multinomial sampling* if `num_beams=1` and `do_sample=True` - *beam-search decoding* if `num_beams>1` and `do_sample=False` - *beam-search multinomial sampling* if `num_beams>1` and `do_sample=True` - *diverse beam-search decoding* if `num_beams>1` and `num_beam_groups>1` - *constrained beam-search decoding* if `constraints!=None` or `force_words_ids!=None` - *assisted decoding* if `assistant_model` or `prompt_lookup_num_tokens` is passed to `.generate()` - *dola decoding* if `dola_layers` is passed to `.generate()` To learn more about decoding strategies refer to the [text generation strategies guide](../generation_strategies). A large number of these flags control the logits or the stopping criteria of the generation. Make sure you check the [generate-related classes](https://huggingface.co/docs/transformers/internal/generation_utils) for a full description of the possible manipulations, as well as examples of their usage. Arg: > Parameters that control the length of the output max_length (`int`, *optional*, defaults to 20): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (`int`, *optional*): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. min_length (`int`, *optional*, defaults to 0): The minimum length of the sequence to be generated. Corresponds to the length of the input prompt + `min_new_tokens`. Its effect is overridden by `min_new_tokens`, if also set. min_new_tokens (`int`, *optional*): The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt. early_stopping (`bool` or `str`, *optional*, defaults to `False`): Controls the stopping condition for beam-based methods, like beam-search. It accepts the following values: `True`, where the generation stops as soon as there are `num_beams` complete candidates; `False`, where an heuristic is applied and the generation stops when is it very unlikely to find better candidates; `"never"`, where the beam search procedure only stops when there cannot be better candidates (canonical beam search algorithm). max_time (`float`, *optional*): The maximum amount of time you allow the computation to run for in seconds. generation will still finish the current pass after allocated time has been passed. stop_strings (`str or List[str]`, *optional*): A string or a list of strings that should terminate generation if the model outputs them. > Parameters that control the generation strategy used do_sample (`bool`, *optional*, defaults to `False`): Whether or not to use sampling ; use greedy decoding otherwise. num_beams (`int`, *optional*, defaults to 1): Number of beams for beam search. 1 means no beam search. num_beam_groups (`int`, *optional*, defaults to 1): Number of groups to divide `num_beams` into in order to ensure diversity among different groups of beams. [this paper](https://arxiv.org/pdf/1610.02424.pdf) for more details. penalty_alpha (`float`, *optional*): The values balance the model confidence and the degeneration penalty in contrastive search decoding. dola_layers (`str` or `List[int]`, *optional*): The layers to use for DoLa decoding. If `None`, DoLa decoding is not used. If a string, it must be one of "low" or "high", which means using the lower part or higher part of the model layers, respectively. "low" means the first half of the layers up to the first 20 layers, and "high" means the last half of the layers up to the last 20 layers. If a list of integers, it must contain the indices of the layers to use for candidate premature layers in DoLa. The 0-th layer is the word embedding layer of the model. Set to `'low'` to improve long-answer reasoning tasks, `'high'` to improve short-answer tasks. Check the [documentation](https://github.com/huggingface/transformers/blob/main/docs/source/en/generation_strategies.md) or [the paper](https://arxiv.org/abs/2309.03883) for more details. > Parameters that control the cache use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. cache_implementation (`str`, *optional*, default to `None`): Name of the cache class that will be instantiated in `generate`, for faster decoding. Possible values are: - `"dynamic"`: [`DynamicCache`] - `"static"`: [`StaticCache`] - `"offloaded_static"`: [`OffloadedStaticCache`] - `"sliding_window"`: [`SlidingWindowCache`] - `"hybrid"`: [`HybridCache`] - `"mamba"`: [`MambaCache`] - `"quantized"`: [`QuantizedCache`] If none is specified, we will use the default cache for the model (which is often [`DynamicCache`]). See our [cache documentation](https://huggingface.co/docs/transformers/en/kv_cache) for further information. cache_config (`CacheConfig` or `dict`, *optional*, default to `None`): Arguments used in the key-value cache class can be passed in `cache_config`. Can be passed as a `Dict` and it will be converted to its repsective `CacheConfig` internally. Otherwise can be passed as a `CacheConfig` class matching the indicated `cache_implementation`. return_legacy_cache (`bool`, *optional*, default to `True`): Whether to return the legacy or new format of the cache when `DynamicCache` is used by default. > Parameters for manipulation of the model output logits temperature (`float`, *optional*, defaults to 1.0): The value used to module the next token probabilities. This value is set in a model's `generation_config.json` file. If it isn't set, the default value is 1.0 top_k (`int`, *optional*, defaults to 50): The number of highest probability vocabulary tokens to keep for top-k-filtering. This value is set in a model's `generation_config.json` file. If it isn't set, the default value is 50. top_p (`float`, *optional*, defaults to 1.0): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. This value is set in a model's `generation_config.json` file. If it isn't set, the default value is 1.0 min_p (`float`, *optional*): Minimum token probability, which will be scaled by the probability of the most likely token. It must be a value between 0 and 1. Typical values are in the 0.01-0.2 range, comparably selective as setting `top_p` in the 0.99-0.8 range (use the opposite of normal `top_p` values). typical_p (`float`, *optional*, defaults to 1.0): Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up to `typical_p` or higher are kept for generation. See [this paper](https://arxiv.org/pdf/2202.00666.pdf) for more details. epsilon_cutoff (`float`, *optional*, defaults to 0.0): If set to float strictly between 0 and 1, only tokens with a conditional probability greater than `epsilon_cutoff` will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the size of the model. See [Truncation Sampling as Language Model Desmoothing](https://arxiv.org/abs/2210.15191) for more details. eta_cutoff (`float`, *optional*, defaults to 0.0): Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either `eta_cutoff` or `sqrt(eta_cutoff) * exp(-entropy(softmax(next_token_logits)))`. The latter term is intuitively the expected next token probability, scaled by `sqrt(eta_cutoff)`. In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See [Truncation Sampling as Language Model Desmoothing](https://arxiv.org/abs/2210.15191) for more details. diversity_penalty (`float`, *optional*, defaults to 0.0): This value is subtracted from a beam's score if it generates a token same as any beam from other group at a particular time. Note that `diversity_penalty` is only effective if `group beam search` is enabled. repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. encoder_repetition_penalty (`float`, *optional*, defaults to 1.0): The paramater for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. length_penalty (`float`, *optional*, defaults to 1.0): Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative), `length_penalty` > 0.0 promotes longer sequences, while `length_penalty` < 0.0 encourages shorter sequences. no_repeat_ngram_size (`int`, *optional*, defaults to 0): If set to int > 0, all ngrams of that size can only occur once. bad_words_ids (`List[List[int]]`, *optional*): List of list of token ids that are not allowed to be generated. Check [`~generation.NoBadWordsLogitsProcessor`] for further documentation and examples. force_words_ids (`List[List[int]]` or `List[List[List[int]]]`, *optional*): List of token ids that must be generated. If given a `List[List[int]]`, this is treated as a simple list of words that must be included, the opposite to `bad_words_ids`. If given `List[List[List[int]]]`, this triggers a [disjunctive constraint](https://github.com/huggingface/transformers/issues/14081), where one can allow different forms of each word. renormalize_logits (`bool`, *optional*, defaults to `False`): Whether to renormalize the logits after applying all the logits processors (including the custom ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits are normalized but some logit processors break the normalization. constraints (`List[Constraint]`, *optional*): Custom constraints that can be added to the generation to ensure that the output will contain the use of certain tokens as defined by `Constraint` objects, in the most sensible way possible. forced_bos_token_id (`int`, *optional*, defaults to `model.config.forced_bos_token_id`): The id of the token to force as the first generated token after the `decoder_start_token_id`. Useful for multilingual models like [mBART](../model_doc/mbart) where the first generated token needs to be the target language token. forced_eos_token_id (`int` or List[int]`, *optional*, defaults to `model.config.forced_eos_token_id`): The id of the token to force as the last generated token when `max_length` is reached. Optionally, use a list to set multiple *end-of-sequence* tokens. remove_invalid_values (`bool`, *optional*, defaults to `model.config.remove_invalid_values`): Whether to remove possible *nan* and *inf* outputs of the model to prevent the generation method to crash. Note that using `remove_invalid_values` can slow down generation. exponential_decay_length_penalty (`tuple(int, float)`, *optional*): This Tuple adds an exponentially increasing length penalty, after a certain amount of tokens have been generated. The tuple shall consist of: `(start_index, decay_factor)` where `start_index` indicates where penalty starts and `decay_factor` represents the factor of exponential decay suppress_tokens (`List[int]`, *optional*): A list of tokens that will be suppressed at generation. The `SupressTokens` logit processor will set their log probs to `-inf` so that they are not sampled. begin_suppress_tokens (`List[int]`, *optional*): A list of tokens that will be suppressed at the beginning of the generation. The `SupressBeginTokens` logit processor will set their log probs to `-inf` so that they are not sampled. forced_decoder_ids (`List[List[int]]`, *optional*): A list of pairs of integers which indicates a mapping from generation indices to token indices that will be forced before sampling. For example, `[[1, 123]]` means the second generated token will always be a token of index 123. sequence_bias (`Dict[Tuple[int], float]`, *optional*)): Dictionary that maps a sequence of tokens to its bias term. Positive biases increase the odds of the sequence being selected, while negative biases do the opposite. Check [`~generation.SequenceBiasLogitsProcessor`] for further documentation and examples. token_healing (`bool`, *optional*, defaults to `False`): Heal tail tokens of prompts by replacing them with their appropriate extensions. This enhances the quality of completions for prompts affected by greedy tokenization bias. guidance_scale (`float`, *optional*): The guidance scale for classifier free guidance (CFG). CFG is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality. low_memory (`bool`, *optional*): Switch to sequential beam search and sequential topk for contrastive search to reduce peak memory. Used with beam search and contrastive search. watermarking_config (`BaseWatermarkingConfig` or `dict`, *optional*): Arguments used to watermark the model outputs by adding a small bias to randomly selected set of "green" tokens. See the docs of [`SynthIDTextWatermarkingConfig`] and [`WatermarkingConfig`] for more details. If passed as `Dict`, it will be converted to a `WatermarkingConfig` internally. > Parameters that define the output variables of generate num_return_sequences (`int`, *optional*, defaults to 1): The number of independently computed returned sequences for each element in the batch. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more details. output_hidden_states (`bool`, *optional*, defaults to `False`): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more details. output_scores (`bool`, *optional*, defaults to `False`): Whether or not to return the prediction scores. See `scores` under returned tensors for more details. output_logits (`bool`, *optional*): Whether or not to return the unprocessed prediction logit scores. See `logits` under returned tensors for more details. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`], as opposed to returning exclusively the generated sequence. This flag must be set to `True` to return the generation cache (when `use_cache` is `True`) or optional outputs (see flags starting with `output_`) > Special tokens that can be used at generation time pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*): The id of the *beginning-of-sequence* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. > Generation parameters exclusive to encoder-decoder models encoder_no_repeat_ngram_size (`int`, *optional*, defaults to 0): If set to int > 0, all ngrams of that size that occur in the `encoder_input_ids` cannot occur in the `decoder_input_ids`. decoder_start_token_id (`int` or `List[int]`, *optional*): If an encoder-decoder model starts decoding with a different token than *bos*, the id of that token or a list of length `batch_size`. Indicating a list enables different start ids for each element in the batch (e.g. multilingual models with different target languages in one batch) > Generation parameters exclusive to assistant generation is_assistant (`bool`, *optional*, defaults to `False`): Whether the model is an assistant (draft) model. num_assistant_tokens (`int`, *optional*, defaults to 20): Defines the number of _speculative tokens_ that shall be generated by the assistant model before being checked by the target model at each iteration. Higher values for `num_assistant_tokens` make the generation more _speculative_ : If the assistant model is performant larger speed-ups can be reached, if the assistant model requires lots of corrections, lower speed-ups are reached. num_assistant_tokens_schedule (`str`, *optional*, defaults to `"constant"`): Defines the schedule at which max assistant tokens shall be changed during inference. - `"heuristic"`: When all speculative tokens are correct, increase `num_assistant_tokens` by 2 else reduce by 1. `num_assistant_tokens` value is persistent over multiple generation calls with the same assistant model. - `"heuristic_transient"`: Same as `"heuristic"` but `num_assistant_tokens` is reset to its initial value after each generation call. - `"constant"`: `num_assistant_tokens` stays unchanged during generation assistant_confidence_threshold (`float`, *optional*, defaults to 0.4): The confidence threshold for the assistant model. If the assistant model's confidence in its prediction for the current token is lower than this threshold, the assistant model stops the current token generation iteration, even if the number of _speculative tokens_ (defined by `num_assistant_tokens`) is not yet reached. The assistant's confidence threshold is adjusted throughout the speculative iterations to reduce the number of unnecessary draft and target forward passes, biased towards avoiding false negatives. `assistant_confidence_threshold` value is persistent over multiple generation calls with the same assistant model. It is an unsupervised version of the dynamic speculation lookahead from Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models . prompt_lookup_num_tokens (`int`, *optional*): The number of tokens to be output as candidate tokens. max_matching_ngram_size (`int`, *optional*): The maximum ngram size to be considered for matching in the prompt. Default to 2 if not provided. assistant_early_exit(`int`, *optional*): If set to a positive integer, early exit of the model will be used as an assistant. Can only be used with models that support early exit (i.e. models where logits from intermediate layers can be interpreted by the LM head). assistant_lookbehind(`int`, *optional*, defaults to 10): If set to a positive integer, the re-encodeing process will additionally consider the last `assistant_lookbehind` assistant tokens to correctly align tokens. Can only be used with different tokenizers in speculative decoding. See this [blog](https://huggingface.co/blog/universal_assisted_generation) for more details. target_lookbehind(`int`, *optional*, defaults to 10): If set to a positive integer, the re-encodeing process will additionally consider the last `target_lookbehind` target tokens to correctly align tokens. Can only be used with different tokenizers in speculative decoding. See this [blog](https://huggingface.co/blog/universal_assisted_generation) for more details. > Parameters related to performances and compilation compile_config (CompileConfig, *optional*): If using a static cache, this controls how `generate` will `compile` the forward pass for performance gains. disable_compile (`bool`, *optional*): Whether to disable the automatic compilation of the forward pass. Automatic compilation happens when specific criteria are met, including using a compileable cache. Please open an issue if you find the need to use this flag. > Wild card generation_kwargs: Additional generation kwargs will be forwarded to the `generate` function of the model. Kwargs that are not present in `generate`'s signature will be used in the model forward pass. )output_attentionsoutput_hidden_states output_scores output_logitsc Ks|dd|_|dd|_|dd|_|dd|_|dd|_|d d|_|d d|_|d d|_|d d |_ |dd |_ |dd|_ |dd|_ |dd|_ |dd|_|dd|_|jdur|jtvrt|j}t|jtr||j|_|dd|_|dd|_|dd|_|dd|_|dd|_|dd|_|dd|_|dd|_|d d|_|d!d|_|d"d|_|d#d|_|d$d|_ |d%d|_!|d&d|_"|d'd|_#|d(d|_$|d)d|_%|d*d|_&|d+d|_'|d,d|_(|d-d|_)|d.d|_*|d/d|_+|d0d|_,|d1d|_-|d2d|_.|d3d|_/|d4d|_0|d5d}|dur_d|_1nt|t2ri||_1nt3||_1|d6d |_4|d7d|_5|d8d|_6|d9d|_7|d:d|_8|d;d|_9|dd|_<|d?d|_=|d@d|_>d|_?|dAd|_@|dBdC|_A|dDdE|_B|dFd|_C|dGd|_D|dHd|_E|dIdJ|_F|dKdJ|_G|dLtH|_I|dMd|_J|dNi|_K|dOd|_L|dPd|_M|dQtN|_O|jLsU|PD]*\}}z tQ|||Wq*tRyT}ztSTdR|dS|dT||d}~ww|jUddUdS)VN max_lengthmax_new_tokens min_lengthrmin_new_tokensearly_stoppingFmax_time stop_strings do_sample num_beamsr(num_beam_groups penalty_alpha dola_layers use_cacheTcache_implementation cache_configreturn_legacy_cacheprefill_chunk_size temperature?top_k2top_pmin_p typical_pepsilon_cutoff eta_cutoffdiversity_penaltyrepetition_penaltyencoder_repetition_penaltylength_penaltyno_repeat_ngram_size bad_words_idsforce_words_idsrenormalize_logits constraintsforced_bos_token_idforced_eos_token_idremove_invalid_values exponential_decay_length_penaltysuppress_tokensbegin_suppress_tokensforced_decoder_ids sequence_bias token_healingguidance_scale low_memorywatermarking_confignum_return_sequencesrPrQrRrSreturn_dict_in_generate pad_token_id bos_token_id eos_token_idencoder_no_repeat_ngram_sizedecoder_start_token_idnum_assistant_tokensnum_assistant_tokens_scheduleconstantassistant_confidence_thresholdg?prompt_lookup_num_tokensmax_matching_ngram_sizeassistant_early_exitassistant_lookbehind target_lookbehindcompile_configdisable_compilegeneration_kwargsrrrz Can't set z with value z for )is_init)VpoprTrVrWrXrYrZr[r\r]r^r_r`rarbrcCACHE_CONFIG_MAPPING isinstancedict from_dictrdrerfrhrjrkrlrmrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrBaseWatermarkingConfigWatermarkingConfigrrPrQrRrSrrrrrr is_assistantrrrrrrrr CompileConfigrrrrrrritemssetattrAttributeErrorloggererrorvalidate)selfkwargscache_config_classrkeyvalueerrrMrMrN__init__s      zGenerationConfig.__init__cCst|jddS)NTignore_metadata)hashto_json_stringrrMrMrN__hash__szGenerationConfig.__hash__cCs2t|tsdS|jddd}|jddd}||kS)NFT)use_diffr)rrOr)rotherself_without_metadataother_without_metadatarMrMrN__eq__s zGenerationConfig.__eq__cCs|jjd|jddS)N Tr __class__r@rrrMrMrN__repr__ szGenerationConfig.__repr__Nassistant_modelrreturncCs|jdus |jdurtj}n?|jdkr8|jdur4|jdur0|jdkr0|jdur0|jdkr0tj}ntj }ntj }n|j dkrAtj }n |jdurJtj }ntj}|dus[|jdus[|jdurg|dvrctj}ntd|jdury|dvrutj}|Std|S) a Returns the generation mode triggered by the [`GenerationConfig`] instance. Arg: assistant_model (`PreTrainedModel`, *optional*): The assistant model to be used for assisted generation. If set, the generation mode will be assisted generation. Returns: `GenerationMode`: The generation mode triggered by the instance. Nr(FrT)r8r9zYou've set `assistant_model`, which triggers assisted generate. Currently, assisted generate is only supported with Greedy Search and Sample.zYou've set `dola_layers`, which triggers DoLa generate. Currently, DoLa generate is only supported with Greedy Search and Sample.)rxrvr6rKr]r\rhr_rDrErFr^rLrJrIrrrG ValueErrorr`rH)rrgeneration_moderMrMrNget_generation_modes@           z$GenerationConfig.get_generation_modeFc Cs|jdvrtd|jd|jdur!|jdkr!td|jd|jdur5|jdkr5td|jdd }|r;d }|jd urd |}|jdurZ|jd krZt|jd|jdt |j durp|j d krpt|jd|j dt |j durt|jd|j dt |j dur|j d krt|jd|j dt |j dur|j dkr|jdurt|jd|j dt |jdur|jdkrt|jd|jdt |jdur|jdkrt|jd|jdt |jdurtdt d|_|jdkrcd|}|jd urt|jd|jdt |jdur |jdkr t|jd|jdt |jdur8|jdkr8t|jd|jdt |jdurP|jd krPt|jd|jdt |jdurbt|jd|jdt n|jduso|jdurd |}|jd!urt|jd"|jd|jdur|jdkrt|jd|jd|jdks|jdkrd#}|jd!urt|d$|j|jdkrt|d%|jdkrt|d&|jdur|jdus|jd'krtd(|jd)t |jdkr|jdkr|jd urtd*|jd+n|j|jkrtd,|jd-|jd+|jdur2|jtvr2td.|jd/t|jdur_t|j}|durLtd0|jd1t|j|sZ| |j|_|j!|j"d urd2}d3D]} t#|| durt$%|j| t#|| d4qi|j&durt|j&t'st|j&t(std5t)t' |j&|_&|j&!t|j*t+std6t,|j*d7|j-d!ur|j.D]} t#|| d!urtd8| d9| d:t qd;} | D]} t/|| rtd<| d=qdS)>aa Validates the values of the attributes of the [`GenerationConfig`] instance. Raises exceptions in the presence of parameterization that can be detected as incorrect from the configuration instance alone. Note that some parameters not validated here are best validated at generate runtime, as they may depend on other inputs and/or the model, such as parameters related to the generation length. Arg: is_init (`bool`, *optional*, defaults to `False`): Whether the validation is performed during the initialization of the instance. >FTneverz6`early_stopping` must be a boolean or 'never', but is .Nrz0`max_new_tokens` must be greater than 0, but is z*`pad_token_id` should be positive but got z. This will cause errors when batch generating, if there is padding. Please set `pad_token_id` explicitly as `model.generation_config.pad_token_id=PAD_TOKEN_ID` to avoid errors in generationz This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.Fz`do_sample` is set to `False`. However, `{flag_name}` is set to `{flag_value}` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `{flag_name}`.rgrf) flag_name flag_valuerjrkrlrirhrnrmroz-`num_beams` is set to None - defaulting to 1.r(z`num_beams` is set to 1. However, `{flag_name}` is set to `{flag_value}` -- this flag is only used in beam-based generation modes. You should set `num_beams>1` or unset `{flag_name}`.rYr^rprsrxaone of `constraints`, `force_words_ids` is not `None`, triggering constrained beam search. However, `{flag_name}` is set to `{flag_value}`, which is incompatible with this generation mode. Set `constraints` and `force_words_ids` to `None` or unset `{flag_name}` to continue.Tr\zu`diversity_penalty` is not 0.0 or `num_beam_groups` is not 1, triggering group beam search. In this generation mode, z"`do_sample` must be set to `False`z4`num_beams` should be divisible by `num_beam_groups`zZ`diversity_penalty` should be greater than `0.0`, otherwise your groups will be identical.g333333?z]`dola_layers` is set to trigger DoLa decoding, but `repetition_penalty` is set to a value of zo, which could induce unwanted repetition. The recommended value for DoLa decoding is `repetition_penalty>=1.2`.z_Greedy methods without beam search do not support `num_return_sequences` different than 1 (got z).z`num_return_sequences` (z-) has to be smaller or equal to `num_beams` (z Invalid `cache_implementation` (z). Choose one of: zJYou provided a `cache_config` but the cache implementation you are using (zl) does not require any config. Make sure to use the correct cache implementation matching your cache config.zrYou have set `use_cache` to `False`, but {cache_arg} is set to {cache_arg_value}. {cache_arg} will have no effect.)rbrcrd) cache_argcache_arg_valuez`watermarking_config` as a dict is deprecated. Please construct `watermarking_config` object with `WatermarkingConfig` or `SynthIDTextWatermarkingConfig` class.z0You provided `compile_config` as an instance of z0, but it must be an instance of `CompileConfig`.z5`return_dict_in_generate` is NOT set to `True`, but `z5` is. When `return_dict_in_generate` is not `True`, `z ` is ignored.)logits_processorstopping_criteriaprefix_allowed_tokens_fn synced_gpusrstreamernegative_prompt_idsnegative_prompt_attention_maskz Argument `zn` is not a valid argument of `GenerationConfig`. It should be passed to `generate()` (or a pipeline) directly.)0rYrrVrwarningswarnr\rfformat UserWarningrjrkrlrhr_rmror]r^rprsrxrvr`rqrrbALL_CACHE_IMPLEMENTATIONSrcrgetrrrragetattrr warning_oncerrSynthIDTextWatermarkingConfig FutureWarningrrtyperextra_output_flagshasattr) rr fix_locationgreedy_wrong_parameter_msgsingle_beam_wrong_parameter_msgconstrained_wrong_parameter_msggroup_error_prefix cache_classno_cache_warningarg_nameextra_output_flaggenerate_argumentsargrMrMrNrNs              $                  zGenerationConfig.validatesave_directoryconfig_file_name push_to_hubc Ksz-tjdd }|Wdn1swYt|dkr,ttdd|DWntyA}ztdt|d}~ww|dd}|durbtd t| d ddur^td ||d <|durh|nt }t j |rxtd |d t j|dd|r|dd}|d|t j jd} |j| fi|} ||} t j ||} |j| ddtd| |r|j|| | || d ddSdS)a) Save a generation configuration object to the directory `save_directory`, so that it can be re-loaded using the [`~GenerationConfig.from_pretrained`] class method. Args: save_directory (`str` or `os.PathLike`): Directory where the configuration JSON file will be saved (will be created if it does not exist). config_file_name (`str` or `os.PathLike`, *optional*, defaults to `"generation_config.json"`): Name of the generation configuration JSON file to be saved in `save_directory`. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. T)recordNrcSsg|]}|jqSrM)message).0wrMrMrN jsz4GenerationConfig.save_pretrained..zThe generation config instance is invalid -- `.validate()` throws warnings and/or exceptions. Fix these issues to save the configuration. Thrown during validation: use_auth_tokenrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.zProvided path (z#) should be a directory, not a file)exist_okcommit_messagerepo_idrzConfiguration saved in )rr)rcatch_warningsrlenrstrrrrrrospathisfileAssertionErrormakedirssplitsep _create_repo_get_files_timestampsjoin to_json_filerinfo_upload_modified_files) rrrrrcaught_warningsexcrrrfiles_timestampsoutput_config_filerMrMrNsave_pretrainedLs^       z GenerationConfig.save_pretrainedmainpretrained_model_name cache_dirforce_downloadlocal_files_onlyrrevisioncKs*|dur|nt}|dd} |dd} |dd} |dd} |dd} |dd }|d d}| durFtd t|durDtd | }d |d}| durS| |d<tj||}t |}tj |}tj tj| |rt|}d}nAt |r|}t |}n6|}zt||||| | ||||| |d }t||}Wntytytd|d|d|dwz ||}||d <Wntjtfytd|dw|rtd|n td|d||ddur|j|fi|\}}t||_||fS|j|fi|}t||_|S)a1 Instantiate a [`GenerationConfig`] from a generation configuration file. Args: pretrained_model_name (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~GenerationConfig.save_pretrained`] method, e.g., `./my_model_directory/`. config_file_name (`str` or `os.PathLike`, *optional*, defaults to `"generation_config.json"`): Name of the generation configuration JSON file to be loaded from `pretrained_model_name`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final configuration object. If `True`, then this functions returns a `Tuple(config, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not configuration attributes: i.e., the part of `kwargs` which has not been used to update `config` and is otherwise ignored. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. kwargs (`Dict[str, Any]`, *optional*): The values in kwargs of any keys which are configuration attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* configuration attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: [`GenerationConfig`]: The configuration object instantiated from this pretrained model. Examples: ```python >>> from transformers import GenerationConfig >>> # Download configuration from huggingface.co and cache. >>> generation_config = GenerationConfig.from_pretrained("openai-community/gpt2") >>> # E.g. config was saved using *save_pretrained('./test/saved_model/')* >>> generation_config.save_pretrained("./test/saved_model/") >>> generation_config = GenerationConfig.from_pretrained("./test/saved_model/") >>> # You can also specify configuration names to your generation configuration file >>> generation_config.save_pretrained("./test/saved_model/", config_file_name="my_configuration.json") >>> generation_config = GenerationConfig.from_pretrained("./test/saved_model/", "my_configuration.json") >>> # If you'd like to try a minor variation to an existing configuration, you can also pass generation >>> # arguments to `.from_pretrained()`. Be mindful that typos and unused arguments will be ignored >>> generation_config, unused_kwargs = GenerationConfig.from_pretrained( ... "openai-community/gpt2", top_k=1, foo=False, do_sample=True, return_unused_kwargs=True ... ) >>> generation_config.top_k 1 >>> unused_kwargs {'foo': False} ```Nresume_downloadproxiesr subfolderr_from_pipeline _from_autoFrrrconfig) file_typefrom_auto_classusing_pipelineT) rrrrrr user_agentrrrz!Can't load the configuration of 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filez"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at return_unused_kwargs)rrrrrrrrrrexistsrrrrrEnvironmentError Exception_dict_from_json_filejsonJSONDecodeErrorUnicodeDecodeErrorrr rrrr)clsrrrrrrrrrrrr from_pipeliner commit_hashr config_pathis_localresolved_config_fileconfiguration_file config_dictr unused_kwargsrMrMrNfrom_pretraineds_                 z GenerationConfig.from_pretrained json_filecCs@t|ddd }|}Wdn1swYt|S)Nrutf-8encoding)openreadr%loads)r(r2readertextrMrMrNr$Ts  z%GenerationConfig._dict_from_json_filer/cKs|dd}|dd|ddd|vr d|vr |d|d<|dii||}|jdi|}td||rA||fS|S) a Instantiates a [`GenerationConfig`] from a Python dictionary of parameters. Args: config_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the configuration object. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the configuration object. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. r FrNrrzGenerate config rM)rupdaterr )r(r/rr rr0rMrMrNrZs    zGenerationConfig.from_dictdcCs^|dddurt|dtst|ddd|d<|D] }t|tr,||q dS)a. Checks whether the passed dictionary and its nested dicts have a *torch_dtype* key and if it's not None, converts torch.dtype to a string of just the type. For example, `torch.float32` get converted into *"float32"* string, which can then be stored in the json format. torch_dtypeNrr()rrrrvaluesrdict_torch_dtype_to_str)rr=rrMrMrNr@|s   z(GenerationConfig.dict_torch_dtype_to_strcCsZ|}t}i}|D]\}}||vs!|dks!|||kr%|||<q|||S)a' Removes all attributes from config which correspond to the default config attributes for better readability and serializes to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance, r)to_dictrOrr@)rr/default_config_dictserializable_config_dictrrrMrMrN to_diff_dicts  zGenerationConfig.to_diff_dictcCsLt|j}d|vr |d=d|vr|d=d|vr|d=t|d<|||S)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. rrrr)copydeepcopy__dict__rr@routputrMrMrNrAs  zGenerationConfig.to_dictTrrcsp|dur |}n|}|rtD]}||dqfddfdd|}|}tj|dddd S) aG Serializes this instance to a JSON string. Args: use_diff (`bool`, *optional*, defaults to `True`): If set to `True`, only the difference between the config instance and the default `GenerationConfig()` is serialized to JSON string. ignore_metadata (`bool`, *optional*, defaults to `False`): Whether to ignore the metadata fields present in the instance Returns: `str`: String containing all the attributes that make up this configuration instance in JSON format. TNcs@t|trfdd|DSt|trfdd|DS|S)Ncsi|] \}}t||qSrM)rrrrconvert_keys_to_stringrMrN zSGenerationConfig.to_json_string..convert_keys_to_string..csg|]}|qSrMrM)ritemrKrMrNrszSGenerationConfig.to_json_string..convert_keys_to_string..)rrrlistobjrKrMrNrLs  z?GenerationConfig.to_json_string..convert_keys_to_stringcs4t|trfdd|DSt|r|S|S)Ncsi|] \}}||qSrMrMrJconvert_dataclass_to_dictrMrNrMszVGenerationConfig.to_json_string..convert_dataclass_to_dict..)rrrrrArQrSrMrNrTs zBGenerationConfig.to_json_string..convert_dataclass_to_dictr indent sort_keys )rDrAMETADATA_FIELDSrr%dumps)rrrr/metadata_fieldrM)rTrLrNrs   zGenerationConfig.to_json_stringjson_file_pathcCsFt|ddd}||j|dWddS1swYdS)a Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this configuration instance's parameters will be saved. use_diff (`bool`, *optional*, defaults to `True`): If set to `True`, only the difference between the config instance and the default `GenerationConfig()` is serialized to JSON file. rr4r5rN)r7writer)rr\rwriterrMrMrNrs "zGenerationConfig.to_json_file model_configcs|}|dddd|D}|j|ddd|jdd}||urMt}|}D]}t|t||k}||vrL|rLt|||q2j durat fd d j Drad_ t _ S) a Instantiates a [`GenerationConfig`] from a [`PretrainedConfig`]. This function is useful to convert legacy [`PretrainedConfig`] objects, which may contain generation parameters, into a stand-alone [`GenerationConfig`]. Args: model_config (`PretrainedConfig`): The model config that will be used to instantiate the generation config. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. rNcSsi|] \}}|dur||qSNrMrJrMrMrNrMrNz6GenerationConfig.from_model_config..FT)r r)decoderc3s|] }t|dVqdS)FN)r)rrgeneration_configrMrN s  z5GenerationConfig.from_model_config..)rArrrget_text_configrOkeysrrranyrrr)r(r_r/decoder_configdefault_generation_configdecoder_config_dictattris_unsetrMrbrNfrom_model_configs(      z"GenerationConfig.from_model_configc sXg|D]\}}t||rt||||q|fdd|D}|S)a Updates attributes of this class instance with attributes from `kwargs` if they match existing attributes, returning all the unused kwargs. Args: kwargs (`Dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `Dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. csi|] \}}|vr||qSrMrMrJ to_removerMrNrM2rNz+GenerationConfig.update..)rrrappendr)rrrrr0rMrnrNr<s    zGenerationConfig.updater`)F)NF)NNFFNr)TF)T)!r@rArBrCrrrrrr r6rrr rrPathLikeboolr classmethodr1r$r rrr@rDrArrrrmr<rMrMrMrNrOdsv%w >  N  :! , +rOc@seZdZdZeddZdeeej ffddZ de ee ffdd Z d d Zd d ZddZddZeddZeddZdS)rzGeneric watermarking configcKs^|di|}g}|D]\}}t||r!t|||||q |D]}||dq$|S)a Constructs a BaseWatermarkingConfig instance from a dictionary of parameters. Args: config_dict (Dict[str, Any]): Dictionary containing configuration parameters. **kwargs: Additional keyword arguments to override dictionary values. Returns: BaseWatermarkingConfig: Instance of BaseWatermarkingConfig constructed from the dictionary. NrM)rrrrpr)r(r/rrrorrrMrMrNr:s    z BaseWatermarkingConfig.from_dictr\cCsZt|ddd}|}tj|dddd}||WddS1s&wYdS) z Save this instance to a JSON file. Args: json_file_path (Union[str, os.PathLike]): Path to the JSON file in which this configuration instance's parameters will be saved. rr4r5r TrUrXN)r7rAr%rZr])rr\r^r/ json_stringrMrMrNrPs  "z#BaseWatermarkingConfig.to_json_filercCst|j}|S)z Serializes this instance to a Python dictionary. Returns: Dict[str, Any]: Dictionary of all the attributes that make up this configuration instance. )rErFrGrHrMrMrNrA]s zBaseWatermarkingConfig.to_dictccs*t|jD] \}}||fVq dSr`rErFrGr)rrkrrMrMrN__iter__gs zBaseWatermarkingConfig.__iter__cCs|jjd|S)NrrrrMrMrNrkszBaseWatermarkingConfig.__repr__cCstj|jdddS)z Serializes this instance to a JSON formatted string. Returns: str: JSON formatted string representing the configuration instance. r )rVrX)r%rZrGrrMrMrNrnsz%BaseWatermarkingConfig.to_json_stringcKs,|D]\}}t||rt|||qdS)z Update the configuration attributes with new values. Args: **kwargs: Keyword arguments representing configuration attributes and their new values. N)rrr)rrrrrMrMrNr<ws   zBaseWatermarkingConfig.updatecCdSr`rMrrMrMrNrzBaseWatermarkingConfig.validatecCrwr`rM)r vocab_sizerMrMrNconstruct_processorrxz*BaseWatermarkingConfig.construct_processorN)r@rArBrCrsrr rrrqrr rrArvrrr<rrrzrMrMrMrNr6s     rc @sheZdZdZ     ddeedeed eed eed eef d d ZddZ deddfddZ dS)ra Class that holds arguments for watermark generation and should be passed into `GenerationConfig` during `generate`. See [this paper](https://arxiv.org/abs/2306.04634) for more details on the arguments. Accepts the following keys: - greenlist_ratio (`float`): Used for watermarking. The ratio of "green" tokens used to the vocabulary size. Defaults to 0.25. - bias (`float`): Used with watermarking. The bias added to the selected "green" tokens' logits. Defaults to 2.0. - hashing_key (`int`): Hashing key used for watermarking. Defaults to 15485863 (the millionth prime). - seeding_scheme (`str`): Algorithm to use for watermarking. Accepts values: - "lefthash" (default): "green" tokens selection depend on the last token (Algorithm 2 from the paper) - "selfhash": "green" tokens selection depends on the current token itself (Algorithm 3 from the paper) The downside of this scheme is that it considers all possible next tokens and can be slower than "lefthash". - context_width(`int`): The context length of previous tokens to use in seeding. Higher context length makes watermarking more robust. ?@Klefthashr(greenlist_ratiobias hashing_keyseeding_scheme context_widthcCs"||_||_||_||_||_dSr`)rrrrr)rrrrrrrMrMrNrs  zWatermarkingConfig.__init__cCsvd}|jdvrt|jdd|jdd|jkrdks)nt|jdd |jd|jd ks9t|jd d |jddS) N}Some of the keys in `watermarking_config` are defined incorrectly. `{key}` should be {correct_value}` but found {found_value})selfhashr~rz[`selfhash`, `lefthash`]r correct_value found_valuernrgrzin range between 0.0 and 1.0r(rza positive integer)rrrrrrwatermark_missing_arg_msgrMrMrNrs6  zWatermarkingConfig.validateryrr*c Cs t|||j|j|j|j|jdS)N)rydevicerrrrr)r*rrrrrrryrrMrMrNrzsz&WatermarkingConfig.construct_processorN)r{r|r}r~r() r@rArBrCr floatintrrrrzrMrMrMrNrs( rc@s`eZdZdZ     ddedeeded ed ed ed efd dZddZdeddfddZ dS)ra Class that holds arguments for watermark generation and should be passed into `GenerationConfig` during `generate`. See [this paper](https://www.nature.com/articles/s41586-024-08025-4) for more details on the arguments. Args: ngram_len (`int`): Ngram length. keys (`List[int]`): A sequence of watermarking keys, one for each depth. context_history_size (`int`, *optional*, defaults to 1024): Size of the tensor to keep track of seen contexts. sampling_table_seed (`int`, *optional*, defaults to 0): Random seed to generate the sampling table. sampling_table_size (`int`, *optional*, defaults to 65536): Size of the sampling table. skip_first_ngram_calls (`bool`, *optional*, defaults to `False`): Whether to skip first ngram calls. debug_mode (`bool`, optional, *optional*, defaults to `False`): Logits are modified to uniform one got before watermarking modification is applied. This is to test the implementation. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, SynthIDTextWatermarkingConfig >>> tokenizer = AutoTokenizer.from_pretrained('google/gemma-2-2b', padding_side="left") >>> model = AutoModelForCausalLM.from_pretrained('google/gemma-2-2b') >>> # SynthID Text configuration >>> watermarking_config = SynthIDTextWatermarkingConfig( ... keys=[654, 400, 836, 123, 340, 443, 597, 160, 57], ... ngram_len=5, ... ) >>> # Generation with watermarking >>> tokenized_prompts = tokenizer(["Once upon a time, "], return_tensors="pt", padding=True) >>> output_sequences = model.generate( ... **tokenized_prompts, watermarking_config=watermarking_config, do_sample=True, max_new_tokens=10 ... ) >>> watermarked_text = tokenizer.batch_decode(output_sequences, skip_special_tokens=True) ``` rF ngram_lenrfcontext_history_sizesampling_table_seedsampling_table_sizeskip_first_ngram_calls debug_modecCs.||_||_||_||_||_||_||_dSr`)rrfrrrrr)rrrfrrrrrrMrMrNrs  z&SynthIDTextWatermarkingConfig.__init__cCs(d}|jdkrt|jdd|jddS)Nrirz< 2**24r)rrrrrMrMrNrs z&SynthIDTextWatermarkingConfig.validateryrr*c Cs&t|j|j|j|j|j||j|jdS)N)rrfrrrrrr)r)rrfrrrrrrrMrMrNrz$sz1SynthIDTextWatermarkingConfig.construct_processorN)rrrFF) r@rArBrCrr rrrrrzrMrMrMrNrs0/ rc@sxeZdZUdZdZeed<dZeeed<dZ e e e fed<dZ e ed <dZeeed <dZd ee effd d ZdS)raU Class that holds arguments relative to `torch.compile` behavior, when using automatic compilation in `generate`. See [`torch.compile`](https://pytorch.org/docs/stable/generated/torch.compile.html) for more details on the arguments. Args: fullgraph (`bool`, *optional*, defaults to `True`): If `True`, requires that the whole forward be capturable in a single graph. dynamic (`bool` or `None`, *optional*): Whether to try to use dynamic shape graphs. backend (`str` or `Callable`, *optional*, defaults to `"inductor"`): Backend to be used. mode (`str`, *optional*, defaults to `"reduce-overhead"`): Controls balance between performance and overhead. options (`dict`, *optional*): A dictionary of options to pass to the backend. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, CompileConfig >>> tokenizer = AutoTokenizer.from_pretrained('google/gemma-2-2b') >>> model = AutoModelForCausalLM.from_pretrained('google/gemma-2-2b').cuda() >>> # Automatic compile configuration, used with static cache >>> compile_config = CompileConfig(dynamic=True) >>> # Generation with static cache and compile config >>> input = tokenizer.encode("Hello there, how", return_tensors="pt").cuda() >>> output = model.generate( ... input, do_sample=False, max_new_tokens=300, cache_implementation="static", compile_config=compile_config ... ) >>> output_text = tokenizer.batch_decode(output, skip_special_tokens=True)[0] ``` T fullgraphNr5inductorbackendzreduce-overheadmodeoptionsrcCstdd|jDS)z0Serializes this instance to a Python dictionary.cSsi|] \}}|dkr||qS)_compile_all_devicesrMrJrMrMrNrM`rNz)CompileConfig.to_dict..rurrMrMrNrA^szCompileConfig.to_dict)r@rArBrCrrr__annotations__r5r rr rrrrrrr rrArMrMrMrNr1s  # r)ArCrEr%rrabcrr dataclassesrrtypingrrrr r r r rrconfiguration_utilsrutilsrrrrrrrrrmodeling_utilsr get_loggerr@rrYr NEED_SETUP_CACHE_CLASSES_MAPPINGQUANT_BACKEND_CLASSES_MAPPINGr cache_utilsrrr r!r"r#r$r%r&r'logits_processr)r*rPrfr6rOrrrrrMrMrMrNsj$  ,  0  [RMY