o  hXG@snddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZddlZddlZddlmZddlmZddlmZddlmZddlmZdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;ddlZ>m?Z?ddl@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZIddl%mJZJmKZKmLZLmMZMddlNmOZOmPZPmQZQmRZRmSZSmTZTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhddlimjZjmkZkmlZlmmZmmnZnmoZompZper+ddlqmrZrddlsmtZtddlumvZve8wexZye4rd?Zd@edA dBdfdCejdDejdEedFedGed2ejf dHdIZdJe edKe eejfdLejd2ejfdMdNZdS)PN) dataclass) TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)version)nn) functional)AssistantVocabTranslatorCache)Cache DynamicCacheEncoderDecoderCacheHybridChunkedCacheOffloadedCacheQuantizedCacheConfig StaticCache)PretrainedConfig)is_deepspeed_zero3_enabled)is_fsdp_managed_module)CausalLMOutputWithPastSeq2SeqLMOutput)isin_mps_friendly)ExtensionsTrie) ModelOutputis_accelerate_availableis_hqq_availableis_optimum_quanto_availableis_torchdynamo_exportinglogging)DisjunctiveConstraintPhrasalConstraint) BeamScorerBeamSearchScorerConstrainedBeamSearchScorer) AssistedCandidateGenerator-AssistedCandidateGeneratorDifferentTokenizersCandidateGeneratorEarlyExitCandidateGeneratorPromptLookupCandidateGenerator%UniversalSpeculativeDecodingGenerator_crop_past_key_values_prepare_attention_mask_prepare_token_type_ids) NEED_SETUP_CACHE_CLASSES_MAPPINGQUANT_BACKEND_CLASSES_MAPPINGGenerationConfigGenerationMode)#EncoderNoRepeatNGramLogitsProcessor'EncoderRepetitionPenaltyLogitsProcessorEpsilonLogitsWarperEtaLogitsWarperExponentialDecayLengthPenaltyForcedBOSTokenLogitsProcessorForcedEOSTokenLogitsProcessorHammingDiversityLogitsProcessorInfNanRemoveLogitsProcessorLogitNormalizationLogitsProcessorListMinLengthLogitsProcessor!MinNewTokensLengthLogitsProcessorMinPLogitsWarperNoBadWordsLogitsProcessorNoRepeatNGramLogitsProcessor PrefixConstrainedLogitsProcessor RepetitionPenaltyLogitsProcessorSequenceBiasLogitsProcessor$SuppressTokensAtBeginLogitsProcessorSuppressTokensLogitsProcessorTemperatureLogitsWarperTopKLogitsWarperTopPLogitsWarperTypicalLogitsWarper.UnbatchedClassifierFreeGuidanceLogitsProcessor)ConfidenceCriteriaEosTokenCriteriaMaxLengthCriteriaMaxTimeCriteriaStoppingCriteriaStoppingCriteriaListStopStringCriteria)PreTrainedModel)PreTrainedTokenizerBase) BaseStreamer)AlignDevicesHookadd_hook_to_module)past_key_values cache_paramsstatememspast_buckets_statesc@seZdZUdZejed<dZee ej ed<dZ ee ej ed<dZ ee e ej ed<dZ ee e ej ed<dZee e e ej ed<dS) GenerateDecoderOnlyOutputaw Outputs of decoder-only generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. sequencesNscoreslogits attentions hidden_statesr])__name__ __module__ __qualname____doc__torch LongTensor__annotations__rdrr FloatTensorrerfrgr]rprpq/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/generation/utils.pyrbs  "rbc@seZdZUdZejed<dZee ej ed<dZ ee ej ed<dZ ee ej ed<dZ ee ej ed<dZee e ej ed<dZee e ej ed <dZee e ej ed <dZee e e ej ed <dS) GenerateEncoderDecoderOutputaI Outputs of encoder-decoder generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. rcNrdreencoder_attentionsencoder_hidden_statesdecoder_attentionscross_attentionsdecoder_hidden_statesr])rhrirjrkrlrmrnrdrr rorersrtrurvrwr]rprprprqrrs  #"rrc@seZdZUdZejed<dZeej ed<dZ ee ej ed<dZ ee ej ed<dZ eejed<dZee e ej ed<dZee e ej ed <dZee e e ej ed <dS) GenerateBeamDecoderOnlyOutputa Outputs of decoder-only generation models, when using beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. rcNsequences_scoresrdre beam_indicesrfrgr])rhrirjrkrlrmrnryrrordr rerzrfrgr]rprprprqrxs  "rxc@seZdZUdZejed<dZeej ed<dZ ee ej ed<dZ ee ej ed<dZ eejed<dZee ej ed<dZee ej ed <dZee e ej ed <dZee e ej ed <dZee e ej ed <dZee e e ej ed <dS) GenerateBeamEncoderDecoderOutputa Outputs of encoder-decoder generation models, when using beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size*num_beams*num_return_sequences, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`tuple(tuple(torch.FloatTensor)))`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. rcNryrdrerzrsrtrurvrwr])rhrirjrkrlrmrnryrrordr rerzrsrtrurvrwr]rprprprqr{s  *"r{c"@seZdZdZdejdeejdeejdeejejffddZ dejdeejdeejdeejejffdd Z ddejd ee d eejdeejdeejf d dZ ddeej deej deeeej fdeej eeeeej fffddZ ddeej deej deeeej fdejfddZdej dedeeefdejfddZdej deededeeeffddZ ddededeeej fdej deejdeejeeej fff d d!Ze " # dd$ed%edeejdeejeeefffd&d'Z # "dd(edeeefd%ed)edeeeff d*d+Zd,d-Zdedejdej d.d/d0ed1d2d3d2dede fd4d5Z! dded6ed7ejd8e"eej ge#efd0eedeedeeeefd9eej d:eej defd;d<Z$ dded=ee%d>ed2de%fd?d@Z&dAe'ee%fdBe'ee%fde'ee%ffdCdDZ( #ddEej dFeej dGeej dHedej f dIdJZ)dKdLZ*dMdNZ+deeeffdOdPZ,dQdRZ-dSdTZ. ddeedUeedVedeeeffdWdXZ/dYdZZ0deeee'eefffd[d\Z1d]eded^edejde f d_d`Z2defdadbZ3deded.d/dedcedejdefdddeZ4defdfdgZ5 ddedheedee'ejeffdidjZ6e7 ddeej deed0eed=ee%d8ee"eej ge#efdkeed.ed/dledmd9eej d:eej dUeede'e8ejffdndoZ9dpedkedejdefdqdrZ: ddejd>ed2dejfdsdtZ;dejdue'ee#efd0ed=e%dedkedldmde'edejd0ed=e%dedkedledmde'e0` and `top_k>1` - *multinomial sampling* if `num_beams=1` and `do_sample=True` - *beam-search decoding* if `num_beams>1` and `do_sample=False` - *beam-search multinomial sampling* if `num_beams>1` and `do_sample=True` - *diverse beam-search decoding* if `num_beams>1` and `num_beam_groups>1` - *constrained beam-search decoding* if `constraints!=None` or `force_words_ids!=None` - *assisted decoding* if `assistant_model` or `prompt_lookup_num_tokens` is passed to `.generate()` To learn more about decoding strategies refer to the [text generation strategies guide](../generation_strategies). input_ids inputs_embedscache_positionreturncCstr ||||S|dur'|jddkr'|dd|jd df}||fS|dus4|d|jdkrF|dd|jd df}||fS|jd|jdkrX|dd|f}||fS)a Generic cache-dependent input preparation The code is put in a separate function to allow granular unit testing as it needs a different implementation to be exportable. If we have cache: let's slice `input_ids` through `cache_position`, to keep only the unprocessed tokens - Exception 1: when passing input_embeds, input_ids may be missing entries - Exception 2: some generation methods do special slicing of input_ids, so we don't need to do it here - Exception 3: with synced GPUs cache_position may go out of bounds, but we only want dummy token in that case. - Excpetion 4: If input_embeds are passed then slice it through `cache_position`, to keep only the unprocessed tokens and generate the first token for each sequence. Later use the generated Input ids for continuation. The current implementation does not rely on ``self`` and could be a class method. It is left as a standard method to be easily rewritten. Nr$r)r",_cache_dependant_input_preparation_exportingshapeselfr}r~rrprprq"_cache_dependant_input_preparation{sz2GenerationMixin._cache_dependant_input_preparationcsv|dur|dd|f}||fSddddddt|jdd kfd d fd d |||g\}}||fS) z This method implements method ``_cache_dependant_input_preparation`` with :func:`torch.cond` to make it exportable with :func:`torch.export.export`. The code is put in a separate function to allow granular unit testing. NcS|dd|jd dfSNrr)r~rrprprqbranch_1zNGenerationMixin._cache_dependant_input_preparation_exporting..branch_1cSrrrr}rrprprqbranch_2rzNGenerationMixin._cache_dependant_input_preparation_exporting..branch_2cSs|dd|fSNrprrprprqbranch_3szNGenerationMixin._cache_dependant_input_preparation_exporting..branch_3r$rcs|||fSrrpr}r~r)rrprqszNGenerationMixin._cache_dependant_input_preparation_exporting..cs.|t|d|jdkfdd||gfS)Nrr$cs(t|jd|jdkdd||gS)Nr$rcSs|SrrprrprprqrszrGenerationMixin._cache_dependant_input_preparation_exporting......rlcondrr)rrprqrs z`GenerationMixin._cache_dependant_input_preparation_exporting....rr)rrrprqrs rrrp)rrrrqrs 0   zu&z9GenerationMixin._prepare_model_inputs..Nz `inputs`: z` were passed alongside z0 which is not allowed. Make sure to either pass z or z=...r}r~zAYou passed `inputs_embeds` to `.generate()`, but the model class z doesn't have its forwarding implemented. See the GPT2 implementation for an example (https://github.com/huggingface/transformers/pull/21405), and feel free to open a PR with it!)rzMYou passed `inputs_embeds` and `input_ids` to `.generate()`. Please pick one.)rrhasattrrmain_input_namerr ValueErrorrrrrrrrrh*_maybe_initialize_input_ids_for_generation)rrrr inputs_kwarghas_inputs_embeds_forwardingrprrq_prepare_model_inputsasH     z%GenerationMixin._prepare_model_inputscCs|dur|S|d}|jjr(|dur(|jdd}tj|tj|jddSd}| D]}t |tj r=|j d}nq.d|vrNtj|dftj|jdS|durVt d tj|dftj|jd|S) z3Initializes input ids for generation, if necessary.Nencoder_outputsrrir$rr~zB`bos_token_id` has to be defined when no `input_ids` are provided.)rrrlast_hidden_statesizerlonesrrvaluesrTensorrr)rrrrrrrrrprprqrs"    z:GenerationMixin._maybe_initialize_input_ids_for_generation inputs_tensorgeneration_configc Cs|j}|j}d|vr|djddkr|d}tj|jddtj|jd}|dur,|St|jdko;|jtj tjfv}|s@|S|duoKt ||d }|dupXt ||d } || } | |} | | || } | S)Nr}r$rrrelements test_elements) _pad_token_tensor_eos_token_tensorrrlrrrrrintranyne) rrrr pad_token_id eos_token_iddefault_attention_mask is_input_idsis_pad_token_in_inputs&is_pad_token_not_equal_to_eos_token_idcan_infer_attention_maskattention_mask_from_paddingrrprprq&_prepare_attention_mask_for_generations* z6GenerationMixin._prepare_attention_mask_for_generationrcs|}t|drt|drd|j_nt|tddgdfdd|D}tt |j j dvp:d v}|sHfd d|D}|j |d <|j |d <|durX|n|j}d|d <|||<|di||d<|S)N hf_device_map_hf_hookT)io_same_device)decoder_ cross_attn use_cachecs,i|]\}tfddDs|qS)c3s|]}|VqdSr) startswithrpargumentrprq z\GenerationMixin._prepare_encoder_decoder_kwargs_for_generation...r)rr)irrelevant_prefixrrqrs zRGenerationMixin._prepare_encoder_decoder_kwargs_for_generation..rrcsi|] \}}|vr||qSrprp)rrr)encoder_signaturerprqrsoutput_attentionsoutput_hidden_states return_dictrrp) get_encoderrrrr\r[rrrrrrr r r)rrrrrrencoder_kwargsencoder_accepts_wildcardrp)rrrq._prepare_encoder_decoder_kwargs_for_generations,       z>GenerationMixin._prepare_encoder_decoder_kwargs_for_generationrdecoder_start_token_idrcCs|durd|vr|d}nd|vr|dkr|d}nd}|dur%|j}|jdkrE|jd|kr>td|d|jd|dd}n tj|dftj|d |}|dur\|}||fSd |j j vss|j j d krxd |j jj vrx ||fS|j j d vr ||fS|dddf|dddfkrtj||gdd }d|vr|d}tjt|ddddf|fdd }||d<||fS)zGPrepares `decoder_input_ids` for generation with encoder-decoder modelsNrr}r$rz1`decoder_start_token_id` expected to have length z but got rrdonutzvision-encoder-decoder)whisperdimr)rrrrrviewrlrrrrhlowerr model_typerallitemcat ones_like)rrrrrrrrrprprq)_prepare_decoder_input_ids_for_generationsF     (z9GenerationMixin._prepare_decoder_input_ids_for_generationr$F expand_sizerc sldkr||fSfdd}|dur|jdd}||}|r2|ddur*td||d|d<||fS) zIExpands tensors from [batch_size, ...] to [batch_size * expand_size, ...]r$csH|D]}|dkr!||dur!t||tjr!||jdd||<q|S)Nrrr)rrlrrepeat_interleave)dict_to_expandrrrprq_expand_dict_for_generationVs zRGenerationMixin._expand_inputs_for_generation.._expand_dict_for_generationNrrrzMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rrr)rrr}rr!rpr rq_expand_inputs_for_generationIs   z-GenerationMixin._expand_inputs_for_generationoutputsnum_new_tokensc CsVtD]}||vr|dvrd}n|}t||||<nqd|vr7|d}tj||dddfdgdd|d<|sUd|vrT|d}tj|||jddfgdd|d<nd |vrp|d } tj| | | jddfgdd|d <|d d r|d dd||d <|S|d } tj | dd| d|d| j d  | j } t| | f|d <|S)N)rar`r]rrrrrr$rrTrr) ALL_CACHE_NAMESrrlr unsqueezenew_onesrrrrrtor) rr#rrr$possible_cache_name cache_namerrrpast_positions new_positionsrprprq#_update_model_kwargs_for_generationlsF (   z3GenerationMixin._update_model_kwargs_for_generationcCstd|jjd|j)NzGMake sure that a `_reorder_cache` function is correctly implemented in z to enable beam search for )NotImplementedErrorrri)rr]beam_idxrprprq_reorder_caches  zGenerationMixin._reorder_cacheassistant_modelrXlogits_processortarget_tokenizerrYassistant_tokenizerc Cstdd|||fD} |jdurt||||||d} | S|jdur0t|j|j|j|jd} | S| ro|jdurQt |||j j |j } t||||||||| d } | S|jdurdt||||||||d } | Std t|jjt||||||d} | S) zU Returns the candidate generator to be used in `assisted_generation` css|]}|duVqdSrrp)rrrprprqrz;GenerationMixin._get_candidate_generator..N)r}r2rrrr3)rnum_output_tokensmax_matching_ngram_size max_lengthT) r}r2rrrr3r4r5atm_translatorF)r}r2rrrr3r4r5z7Invalid value for `do_sample`: expected a boolean, got )rassistant_early_exitr-prompt_lookup_num_tokensr.rr8r9 do_samplerget_translatorr vocab_sizerr/r+rtyperhr*) rrr}rr2r3r4r5rdifferent_tokenizerscandidate_generatorr:rprprq_get_candidate_generatorsx 7 . # z(GenerationMixin._get_candidate_generatorinput_ids_seq_lengthencoder_input_idsprefix_allowed_tokens_fnnegative_prompt_idsnegative_prompt_attention_maskc Cst} |jdur|jdkr| t|j||| |jd|jdur)| t|jd|jdur@|jdkr@| t|j|j |j d|j durb|j dkrbt |j dkr\| t|j |d ntd t|jduru|jdkru| t|jd |jdur|jd kr| t|j|jdur|jd krt |j dkr| t|j|ntd t|jdur| t|j|j|jdur|jdur|jd kr| t|j|j|d|jdur|jdur|jd kr| t||j|j|d|dur| t||j |j |j dur | t!|j |j"dur| t#|j$|j"|d|j%dur*| t&|j'dur;| t(|j'|j||j)durK| t*|j)|d|j+duro|} |dks^|j dur`| n| d} | t,|j+| |d|j-duryt.d|/| |} |j0rS|j dkrt1|jt2rt |jd} nt1|jt3j4r|jj d d} nd} nd} |j5dur|j5dkr| t6|j5|j7dur|j7d kr| t8|j7| d|j9dur|j9dkr| t:|j9| d|j;dur| t<|j;| d|j=dur|j=dkr| t>|j=| d|j?dur3d|j?kr'dkr3nn | t@|j?| d|jAdurSd|jAkrFdkrSnn | tB|jA| |d|jCdure| |jCD|jEjF||jGdurq| tH| S)z This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. Nr$)unconditional_idsunconditional_attention_maskr sequence_bias)diversity_penalty num_beamsnum_beam_groups?r)penaltyrEzyPassing `encoder_repetition_penalty` requires some form of `input_ids` to be passed to `generate`, ignoring the argument.)rRrz{Passing `encoder_no_repeat_ngram_size` requires some form of `input_ids` to be passed to `generate`, ignoring the argument.rTzYou have explicitly specified `forced_decoder_ids`. Please remove the `forced_decoder_ids` argument in favour of `input_ids` or `decoder_input_ids` respectively.)top_kmin_tokens_to_keep)top_prU)min_prU)massrU)epsilonrU)rYrUr)IrAguidance_scaleappendrPrrLrIrNr>rOrPencoder_repetition_penaltyrrr8warningswarn UserWarningrepetition_penaltyrHno_repeat_ngram_sizerFencoder_no_repeat_ngram_sizer7 bad_words_idsrEr min_lengthrBmin_new_tokensrCrGforced_bos_token_idr<forced_eos_token_idr=r9remove_invalid_valuesr? exponential_decay_length_penaltyr;suppress_tokensrKbegin_suppress_tokensrJforced_decoder_idsr_merge_criteria_processor_listr=rlistrlr temperaturerLrTrMrVrNrWrD typical_prOepsilon_cutoffr9 eta_cutoffr:watermarking_configconstruct_processorrr?renormalize_logitsr@) rrrDrErFr3rrrGrH processors begin_indexrUrprprq_get_logits_processorsh                            **   z%GenerationMixin._get_logits_processorstopping_criteria tokenizercKst}|jdurt|jdd}|t|j|d|jdur'|t|jd|jdur>|dur4t d|t |j|d|j durL|t |j d|j rb|jdurb|jdkrb|t|jd|||}|S) Nmax_position_embeddings)r9r{)max_timeaThere are one or more stop strings, either in the arguments to `generate` or in the model's generation config, but we could not locate a tokenizer. When generating with stop strings, you must pass the model's tokenizer to the `tokenizer` argument of `generate`.) stop_stringsrz)rr)assistant_confidence_threshold)rVr9rrr[rSr|rTr}rrWrrR is_assistantr~rQrm)rrryrzrcriteriar{rprprq_get_stopping_criterias8        z&GenerationMixin._get_stopping_criteria default_list custom_listc Cst|dkr|St|}|D]B}d}|D]4}t|t|urIt|tr&dnd}td|dt|dt|dt|d ||d }nq|sQ||q|D] }||vr_||qT|S) a3 Merge user-defined processors/criteria with the ones instantiated inside `generate`. In case the same processor/criteria is present on both lists, use the user-defined one. (Note: up to v4.49.0, this funtion threw an exception is the same logit processor was found twice.) rFzstopping criteriazlogits processorz A custom z of type zt has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom z5 will take precedence. Please check the docstring of z$ to see related `.generate()` flags.T)rr@rrUrrr[)rrr final_listdefault using_customcustom object_typerprprqrms6     z.GenerationMixin._merge_criteria_processor_listrcrdrznormalize_logitsc Cs6|durt|djddd|j}|dt|}t| t|d dd}|rM| d|j j |jd}tj jj|dd}| d|jd}|dk}d|d}|ddd|f}|ddd|f}d||<||j j }|jd|}|dd|df|} |d| } d| |<| S)a Computes the transition scores of sequences given the generation scores (and beam indices, if beam search was used). This is a convenient method to quickly obtain the scores of the selected tokens at generation time. Parameters: sequences (`torch.LongTensor`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)`): Transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. Only required if a `num_beams>1` at generate-time. normalize_logits (`bool`, *optional*, defaults to `False`): Whether to normalize the logits (which, for legacy reasons, may be unnormalized). Return: `torch.Tensor`: A `torch.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)` containing the transition scores (logits) Examples: ```python >>> from transformers import GPT2Tokenizer, AutoModelForCausalLM >>> import numpy as np >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> tokenizer.pad_token_id = tokenizer.eos_token_id >>> inputs = tokenizer(["Today is"], return_tensors="pt") >>> # Example 1: Print the scores for each token generated with Greedy Search >>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, normalize_logits=True ... ) >>> # input_length is the length of the input prompt for decoder-only models, like the GPT family, and 1 for >>> # encoder-decoder models, like BART or T5. >>> input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1] >>> generated_tokens = outputs.sequences[:, input_length:] >>> for tok, score in zip(generated_tokens[0], transition_scores[0]): ... # | token | token string | log probability | probability ... print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}") | 262 | the | -1.414 | 24.33% | 1110 | day | -2.609 | 7.36% | 618 | when | -2.010 | 13.40% | 356 | we | -1.859 | 15.58% | 460 | can | -2.508 | 8.14% >>> # Example 2: Reconstruct the sequence scores from Beam Search >>> outputs = model.generate( ... **inputs, ... max_new_tokens=5, ... num_beams=4, ... num_return_sequences=4, ... return_dict_in_generate=True, ... output_scores=True, ... ) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False ... ) >>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores. >>> # Tip 1: recomputing the scores is only guaranteed to match with `normalize_logits=False`. Depending on the >>> # use case, you might want to recompute it with `normalize_logits=True`. >>> # Tip 2: the output length does NOT include the input length >>> output_length = np.sum(transition_scores.numpy() < 0, axis=1) >>> length_penalty = model.generation_config.length_penalty >>> reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty) >>> print(np.allclose(outputs.sequences_scores, reconstructed_scores)) True ```Nrrr$r)rlrrrr)rexpandrstackreshape transposerr?r r log_softmaxrsummaxrgather) rrcrdrzrbeam_indices_maskmax_beam_lengthbeam_sequence_indicescut_idxindicestransition_scoresrprprqcompute_transition_scores s&T$  z)GenerationMixin.compute_transition_scorescCs.|sgd}td|jjd|ddS)z Confirms that the model class is compatible with generation. If not, raises an exception that points to the right class to use. ) ForCausalLMForConditionalGenerationForSpeechSeq2Seq ForVision2SeqzThe current model class (z) is not compatible with `.generate()`, as it doesn't have a language model head. Classes that support generation often end in one of these names: .N) can_generate TypeErrorrrh)r$terminations_with_generation_supportrprprq_validate_model_classs z%GenerationMixin._validate_model_classcsdurdSjjr0jjs0gdfddtjDtfddD}|s0tdd}jjjjkrL|durJtd|d dS|dusT|dur\td |d dS) N)encoder_attention_headsencoder_ffn_dimencoder_layerscg|]}|vr|qSrprprattr)attributes_to_checkrprq z7GenerationMixin._validate_assistant..c3s(|]}tj|tj|kVqdSr)rrr)r2rrprqrs z6GenerationMixin._validate_assistant..zThe main model and the assistant don't have compatible encoder-dependent input shapes. Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.zc(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)z`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` rzThe main and assistant moedels have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` )rrdirrrget_text_configr?)rr2rzr5 are_equal doc_referencerp)r2rrrq_validate_assistants2  z#GenerationMixin._validate_assistantc Cs|t|ddtr|jst|jjd|jjr$dD]}| |dqg}t t |j j}d|vs7d|vrB|t t |jjO}|jjrt||jd}t|dd}|dura|durat|dd}|durrt t |jj}||O}t|dd}|dur|durt|dd}|durt t |jj} |d d | DO}|D]\}} | dur||vr||q|rtd |d dS) zXValidates model kwargs for generation. Generate argument typos will also be caught here.r]Nz does not support an instance of `Cache` as `past_key_values`. Please check the model documentation for supported cache formats.)rrrrdecodercSsh|]}d|qS)rrp)rxrprprq z9GenerationMixin._validate_model_kwargs..z8The following `model_kwargs` are not used by the model: zG (note: typos in the generate arguments will also show up in this list))rrrrrrrhrrrrrrrrrrrrr[) rrrunused_model_args model_argsrrencoder_model_argsrdecoder_model_argsrrprprq_validate_model_kwargssD       z&GenerationMixin._validate_model_kwargsc Cs |r|jdur|jdkrtd|jdt||jkr3|jjr"dnd}td|d|d |jd d }|r@|d |jd 7}|jdur\|j|jkr\td|jd|jd|t|j dur|j |}||jkrtd|j d|d|jd|tdSdSdS)z=Performs validation related to the resulting generated lengthNz0Using the model-agnostic default `max_length` (=zz) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.rr}zInput length of z is z, but `max_length` is set to z}. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`.z Generation will stop at the defined maximum length. You should decrease the minimum length and/or increase the maximum length.z" Note that `max_length` is set to z, its default value.z-Unfeasible length constraints: `min_length` (z.) is larger than the maximum possible length (z).z1Unfeasible length constraints: `min_new_tokens` (z$), when added to the prompt length (z/), is larger than the maximum possible length () max_new_tokensr9r]r^r_rrrrdre)rrinput_ids_lengthhas_default_max_lengthinput_ids_stringmin_length_error_suffixrdrprprq_validate_generated_lengthsT        z*GenerationMixin._validate_generated_lengthcCs,|jdur!|s|jdurtd|jd|jd|j||_n;|dkr;||jdkr;|jjs;|j|jd8_n!|r\|jtjkr\|j||_t|jdd}|dur\t |j||_|j dury|sqtd|j d |j d |j ||_ |S|dkr||jdkr|jjst |j |jdd |_ |S) z]Prepared max and min length in generation configs to avoid clashes between similar attributesNzBoth `max_new_tokens` (=z) and `max_length`(=z) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)r~r$r{zBoth `min_new_tokens` (=z) and `min_length`(=z) seem to have been set. `min_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)r) rr9rwarningrrrr5rminrerdr)rrrhas_default_min_lengthrrrr{rprprq_prepare_generated_length1sD       z)GenerationMixin._prepare_generated_lengthuse_model_defaultsrcKsd}|dur5|jjr0|jjt|jkr0t|jdkr0t|j}||jkr0t dt ||_|j}d}t |}|stt|jjj}|dusV|dur|tdkri}t}|jD]+\} } | dsm| dkrnq`t|| } t|j| } | | kr| | kr| || <t|| | q`t|dkrtd |d n(|jdur|jj|_|jdur|jj|_|jdur|jj|_|jdur|jj|_|jd i|} || fS) z Prepares the base generation config, then applies any generation configuration options from kwargs. This function handles retrocompatibility with respect to configuration files. FNra?You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed in v5. Please use and modify the model generation configuration (see https://huggingface.co/docs/transformers/generation_strategies#default-text-generation-configuration )Tz4.50.0rtransformers_versionzX`generation_config` default values have been modified to match model-specific defaults: z=. If this is not desired, please set these values explicitly.rp)r_from_model_config_original_object_hashhashrr&_get_non_default_generation_parametersr5from_model_configr]r^r_copydeepcopyr parser base_version__dict__rrrsetattrrrrrrrupdate)rrrrusing_model_generation_confignew_generation_configmodel_base_versionmodified_valuesdefault_generation_configr default_valuecustom_gen_config_valuemodel_gen_config_valuerrprprq_prepare_generation_configis`                z*GenerationMixin._prepare_generation_configcCsd|vr|jjstj|dddddftjddd}n2d|vr>|jjr>tj|dddddftjddd}ntj|dddftjddd}d}|ddur|d}d}t|tso|ddj d}nt |d r~| dur~| }||d}||d <|S) zbCalculates `cache_position` for the pre-fill stage based on `input_ids` and optionally past lengthr~rNr%r$decoder_inputs_embedsr]rget_seq_lengthr) rrrlrint64rrrrrrr)rr}rrrcacherprprq_get_initial_cache_positions".*&  z+GenerationMixin._get_initial_cache_positionc sd}t|dr7t|jdhkst|jddhkrdn dd|jDdfdd |jD}|dur=dS|jj}t|d krWd |vrWt t ||d Si}t|d rd| D]\}}|| urr|nqddur{td fdd|D}t||krt |D]}|D]}d|d|dvr||||<nqqnF |vrt t ||}ndvrɈdd dntddqn|D]} t |D]}d|d| dvr|| ||<nqqt |D]}||vrtd|dq|S)z Returns the device map for each decoder layer, to allocate the cache on the right device. Inspired from `dispatch_model` in accelerate. NrcpudiskcSsg|]}|dvr|qS)rrrp)rdrprprqrrzHGenerationMixin._get_layer_device_map_for_cache_init..rcs"i|] \}}||dvr n|qSrrp)rnamer) main_devicerprqrszHGenerationMixin._get_layer_device_map_for_cache_init..r$ get_decoderzw`model.get_decoder()` is not returning a named module of the model. This is unexpected, please open an issue on GitHub.csg|]}|vr|qSrprp)r module_name) decoder_namerprqrsrTz Decoder name z" not found in execution device mapzlayer z! has not been mapped to a device.)rrrrrrrnum_hidden_layersrdictfromkeysrange named_modulesr RuntimeErrorrrsplit) rexecution_device_maprlayer_device_maprmoduledecoder_mapped_modulesidxrlayerrp)rrrq$_get_layer_device_map_for_cache_initst *             z4GenerationMixin._get_layer_device_map_for_cache_initcache_implementation max_cache_lencCs~|dkrdt|jddvrd}t|}|jjp|ddu}t|dr,|r)|jjn|j}|d kr7t|jj |}t|d pLt || pL|j |kpLt |t } |d krX| pW|j |k} |rot|dro| pn|jjj |dd jd k} | rt|jd r||jj} n|j} |} |j||| || d} |di| |_|r| } |dd jd | d<t|j|di| |_|jS|j|jS)z Sets a cache for `generate`, that will persist across calls. A new cache will only be initialized a new `generate` call requires a larger cache or uses a different batch size. Returns the resulting cache object. hybridllama4rrhybrid_chunkedrN_cachesliding_windowmambarr$_pre_quantization_dtype)rmax_batch_sizerrrrrrp)rrr3rrrrself_attention_cacherrrrrrcross_attention_cacherrrrrrrreset)rrrrrr cache_clsrequires_cross_attention_cachecache_to_checkneed_new_cache cache_dtyper cache_kwargsr rprprq _get_cache!sT       zGenerationMixin._get_cachecCs6|jod|jjvod|jjvod|jjvS)a Return `True` if current model can use a `DynamicCache` instance when initializing the `past_key_values`. This is mostly the same as `_supports_cache_class` attribute, but add exception for `Jamba` model which uses its own `HybridMambaAttentionDynamicCache` and do not need to initialize the Cache in advance in order to save memory (because no back and forth `to_legacy_cache` and `from_legacy_cache` will be performed for `HybridMambaAttentionDynamicCache`). jambazambabamba)rrrhrrrprprq_supports_default_dynamic_cache_s z/GenerationMixin._supports_default_dynamic_cachemax_cache_lengthc Cs6d|jjvr dnd}|jjp|ddu}||} | durF|jdur-td|dt| t rD| rD|s=t | nt | ||<dS|jdurMdS| sc|jduratd |jd tdS|dury|jdurytd |jd d|_|jpt|jd d|_|jdur |jtvr|jdkr|jstd|j|jt|j|j||||d||<dS|jdkr|jstd|jdur|jnt} t| j } | j dkrt!st"d| j dkrt#st"d| | ||<dS|jdkrt$||<dS|jdkrt ||<dSdS|st nt t t ||<dS)z Prepares the cache for generation (if applicable), given `generate`'s parameterization. If a cache is instantiated, writes it to `model_kwargs`, under the name expected by the model. rr]r^rNzMPassing both `cache_implementation` (used to initialize certain caches) and `zB` (a Cache object) is unsupported. Please use only one of the two.FzThis model does not support `Cache` instances, it only supports the legacy cache format (tuple of tuples). `cache_implementation` (set to z) will be ignored.zRAn assistant model is provided, using a dynamic cache instead of a cache of type='z'.rstaticzThis model does not support `cache_implementation='static'`. Please check the following issue: https://github.com/huggingface/transformers/issues/28981)rrrrr quantizedzThis model does not support the quantized cache. If you want your model to support quantized cache, please open an issue and tag @zucchini-nlp.quantozYou need to install optimum-quanto in order to use KV cache quantization with optimum-quanto backend. Please install it via with `pip install optimum-quanto`HQQzYou need to install `HQQ` in order to use KV cache quantization with HQQ backend. Please install it via with `pip install hqq` offloadeddynamic)%rrhrrrrrrrtuplerrfrom_legacy_cacherrr]r^r_rrrrr3_supports_static_cacher rrOnum_return_sequences_supports_quantized_cache cache_configrr4backendr! ImportErrorr r) rrrr2rrrr+ruser_defined_cacher cache_classrprprq_prepare_cache_for_generationns                z-GenerationMixin._prepare_cache_for_generationcCsdtt|jjvS)z Return True if the current model supports the keyword argument `logits_to_keep` in forward() to save memory. Checking it in this way allows to avoid using a new model attribute. logits_to_keep)rrrrrrrrprprq_supports_logits_to_keepsz(GenerationMixin._supports_logits_to_keepkwargs_has_attention_maskc sLdfdd }||j|d}||j|d}||j|d}||j|d}jjr/|dur-|n|}|dur=|jdkr=|d}|dur]|dur]|durP|sPt d|d}t d|djjri|durit d |durt ||d  r|dur|st d |durt|s|dk rt d |d ||_||_||_||_dS)a Prepares the special tokens for generation, overwriting the generation config with their processed versions converted to tensor. Note that `generation_config` is changed in place and stops being serializable after this method is called. That is no problem if called within `generate` (`generation_config` is a local copy that doesn't leave the function). However, if called outside `generate`, consider creating a copy of `generation_config` first. NcsF|dur|S|dur |nj}t|tjr||Stj||tjdS)Nrr)rrrlrr)tensorr)tokenrrrprq_tensor_or_nones   z@GenerationMixin._prepare_special_tokens.._tensor_or_nonerSrzThe attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.z)Setting `pad_token_id` to `eos_token_id`:z for open-end generation.z\`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.rzThe attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.z;`eos_token_id` should consist of positive integers, but is zq. Your generation will not stop until the maximum length is reached. Depending on other flags, it may even crash.r)rrrrrrrr'rrrrrrrlis_floating_point_bos_token_tensorrr_decoder_start_token_tensor) rrr%rr)bos_token_tensoreos_token_tensorpad_token_tensordecoder_start_token_tensorrprrq_prepare_special_tokenssP        z'GenerationMixin._prepare_special_tokens synced_gpusstreamerrZc ' s|| dd} | dd}|j| fi| \}||||| ||dur;ts5t|o:t dk}|durA|nt }|durJ|nt }dt t |jjv}d|v}|dddu}||j|\}}}|jd}|j}|j||d|jjsjdur|dkrt|jd krt|ddd fjkdkrtd |jjs|d krd _ |s|r|r|!|||d<n|r|dkrt|djd krt"d|jjrd|vr|#|||}|jjr|j$|||j%|jd\}}n |dkr|n|d}j&r|'|| }|dur|(|)|jd }| dduo.j*du}| dduo;j+du}|j,|||||d|-rUd|vrUd|d<|.||j*d}|jd|krz|d krz|jjsz||jd7}|/|||||0|}|durj1dkrt"d|jj2|jj2krt34d|jj2d|jj2d|jj2dt5|j6|||||j|| | d }|j7d8|| d| }j |d<|t8j9kr4j:dkrt"dj:d|dkrt"d|dst"d j;d!vrt"d"|j<rt"d#|j=j>|j?||||| ||d$}|j@|f|||||d%|} n|t8jAkrZ|j<rGt"d&|j=j>|jB|fjC||||d'|} n|t8jDkr|dsit"d(|j<rvt"d)|j=j>|jE|f||||d*|} nS|t8jFt8jGfvr|jHd8|j:|jjd+|\}}|jI|f||||d*|} n(|t8jJt8jKfvr|jHd8|j1|jjd+|\}}|jL|f|||d,|} n|t8jMkrtN|j1|jjOjPj:jQj*d-}!|jHd8|j1|jjd+|\}}|jR||!f|||d,|} n|t8jSkrg}"jTdur&jT}"jUdurfd.d/}#tVjUtWrAtjUdkrD|#jUD]`}$tV|$dtWrtV|$tWr^t|$dkra|#tXd0d1|$Drn|#tXd2d1|$Dr{|#tY|$}%n!tV|$tWrt|$dkr|#tXd3d1|$Dr|#tZ|$}%|"[|%qGt\|"|j1|jjOjPj:j*d4}&|jHd8|j1|jjd+|\}}|j]|f|&|||d5|} j^d urt_| d6rt`| jad7dur| jab| _a| S)9a Generates sequences of token ids for models with a language modeling head. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](../generation_strategies). Parameters: inputs (`torch.Tensor` of varying shape depending on the modality, *optional*): The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs` should be in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of `input_ids`, `input_values`, `input_features`, or `pixel_values`. generation_config ([`~generation.GenerationConfig`], *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complements the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. If your stopping criteria depends on the `scores` input, make sure you pass `return_dict_in_generate=True, output_scores=True` to `generate`. This feature is intended for advanced users. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], List[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and `input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://arxiv.org/abs/2010.00904). synced_gpus (`bool`, *optional*): Whether to continue running the while loop until max_length. Unless overridden, this flag will be set to `True` if using `FullyShardedDataParallel` or DeepSpeed ZeRO Stage 3 with multiple GPUs to avoid deadlocking if one GPU finishes generating before other GPUs. Otherwise, defaults to `False`. assistant_model (`PreTrainedModel`, *optional*): An assistant model that can be used to accelerate generation. The assistant model must have the exact same tokenizer. The acceleration is achieved when forecasting candidate tokens with the assistant model is much faster than running generation with the model you're calling generate from. As such, the assistant model should be much smaller. streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. negative_prompt_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The negative prompt needed for some processors such as CFG. The batch size must match the input batch size. This is an experimental feature, subject to breaking API changes in future versions. negative_prompt_attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Attention_mask for `negative_prompt_ids`. use_model_defaults (`bool`, *optional*): When it is `True`, unset parameters in `generation_config` will be set to the model-specific default generation configuration (`model.generation_config`), as opposed to the global defaults (`GenerationConfig()`). If unset, models saved starting from `v4.50` will consider this flag to be `True`. kwargs (`Dict[str, Any]`, *optional*): Ad hoc parametrization of `generation_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `torch.LongTensor`. If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateDecoderOnlyOutput`], - [`~generation.GenerateBeamDecoderOnlyOutput`] If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateEncoderDecoderOutput`], - [`~generation.GenerateBeamEncoderDecoderOutput`] rzNr5r$rrrrSrrzA decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.r~Tr}z1`attention_mask` passed to `generate` must be 2D.)rrrrrr9rd)rrrrrrr#zZ`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1.z~You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on z, whereas the model is on z. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('z ') before running `.generate()`.) rrDrErFr3rrrGrH)rryrzrzFnum_return_sequences has to be 1 when doing assisted generate, but is rz6assisted generate is only supported for batch_size = 1z+assisted generate requires `use_cache=True`)rrrz=assisted generate is not supported with Static cache classes`zCassisted generation is not supported with stateful models, such as )rr}rr2r3r4r5r)rBr3ryrr2r3z=dola decoding is not supported with stateful models, such as ) dola_layersr3ryrr2r3z,Contrastive search requires `use_cache=True`zBcontrastive search is not supported with stateful models, such as )r3ryrr2r3r}rr)r3ryrr2)rrOrlength_penaltydo_early_stoppingnum_beam_hyps_to_keeprPr9cstdjd)Nzo`force_words_ids` has to either be a `List[List[List[int]]]` or `List[List[int]]` of positive integers, but is r)rforce_words_idsrprrprq typeerror s z+GenerationMixin.generate..typeerrorcss|] }t|t VqdSr)rrnr token_idsrprprqr sz+GenerationMixin.generate..css"|] }tdd|DVqdS)cs$|] }t|t p |dkVqdSrNrrrtoken_idrprprqr "z5GenerationMixin.generate...Nrr<rprprqr s  csr>r?r@rArprprqr rC) constraintsrrOrr6r7r8r9)constrained_beam_scorerr3ryrr2r]to_legacy_cacherp)crrrrrrrrdistget_world_sizerArVrrrrrrrrrrrr1rrrrrlrrrrrrrrr, token_healing heal_tokensputrr9rdrr$rr"get_generation_moderOr@r]r^r_rxrr6ASSISTED_GENERATIONrr _is_statefulrrhrC_assisted_decodingDOLA_GENERATION_dola_decodingr4CONTRASTIVE_SEARCH_contrastive_searchSAMPLE GREEDY_SEARCHr"_sample BEAM_SAMPLE BEAM_SEARCH _beam_searchGROUP_BEAM_SEARCHr(r6early_stoppingrP_group_beam_searchCONSTRAINED_BEAM_SEARCHrDr9rrnrr%r&r[r)_constrained_beam_searchreturn_legacy_cacherrr]rF)'rrrr3ryrFr2r2r3rGrHrrrzr5raccepts_attention_maskrequires_attention_maskr%rrrrr}rrrrgeneration_modeprepared_logits_processorprepared_stopping_criteriarBresult beam_scorerfinal_constraintsr;word_ids constraintrErpr:rqgenerate@stj                                                    zGenerationMixin.generatethis_peer_finishedcCsL|r tj|rdnd|d}tj|tjjd|dkrdSdS|r$dSdS)z Returns whether there are still unfinished sequences in the device. The existence of unfinished sequences is fed through `this_peer_finished`. ZeRO stage 3-friendly. rMrQrS)opFT)rlr'rG all_reduceReduceOpSUMr)rrkr2rthis_peer_finished_flagrprprq_has_unfinished_sequences* s z)GenerationMixin._has_unfinished_sequencescsdurtdjj}}t}td|d}ddj|ddD}|d dd j|j }t ||k||} | d krE|S|ddd f }d d fdd|D} tt|| D]`\} \} } || } t | |kr~qj fdd|j| dD}t|dkrqj|| fd7<|j|d| dd } | d krqjt| | |kdkr||d <|j|d |d|| <qj|S)a Generates sequences of token ids for models with a language modeling head. Parameters: input_ids (`torch.LongTensor`): The sequence used as a prompt for the generation. tokenizer (`PreTrainedTokenizerBase`, *optional*): The tokenizer used to decode the input ids. Return: `torch.LongTensor` where each sequence has its tail token replaced with its appropriate extension. Nzs When generating with token healing, you must pass the model's tokenizer to the `tokenizer` argument of `generate`.r$)rrcSsg|]}|qSrp)striprrprprqrS z/GenerationMixin.heal_tokens..T)skip_special_tokenspt)return_tensorspaddingrr c3s"|] }|dVqdS)rxN)decodereplace)rt space_tokrzrprqri s z.GenerationMixin.heal_tokens..csi|] }|fdqS)g$@)convert_tokens_to_ids)ralt_tok)rzrprqru sz/GenerationMixin.heal_tokens..)prefixrQrKr:)rrrr get_vocabr5 batch_decoder}r)rrlwherenumeltolistconvert_ids_to_tokensr~ enumerateziprr extensionsrrrjr')rr}rzrr vocab_trierpromptstail_ids tail_toks batch_idxtail_idtail_tok batch_idsseq_bias trimmed_idsrpr|rqrJ< sT           zGenerationMixin.heal_tokensr4c& sP|jjrtd|j} |j} |j} |j} |j} |j}t dd|D}|j }|r,| r,dnd}|r4| r4dnd}|r<| r.rpNrrFrr$low(rhighcsg|]}|kr|qSrprpri final_layerrprqr rz2GenerationMixin._dola_decoding..z?dola_layers must be either 'low', 'high' or a list of integers.zCDoLa is not supported for models that don't have output embeddings.rST)r r r r)rrrr num_samplesr]rcrdrerfrgr])4rrrrr r  output_scores output_logitsreturn_dict_in_generaterr=rrlrrrrrrtie_word_embeddingsrstrrnrget_output_embeddingsrqrredetachr)float32floatrgr._dola_select_contrastrurfrvrwr r softmax multinomialsqueezeargmaxrrKrrendrbr)&rr}r4r3ryrr2r3rrr r rrrhas_eos_stopping_criteriar=rd raw_logitsrurvrwrunfinished_sequencesrk start_layercandidate_premature_layerslm_headrr#final_layer_next_token_logits final_logitscandidate_premature_logitscandidate_premature_layernext_token_logitsnext_token_scoresprobs next_tokensrprrqrQ s2      (            L zGenerationMixin._dola_decodingc> s tdd|D}|j|j} |j} |j} |j} |j} |j}|j}|j }|r*|r*dnd}|r2| r2dnd}|r:| r:dnd}|rB| rBdnd}|rJ| rJdnd}|rh|j j rh| r[|d dnd}| rf|d dnd}|j d}tj|tj|jd }|||}tj|tjd }|j j rd |vr|d dur|d }n|d }|jdd }d}|j|||jdrz| ddust|dttfrV|ddkrVd|d<|j|fi|}|di|dd| d}|j j r|jd}n|jd}|jdddddfjdtj|jd}|j |||j j d}|s#|j!d||j j d|\} }| d}!|!dur6t"|j#j$dt|!dt%tj&frM|!ddj d|krVt"|j#j$d|||}"t'j(j)|"dd }#tj*|#dd\}$}%|r|rx||f7}| r||"f7}| r||j j r|j+fn|j,f7}|j j r||j-f7}| r||j j r|jfn|jf7}~|s|d}&t|&t.st|&trt|&j/t.r|&0n%g}'|&D]}(g})|(D] }*|)1|*jdd q|'1t%|)qt%|'}&|&|d<|rXg}+t2D]L},|j|%dd|,f3ddfi|}-|di|-dd| d}t|dt.s=t|dtrHt|dj/t.rHd|d<|d4d|+1|qt5|+|j 6}n|j|%3ddfi|}-|di|-dd| d}~-|j j r|jd}.|j}/n|jd}.|j}/|jdddddf7}0|jdd }1t8|1|.|$|| }2tj9||:|j ddfgdd }|2d}2t;fddt<|2D}3|%t2t=|%|2f}4t>t?|.j@dd }.|.t2||2ddf}.tj9||.Adgdd }d}5|/D]}(t>t?|(t2||2ddf}(|5|(f7}5q|r@|j|%dd|2f3ddfi|}6|di|6dddd}7|7d}8nNd}8tBD] }9|8pNtC||9d}8qDt|8t.sdt|8trjt|8j/t.rj|8D|3n$g}'|8D]}(g})|(D] }*|)1|*|3dfqt|'1t%|)qnt%|'}8t>t?|0t2||2ddf}||j}|j j rd}:d};| r|j-D]}(t>tj?|(dd t2||2df}(|:|(f7}:q|j+D]}(t>tj?|(dd t2||2df}(|;|(f7};qtE|8|5|;pd|:pdd }n-d}<| r |j,D]}(t>tj?|(dd t2||2df}(|<|(f7}<qtF|8|5|<p'dd!}|j |||j j d}|r;|r;q|rH|4|| d|}4tj9||4dddfgdd }|durc|G|4H||||@}|Idk}|j|||jds|dur|J|r| ddurt|dt.st|dtrt|dj/t.r|d4dn-g}!|dD] }(g}=|(D]}*|=1|*dddddfq|!1t%|=qt%|!|d<|j j rtK||||||||| dd" StL|||||| dd#S|S)$a Generates sequences of token ids for models with a language modeling head using **contrastive search** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. model_kwargs: Additional model specific keyword arguments will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. csrrrrrprprqr rz6GenerationMixin._contrastive_search..rpNrrfrgrrr%rrrFrSr]Tr)r r r rrrrrr5zQ does not support caching and therefore **can't** be used for contrastive search.z| does not have a standard cache format and therefore **can't** be used for contrastive search without further modifications.)rrr$rcsg|] \}}||qSrprp)rrrrTrprqrn z7GenerationMixin._contrastive_search...)r]rwrurv)r]rgrf rcrdrersrtrurvrwr]r)MrrT penalty_alpharr r rrr low_memoryrrrrrlrrrrrrrqrrrrrrwrgrer)rr.r"rrrhrrr r rtopkrurfrvrrbatch_repeat_interleaver[rrcropstack_model_outputsrr _ranking_fastrr(r'rrrsplitrr'r&rbatch_select_indicesrrrKrrrrrrb)>rr}r3ryrr2r3rrrrr r rrr sequentialrrdrurvrwrsrtrrcosine_matrix_maskrkrr#last_hidden_stateslogit_for_next_steprr]processed_logit_for_next_step next_probs top_k_probs top_k_idspastnew_key_valuesrrr all_outputsrnext_model_inputs next_hiddenfull_hidden_statesrecontext_hidden selected_idx augmented_idxrnext_decoder_hidden_statesnext_model_inputselected_outputsnext_past_key_valuesr*next_step_cross_attentionsnext_step_decoder_attentionsnext_step_attentionslayer_past_key_valuesrprrqrSe s(,                     &        $    $   $ $ $           z#GenerationMixin._contrastive_searchc$ Ks|j}|j} |j} |j} |j} |j} tdd|D}|j}| r$| r$dnd}| r,| r,dnd}| r4| r4dnd}| r<| r.rpNrrfrgFrr] hf_quantizercuda0TOKENIZERS_PARALLELISMTrSr r r rrrrr$rrrr):rr r rrrrr=rrrrrlrrrr__call__rris_compileablerrrdisable_compiler@compile_config_compile_all_devicesosenvironget_compiled_callprefill_chunk_size_prefill_chunkingrqrrr.rer)rrurfrvrwrgr r rrrrrrKrrrrrrb)$rr}r3ryrr2r3rrr r rrrrr=rdrrurvrwrsrtrcur_lenrkr model_forwardr is_prefillrr#rrrrrprprqrV s+         (         K   zGenerationMixin._samplecCsx|jj}t|ttfr|||}|Sd|vr5t|ttfs(t d|d|||}t |}|S| ||S)aC Temporary function to handle the different types of cache reordering processes while we roll out `Cache`. TODO: standardize cache formats and make all models compatible with `Cache`. It would remove the need for this function, with `Cache.reorder_cache` being the sole remaining code path gptbigcodez'Using an unsupported cache format with zG. Currently, it only supports the legacy tuple format or `DynamicCache`) rrhrrrrnr1rrrr reorder_cache)rr]r0 model_classrprprq_temporary_reorder_cache s      z(GenerationMixin._temporary_reorder_cacher'cCs0t|j}t||d|dg|ddS)z=[batch_size, num_beams, ...] -> [batch_size * num_beams, ...]rr$rNrnrrlr)r'rrprprq_flatten_beam_dim s &z!GenerationMixin._flatten_beam_dimrOcCs&t|j}t|||g|ddS)z=[batch_size * num_beams, ...] -> [batch_size, num_beams, ...]r$Nr)r'rrOrrprprq_unflatten_beam_dim s z#GenerationMixin._unflatten_beam_dimcCsFt|jt|jkr|d}t|jt|jks tj||dd}|S)a Gathers the beam slices indexed by beam_indices into new beam array. Args: tensor (`torch.Tensor`): A tensor containing data to be gathered. The tensor is a 2D or a 3D tensor with the two first dimensions depicting the batch and the beam dimensions. beam_indices (`torch.Tensor` of shape `(batch_size, num_beams_to_select)`): The indices of the beams to select . Returns: A tensor with the selected beams rr$)inputrr)rrr'rltake_along_dim)r'rzgathered_tensorrprprq _gather_beams s  zGenerationMixin._gather_beamsrunning_beam_scores beam_scoresis_sent_finished!next_token_hits_stopping_criteriarr9decoder_prompt_lenr[r6c Cs|dkr |dkr ||} n||} |ddddf| |} t|tj|ddddd} t| | k} t||du@} t|}| | @|@S) zv Beam Search stopping condition -- halts the generation loop if any of these conditions becomes False neverrMNr$Trkeepdimre)rlrrrr)rrrrrr9rr[r6best_hypothetical_lengthbest_possible_running_scoreworst_finished_scoreimprovement_possibleexists_open_beamvalid_continuationsrprprq%_beam_search_has_unfinished_sequencess   z5GenerationMixin._beam_search_has_unfinished_sequencesaccumulated_log_probsrunning_sequencesrunning_beam_indicesr= beams_to_keepr?c Cs|rtjtjj|dd|d} tj|d| d} n tj||d\} } | | } ||| }||| }| | }||dddd|f<tj| |j d dd|}| |}||dddd||f<| ||fS) a' Get top-K continuations given the accumulated log probs on the next token. A few notes to understand what's going on: 1. Each item in batch has `num_beams` * `vocab_size` candidate continuations. For each item, get the top K [K = (number of EOS tokens + 1) * `num_beams`] candidates with the highest accumulated log-probabilities, or sample them without replacement using the accumulated scores 2. We gather the top K (as opposed to `num_beams`, or any number lower than K) here so that we have at least `num_beams` sequences remaining to continue the live beam search. 3. Note that other stopping criteria might result in impossible to continue beams, i.e. all continuations selected in this step hit the stopping criteria. rrrr$)rrindexrNrS) rlrr r rrrrrrr)rrrrrrr=rrOr?r topk_indicestopk_log_probstopk_current_beam_indicestopk_running_beam_indicestopk_running_sequencestopk_ids batch_offsetbatch_modified_indicesrprprq_get_top_k_continuations(s   z(GenerationMixin._get_top_k_continuationsrrrc CsT||tjd}tj||dd}|||}|||} |||} || | fS)z Given the top-K continuations, their scores, and whether they hit a stopping criteria, select the best non-finished beams to continue beam search in the next iteration. r rr$)r)rlrrr) rrrrrrOtopk_running_log_probsnext_topk_indicesrrrrprprq%_get_running_beams_for_next_iteration^s     z5GenerationMixin._get_running_beams_for_next_iterationtop_num_beam_maskcCs|| dddf@}|| d| | }tj|ddd|du@}||tjd7}||d7}tj||fdd}tj||fdd}tj||fdd}tj||fdd}tj|| dd}|||}|||}|||}|||}||||fS) z Updates the finished beams if (and only if) there are new completed sequences that have a higher score than the current finished sequences. Nr$rT)axiskeepdimsr rr)rlrr)rrrr)rrcrrrrzrrrr$rOrrr6r[did_top_num_beams_just_finishedbeams_in_batch_are_fullmerged_sequences merged_scoresmerged_beam_indicesmerged_is_sent_finishedtopk_merged_indicesrprprq_update_finished_beamsts     z&GenerationMixin._update_finished_beamsc8Ks|j}|j}|j} |j} |j} |j} |j} |j}|j}|j }|j }|j }|j }|j \}}||}|jjdkr;|jj}n|jjdkrG|j}n|jj}|}d}|durZ|j dnd}tdd||}tjtj|tjdtj||tjdfdd |j}|||}|j}|rt d | r| rd nd}| r| rd nd}| r| rd nd} | r| rd nd}!| r| rd nd}"| r| rd nd}#| r|jj!r| r|d "d nd}$| r|d "dnd}%|dur|p|dnd}&tj#|||f|&tj$|jd}'|%||||'ddddd|f<|'&'}(tj||ftj(|jd})d|)ddddf<tj#||fdtj(|jd}*tj||ftj|jd}+tj||ftj|jd},tj#||||fdtj)|jd}-|-&'} |j*|||jdr|+|'ddddd|f}.|j,|.fi|}/|/-| rd| ini|/-| rd| ini|d i|/ddi}0|j.|0||jj!d}|r|rq^|0j/dddddfjdtj0|jd}1t1j2j3|1dd }2||.|2}2| r | r||1'f7}| r| r||2'f7}| r|!|jj!r|0j4fn|0j5f7}!|jj!r|"|0j6f7}"| r |#|jj!r|0j7fn|0j8f7}#~0|%|2||}2|2|)dddddf}2t9|2|||f}2|j:|2|'|-|||||||d \}3}4}5||+|4ddddd|df|},|%|,||},|j;|3|4|5|,|d\}'})}-|j<|(|4|*|3| |5|+|,||||||d\}(}*} }+|"dddur|j=|d|+|-d||fd|d<|d}|>|)|*|+|,||||| }|j*|||jdsh|+|(ddd|ddf}(|+|*ddd|f}*|+| ddd|ddf} | dj?dd }6||6}7|(ddd|7f}(| ddd|6f} | rI| s#d}*|jj!r:t@|(|*||| |$|%|!|"|#|"dd StA|(|*||| |!|#|"dd!S|(S)"a Generates sequences of token ids for models with a language modeling head using **beam search decoding** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. If it's the first time you're diving into Beam Search, we recommend you read the following blog post: https://huggingface.co/blog/how-to-generate (especially the beam search section). You can recompute the sequence scores from the individual scores using the `compute_transition_scores` function (https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationMixin.compute_transition_scores) Parameters: input_ids (`torch.LongTensor` of shape `(batch_size*num_beams, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`: An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). model_kwargs: Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`generation.GenerateBeamDecoderOnlyOutput`], [`~generation.GenerateBeamEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateBeamDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateBeamEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. MoshiDepthDecoderImageGPTForCausalImageModelingFNrrr$r%rz`low_memory=True` is not supported after the beam search refactor. Please check the discussion in #35802 *after the PR got merged*, and add a comment there if your questions are not yet answered.rprrfrgr) fill_valuerrrr rSr r r Trr) rrrrrr=rrOr?r)rrrrrO)rcrrrrzrrrr$rOrrr6r[r].)r]r0 rcryrdrerzrsrtrurvrwr]rcryrdrerzrfrgr])Brrr r rrrr=r[r6r9rOrrrrhraudio_vocab_sizer out_featuresrr?rrlrrboolzerosr)rrrrrrfullrrrrrint32rqrrrr.rerr r rrurfrvrwrgrr r#r.rrrr{rx)8rr}r3ryrr2rrrr r rrrr=r[r6r9rOrbatch_size_unflattenedrrr?rrk n_eos_tokensrr$r all_scoresrrzrurvrwrsrtoutput_fill_valuerrcrrrrrflat_running_sequencesr model_outputsre log_probsrrrmax_generated_length output_lengthrprprqrYs.      "  $    (       "       zGenerationMixin._beam_searchrfc4 s||j}|j} |j} |j} |j} |j} |j}|j|j}|t |j ||j }|j \}}| ||}|rE| rEfddt|Dnd|krZtdd|d|r`| r`dnd}|rh| rhdnd}|rp| rpdnd}|rx| rxdnd}|r| rdnd}|r|jjr| r|dd nd}| r|dd nd}tjfd tj|d }d |ddddf<|f}d}|j d}|j|||j drtj|j|d }tjtj|d }|j|fi|}|| rd| ini|| rd| ini|di|ddi} |j| ||jjd}|r$|r$|d}q| r6t| jdddddf}!| rK| jdddddfj d|j d}"t|D]}#t!|#}$|$|#}%g}&tD]|&"fddt|#|$Dqg||&}'| j|&dddfj tj#|j d }(t$j%j&|(dd})|)j d}*||'|)|d}+|+||&'d})|)(|+})| r|+|!|&<|)|%|*})| dur| j d nd },tj)|)t*dd|,|%dddd\})}-tj+|-|*dd}.|-|*}-durt,dnd}/|j-|'|)|-|.|| |/|d }0|0d ||&<|0d!}1|0d"|r0| r0t.fd#d$tt d D<|'||&<tj/|'ddf|1'dgdd}'|'dddf||&<tj+|%dd|#|%||&<qO|r| rq||!f7}| ry||"f7}| r||jjr| j0fn| j1f7}|jjr|| j2f7}| r||jjr| j3fn| j4f7}tj/||'dgdd}~ |d%ddur|5|d%||d%<|d}|j6st7|||rd}|j|||j dsΈdurt,dnd}2|j8|||-|.|| |j9|2|d& }3|r:| sd|3d'<|jjr%t:|3d(|3d'|||3d)||||||d%d* St;|3d(|3d'|||3d)|||d%d+S|3d(S),a Generates sequences of token ids for models with a language modeling head using **diverse beam search decoding** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size*num_beams, sequence_length)`): The sequence used as a prompt for the generation. beam_scorer (`BeamScorer`): An derived instance of [`BeamScorer`] that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of [`BeamScorer`] should be read. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). model_kwargs: Additional model specific kwargs that will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateBeamDecoderOnlyOutput`], [`~generation.GenerateBeamEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateBeamDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateBeamEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. cs&g|]}tddtDqS)cs|]}dVqdSrpNrprrrprprqr(z@GenerationMixin._group_beam_search...)rrrE)r num_sub_beamsrprqr(rz6GenerationMixin._group_beam_search..N)Batch dimension of `input_ids` should be , but is rrprrfrgr rrFrrSr r r Trr$)rrcsg|]}|qSrprp)rr)rrOrprqrsrr)current_tokensbeam_group_idxrrlargestsortedfloor) rounding_mode)rrrz group_indexrnext_beam_scoresnext_beam_tokensnext_beam_indicesc3s*|]}||fVqdSrrpr)rKr0rzrprqrs z5GenerationMixin._group_beam_search..r]rrr9rzrsequence_scoresrcrzr2r3).rrfrgrr r$FrrSr r r TrrrrrrL)rrrzrrRrSrTr]c3s&|]}||fVqdSrrprr0rzrprqr$rUrVrcrzr2r3)5rrr r rrrrrWrOrrrrrrrrrlr7rrrrqrrr.rer)rr r rrZrrurfrvrwrgrrrr\rr'rr]rr^r9r{rx)*rr}rEr3ryrr2rrrr r rrrrrOr_rrdrrurvrwrsrtrrkrrr#rrrhscores_for_all_vocabr?r;rrirkrlrnrprorqr^s,       (         $  j   z(GenerationMixin._constrained_beam_searchrBc+ s|j} |j} |j} |j} |j} |j}|r| rdnd}|r | r dnd}|r(| r(dnd}|r0| r0dnd}|r8| r8dnd}|rV|jjrV| rI|ddnd}| rT|ddnd}|j d}t j |t j |j d}|||}d}d }|j|||j d r|j d }||\}}||j }|dur||j }|j d |j d }||d}t|}t||j d |jj}t||j d }d |vrt j|d t j||||j t j dfdd|d <|j|fi|} d| vr|d | d<| | rd| ini| | rd| ini|di| }!|!jdd| d dfjt j|j dt|dkrSt|d D]$}"||ddd||"fdd|"ddfdd|"ddf<q.| rf|durft||||\}#}$n`| rjd d}%t j |%dddddfd d!d dddf}&nj"d d}&|dd|df}'|'|&dddd fkj#d dd k$}$|r|$|kr|$d 8}$|&ddd|$d f}#t j||#fd d}|dur|%|#&|j d }(|(d })t'||!j(|)|!_(|)||$|j*|!||jj|$d d}|r|rqo|r|$d }*| r |t+fddt|*D7}| r2|t+fddt|*D7}|r7|(n|*}*| rf|jjrTt,||!j-||*}t,||!j.||*d d}n|!j/ddurft,||!j/||*d d}| r|jjrwt,||!j0||*}nt,||!j1||*}||||@}|2dk}d}|j|||j d sy|dur|3t4|dr|j5j6j7dkr|j8|j5j6_8|r|jjrt9|||||||||dd St:||||||ddS|S)a Generates sequences of token ids for models with a language modeling head using **greedy decoding** or **sample** (depending on `do_sample`), assisted by candidate sequences. Assisted generation is an example of a candidate decoding strategy. Can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. candidate_generator (`CandidateGenerator`): A derived instance of [`CandidateGenerator`] that defines how candidate sequences are generated. For more information, the documentation of [`CandidateGenerator`] should be read. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. model_kwargs: Additional model specific keyword arguments will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. rpNrrfrgrrFTrSrr$rr&rr#r r r)rr$c3&|]}dd|ddfVqdSrrpr) new_logitsrprqrrpz5GenerationMixin._assisted_decoding..c3rrrrpr)rrprqrrp)is_decoder_attentionr2 heuristicr]rr);r=r r rrrrrrrrlrrrrrqget_candidatesr)rr1r2rrrrrerrrr_speculative_samplingrrrrrrrKrr0r]update_candidate_strategyr.r_split_model_outputsrvrurfrwrgrrrr2rnum_assistant_tokens_schedulenum_assistant_tokensrrrb)+rr}rBr3ryrr2r3rr=r r rrrrdrrurvrwrsrtrrrkis_first_iterationrcandidate_input_idscandidate_logitscandidate_lengthis_done_candidatecandidate_kwargsrr#r valid_tokens n_matchesrselected_tokenscandidate_new_tokens new_cur_lennew_cache_sizenewly_added_lengthrp)rsrrqrOs61          F  4 *             z"GenerationMixin._assisted_decodingcKs&dtjj_|j}tj|ddddf|dd}d|vr td||j}| dd}d}|D]J} || j d} |durI|ddd| f|d<tj || tj | j d|d <|d d|d <|j| fi|} |di| d d i} | j|d<| }q0||d<|d ddd |d <| d d} |S)N@rrr]z+Cannot use prefill chunkink without a cacherrrrrr Tr$rp)rl_dynamorcache_size_limitrrrrrrrrrrr'rr])rr}rr chunk_size input_chunksrrr input_chunkcurrent_lengthrr#rrprprqrs0        z!GenerationMixin._prefill_chunking)NNNN)NNNr)r$FN)Fr$)NF)NN) NNNNNNNNNNN)Qrhrirjrkrlrmrror rrrrrrrrrr5rrrrrr staticmethodr6r"rr.r1rAr,rCrrrxrVrr rmrrrrrrrrrr rr"r$r1no_gradGenerateOutputrjrqrJGenerateNonBeamOutputrQrSrVrrrrrrr r#r.GenerateBeamOutputrYr'r\r)r^rOrrprprprqr|[s " A   B    $  /  ; &   1  Q   l '    ) z8,9  P"N > y  R      l U U ! C   '    6        3  F % o  r|cCs|dd| df}|jdd}|ddt||fdd}|jdd}|ddt||fdd} | |} t| } | | k} | jdddk} |ri| |kri| d8} |ddd| df}|| fS|jd}|dd| ddf}| |kr|dd| ddf}tj||dd}| |n|}tj |ddddddf}| dkrtj |ddd| f|fdd}|| fS|}|| fS)a Applies sampling as in the speculative decoding paper (https://arxiv.org/pdf/2211.17192.pdf, algorithm 1). Returns the selected tokens, as well as the number of candidate matches. NOTE: Unless otherwise stated, the variable names match those in the paper. Nrrrr$)rr) rrlrr rand_likerrrclampdiv_rr)r}r~rrsrnew_candidate_input_idsqq_irp_iprobability_ratior_i is_acceptedrrgamma p_n_plus_1 q_n_plus_1p_primer{rprprqrw/s4        "rwFc Cst|dkr1d}|D]}|r|n|jd}||dd|d|ff7}q ||f7}|d7}||8}t|D])}d}|D]}|rC||n|jd}||d||dd|ff7}q;||f7}q5|S)z Given the (decoder/cross attentions)/(decoder hidden states) for multiple generated tokens, splits it into a tuple where each member corresponds to a single generated token. rrpr.Nr$)rrr) r# new_outputsr added_lenrt new_tupler last_dim_sizerrprprqrygs   " ryrrnext_top_k_probsralpha beam_widthrc Cs||jddd}||jddd}t||ddd}|j|jd}d|t|jj}||}tj |dd\} } | d}d|||| } t t | |} | j dd\} } | S) a  Reranks the top_k candidates based on a degeneration penalty (cosine similarity with previous tokens), as described in the paper "A Contrastive Framework for Neural Text Generation". Returns the index of the best candidate for each row in the batch. rTrr$rr%rrQ) normrlmatmulrrr)rfinforrrrr) rrrrrrnorm_context_hiddennorm_next_hidden cosine_matrixdegeneration_penaltyrcontrastive_scorerrprprqrs  rfull_batch_size split_sizecsdur dg|Sttjrfddtd|DStts/ttr5tjtr5|Sttr]tdtrOfddtd|DSfddtd|DSt dt )a Takes care of three cases: 1. data is a tensor: e.g. last_hidden_state, pooler_output etc. split them on the batch_size dim 2. data is a tuple: e.g. hidden_states, attentions etc. Keep the tuple as it is and split each tensor in it and return a list of tuples 3. data is a tuple of tuples, e.g. past_key_values. Keep the tuple as it is and split each tuple in it and return a list of tuples of tuples (see documentation of ModelOutput) Ncsg|] }||qSrprprdatarrprqrrz_split..rc$g|]tfddDqS)c3s(|]}tfdd|DVqdS)c3 |] }|VqdSrrp)rr'rrrprqrz._split....Nr)r inner_tuplerrprqrs&$_split...rrrrrqrcr)c3rrrp)r sub_tensorrrprqrrrrrrrrqrrUnexpected attribute type: ) rrlrrrrr batch_splitrrr@)rrrrprrq_splits&          rrrcs6dur gStdkrtdkr!tdtdr+jn}fdd|D}fdd|D}gd fd d|Dfd dtD}fd d |Ddvrtd|fddt|D}dvrfdd|D}fdd|D}|S)a  Split a ModelOutput object (or its subclasses) or Dict into a list of same-class objects based on a specified split size. The input object is dict when it was prepared for forward pass and ModelOutput when it was returned from previous forward pass. Nrz3`full_batch_size` must be divisible by `split_size`z:`split_size` must be smaller or equal to `full_batch_size`__dataclass_fields__crrprprrrrprqrrz'_split_model_inputs..cs&g|]}t|ts|dkr|qS)rrr6rrrprqrr)rrr#cs&g|]}t|ts|vr|qSrprr)keys_to_ignorerrprqrrcs$g|]fddDqS)cs"i|] }|t|qSrp)rr)rrrrrprqrs"z2_split_model_inputs...rpr)rr non_bool_keysrrrqrrcsi|]}||qSrprprrrprqrrz'_split_model_inputs..rcs$g|]\}}i|d|iqS)rrp)rr data_split)encoder_outputs_splitrprqrsr#cs g|] }i|ddiqS)r#rprrrrprqrscsg|] }di|qS)rprpr) bool_datamodel_output_clsrprqrs) r@rrrrr_split_model_inputsrr)rrrrr bool_keysdata_split_listsplit_model_inputsrp)rrrrrrrrrqrs@      rr?csfstdtdtfddDstdddfdd jD}d i|S) z Stack a list of ModelOutput objects (or its subclasses) along the batch_size dimension. The function infers the specific ModelOutput subclass from the list provided. zInput list is empty.rc3s|]}t|VqdSr)r)robj)rrprqrrz&stack_model_outputs..z4All elements in the list should be of the same type.cstddDr dStdtjrtjddStdtr&tStdtr2tStdtrdtddtrStfddt t dDStfddt t dDStdt t frrt Stdtd) z5 Reverse of `_split` function above. css|]}|duVqdSrrp)rrrprprqrr6z7stack_model_outputs.._concat..Nrrc3s8|]tfddttddDVqdS)c3s.|]tjfddDddVqdS)csg|]}|qSrprpr)rjrprqr%rzLstack_model_outputs.._concat....rrNrlrr)rr)rrqr%s,zAstack_model_outputs.._concat...rN)rrrrrrrqr$s & c3s,|]tjfddDddVqdS)csg|]}|qSrprprrrprqr)rszBstack_model_outputs.._concat...rrNrrrrrqr)s*r)rrrlrrrfrom_batch_splitsrrrrrrr'rr@rrprrq_concats"   " z$stack_model_outputs.._concatcs$i|]fddDqS)csg|]}t|qSrp)r)r model_outputrrprqr2rz2stack_model_outputs...rpr)rr?rrqr1rz'stack_model_outputs..Nrp)rr@rrr)r?rconcatenated_datarp)rrr?rqrs  rg?InfgMbPrdbaseline_scores relative_top filter_valuerUc Cs|jdd}|jdd}tj|dd\}} |d|df} tj|ddj} | t|} t| | } | d} |||| k<|||| k<||fS)a] Reference: https://github.com/XiangLi1999/ContrastiveDecoding/blob/170e9142e92159c1237d731e240f5eb14aabf428/transformers/src/transformers/generation_logits_process.py#L235 Apply filtering to only keep tokens with a probability above a certain threshold. The threshold is defined as `relative_top` * max probability in the distribution. rrT) descending.r$) rrlsortrrnplogrr') rdrrrbase_filter_valuerUscores_normalizedbaseline_scores_normalized sorted_logitssorted_indices min_thresh probs_max probs_threshrprprq_relative_top_filter:s     rrrrcs.t|dkr|d}t||\}}||}|Stjfdd|Ddd}tj|dd}tj|dd}d|dddddf|}tj|dd} tj|dd} tj| dddddf|dd d} tj| |dd d} d| | } | d} |t | }|}t||\}}||}|S) Nr$rcsg|]}|qSrprprrrprqr_rsz)_dola_select_contrast..rrg?none) reduction) rrrlrFrrkl_divmeanrrr)rrr base_logitsrestacked_premature_layerssoftmax_mature_layersoftmax_premature_layersavg_distlog_softmax_mature_layerlog_softmax_premature_layerskl1kl2js_divspremature_layerrprrqrSs(  (  r)F)rrrr] dataclassesrtypingrrrrrrr r numpyrrltorch.distributed distributedrG packagingr r torch.nnr r+transformers.generation.candidate_generatorr cache_utilsrrrrrrrconfiguration_utilsrintegrations.deepspeedrintegrations.fsdprmodeling_outputsrr pytorch_utilsrtokenization_utilsrutilsrrr r!r"r#beam_constraintsr%r& beam_searchr'r(r)rBr*r+r,r-r.r/r0r1r2r3r4r5r6logits_processr7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPryrQrRrSrTrUrVrWmodeling_utilsrXtokenization_utils_baserY streamersrZ get_loggerrhraccelerate.hooksr[r\r&rbrrrxr{GreedySearchDecoderOnlyOutput"ContrastiveSearchDecoderOnlyOutputSampleDecoderOnlyOutput%ContrastiveSearchEncoderDecoderOutput GreedySearchEncoderDecoderOutputSampleEncoderDecoderOutputBeamSearchDecoderOnlyOutputBeamSampleDecoderOnlyOutputBeamSearchEncoderDecoderOutputBeamSampleEncoderDecoderOutputGreedySearchOutput SampleOutputBeamSearchOutputBeamSampleOutputContrastiveSearchOutputrrrr|rwryrormrrrrrrrrrprprprqs2 (     $     , p$     #/+:        w 8 $  @9