o  h. @sXddlZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl ZddlmZddlmZmZddlmZmZmZmZddlmZmZdd lmZmZd d lm Z d d l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.e/e0Z1eGd ddeZ2eGdddeZ3eGdddeZ4eGdddeZ5eGdddeZ6eGdddeZ7eGdddeZ8eGdddeZ9eGdddeZ:eGdd d eZ;e e3e2fZe e9e8fZ?e e;e:fZ@e ee?e@fZAGd!d"d"ZBd#d$ZCd%d&ZDd'ejEd(ejEd)ejEd*eFd+eGd,ejEf d-d.ZHdS)/N) dataclass)AnyDictOptionalTupleUnion)dynamic_update_slice)TFCausalLMOutputWithPastTFSeq2SeqLMOutput)TF_MODEL_FOR_CAUSAL_LM_MAPPING)TF_MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING%TF_MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING!TF_MODEL_FOR_VISION_2_SEQ_MAPPING) shape_liststable_softmax) ModelOutputlogging)GenerationConfig) TFForcedBOSTokenLogitsProcessorTFForcedEOSTokenLogitsProcessorTFForceTokensLogitsProcessorTFLogitsProcessorListTFMinLengthLogitsProcessorTFNoBadWordsLogitsProcessorTFNoRepeatNGramLogitsProcessor"TFRepetitionPenaltyLogitsProcessor&TFSuppressTokensAtBeginLogitsProcessorTFSuppressTokensLogitsProcessorTFTemperatureLogitsWarperTFTopKLogitsWarperTFTopPLogitsWarperc@neZdZUdZdZeejed<dZ ee ejed<dZ ee e ejed<dZ ee e ejed<dS)TFGreedySearchDecoderOnlyOutputa Base class for outputs of decoder-only generation models using greedy search. Args: sequences (`tf.Tensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, generated_length, hidden_size)`. N sequencesscores attentions hidden_states __name__ __module__ __qualname____doc__r%rtfTensor__annotations__r&rr'r(r1r1t/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/generation/tf_utils.pyr$9 r$c@eZdZUdZdZeejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dZ ee e ejed<dZee e ejed<dZee e ejed <dS) "TFGreedySearchEncoderDecoderOutputa Base class for outputs of encoder-decoder generation models using greedy search. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder_attentions and the encoder_hidden_states attributes (respectively the decoder_attentions and the decoder_hidden_states attributes) Args: sequences (`tf.Tensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. encoder_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, generated_length, hidden_size)`. Nr%r&encoder_attentionsencoder_hidden_statesdecoder_attentionscross_attentionsdecoder_hidden_statesr*r+r,r-r%rr.r/r0r&rr6r7r8r9r:r1r1r1r2r5U r5c@r#)TFSampleDecoderOnlyOutputaE Base class for outputs of decoder-only generation models using sampling. Args: sequences (`tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_return_sequences, config.vocab_size)`. attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(num_return_sequences*batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(num_return_sequences*batch_size, generated_length, hidden_size)`. Nr%r&r'r(r)r1r1r1r2r=r3r=c@r4) TFSampleEncoderDecoderOutputaZ Base class for outputs of encoder-decoder generation models using sampling. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder_attentions and the encoder_hidden_states attributes (respectively the decoder_attentions and the decoder_hidden_states attributes) Args: sequences (`tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_return_sequences, config.vocab_size)`. encoder_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer of the decoder) of shape `(batch_size*num_return_sequences, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size*num_return_sequences, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_return_sequences, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_return_sequences, generated_length, hidden_size)`. Nr%r&r6r7r8r9r:r;r1r1r1r2r>r<r>c@eZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ eejed<dZ ee e ejed<dZee e ejed<dS) TFBeamSearchDecoderOnlyOutputa Base class for outputs of decoder-only generation models using beam search. Args: sequences (`tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`tf.Tensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed beam scores for each vocabulary token at each generation step. Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams*num_return_sequences, config.vocab_size)`. beam_indices (`tf.Tensor`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Beam indices of generated token id at each generation step. `tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. Nr%sequences_scoresr& beam_indicesr'r(r*r+r,r-r%rr.r/r0rAr&rrBr'r(r1r1r1r2r@ r@c@eZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ eejed<dZ ee ejed<dZee ejed<dZee e ejed <dZee e ejed <dZee e ejed <dS) TFBeamSearchEncoderDecoderOutputa Base class for outputs of encoder-decoder generation models using beam search. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder_attentions and the encoder_hidden_states attributes (respectively the decoder_attentions and the decoder_hidden_states attributes) Args: sequences (`tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`tf.Tensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed beam scores for each vocabulary token at each generation step. Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam. `Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`tf.Tensor`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Beam indices of generated token id at each generation step. `tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`. encoder_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size*num_beams*num_return_sequences, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams*num_return_sequences, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. Nr%rAr&rBr6r7r8r9r:r*r+r,r-r%rr.r/r0rAr&rrBr6r7r8r9r:r1r1r1r2rFs %rFc@r?) TFBeamSampleDecoderOnlyOutputa Base class for outputs of decoder-only generation models using beam sample. Args: sequences (`tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`tf.Tensor` of shape `(batch_size * num_return_sequence)`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed beam scores for each vocabulary token at each generation step. Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams*num_return_sequences, config.vocab_size)`. beam_indices (`tf.Tensor`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Beam indices of generated token id at each generation step. `tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams, generated_length, hidden_size)`. Nr%rAr&rBr'r(rCr1r1r1r2rHrDrHc@rE) TFBeamSampleEncoderDecoderOutputa~ Base class for outputs of encoder-decoder generation models using beam sampling. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder_attentions and the encoder_hidden_states attributes (respectively the decoder_attentions and the decoder_hidden_states attributes) Args: sequences (`tf.Tensor` of shape `(batch_size*num_beams, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`tf.Tensor` of shape `(batch_size * num_return_sequence)`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed beam scores for each vocabulary token at each generation step. Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`tf.Tensor`, *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Beam indices of generated token id at each generation step. `tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`. encoder_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size*num_beams, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size*num_beams, generated_length, hidden_size)`. Nr%rAr&rBr6r7r8r9r:rGr1r1r1r2rI=s $rIc@r#)$TFContrastiveSearchDecoderOnlyOutputa Base class for outputs of decoder-only generation models using contrastive search. Args: sequences (`tf.Tensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, generated_length, hidden_size)`. Nr%r&r'r(r)r1r1r1r2rJns rJc@r4) 'TFContrastiveSearchEncoderDecoderOutputa Base class for outputs of encoder-decoder generation models using contrastive search. Hidden states and attention weights of the decoder (respectively the encoder) can be accessed via the encoder_attentions and the encoder_hidden_states attributes (respectively the decoder_attentions and the decoder_hidden_states attributes) Args: sequences (`tf.Tensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. encoder_attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_attentions=True` is passed or `config.output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(tf.Tensor))`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `tf.Tensor` of shape `(batch_size, generated_length, hidden_size)`. Nr%r&r6r7r8r9r:r;r1r1r1r2rKs rKc"@sFeZdZdZdZeddZdZddZ  d_d e j d e e j d e e j d e d e j f ddZddZdeeeffddZ    d`de e j de ede ed eee j ffddZde j de ede ed e j fddZ dade j de ed eeeffd d!Z  dbd"ededeee j fd#e ed$e ed e e j eee j fff d%d&Z dbd#e ed$e ed efd'd(Ze )   dcd*ed+e d,e e j d-e d e e j eeefff d.d/Z   ddde e j d$e ede eee j fd e e j e eeee j fffd0d1Z   ddde e j d$e ede eee j fd e j fd2d3Z!ed4e"fd5d6Z# ded4e"deeefd+e d eeeffd7d8Z$  9dfd:e"deeefd;edd?Z%ded efd@dAZ&dedBede ed efdCdDZ'dEedFed efdGdHZ(        dgd,e j d0` and `top_k>1` - *multinomial sampling* by calling [`~generation.TFGenerationMixin.sample`] if `num_beams=1` and `do_sample=True` - *beam-search decoding* by calling [`~generation.TFGenerationMixin.beam_search`] if `num_beams>1` You do not need to call any of the above methods directly. Pass custom parameter values to 'generate' instead. To learn more about decoding strategies refer to the [text generation strategies guide](../generation_strategies). NcCs*tdt|jdurtjj|_|jS)NzG`seed_generator` is deprecated and will be removed in a future version.)warningswarn UserWarning_seed_generatorr.random Generatorfrom_non_deterministic_state)selfr1r1r2seed_generators  z TFGenerationMixin.seed_generatorTcOstd)NzbA model class needs to define a `prepare_inputs_for_generation` method in order to use `generate`.)NotImplementedError)rTargskwargsr1r1r2prepare_inputs_for_generationsz/TFGenerationMixin.prepare_inputs_for_generationFr%r&rBnormalize_logitsreturnc Cs`|durttjt|djddddt|g}ttt|t|dfd}t|d|j j |jdf}|rDtj j |dd}|dk}tj tj jdtj|tjddd}|dd| df}|dd| df}t|d|}|jd|}|dd|df}tt|jd|j} tj||| gdd} t|| } t|d| } | S)a Computes the transition scores of sequences given the generation scores (and beam indices, if beam search was used). This is a convenient method to quickly obtain the scores of the selected tokens at generation time. Parameters: sequences (`tf.Tensor`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(tf.Tensor)`): Transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens Tuple of `tf.Tensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`tf.Tensor`, *optional*): Beam indices of generated token id at each generation step. `tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)`. Only required if a `num_beams>1` at generate-time. normalize_logits (`bool`, *optional*, defaults to `False`): Whether to normalize the logits (which, for legacy reasons, may be unnormalized). Return: `tf.Tensor`: A `tf.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)` containing the transition scores (logits) Examples: ```python >>> from transformers import GPT2Tokenizer, TFAutoModelForCausalLM >>> import numpy as np >>> tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2") >>> model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> tokenizer.pad_token_id = tokenizer.eos_token_id >>> inputs = tokenizer(["Today is"], return_tensors="tf") >>> # Example 1: Print the scores for each token generated with Greedy Search >>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, normalize_logits=True ... ) >>> # input_length is the length of the input prompt for decoder-only models, like the GPT family, and 1 for >>> # encoder-decoder models, like BART or T5. >>> input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1] >>> generated_tokens = outputs.sequences[:, input_length:] >>> for tok, score in zip(generated_tokens[0], transition_scores[0]): ... # | token | token string | logits | probability ... print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}") | 262 | the | -1.414 | 24.33% | 1110 | day | -2.609 | 7.36% | 618 | when | -2.010 | 13.40% | 356 | we | -1.859 | 15.58% | 460 | can | -2.508 | 8.14% >>> # Example 2: Reconstruct the sequence scores from Beam Search >>> outputs = model.generate( ... **inputs, ... max_new_tokens=5, ... num_beams=4, ... num_return_sequences=4, ... return_dict_in_generate=True, ... output_scores=True, ... ) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False ... ) >>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores. >>> # Tip: recomputing the scores is only guaranteed to match with `normalize_logits=False`. Depending on the >>> # use case, you might want to recompute it with `normalize_logits=True`. >>> output_length = np.sum(transition_scores.numpy() < 0, axis=1) >>> length_penalty = model.generation_config.length_penalty >>> reconstructed_scores = np.sum(transition_scores, axis=1) / (output_length**length_penalty) >>> print(np.allclose(outputs.sequences_scores, reconstructed_scores)) True ```Nrraxis)rrdtype)r.tile expand_dimsrangeshapelen transposereshapestackconfig vocab_sizenn log_softmaxmath reduce_max reduce_sumcastint32where broadcast_to gather_nd) rTr%r&rBrZbeam_indices_maskmax_beam_lengthcut_idx token_indices gen_step_idxindicestransition_scoresr1r1r2compute_transition_scoress(S." z+TFGenerationMixin.compute_transition_scorescCsz|s;ttttg}t}|D]}|jt|jdd}|dur%| |j qd|j j d}|r7|d|7}t |dS)z Confirms that the model class is compatible with generation. If not, raises an exception that points to the right class to use. N)defaultzThe current model class (zQ) is not compatible with `.generate()`, as it doesn't have a language model head.z2 Please use one of the following classes instead: ) can_generater rr rsetgettyperiaddr* __class__ TypeError)rTgenerate_compatible_mappingsgenerate_compatible_classes model_mappingsupported_modelsexception_messager1r1r2_validate_model_classUs$ z'TFGenerationMixin._validate_model_class model_kwargscCs|jjrdD]}||dqg}tt|jj}d|vs"d|vr-|tt|jjO}| D]\}}|durB||vrB| |q1|rMt d|ddS)zXValidates model kwargs for generation. Generate argument typos will also be caught here.)decoder_input_idsNrXrz8The following `model_kwargs` are not used by the model: zG (note: typos in the generate arguments will also show up in this list)) riis_encoder_decoderpoprinspect signaturerY parameterscallitemsappend ValueError)rTrkeyunused_model_args model_argsvaluer1r1r2_validate_model_kwargsns   z(TFGenerationMixin._validate_model_kwargsinputsgeneration_configlogits_processorcKs||dur+|jjr(|jjt|jkr(t|j}||jkr(t d||_|j}t |}|j d0i|}| | |durdt|tjrN|jjrNnt|tjr]t|jtjr]nt|tj}|ddurvt|dtj|d<d|vrt|dtjr|djjrnt|dtjrt|djtjrn t|dtj|d<|dur|nt}|jdur|jdur|ddurtd|j}t|tr|d}||_t } | r|j!st"d|#||j$|\} } }t%| d} |j&|d<|j'|d <|j(|d <dt)t*+|j,j-.v} d |v}|dddur.|r.| r.|/| |j|j|d<|jj0sN|jdurNtj12| ddd f|jkrNtd |jj0r_d |vr_|3| || }|jj0rs|j4| | ||j5|j$d\}}n | dkrz| n|6d}t%|d }|dduo|j7du}|r|j8dur|j7dkrt d|j7dt9n#|j8dur|s|j7durtd|j8d|j7d|j8||_7t|tjs|j:dur|j:|j7krt"d|j:d|j7d||j7kr|jj0rdnd}td|d|d|j7d|j;duo,|j;dko,|jdko;|jdkoJ|jdkoU|jdko`|j|j@krt"d*|j>d+|j@d,|jFd0||j>|jj0d d-|\}}|jH|f|j7|j|j|jI|jJ||jB|jC|j@d. |S|r||j>|j@krGt"d*|j>d+|j@d,|jE|d'}|jFd0||j>|jj0d d-|\}}|jH|fd |j7|j|j|jI|jJ|||jB|jC|j@d/ |SdS)1a Generates sequences of token ids for models with a language modeling head. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate, e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](../generation_strategies). Parameters: inputs (`tf.Tensor` of varying shape depending on the modality, *optional*): The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs` should of in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of `input_ids`, `input_values`, `input_features`, or `pixel_values`. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which had the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. seed (`List[int]`, *optional*): Random seed to control sampling, containing two integers, used when `do_sample` is `True`. See the `seed` argument from stateless functions in `tf.random`. kwargs (`Dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `tf.Tensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `tf.Tensor`. If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible [`~utils.ModelOutput`] types are: - [`~generation.TFGreedySearchDecoderOnlyOutput`], - [`~generation.TFSampleDecoderOnlyOutput`], - [`~generation.TFBeamSearchDecoderOnlyOutput`], - [`~generation.TFBeamSampleDecoderOnlyOutput`] If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible [`~utils.ModelOutput`] types are: - [`~generation.TFGreedySearchEncoderDecoderOutput`], - [`~generation.TFSampleEncoderDecoderOutput`], - [`~generation.TFBeamSearchEncoderDecoderOutput`], - [`~generation.TFBeamSampleEncoderDecoderOutput`] NaSYou have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use and modify the model generation configuration (see https://huggingface.co/docs/transformers/generation_strategies#default-text-generation-configuration )attention_maskrzThe attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.rz[The selected model does not support Graph mode nor XLA generation (e.g. from tf.function())output_attentionsoutput_hidden_states use_cacheencoder_outputsr^zA decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.) batch_sizemodel_input_namerdecoder_start_token_id bos_token_id input_ids max_lengthz0Using the model-agnostic default `max_length` (=zx) to control the generation length. recommend setting `max_new_tokens` to control the maximum length of the generation.zBoth `max_new_tokens` (=z) and `max_length`(=z) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)z3Unfeasable length constraints: the minimum length (z%) is larger than the maximum length ()zInput length of z is z, but `max_length` is set to zW. This can lead to unexpected behavior. You should consider increasing`max_new_tokens`.rFT)rinput_ids_seq_lengthrz)num_return_sequences has to be 1, but is z when doing greedy search.)r pad_token_id eos_token_idr output_scoresreturn_dict_in_generatez when doing contrastive search.)top_k penalty_alpharrrrrr)r)r expand_sizer)r logits_warperrrrseedrrzwBeam search decoding cannot return more sequences than it has beams. Please set num_beams >= num_return_sequences, got z and z (respectivelly))rrrexpand_in_new_axis) rrrlength_penaltyearly_stoppingrrrnum_return_sequences) do_samplerrrrrrrrrrr1)Krr_from_model_config_original_object_hashhashrfrom_model_configrirMrNcopydeepcopyupdater isinstancer.r/r` is_floatingnpndarray issubdtypefloatingrprqrrrrloggerwarninglistexecuting_eagerlysupports_xla_generationr_prepare_model_inputsrrrrrrrrrrkeys&_prepare_attention_mask_for_generationrrm reduce_any._prepare_encoder_decoder_kwargs_for_generation)_prepare_decoder_input_ids_for_generationrrrmax_new_tokensrO min_lengthrrr num_beams_get_logits_processorr greedy_searchrrcontrastive_search_get_logits_warper_expand_inputs_for_generationsample beam_searchrr)rTrrrrrXnew_generation_configrruse_xla inputs_tensorrraccepts_attention_maskrequires_attention_maskrrhas_default_max_lengthinput_ids_stringis_contrastive_search_gen_modeis_greedy_gen_modeis_beam_gen_modeis_sample_gen_modeis_beam_sample_gen_moderr1r1r2generatesF                                         zTFGenerationMixin.generaterrcCst|jdko|jtjtjfv}|duotj||k}|dup#||k}|r7|r7|r7tjtj ||tjdStj |jddtjdS)Nr r_) rerdr`r.rqint64rmrrp not_equalones)rTrrr is_input_idsis_pad_token_in_inputs&is_pad_token_not_equal_to_eos_token_idr1r1r2rs  z8TFGenerationMixin._prepare_attention_mask_for_generationrrcs|}gdfdd|D}tt|jjdvp#dv}|s1fdd|D}d|d<|||<||jkrCd||j<|d i|}||d <|S) N)decoder_ cross_attnrcs,i|]\}tfddDs|qS)c3s|]}|VqdSN) startswith).0pargumentr1r2 sz^TFGenerationMixin._prepare_encoder_decoder_kwargs_for_generation...)any)rr)irrelevant_prefixrr2 s zTTFGenerationMixin._prepare_encoder_decoder_kwargs_for_generation..rXrcsi|] \}}|vr||qSr1r1)rrr)encoder_signaturer1r2r%sT return_dictrr1) get_encoderrrrrrrmain_input_name)rTrrrencoderencoder_kwargsencoder_accepts_wildcardrr1)rrr2rs$    z@TFGenerationMixin._prepare_encoder_decoder_kwargs_for_generationrrrc Cs|durd|vr|d}nd|vr|dkr|d}nd}|||}tj|dftjd|}|dur:|}||fSt|dddf|krptj||gdd}d |vrp|d }tjt|ddddf|fdd}||d <||fS) zGPrepares `decoder_input_ids` for generation with encoder-decoder modelsNrrrr_rr^r\decoder_attention_mask)r_get_decoder_start_token_idr.rrq reduce_allconcat ones_like) rTrrrrrrdecoder_input_ids_startrr1r1r2r3s(    z;TFGenerationMixin._prepare_decoder_input_ids_for_generationcCsH|dur|n|jj}|dur|n|jj}|dur|S|dur |Std)Nz\`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.)rrrr)rTrrr1r1r2rZsz-TFGenerationMixin._get_decoder_start_token_idrrrrrc sldtjffdd fdd}|dur|}||}|r2|ddur*td||d|d<||fS) a Expands tensors from [batch_size, ...] to [batch_size * expand_size, ...] or [batch_size, expand_size, ...], depending on `expand_in_new_axis`. Beam-based approaches expect this function to be used with `expand_in_new_axis=True` tensorcsLrt|}t|dddf|dft|ddStj|ddS)Nrrr\)rr.rstuplerepeat)rrd)rrr1r2_expand_tensor|s0zGTFGenerationMixin._expand_inputs_for_generation.._expand_tensorcs:|D]}||durt||tjr||||<q|Sr)rr.r/)dict_to_expandr)r r1r2_expand_dict_for_generations zTTFGenerationMixin._expand_inputs_for_generation.._expand_dict_for_generationNrzMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)r.r/rr)rrrrrr r1)r rrr2rns z/TFGenerationMixin._expand_inputs_for_generationc s2|jjrt|drt|jdr|jj|jkr|jjn|jfdd|D}|d}|durH|durHtd|dd|d d |durN|}d krd |vr|jjs~d tt |j j v}|sstd |jjd|j|||d|d <n|durtd|d d }||||}||fS)zT This function extracts the model-specific `inputs` for generation. rrcs&i|]\}}|dus|kr||qSrr1)rkv input_namer1r2rs&z;TFGenerationMixin._prepare_model_inputs..Nz `inputs`: z` were passed alongside z0 which is not allowed. Make sure to either pass z or z=...r inputs_embedszAYou passed `inputs_embeds` to `.generate()`, but the model class z doesn't have its forwarding implemented. See the GPT2 implementation for an example (https://github.com/huggingface/transformers/pull/21405), and feel free to open a PR with it!)rzMYou passed `inputs_embeds` and `input_ids` to `.generate()`. Please pick one.)rirhasattrrrrrrrrrrYrrrr**_maybe_initialize_input_ids_for_generation)rTrrr inputs_kwarghas_inputs_embeds_forwardingr1rr2rsL      z'TFGenerationMixin._prepare_model_inputscCs|dur|S|d}|jjr%|dur%|jjdd}tj|tjddS|dur-tdd}| D]}t |tj rB|jd}nq3tj|dftjd|S) z3Initializes input ids for generation, if necessary.Nrr^r_izB`bos_token_id` has to be defined when no `input_ids` are provided.rr) rrirlast_hidden_staterdr.rrqrvaluesrr/)rTrrrrrdrrr1r1r2rs    z._initialize_attentionc stjddgtjd|}|r)|d}tjdf|jd}t|||}d|i}|S|d}tjdf|jd}t|||}d|i}|S)z]updates the appropriate attention mask -- encoder-decoder models use `decoder_attention_mask`rrr_rr)r.constantrqrrr`r) rnew_past_indexr update_startr#decoder_attention_mask_update_slicer$rattention_mask_update_slicer%r1r2_update_attention4s   zTTFGenerationMixin._update_model_kwargs_for_xla_generation.._update_attentioncSs|dkrCtjddgddgd|gddggtjd}d}|D]$}t|}tt|ddD] }t|||||<q,|t|f7}q|Stjddgg|gdd }t|}tt|D] }t|||||<qY|S) zNinitialize past_key_values with zeros -- the structure depends on `batch_axis`rr_r1Nr r)r )rzupdatesrd) r.r'rqrrcrepadr  scatter_nd)rr#r!padding_valuesnew_past past_layernew_past_layerir1r1r2_initialize_pastEs(zSTFGenerationMixin._update_model_kwargs_for_xla_generation.._initialize_pastc Ss6|dkrRtgd}d}|D]@}t|}tt|ddD](}||ddddddf}t||ddddddf|||||<q|t|f7}q|Stgd}ddtt|D}tt|D].}||ddddddddf}t||ddddddddf|||||<qj|S) Nr)rrrrr1r r^)rrrrrcSsg|]}dqSrr1)r_r1r1r2 fzcTFGenerationMixin._update_model_kwargs_for_xla_generation.._update_past..)r.r'rrcrerr ) rr(r!slice_start_baser3r4r5r6 update_slicer1r1r2 _update_pastVs($  $* zOTFGenerationMixin._update_model_kwargs_for_xla_generation.._update_pastzPNo known `past_key_values variable` found in model outputs (model outputs keys: rrrr )rrrrrrr )rTrrr rrrr!r&r,r7r=ris_past_initializedr#r$r3r(r1r%r2'_update_model_kwargs_for_xla_generations,         z9TFGenerationMixin._update_model_kwargs_for_xla_generationcCst}|jdkrt|jtrt|jd}nd}nd}|jdur-|jdkr-|t|j|j durA|j dkrA|t |j |d|j durU|j dkrU|t |j |d|S)z This class returns a [`TFLogitsProcessorList`] list object that contains all relevant [`TFLogitsWarper`] instances used for multinomial sampling. rr N?r)rmin_tokens_to_keep)top_prA) rrrrrre temperaturerr rr!rBr")rTrwarpersrAr1r1r2rs  z$TFGenerationMixin._get_logits_warperrcCsvt}|jdur|jdkr|t|jd|jdur(|jdkr(|t|j|jdur7|t|j|j|j durP|jdurP|j dkrP|t |j |j|j dur]|t |j |j durl|t|j|j |jdury|t|j|jdur|}|dks|j dur|n|d}|jdur||jdd7}|t|j||jdur|t|j|||}|S)z This class returns a [`TFLogitsProcessorList`] list object that contains all relevant [`TFLogitsProcessor`] instances used to modify the scores of the language model head. Nr@)penaltyrrr^)rrepetition_penaltyrrno_repeat_ngram_sizer bad_words_idsrrrrforced_bos_token_idrforced_eos_token_idrrsuppress_tokensrbegin_suppress_tokensforced_decoder_idsrr_merge_criteria_processor_list)rTrrr processors begin_indexr1r1r2rsL                z'TFGenerationMixin._get_logits_processor default_list custom_listcCs~t|dkr|S|D]-}|D](}t|t|ur6d}td|dt|d|d|d|d|d |d qq |||S) Nrzlogits processorz A custom z of type z with values zP has been passed to `generate`, but it has already been created with the values z. z has been created by passing the corresponding arguments to generate or by the model's config default values. If you just want to change the default values of zL consider passing them as arguments to `generate` instead of using a custom .)rerrextend)rTrQrRr}custom object_typer1r1r2rNs*  z0TFGenerationMixin._merge_criteria_processor_listrrrrc  sdurntdurnjj dur njj dur#njjttr/g dur5 njj dur? njj durI njj durSnjj | djj t  dtvrotjnttfdddDrdndd ttjjv r rgndr rgndr rgndr rgndt|\} t j| ft jd  pd} t j|| gd d } t jft jd }d d} fdd}|| || | \} }} } | }t j||| || | f|d\} }} }s| ddd| f} r~jj rv r1| d!dnd} r=| d!dnd}durHt"nddurSt"nddur^t"nddurit"ndt#| ||dSt$| dS| S)a Generates sequences for models with a language modeling head using greedy decoding. Parameters: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`TFLogitsProcessorList`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. max_length (`int`, *optional*, defaults to 20): The maximum length of the sequence to be generated. pad_token_id (`int`, *optional*): The id of the *padding* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more details. output_hidden_states (`bool`, *optional*, defaults to `False`): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more details. output_scores (`bool`, *optional*, defaults to `False`): Whether or not to return the prediction scores. See `scores` under returned tensors for more details. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. model_kwargs: Additional model specific keyword arguments will be forwarded to the `call` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.TFGreedySearchDecoderOnlyOutput`], [`~generation.TFGreedySearchEncoderDecoderOutput`] or `tf.Tensor`: A `tf.Tensor` containing the generated tokens (default behaviour) or a [`~generation.TFGreedySearchDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.TFGreedySearchEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. Examples: ```python >>> from transformers import ( ... AutoTokenizer, ... TFAutoModelForCausalLM, ... TFLogitsProcessorList, ... TFMinLengthLogitsProcessor, ... ) >>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> # set pad_token_id to eos_token_id because GPT2 does not have a PAD token >>> model.generation_config.pad_token_id = model.generation_config.eos_token_id >>> input_prompt = "Today is a beautiful day, and" >>> input_ids = tokenizer(input_prompt, return_tensors="tf").input_ids >>> # instantiate logits processors >>> logits_processor = TFLogitsProcessorList( ... [ ... TFMinLengthLogitsProcessor(15, eos_token_id=model.generation_config.eos_token_id), ... ] ... ) >>> outputs = model.greedy_search(input_ids, logits_processor=logits_processor) >>> tokenizer.batch_decode(outputs, skip_special_tokens=True) ["Today is a beautiful day, and I'm so happy to be here. I'm so happy to"] ```NrEncoderDecoderc3|]}|vVqdSrr1r model_prefix model_namer1r2r`z2TFGenerationMixin.greedy_search..TFGPT2TFCTRLrruse_memsr_r^r\cS t|Szstate termination condition fn.r.r generatedfinished_sequencesr rr1r1r2greedy_search_cond_fnts z>TFGenerationMixin.greedy_search..greedy_search_cond_fnc sD|ddus r|ddd|f}nt|dd|dfd}j|fdi|}di|d d}|jdddf}|||}s r rT| rajjra|jn rwjjsw|j jjrw|j  rjjr|j n rjjr|j tj |dtjd} durԈ durtd dt|tj} | |  d| } tjjtt| tftdd d } || B}tjtt|ggdd } tj|| | d }|d7}rj|||jjd }nj||jjd}|dddur|dd||||fS)state update fn.rNrr^rTrrr)r] output_typeGIf `eos_token_id` is defined, make sure that `pad_token_id` is defined.rr\rrzr/rrr rrrr!rr1)rr.rbrYlogitsrrirr8r'r9r:r(argmaxrqrrprmrequalrsrerhrctensor_scatter_nd_updater?rr) rfrgr rr model_inputsrnext_token_logitsnext_tokens_scores next_tokensunfinished_seqnext_token_is_eosupdate_indices)rcache_batch_axisr9r8r:rrrneeds_full_inputrrrrrr&rTrrr1r2greedy_search_body_fnyst          "   z>TFGenerationMixin.greedy_search..greedy_search_body_fnmaximum_iterationsrr'r(r%r&r6r7r8r9r:r%r&r'r()%rrrrrrintrrrrrrr.rstrdecoderrrrrrYrrrrrqrr"bool while_looprirrr r5r$)rTrrrrrrrrrrr input_ids_paddingrfrgrhr}rr8r6r7r1)rr{r9r8r:rrrr\r|rrrrrr&rTrrr2rsQ    .P    zTFGenerationMixin.greedy_searchrrc  sdurntdurntdurnjjdur"njjdur,njjttr8g dur> njj durH njj durR njj dur\njj | djj t  dtvrxtjnt t fdddDrdndd ttjjv r rgndr rgndr rgndr rgndt|\} t j| ft jd pd}t j||gd d }t jft jd }d d} fdd}|||| | \}}} } | }t j||||| | f|d\}}} }s*|ddd| f}rjj r r<| d!dnd} rH| d!dnd}durSt"nddur^t"nddurit"nddurtt"ndt#|||dSt$|dS|S)ar Generates sequences for models with a language modeling head using multinomial sampling. Parameters: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`TFLogitsProcessorList`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. logits_warper (`TFLogitsProcessorList`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsWarper`] used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. max_length (`int`, *optional*, defaults to 20): The maximum length of the sequence to be generated. pad_token_id (`int`, *optional*): The id of the *padding* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. seed (`List[int]`, *optional*): Random seed to control sampling, containing two integers, used when `do_sample` is `True`. See the `seed` argument from stateless functions in `tf.random`. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more details. output_hidden_states (`bool`, *optional*, defaults to `False`): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more details. output_scores (`bool`, *optional*, defaults to `False`): Whether or not to return the prediction scores. See `scores` under returned tensors for more details. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. model_kwargs: Additional model specific kwargs will be forwarded to the `call` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.TFSampleDecoderOnlyOutput`], [`~generation.TFSampleEncoderDecoderOutput`] or `tf.Tensor`: A `tf.Tensor` containing the generated tokens (default behaviour) or a [`~generation.TFSampleDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.TFSampleEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. Examples: ```python >>> import tensorflow as tf >>> from transformers import ( ... AutoTokenizer, ... TFAutoModelForCausalLM, ... TFLogitsProcessorList, ... TFMinLengthLogitsProcessor, ... TFTopKLogitsWarper, ... TFTemperatureLogitsWarper, ... ) >>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token >>> model.generation_config.pad_token_id = model.generation_config.eos_token_id >>> input_prompt = "Today is a beautiful day, and" >>> input_ids = tokenizer(input_prompt, return_tensors="tf").input_ids >>> # instantiate logits processors >>> logits_processor = TFLogitsProcessorList( ... [ ... TFMinLengthLogitsProcessor(15, eos_token_id=model.generation_config.eos_token_id), ... ] ... ) >>> # instantiate logits processors >>> logits_warper = TFLogitsProcessorList( ... [ ... TFTopKLogitsWarper(50), ... TFTemperatureLogitsWarper(0.7), ... ] ... ) >>> tf.random.set_seed(0) >>> outputs = model.sample(input_ids, logits_processor=logits_processor, logits_warper=logits_warper) >>> tokenizer.batch_decode(outputs, skip_special_tokens=True) ['Today is a beautiful day, and I love my country. But when I look at Donald Trump,'] ```NrrWc3rXrr1rYr[r1r2r|r]z+TFGenerationMixin.sample..r^rrrar_r^r\cSrbrrdrer1r1r2sample_cond_fns z0TFGenerationMixin.sample..sample_cond_fnc s|ddus r|ddd|f}nt|dd|dfd}j|fdi|}di|d d}|jdddf}|||}|||}sr rZ| rgjjrg|jn r}jjs}|j jjr}|j  rjjr|j n rjjr|j dur} ntj jjjtjjtjjdtjd} tjtjj|d| tjd dd } dur durtd dt|tj} | |  d| } tjjtt| tftdd d } || B}tjtt|ggdd } tj|| | d }|d7}r(j |||jjd}nj!||jjd}|dddurA|"dd||||fS)Nrrr^rTrj)r r_)rp num_samplesrr`r\rlrrmrnror1)#rr.rbrYrprrirr8r'r9r:r( experimentalnumpyrQrandintrqminmaxsqueezestateless_categoricalrrprmrrrrsrerhrcrsr?rr)rfrgr rrrtrrurv sample_seedrwrxryrz)rr{r9r8r:rrrrr|rrrrrr&rrTrrr1r2sample_body_fns           " "   z0TFGenerationMixin.sample..sample_body_fnr~rr'r(rr)%rrrrrrrrrrrrrr.rrrrrrrrYrrrrrqrr"rrrirrr r>r=)rTrrrrrrrrrrrrr rrfrgrrrr8r6r7r1)rr{r9r8r:rrrrr\r|rrrrrr&rrTrrr2rsf    2Y    zTFGenerationMixin.samplecfdd}tj||S)zDGathers the beam slices indexed by beam_indices into new beam array.csdkr!tjtt|dtfdd}tj||d}tj|ddd}dkrQtjtt|dtfdd}tj|}tj||d}|S)Nrr\)permr)paramsrzr] batch_dims)r.rrcrankrfgatherrminvert_permutation)rrgathered_tensorr!rBr1r2 gather_fn#s,, z2TFGenerationMixin._gather_beams..gather_fnr.nest map_structure)nestedrBr!rr1rr2 _gather_beamsszTFGenerationMixin._gather_beamsrrrrc" s@ddd ddd dur nt dur nt dur" njj |dur,|njj}dur6njjttrBg| durH| njj} durRnjjdur\njj durfnjj durpnjj  durz njj durnjj |djjt dtvrtjnttfd d d Drd ndd ttjjvrƈrgndrΈrgndrֈrgndrވrgndt|\}|}tj |ftjd|pd}tj||gdd}tj ftjd|pd}tjftj d}t!tj"t#dgdgd ddd g}tfd}tj |ftjdd}tj |ftjdd}d|vrm |dd|dd<d|vrz |d|d< fdd} fdd}||||||||||| \ }}}}}}}}}} |}tj$||||||||||||f |d\ }}}}}}}}}}tj%j&|d d}t'|ddddf||}t'|ddddf||}|tj(||tj)d }t'|dddf||} |ddd| ddf} |ddd| f} |ddd| ddf}sY|ddd|f}|ddd||f}rj*j+rrk|d,dnd}rw|d,dnd} r~t-nt.}!|!||||| d Srt/nt0}!|!|||dS|S) a Generates sequences for models with a language modeling head using beam search. If `do_sample` is `False`, uses a greedy approach, otherwise does multinomial sampling without replacement. Parameters: input_ids (`tf.Tensor` of shape `(batch_size, num_beams, sequence_length)`): The sequence used as a prompt for the generation. do_sample (`bool`, *optional*, defaults to `False`): Whether or not to use sampling ; use greedy decoding otherwise. max_length (`int`, *optional*, defaults to 20): The maximum length of the sequence to be generated. pad_token_id (`int`, *optional*): The id of the *padding* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. length_penalty (`float`, *optional*, defaults to 1.0): Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative), `length_penalty` > 0.0 promotes longer sequences, while `length_penalty` < 0.0 encourages shorter sequences. early_stopping (`bool` or `str`, *optional*, defaults to `False`): Controls the stopping condition for beam-based methods, like beam-search. It accepts the following values: `True`, where the generation stops as soon as there are `num_beams` complete candidates; `False`, where an heuristic is applied and the generation stops when is it very unlikely to find better candidates; `"never"`, where the beam search procedure only stops when there cannot be better candidates (canonical beam search algorithm). logits_processor (`[TFLogitsProcessorList]`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. logits_warper (`TFLogitsProcessorList`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsWarper`] used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. num_return_sequences(`int`, *optional*, defaults to 1): The number of independently computed returned sequences for each element in the batch. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more details. output_hidden_states (`bool`, *optional*, defaults to `False`): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more details. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. model_kwargs: Additional model specific kwargs will be forwarded to the `call` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.TFBeamSearchDecoderOnlyOutput`], [`~generation.TFBeamSearchEncoderDecoderOutput`] or `tf.Tensor`: A `tf.Tensor` containing the generated tokens (default behaviour) or a [`~generation.TFBeamSearchDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.TFBeamSearchEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. Examples: ```python >>> from transformers import ( ... AutoTokenizer, ... TFAutoModelForSeq2SeqLM, ... TFLogitsProcessorList, ... TFMinLengthLogitsProcessor, ... ) >>> import tensorflow as tf >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base") >>> model = TFAutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base") >>> encoder_input_str = "translate English to German: How old are you?" >>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="tf").input_ids >>> # lets run beam search using 3 beams >>> num_beams = 3 >>> # define decoder start token ids >>> input_ids = tf.ones((1, num_beams, 1), dtype=tf.int32) >>> input_ids = input_ids * model.generation_config.decoder_start_token_id >>> # add encoder_outputs to model keyword arguments >>> encoder_outputs = model.get_encoder()(encoder_input_ids, return_dict=True) >>> encoder_outputs.last_hidden_state = tf.repeat( ... tf.expand_dims(encoder_outputs.last_hidden_state, axis=0), num_beams, axis=1 ... ) >>> model_kwargs = {"encoder_outputs": encoder_outputs} >>> # instantiate logits processors >>> logits_processor = TFLogitsProcessorList( ... [TFMinLengthLogitsProcessor(5, eos_token_id=model.generation_config.eos_token_id)] ... ) >>> outputs = model.beam_search(input_ids, logits_processor=logits_processor, **model_kwargs) >>> tokenizer.batch_decode(outputs, skip_special_tokens=True) ['Wie alt bist du?'] ```rcSsBt|}t||d|||||dg||ddS)z8Flattens the first two dimensions of a non-scalar array.Nrr rr.rg)rr!rdr1r1r2flatten_beam_dims 0z7TFGenerationMixin.beam_search..flatten_beam_dimcSs4t|}t||d|d|g||ddS)zFUnflattens the first, flat batch*beam dimension of a non-scalar array.Nr^rr)rrr!rdr1r1r2unflatten_beam_dims,z9TFGenerationMixin.beam_search..unflatten_beam_dimNrrWc3rXrr1rYr[r1r2rr]z0TFGenerationMixin.beam_search..r^rrar_r^r\errrc s|k} dkrdkr|ddddf|} n|ddddftj||tjd} t|tjj|dddd} tj| | k} tj|du@}| |@| @S) z Beam Search termination condition function -- halts the generation loop if any of these conditions becomes False neverrNrr_Tr]keepdimsr)r.rpfloat32rrrm reduce_minrr)r running_sequencesrunning_scoresrunning_beam_indicesr%r&rBis_sent_finisheddecoder_prompt_lenrnot_max_length_yetbest_running_scoreworst_finished_scoreimprovement_still_possiblestill_open_beam)rrrr1r2beam_search_cond_fn s" z:TFGenerationMixin.beam_search..beam_search_cond_fnc 4 s| ddus r|ddddd|f} nt|dddd|dfd} j | fdi| } di| dd} | jdddf} tj| } | ||}|}rt | ||}|}|}|tj|dd }|jd}t||f}s݈r݈r | ||rj j r | j nrĈj j sĈ | j j j rĈ | jrшj j rш | jn r݈j j r݈ | jd}rt||}tj||ddd }n tjj||d \}}||}||}||}||}tt|g}tt|g}tj||t||ggdd }tj||t||gd }|ttjtdd |j}tj||t|||ggdd }tj||t||gd }durtj|dddd|fjtjd }n0tjjtt|dddd|ft g|dddd|fjttdddd }|ttj!tj"tjd tjtjd fdd t#|@}|t$|tj%d} tjj| d d}!|| |g|!\}"}#}$|tj$|d|tj%d  }ttjj&|dddt#|du@}%||%B}&|t$|&tj%d7}tj!||gdd }'tj!||gdd }(tj!||gdd })tj!||gdd }*tjj|(d d}+|'|(|)|*g|+\},}-}.}/|d}d| vrtj'(fdd| j)}0||!}1j|0|1d}2tj'( fdd|2| d<rj*| | | j j d}3nj+| | j j d}3| dddur| ,dd||"|#|$|,|-|.|/||3f S)z Beam Search iterative update function -- each iteration adds a new token and updates the best sequences seen so far rNrr^rTrjr r\r]rrrmr_rrrcs|dSNr!r1r)r{rrr1r2 szLTFGenerationMixin.beam_search..beam_search_body_fn..rcs |dSrr1r)r{rr1r2r s rnror1)-rr.rbrYrprkrlrdrgrrirr8r'r9r:r(sample_without_replacementrrmrrr rcrarhrsrsr"rrrrrerrrrprrrrrr?rr)4r rrrr%r&rBrrrrrtrrp log_probslog_probs_processedrj beams_to_keep topk_indicestopk_log_probstopk_current_beam_indicestopk_running_beam_indicestopk_running_sequencestopk_ids indices_batch indices_beamrztopk_sequencesbatch_modified_indicestopk_beam_indiceseos_in_next_tokendid_topk_just_finishedrunning_topk_log_probsnext_topk_indicesnext_running_sequencesnext_running_scoresnext_running_beam_indicesbeams_in_batch_are_full add_penaltymerged_sequences merged_scores merged_beamsmerged_is_sent_finishedtopk_merged_indicesnext_sequences next_scoresnext_beam_indicesnext_is_sent_finishedcachenext_running_indices next_cachenext_model_kwargs) all_scoresrr{r9r8r:rrrrrrrrr|rrrrrrTrrrr1r2beam_search_body_fn) s:"                 &  (          z:TFGenerationMixin.beam_search..beam_search_body_fnr~r'r() r%rAr&rBr6r7r8r9r:)r%rAr&rBr'r(r)1rrrrrrrrrrrrrrrrr.rrrrrrrrYrrrrrqrr"rrarbconvert_to_tensorrrmrrrrprrirrrIrFrHr@)"rTrrrrrrrrrrrrrrrr rrrr%rrr&rrBrrrr8 none_finishedr6r7 output_clsr1)rrr{r9r8r:rrrrrrrrr\r|rrrrrrTrrrr2r4s p    (   :)q     zTFGenerationMixin.beam_searchrrc  sddddur ntdurnt dur njj dur'njjdur1njjttr=g durC njj durM njj durW njj duranjj d| ddt dtvr}tjnt t fdd d Drd ndr rgndr rgndr rgndr rgndt|\}t j |ft jd pd}t j||gd d}t jft jd }dd} fdd}||||| d\}}}} } |}t j|||||| |f|d\}}}}}s'|ddd|f}rjjr~ r9| ddnd} rE| ddnd}durPtnddur[tnddurftnddurqtndt|||dSt|dS|S)u Generates sequences of token ids for models with a language modeling head using **contrastive search** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. top_k (`int`, *optional*, defaults to 1): The size of the candidate set that is used to re-rank for contrastive search penalty_alpha (`float`, *optional*, defaults to 0): The degeneration penalty for contrastive search; activate when it is larger than 0 logits_processor (`TFLogitsProcessorList`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. logits_warper (`TFLogitsProcessorList`, *optional*): An instance of [`TFLogitsProcessorList`]. List of instances of class derived from [`TFLogitsWarper`] used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. max_length (`int`, *optional*, defaults to 20): The maximum length of the sequence to be generated. pad_token_id (`int`, *optional*): The id of the *padding* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more details. output_hidden_states (`bool`, *optional*, defaults to `False`): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more details. output_scores (`bool`, *optional*, defaults to `False`): Whether or not to return the prediction scores. See `scores` under returned tensors for more details. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. model_kwargs: Additional model specific keyword arguments will be forwarded to the `call` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.TFContrastiveSearchDecoderOnlyOutput`], [`~generation.TFContrastiveSearchEncoderDecoderOutput`] or `tf.Tensor`: A `tf.Tensor` containing the generated tokens (default behaviour) or a [`~generation.TFContrastiveySearchDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.TFContrastiveSearchEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. Examples: ```python >>> from transformers import AutoTokenizer, TFAutoModelForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m") >>> model = TFAutoModelForCausalLM.from_pretrained("facebook/opt-125m") >>> # set pad_token_id to eos_token_id because OPT does not have a PAD token >>> model.config.pad_token_id = model.config.eos_token_id >>> input_prompt = "DeepMind Company is" >>> input_ids = tokenizer(input_prompt, return_tensors="tf") >>> outputs = model.contrastive_search(**input_ids, penalty_alpha=0.6, top_k=4, max_length=64) >>> tokenizer.batch_decode(outputs, skip_special_tokens=True) ['DeepMind Company is a company that focuses on the development and commercialization of artificial intelligence (AI). DeepMind’s mission is to help people understand and solve problems that are difficult to solve in the world today.\n\nIn this post, we talk about the benefits of deep learning in business and how it'] ```rcr)zbGathers the slices indexed by selected_idx_stacked from a potentially nested structure of tensors.cstj|d}|S)N)rrzr])r.r)rrr!selected_idx_stackedr1r2r szVTFGenerationMixin.contrastive_search..gather_best_candidate..gather_fnr)rrr!rr1rr2gather_best_candidate szCTFGenerationMixin.contrastive_search..gather_best_candidateNTrrWc3rXrr1rYr[r1r2r r]z7TFGenerationMixin.contrastive_search..r^rr_r^r\cSrbrcrd)rfrgr rnext_step_cached_variablesr1r1r2contrastive_search_cond_fn s zHTFGenerationMixin.contrastive_search..contrastive_search_cond_fnc s>|ddurj|ddd|ffdi|}di|dd d}jjr1|jd}n|jd}rIt|ddgd |gddgg}|jdddddf}rfj ||| jjd}n j ||jjd }j djjd |\} }|d} | durt j jd t| dttjfr| ddjdkrt j jd n |d }|d}|d}|||}|||}t|dd} tjj| d\} } s(r( r| rjjr|jn r jjs |jjjr |j rjjr|jn r(jjr(|jtjfdd|d|d<jt| ddgfdi|}di|dd d}|}|jdddddf}jjru|jd}|j}n|jd}|j}tj|ddd|ddfdd}t||| }|tj |jdtj!d}tj"| |ddd}||}rt#||d|dg}n tj$||gdd}||}||d}||}jjrd}d} r|j|}|j|}t%|||pd|pdd}nd} r|j|}t&|||pdd}durU dur&t ddt'|tj(}|| d|}tjj)t*t+|t,ft-ddd}||B}tj.t t+|ggdd}tj/|||d}|d7}rj |||d jjd}n j ||jjd }|||d}|||||fS)rirNrT)rrrr^rrnro)rrzQ does not support caching and therefore **can't** be used for contrastive search.z| does not have a standard cache format and therefore **can't** be used for contrastive search without further modifications.logit_for_next_steplast_hidden_statesrr\rcstj|dS)Nr\)r.r r)r{rr1r2rg r:zZTFGenerationMixin.contrastive_search..contrastive_search_body_fn..rr_rrr1)rr:r8r9)rr(r'rlrm)rrr)0rrYrirr:r(r.r0rpr?rrrrr*rr r/rdrrmrrr8r'r9rrrgrr  _ranking_fastrcrrrrr r rprqrrrrsrerbrhrs)rfrgr rrrtrrrr8r next_probs top_k_probs top_k_idsnext_model_inputsnext_past_key_valuesrp next_hiddenfull_hidden_statescontext_hidden selected_idxrrwnext_decoder_hidden_statesnext_step_cross_attentionsnext_step_decoder_attentionsnext_step_attentionsrxryrz)rr{r9r8r:rrrrrrrrrrrr&rTrrrr1r2contrastive_search_body_fn s>   "                  &        " zHTFGenerationMixin.contrastive_search..contrastive_search_body_fnr~rr'r(rrr)rrrrrrrrrrrrr.rrrrrrrqrr"rrrirrr rKrJ)rTrrrrrrrrrrrrrr rrfrgrrrrr8r6r7r1)rr{r9r8r:rrrrrr\rrrrrrr&rTrrrr2r| s J      4]    z$TFGenerationMixin.contrastive_search)NF)NNNNr)NN)rFNF)NNN)F)Fr)NNNNNNNN) NNNNNNNNNNr) FNNNNNNNNNNNN) rrNNNNNNNNN)4r*r+r,r-rPpropertyrUrrYr.r/rrrr|rrrrrrrrTFGenerateOutputrrrrrr staticmethodrrrrrrr?rrrNTFGreedySearchOutputrTFSampleOutputrrfloatTFBeamSearchOutputTFBeamSampleOutputrTFContrastiveSearchOutputrr1r1r1r2rLs   x     #  (  ( D         7           $       O      rLc Csnt|}tttjt|ddd|ddg}tt|t|ddggd}t|t|dg|S)Nrr^r\r) rr.rgrsrbrcrfrr1)r batch_indicesrdbroad_casted_batch_dims pair_indicesr1r1r2scatter_values_on_batch_indices s,"r c CsBtjtjtjt|dd  }tj|||\}}|S)z categorical sampling without replacement is currently not implemented the gumbel-max trick will do for now see https://github.com/tensorflow/tensorflow/issues/9260 for more info rr)r.rmlogrQuniformrrkr)rprzr8rzr1r1r2r s(rrrnext_top_k_probsalpha beam_widthr[c Cs|tj|ddd}|tj|ddd}tjtjj||dddd}tj|dd}tj|dgd}d||||} tj| d|gd} tj| d d} | S) a  Reranks the top_k candidates based on a degeneration penalty (cosine similarity with previous tokens), as described in the paper "A Contrastive Framework for Neural Text Generation". Returns the index of the best candidate for each row in the batch. r Tr) transpose_br^r\)rdr@r)r.normrlinalgmatmulrnrgrq) rrrrrnorm_context_hiddennorm_next_hidden cosine_matrixdegeneration_penaltycontrastive_scorerr1r1r2r( s r)IrrrM dataclassesrtypingrrrrrrr tensorflowr.%tensorflow.compiler.tf2xla.python.xlarmodeling_tf_outputsr r models.autor r rrtf_utilsrrutilsrrconfiguration_utilsrtf_logits_processrrrrrrrrrrr r!r" get_loggerr*rr$r5r=r>r@rFrHrIrJrKrrrrrrrLr rr/rrrr1r1r1r2s   < ))"1"0 (     l