o  hq@s2ddlZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZdd lmZdd lmZmZeeZeZd ZGd d d eZ Gddde Z!Gddde Z"Gddde Z#Gddde Z$Gddde Z%Gddde&Z'de'de(de'fddZ)dS)N)ABC) OrderedDict)deepcopy)DictListOptionalTupleUnion) functional)isin_mps_friendly)PreTrainedTokenizerBase)add_start_docstringsloggingaL Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) scores (`torch.FloatTensor` of shape `(batch_size, config.vocab_size)`): Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax or scores for each vocabulary token after SoftMax. If this stopping criteria depends on the `scores` input, make sure you pass `return_dict_in_generate=True, output_scores=True` to `generate`. kwargs (`Dict[str, Any]`, *optional*): Additional stopping criteria specific kwargs. Return: `torch.BoolTensor`. (`torch.BoolTensor` of shape `(batch_size, 1)`), where `True` indicates we stop generation for a particular row, `True` indicates we should continue. c@s4eZdZdZeedejdejdej fddZ dS)StoppingCriteriazAbstract base class for all stopping criteria that can be applied during generation. If your stopping criteria depends on the `scores` input, make sure you pass `return_dict_in_generate=True, output_scores=True` to `generate`. input_idsscoresreturncKstd)Nz'StoppingCriteria needs to be subclassed)NotImplementedError)selfrrkwargsr}/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/generation/stopping_criteria.py__call__5szStoppingCriteria.__call__N) __name__ __module__ __qualname____doc__r"STOPPING_CRITERIA_INPUTS_DOCSTRINGtorch LongTensor FloatTensor BoolTensorrrrrrr.s"rc@LeZdZdZd dedeefddZeede j de j d e j fd d Z dS) MaxLengthCriteriaa This class can be used to stop generation whenever the full generated number of tokens exceeds `max_length`. Keep in mind for decoder-only type of transformers, this will include the initial prompted tokens. Args: max_length (`int`): The maximum length that the output sequence can have in number of tokens. max_position_embeddings (`int`, *optional*): The maximum model length, as defined by the model's `config.max_position_embeddings` attribute. N max_lengthmax_position_embeddingscCs||_||_dSN)r%r&)rr%r&rrr__init__Fs zMaxLengthCriteria.__init__rrrcKs^|jd}||jk}|jdur |s ||jkr td|jdtj|jdf||jtjdS)NzrThis is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (zb). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.rdevicedtype) shaper%r&logger warning_oncerfullr+bool)rrrrcur_lenis_donerrrrJs  zMaxLengthCriteria.__call__r')rrrrintrr(rrrr r!r"rrrrrr$:s  "r$c@r#) MaxTimeCriteriaa This class can be used to stop generation whenever the full generation exceeds some amount of time. By default, the time will start being counted when you initialize this function. You can override this by passing an `initial_time`. Args: max_time (`float`): The maximum allowed time in seconds for the generation. initial_time (`float`, *optional*, defaults to `time.time()`): The start of the generation allowed time. Nmax_timeinitial_timestampcCs&||_|durt|_dS||_dSr')r6timer7)rr6r7rrrr(ds zMaxTimeCriteria.__init__rrrcKs2t|j|jk}tj|jdf||jtjdS)Nrr*)r8r7r6rr0r-r+r1rrrrr3rrrrhszMaxTimeCriteria.__call__r')rrrrfloatrr(rrrr r!r"rrrrrr5Ws  "r5c @seZdZdZdedeeeeffddZddZ e dd d Z e d e e ee eeeffe ee eeeffffd d Ze d e eejffddZeedejdejd ejfddZdS)StopStringCriteriaa This class can be used to stop generation whenever specific string sequences are generated. It preprocesses the strings together with the tokenizer vocab to find positions where tokens can validly complete the stop strings. Generation is stopped as soon as a token is generated that completes any of the stop strings. We want to catch any instance in which the stop string would be present in the decoded output, which means we must also catch cases with "overhangs" off one or both ends. To make this more concrete, for the stop string "stop", any of the following token sequences would trigger the match: - ["st", "op"] - ["stop"] - ["st", "opera"] - ["sto", "pper"] - ["las", "topper"] - ["s", "to", "pped"] Note that a match will only be triggered if the stop string is at the end of the generated sequence. In other words, these sequences will not trigger a match: - ["stop", "at"] - ["st", "op", "at"] - ["st", "opera", "tion"] The reason these are not a match is that the stop string does not overlap with the final token. If you can remove one or more tokens from the end of the sequence without destroying the stop string, then this criterion will not match that stop string. This is by design; because this check is run after each token is generated, we can't miss a valid stop string if one is generated, but we don't want to halt generation just because the stop string exists somewhere in the past input_ids. How is the match actually performed, though? We do it in quite a confusing way, because we want the entire match process to be compilable with Torch or XLA, which means we cannot use standard string methods. However, it is possible, with some work, to do string matching with pure tensor operations. We'll begin by describing the algorithm we use with standard string operations, and then at the end we'll explain how this is converted to pure tensor operations. The key to the algorithm is an observation: Because the stop string must overlap with the end of the token sequence, we can start at the end of the sequence and work backwards. Specifically, we check that there is an overlap between the start of the final token and the end of the stop_string, or to put it another way, stop_string[-i:] == token[:i] for some i > 0. If you look at the positive examples above, you'll see the last token in all of them fulfills this property: - ["st", "op"] (overlap is "op", overlap length == 2) - ["stop"] (overlap is "stop", overlap length == 4) - ["st", "opera"] (overlap is "op", overlap length == 2) - ["sto", "pper"] (overlap is "p", overlap length == 1) - ["las", "topper"] (overlap is "top", overlap length == 3) - ["s", "to", "pped"] (overlap is "p", overlap length == 1) It's impossible to construct a matching sequence that does not have this property (feel free to verify this yourself). However, although this overlap between the start of the final token and the end of the stop string is necessary for a match, it is not sufficient. We also need to check that the rest of the token sequence is consistent with the stop string. How do we do that? Let's use ["s", "to", "pped"] as an example. We know that the final token, "pped", has an overlap of 1 with the stop string, "stop". We then go back to the previous token, "to". Since we have already matched 1 character from the stop string, the remainder to check is "sto". We check that the next token "to" matches the end of the remainder, which it does. We have now matched 3 characters from the stop string, and the remainder to match is "s". We go back to the previous token again, which is also "s". This is a match, and so we have matched the entire stop string. How does it work when the tokens run off the start of the stop string, though? Let's consider the example of ["las", "topper"]. The final token, "topper", has an overlap of 3 with the stop string, "stop". Therefore, the remaining stop string to match is "s". We go back to the previous token, "las". Because the remainder to match is just "s", with length 1, we consider only the final 1 character from the token, which is "s". This matches the stop string, and so the entire string is matched. How do we compute these matches with tensor operations, though? Simply: we efficiently precompute the necessary information for all tokens! For every token, we compute: - Its overlap with the end of the stop string, if any - The positions inside the stop string where the token matches, including matches that run off the start. - The total length of the token For example, for the token "pped", we would compute an end overlap of 1, no internal matching positions, and a length of 4. For the token "to", we would compute no end overlap, a single internal matching position of 1 (counting from the end), and a length of 2. For the token "s", we would compute no end overlap, a single internal matching position of 3 (again counting from the end) and a length of 1. As long as we have this information, we can execute the algorithm above without any string comparison operations. We simply perform the following steps: - Check if the final token has an end-overlap with the start string - Continue backwards, keeping track of how much of the stop string we've matched so far - At each point, check if the next token has the current position as one of its valid positions - Continue until either a match fails, or we completely match the whole stop string Again, consider ["s", "to", "pped"] as an example. "pped" has an end overlap of 1, so we can begin a match. We have matched 1 character so far, so we check that the next token "to", has 1 as a valid position (again, counting from the end). It does, so we add the length of "to" to our position tracker. We have now matched 3 characters, so we check that the next token "s" has 3 as a valid position. It does, so we add its length to the position tracker. The position tracker is now 4, which is the length of the stop string. We have matched the entire stop string. In the second case, ["las", "topper"], "topper" has an end overlap of 3, so we can begin a match. We have matched 3 characters so far, so we check that the next token "las" has 3 as a valid position. It does, because we allow tokens to match positions that run off the start of the stop string. We add its length to the position tracker. The position tracker is now 6, which is greater than the length of the stop string! Don't panic, though - this also counts as a match of the stop string. We have matched the entire stop string. Args: tokenizer (`PreTrainedTokenizer`): The model's associated tokenizer (necessary to extract vocab and tokenize the termination sequences) stop_strings (`Union[str, List[str]]`): A list of strings that should end generation. If a string is passed, it will be treated like a list with a single element. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2") >>> inputs = tokenizer("The biggest states in the USA by land area:", return_tensors="pt") >>> gen_out = model.generate(**inputs) >>> print(tokenizer.batch_decode(gen_out, skip_special_tokens=True)[0]) The biggest states in the USA by land area: - Alaska - Texas - California >>> # Passing one or more stop strings will halt generation after those strings are emitted >>> # Note that generating with stop strings requires you to pass the tokenizer too >>> gen_out = model.generate(**inputs, stop_strings=["Texas"], tokenizer=tokenizer) >>> print(tokenizer.batch_decode(gen_out, skip_special_tokens=True)[0]) The biggest states in the USA by land area: - Alaska - Texas ``` tokenizer stop_stringscCst|tr|g}t||_|}t|t|}}||||\|_|_ |_ t dd|jD|_ t |j|_tjdd|Dtjd|_dS)NcSg|]}t|qSrlen.0 stop_stringrrr z/StopStringCriteria.__init__..cSr>rr?rArrrrDrEr,) isinstancestrtupler= get_vocabkeysvalues!clean_and_embed_tokens_with_cache embedding_vecmax_valid_positionsmax_valid_end_lensmaxmaximum_token_lenr@num_stop_stringsrtensorint32 target_lens)rr<r=vocab token_list token_indicesrrrr(s    zStopStringCriteria.__init__c Cs|||jftvrt|||jf\}}}t|||jfn)||\}}||||j\}}}|||ft|||jf<ttdkrFtjdd|||fS)NF)last)r=STOP_STRING_EMBEDDING_CACHE move_to_endclean_tokenizer_vocab!_stop_string_create_embedding_vecr@popitem) rrXrYr<rNrOrPclean_token_listclean_token_indicesrrrrMs       z4StopStringCriteria.clean_and_embed_tokens_with_cacheabcdefc s}g}g}|ddd}fdd|D}|D]#\}}||g} | | |t|d} || ||qt|t|fS)u? This method turns a tokenizer vocab into a "clean" vocab where each token represents the actual string it will yield, without any special prefixes like "##" or "Ġ". This is trickier than it looks - the method tokenizer.convert_tokens_to_string() does not always return the correct string because of issues with prefix space addition/removal. To work around this, we add a static prefix to the start of the token, then remove it (and any prefix that may have been introduced with it) after calling convert_tokens_to_string(). F)add_special_tokensrcsg|]}|qSr)_convert_id_to_token)rBtokr<rrrD!sz.N)rJitemsconvert_tokens_to_stringindexr@appendrI) r< static_prefixrWrarb sentence_base tokens_basetoken token_idx token_stringrrgrr^s   z(StopStringCriteria.clean_tokenizer_vocabrc Csi}i}|D]y}|ddd}i||<i||<t||D]b\}}|ddd} g} g} tdt|t|D]7} | dkrF| | d} d} n| } || | t| }| |rm| dkrh| tt| t|q6| | q6| rv| |||<| r~| |||<qq||fS)azThis function preprocesses stop strings and the tokenizer vocabulary to determine where tokens can validly appear in the stop strings. For each token, it computes a list of positions in the stop string where the token appears, as well as a list of the possible "end overlaps" for that token - that is, the number of characters from the end of the stop string that overlap with the start of the token, which can have more than one value. The reason for computing these may seem a bit cryptic - please see the docstring for StopStringCriteria for a full explanation of what these values are for!Nr)r)zipranger@ startswithrkmin)rXrYr=token_valid_positionstoken_end_overlapsrCreversed_stop_stringrotok_idxreversed_tokenmatching_positionspossible_end_lengthsirfstoprrr#_stop_string_get_matching_positions)s8     z6StopStringCriteria._stop_string_get_matching_positionsc Csht|||\}}dd|D}|rt|nd}dd|D}|s)tdt|}t|||d} tjt|d| ftjdd} t |D]\\} } || } || }| D]\}}|| ||| || t|f<qZ| D] \}}|| ||t||| |t||| t|f<qst ||D] \}}t|| |df<qqJt j | t jd } | ||fS) aGThis function precomputes everything needed for the run-time checks in StopStringCriteria, and packs them into an embedding tensor that can be accessed with pure tensor operations. For the specifics of the values that are precomputed and what they are used for, please refer to the StopStringCriteria docstring!cS"g|] }|D]}t|qqSrrLr@rB positionsvalrrrrDZ"zHStopStringCriteria._stop_string_create_embedding_vec..rrcSrrrrrrrrD_rzStop string preprocessing was unable to identify tokens matching one or more of the supplied stop string(s). This is most often caused by the stop strings containing unusual characters that are not in the tokenizer vocabulary.r r))r, fill_valuerF)r;rrLrQ ValueErrorr@npr0rU enumeraterhrsrrT)rXrYr=rwrxall_valid_positionsrOvalid_end_lensrPvec_size gather_vecr~rCrend_lensrpvalid_positionspossible_end_lensrorrrr_RsJ  z4StopStringCriteria._stop_string_create_embedding_vecrrcKs|j|j|_|j|j|_|dd|j df}t|d}tj||jddd}|j }t ||j}|ddddd||j f d|j df}|dddd||j df d|j df}|dddddddf} | dd|jd|jdf} tj|| gdd} | jdd} |dk} tj| dddddddf|dddddddddfkdd} tj| | gdd}|jdtjd }|dk}tj| |d d|jdddfk}tj|ddS) N)rrrrr)rQr))dim)axis)rr,)rrr))rNtor+rVrRrflipclampsizerOF embeddingrS unflattenexpandr-catcumsumanyrUamax)rrrr flipped_idsrOembeddedr end_lengthslengthslengths_with_endsr initial_match later_matchmatchmaskstring_matchesrrrrs0      H$zStopStringCriteria.__call__N)rc)rrrrr r rHrr(rM staticmethodr^rrr4rrrTr_rrr r!Tensorrrrrrr;ns 2 (2"r;c@sReZdZdZdeeeeejffddZ e e dej dej dejfdd Zd S) EosTokenCriteriaa) This class can be used to stop generation whenever the "end-of-sequence" token is generated. By default, it uses the `model.generation_config.eos_token_id`. Args: eos_token_id (`Union[int, List[int], torch.Tensor]`): The id(s) of the *end-of-sequence* token. eos_token_idcCs0t|tjst|tr|g}t|}||_dSr')rGrrr4rTr)rrrrrr(s    zEosTokenCriteria.__init__rrrcKs,|j|j|_t|dddf|j}|S)Nr))rrr+r r9rrrrszEosTokenCriteria.__call__N)rrrrr r4rrrr(rrr r!r"rrrrrrs  "rc@s4eZdZdZddZdejdejdejfddZ d S) ConfidenceCriteriaa This class can be used to stop generation whenever assistant model's confidence in its prediction for the current token is lower than the threshold `model.generation_config.assistant_confidence_threshold` even if the number of speculative tokens (defined by `num_assistant_tokens`) is not yet reached. Args: assistant_confidence_threshold (`float`): The value of the threshold. cCs ||_dSr')assistant_confidence_threshold)rrrrrr(s zConfidenceCriteria.__init__rrrcKs4|dd}|d|df}||jkrdSdS)Nr)r)rr)TF)softmaxitemr)rrrrprobsprrrrs  zConfidenceCriteria.__call__N) rrrrr(rr r!r"rrrrrrs rc@sFeZdZeedejdejdejfddZ e de e fddZ dS) StoppingCriteriaListrrrcKsBtj|jdfd|jtjd}|D] }||||fi|B}q|S)NrFr*)rr0r-r+r1)rrrrr3criteriarrrrszStoppingCriteriaList.__call__cCs"|D] }t|tr|jSqdSr')rGr$r%)rstopping_criteriumrrrr%s   zStoppingCriteriaList.max_lengthN)rrrrrrr r!r"rpropertyrr4r%rrrrrs rstopping_criteriar%rcCsJ|j}t|}|dur||krtdt|S|dur#|t|d|S)NzOYou set different `max_length` for stopping criteria and `max_length` parameter)r%)r%rwarningswarn UserWarningrkr$)rr%stopping_max_lengthnew_stopping_criteriarrrvalidate_stopping_criterias r)*r8rabcr collectionsrcopyrtypingrrrrr numpyrrtorch.nnr r pytorch_utilsr tokenization_utils_baser utilsrr get_loggerrr.r\rrr$r5r;rrlistrr4rrrrrs2        W