o  h50@s@dZddlZddlZddlZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZddlmZddlmZmZmZmZmZmZmZmZmZmZddlZdd lmZd d l m!Z!d d l"m#Z#d d l$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@d dlAmBZBmCZCd dlDmEZEere;rddlFZFe8rddlGZHe1rddlImZJd=ddZKe:rddlLmMZMddlLmNZOnedddGdddZMeGdddZOe>PeQZReSdZTeSdZUeVZWeeVZXeeSZYeeVeVfZZeeeVeeVfZ[eeeSeeSfZ\eddededfZ]d Z^d!Z_d"Z`d#Zad$Zbecd%ZdGd&d'd'e%ZeGd(d)d)eZfGd*d+d+eZgGd,d-d-eZhGd.d/d/Zid0Zjd1Zkd2Zle)elGd3d4d4eie'Zmd5eeVd6eVfd7d8Zne-emjoem_oemjojduremjojjpd9d:d;d<emjo_dSdS)>a" Base classes common to both the slow and the fast tokenization classes: PreTrainedTokenizerBase (host all the user fronting encoding methods) Special token mixing (host the special tokens logic) and BatchEncoding (wrap the dictionary of output with special method for the Fast tokenizers) N)UserDict)MappingSized)contextmanager) dataclass) isfunction) TYPE_CHECKINGAnyCallableDictList NamedTupleOptionalSequenceTupleUnion)version) __version__)custom_object_save) ExplicitEnumPaddingStrategyPushToHubMixin TensorTypeadd_end_docstringsadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_func download_urlextract_commit_hashget_json_schemais_flax_available is_jax_tensoris_mlx_availableis_numpy_arrayis_offline_modeis_protobuf_available is_remote_urlis_tf_available is_tf_tensoris_tokenizers_availableis_torch_availableis_torch_deviceis_torch_tensorloggingrequires_backends to_py_obj)_compile_jinja_template_render_with_assistant_indices)PROTOBUF_IMPORT_ERRORcCs$tr ddlm}|Stt|)Nr) DecodeError)r'google.protobuf.messager6 ImportErrorr4format) error_messager6r;x/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.pyimport_protobuf_decode_errorQs r=) AddedToken)EncodingFT)frozeneqc@s2eZdZdZ d defddZddZd d ZdS) r>a! AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave. The `normalized` will default to `not special` if it is not specified, similarly to the definition in `tokenizers`. FNcontentcCs<||_||_||_||_||_|dur||_dS| |_dSN)rB single_wordlstriprstripspecial normalized)selfrBrDrErFrGrHr;r;r<__init__is zAddedToken.__init__cC|jSrC__dict__rIr;r;r< __getstate__szAddedToken.__getstate__cCrKrC)rBrNr;r;r<__str__vrPzAddedToken.__str__)FFFFN)__name__ __module__ __qualname____doc__strrJrOrQr;r;r;r<r>_s   r>c@seZdZdZdS) EncodingFastz_This is dummy class because without the `tokenizers` library we don't have these objects anywayN)rRrSrTrUr;r;r;r<rWysrWgꌠ9Y>)Fg@xD np.ndarray torch.Tensorzspecial_tokens_map.jsonzadded_tokens.jsonztokenizer_config.jsonzchat_template.jinjaztokenizer.jsonztokenizer\.(.*)\.jsonc@s eZdZdZdZdZdZdZdS)TruncationStrategyz Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. only_first only_second longest_firstdo_not_truncateN)rRrSrTrU ONLY_FIRST ONLY_SECOND LONGEST_FIRSTDO_NOT_TRUNCATEr;r;r;r<rZs rZc@"eZdZUdZeed<eed<dS)CharSpanz Character span in the original string. Args: start (`int`): Index of the first character in the original string. end (`int`): Index of the character following the last character in the original string. startendNrRrSrTrUint__annotations__r;r;r;r<rd  rdc@rc) TokenSpanz Token span in an encoded string (list of tokens). Args: start (`int`): Index of the first token in the span. end (`int`): Index of the token following the last token in the span. rerfNrgr;r;r;r<rkrjrkc seZdZdZ     dHdeeeefdeee e e fdedee fde dee f fd d Zed ee fd d Zed e fddZdee efd eee ffddZdefddZddZddZddZddZddZed eee fdd ZdId"e d eefd#d$ZdId"e d eee fd%d&ZdId"e d eee fd'd(ZdId"e d eee fd)d*ZdJd+e d,ee d e fd-d.ZdJd+e d,ee d e fd/d0Z !dKd1e d2ee d3e d ee!fd4d5Z"dJd+e d,ee d ee#fd6d7Z$ !dKd8e d9ee d3e d e fd:d;Z% !dKd1e d2ee d3e d e#fdd?Z' dLdeeee fde fd@dAZ(ddBdCeedDfdEe d dfdFdGZ)Z*S)M BatchEncodinga Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc). This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space. Args: data (`dict`, *optional*): Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.). encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*): If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information. tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. prepend_batch_axis (`bool`, *optional*, defaults to `False`): Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). Note that this parameter has an effect if the parameter `tensor_type` is set, *otherwise has no effect*. n_sequences (`Optional[int]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. NFdataencoding tensor_typeprepend_batch_axis n_sequencescs\t|t|tr|g}||_|dur"|dur"t|r"|dj}||_|j||ddS)Nrrorp) superrJ isinstancerW _encodingslenrq _n_sequencesconvert_to_tensors)rIrmrnrorprq __class__r;r<rJs   zBatchEncoding.__init__returncCrK)z `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of sentences) )rwrNr;r;r<rqszBatchEncoding.n_sequencescCs |jduS)z `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`] or not. NrurNr;r;r<is_fasts zBatchEncoding.is_fastitemcsTttr jSjdurjSttr&fddjDStd)a If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.). If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`. If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.) with the constraint of slice. Ncsi|] }|j|qSr;)rm).0keyr~rIr;r< z-BatchEncoding.__getitem__..zInvalid key. Only three types of key are available: (1) string, (2) integers for backend Encoding, and (3) slices for data subsetting.)rtrVrmruslicekeysKeyErrorrIr~r;rr< __getitem__s    zBatchEncoding.__getitem__cCs z|j|WStytwrC)rmrAttributeErrorrr;r;r< __getattr__s   zBatchEncoding.__getattr__cCs|j|jdS)N)rm encodingsrmrurNr;r;r<rO szBatchEncoding.__getstate__cCs,d|vr |d|_d|vr|d|_dSdS)Nrmrr)rIstater;r;r< __setstate__#s  zBatchEncoding.__setstate__cC |jSrC)rmrrNr;r;r<r* zBatchEncoding.keyscCrrC)rmvaluesrNr;r;r<r-rzBatchEncoding.valuescCrrC)rmitemsrNr;r;r<r0rzBatchEncoding.itemscCrK)z `Optional[List[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if the input was tokenized through Python (i.e., not a fast) tokenizer. r|rNr;r;r<r7szBatchEncoding.encodingsr batch_indexcC|jstd|j|jS)a Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to integer indices) at a given batch index (only works for the output of a fast tokenizer). Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[str]`: The list of tokens at that index. zgtokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)ru ValueErrortokensrIrr;r;r<r?s   zBatchEncoding.tokenscCr)a Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequences, - `0` for tokens corresponding to words in the first sequence, - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly encoded. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding sequence. zmsequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)rur sequence_idsrr;r;r<rQs  zBatchEncoding.sequence_idscCs$|jstdtdt||S)> Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). zfwords() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).z`BatchEncoding.words()` property is deprecated and should be replaced with the identical, but more self-explanatory `BatchEncoding.word_ids()` property.)rurwarningswarn FutureWarningword_idsrr;r;r<wordsis  zBatchEncoding.wordscCr)rziword_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)rurrrr;r;r<rs   zBatchEncoding.word_idsbatch_or_token_index token_indexcCX|jstd|dur|}nd}|}|dkr|j|}|dkr$|j|}|j||S)a Get the index of the sequence represented by the given token. In the general use case, this method returns `0` for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair Can be called as: - `self.token_to_sequence(token_index)` if batch size is 1 - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zGtoken_to_sequence() is not available when using Python based tokenizersNr)rur _batch_size_seq_lentoken_to_sequencerIrrrr;r;r<rs  zBatchEncoding.token_to_sequencecCr)a  Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as: - `self.token_to_word(token_index)` if batch size is 1 - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zCtoken_to_word() is not available when using Python based tokenizersNr)rurrr token_to_wordrr;r;r<rs  zBatchEncoding.token_to_wordbatch_or_word_index word_indexsequence_indexcCsn|jstd|dur|}nd}|}|dkr|j|}|dkr$|j|}|j|||}|dur5t|SdS)a- Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with: - **start** -- Index of the first token. - **end** -- Index of the token following the last token. Can be called as: - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1 - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence. word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns `None` if no tokens correspond to the word. This can happen especially when the token is a special token that has been used to format the tokenization. For example when we add a class token at the very beginning of the tokenization. zDword_to_tokens() is not available when using Python based tokenizersNr)rurrrword_to_tokensrk)rIrrrrspanr;r;r<rs'  zBatchEncoding.word_to_tokenscCsH|jstd|dur|}nd}|}|j||}|dur"t|SdS)a Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with: - **start** -- Index of the first character in the original string associated to the token. - **end** -- Index of the character following the last character in the original string associated to the token. Can be called as: - `self.token_to_chars(token_index)` if batch size is 1 - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1 Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence. Returns: [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token (e.g. , ) doesn't correspond to any chars in the origin string. zDtoken_to_chars() is not available when using Python based tokenizersNr)rurtoken_to_charsrd)rIrrr span_indicesr;r;r<rszBatchEncoding.token_to_charsbatch_or_char_index char_indexcC6|jstd|dur|}nd}|}|j|||S)a Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch. Can be called as: - `self.char_to_token(char_index)` if batch size is 1 - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int`: Index of the token, or None if the char index refers to a whitespace only token and whitespace is trimmed with `trim_offsets=True`. zCchar_to_token() is not available when using Python based tokenizersNr)rur char_to_tokenrIrrrrr;r;r<r=s!zBatchEncoding.char_to_tokencCs:|jstd|dur|}nd}|}t|j|||S)a/ Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character following the last character in the original string Can be called as: - `self.word_to_chars(word_index)` if batch size is 1 - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1 Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: `CharSpan` or `List[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with: - start: index of the first character associated to the token in the original string - end: index of the character following the last character associated to the token in the original string zCword_to_chars() is not available when using Python based tokenizersNr)rurrd word_to_chars)rIrrrrr;r;r<rgs$zBatchEncoding.word_to_charscCr)a@ Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be called as: - `self.char_to_word(char_index)` if batch size is 1 - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string. char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int` or `List[int]`: Index or indices of the associated encoded token(s). zBchar_to_word() is not available when using Python based tokenizersNr)rur char_to_wordrr;r;r<rszBatchEncoding.char_to_wordc st|dur|St|tst|}|tjkr&tstdddl}|j|j}nX|tjkrAt s2tdddl j}dfdd n=|tj krYt sMtdddl m}|jt}n%|tjkrutsetdddlmjfd d }n dfd d t}|D]5\}}z|r|g}||s|}|||<Wqty} z|d krtd | td|d| d} ~ ww|S)a Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. NzSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.cs4t|trt|dtjrt|S|SNr)rtlistnpndarray from_numpyarraytensor)valuedtype)torchr;r< as_tensors z3BatchEncoding.convert_to_tensors..as_tensorzEUnable to convert output to JAX tensors format, JAX is not installed.zEUnable to convert output to MLX tensors format, MLX is not installed.cs t|jSrC)rtr)obj)mxr;r< is_tensors z3BatchEncoding.convert_to_tensors..is_tensorcsnt|ttfr0t|dtttjfr0dd|D}tt|dkr0|dur0dd|Dtd}tj||dS)NrcSg|]}t|qSr;rvrvalr;r;r< zGBatchEncoding.convert_to_tensors..as_tensor..rcSsg|]}t|qSr;)rasarrayrr;r;r<r)r) rtrtuplerrrvsetobjectr)rr value_lens)rr;r<rs $overflowing_tokenszUnable to create tensor returning overflowing tokens of different lengths. Please see if a fast version of this tokenizer is available to have this feature available.zUnable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`zY` in this case) have excessive nesting (inputs type `list` where type `int` is expected).rC)rtr TENSORFLOWr)r8 tensorflowconstantrPYTORCHr,rJAXr" jax.numpynumpyrr#MLXr$mlx.corecorer%r Exceptionr) rIrorptfrjnprrrer;)rrrr<rxsr          z BatchEncoding.convert_to_tensors) non_blockingdevicez torch.devicercslt|dgddlttststtr)fdd|jD|_|St dtd|S)au Send all values to device by calling `v.to(device, non_blocking=non_blocking)` (PyTorch only). Args: device (`str` or `torch.device`): The device to put the tensors on. non_blocking (`bool`): Whether to perform the copy asynchronously. Returns: [`BatchEncoding`]: The same instance after modification. rrNcs0i|]\}}|t|jr|jdn|qS))rr)rtTensortorkvrrrr;r<r4sz$BatchEncoding.to..z+Attempting to cast a BatchEncoding to type z. This is not supported.) r0rrtrVr-rhrmrloggerwarning)rIrrr;rr<r"s zBatchEncoding.to)NNNFN)rrCrNF)+rRrSrTrUrr rVr rrWrrboolrhrJpropertyrqr}rrrOrrrrr rrrrrrrrkrrdrrrrrxr __classcell__r;r;ryr<rls "'' 5( + -( ,grlc seZdZdZgdZd%ddZdefddZ d&d ee e e e ffdefd d Z d%d e e e e e e e ffdedefddZd%d e e e e e fdedefddZedefddZfddZfddZedee e e e e fffddZedee e e e e e e e ffffddZede e e e ffddZede e fdd Zede efd!d"Zde e fd#d$ZZS)'SpecialTokensMixina A mixin derived by [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] to handle specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access these special tokens in a model-independent manner and allow to set and update the special tokens. Args: bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional tokens, which will be marked as `special`, meaning that they will be skipped when decoding if `skip_special_tokens` is set to `True`. ) bos_token eos_token unk_token sep_token pad_token cls_token mask_tokenadditional_special_tokensFcKsd|_||_t|j|_g|jd<|D]O\}}|durq||jvre|dkrKt|tt fs7Jd|dt dd|DsDJdt |||qt|t t frYt |||qtd|d t|qdS) NrrzValue z is not a list or tuplecs|] }t|ttfVqdSrCrtrVr>rtr;r;r< tz.SpecialTokensMixin.__init__..z2One of the tokens is not a string or an AddedTokenSpecial token - has to be either str or AddedToken but got: )_pad_token_type_idverbosedictfromkeysSPECIAL_TOKENS_ATTRIBUTES_special_tokens_maprrtrrallsetattrrVr> TypeErrortype)rIrkwargsrrr;r;r<rJds&  zSpecialTokensMixin.__init__r{cCstd|j|jddS)z The `sanitize_special_tokens` is now deprecated kept for backward compatibility and will be removed in transformers v5. zAThe `sanitize_special_tokens` will be removed in transformers v5.Tspecial_tokens)r warning_once add_tokensall_special_tokens_extendedrNr;r;r<sanitize_special_tokens}s z*SpecialTokensMixin.sanitize_special_tokensTspecial_tokens_dictc Cs|sdSg}|D]\}}||jvsJd|d|jr*td|d|d|dkrt|ttfr>tdd |DsIJd |d |d g}|D]}t|t r]t |d d d dd}|sgt ||j vrgqM| |qM|r~t |dkr~t||t|n|jd|||7}q t|t t fstd|d |dt|t rt |d d d dd}t|t rt|||||vr| |q |j|dd}|S)a Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Using `add_special_tokens` will ensure your special tokens can be used in several ways: - Special tokens can be skipped when decoding using `skip_special_tokens = True`. - Special tokens are carefully handled by the tokenizer (they are never split), similar to `AddedTokens`. - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (for instance [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be `''`). Args: special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`): Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`): If `True`, the existing list of additional special tokens will be replaced by the list provided in `special_tokens_dict`. Otherwise, `self._special_tokens_map["additional_special_tokens"]` is just extended. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens. Remember, this only affects which tokens are skipped during decoding, not the `added_tokens_encoder` and `added_tokens_decoder`. This means that the previous `additional_special_tokens` are still added tokens, and will not be split by the model. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2") model = GPT2Model.from_pretrained("openai-community/gpt2") special_tokens_dict = {"cls_token": ""} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) assert tokenizer.cls_token == "" ```rzKey z is not a special tokenz Assigning z to the z key of the tokenizerrcsrrCrrr;r;r<rrz8SpecialTokensMixin.add_special_tokens..zTokens z for key z* should all be str or AddedToken instancesFTrFrErHrGzToken z* should be a str or an AddedToken instancer)rrrrinfortrrrrVr>rappendrvrrextendrr )rIr!replace_additional_special_tokens added_tokensrrto_addtokenr;r;r<add_special_tokenssB:"       z%SpecialTokensMixin.add_special_tokens new_tokensr cCs*|sdSt|ttfs|g}|j||dS)a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary and will be isolated before the tokenization algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore not treated in the same way. Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Args: new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`): Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string token to let you personalize its behavior: whether this token should only match against a single word, whether this token should strip all potential whitespaces on the left side, whether this token should strip all potential whitespaces on the right side, etc. special_tokens (`bool`, *optional*, defaults to `False`): Can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or [MASK] are usually not lower-cased for instance). See details for `tokenizers.AddedToken` in HuggingFace tokenizers library. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizerFast.from_pretrained("google-bert/bert-base-uncased") model = BertModel.from_pretrained("google-bert/bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```rr)rtrr _add_tokensrIrr r;r;r<r s )zSpecialTokensMixin.add_tokenscCtrCNotImplementedErrorrr;r;r<rzSpecialTokensMixin._add_tokenscCrK)zH `int`: Id of the padding token type in the vocabulary. )rrNr;r;r<pad_token_type_idsz$SpecialTokensMixin.pad_token_type_idcs|}|dp |d}|r|ds|ddn|dd}jdddurltfdd||fDrl|rO|durM|dkrD|nfd d |D}|}|dkret|ttfse|duretd ||j |<dSt ||dS) N_id_idsrc3|] }|jdvVqdSrNrLrnamerNr;r<r, z1SpecialTokensMixin.__setattr__..rcsg|]}|qSr;)convert_ids_to_tokensrrNr;r<r4rz2SpecialTokensMixin.__setattr__..z%Cannot set a non-string value as the ) endswithrMgetanyr)rtrVr>rrrs __setattr__)rIrrkey_without_idkey_is_special_idryrNr<r-&s$" zSpecialTokensMixin.__setattr__cs|}|dp |d}|r|ds|ddn|dd}jdddurstfdd||fDrsjd}|sc||durPjrNtd|d dS||}|d kr\t|Sd d |DSt|}|durq |SdS|jvrt j j d |t |S)Nr r!r"r#rc3r$r%rLr&rNr;r<rDr(z1SpecialTokensMixin.__getattr__..zUsing z, but it is not set yet.rcSrr;rV)rtokr;r;r<rNrz2SpecialTokensMixin.__getattr__..z has no attribute )r*rMr+r,rrerrorrVgetattrconvert_tokens_to_idsrrzrRrsr)rIrr.r/rrattr_as_tokensryrNr<r>s("     zSpecialTokensMixin.__getattr__cCs*i}|jD] }t||}|r|||<q|S)a  `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Convert potential tokens of `tokenizers.AddedToken` type to string. )rr3rIset_attrattr attr_valuer;r;r<special_tokens_mapXs  z%SpecialTokensMixin.special_tokens_mapcCs*i}|jD] }|j|}|r|||<q|S)a `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. )rrr6r;r;r<special_tokens_map_extendedgs  z.SpecialTokensMixin.special_tokens_map_extendedcspg}t|jD]+}t|ttfrfdd|D}n t|vr&|gng}tt|| |q |S)a `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.), the order has nothing to do with the index of each tokens. If you want to know the correct indices, check `self.added_tokens_encoder`. We can't create an order anymore as the keys are `AddedTokens` and not `Strings`. Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. csg|] }t|vr|qSr;r0rrseenr;r<rrzBSpecialTokensMixin.all_special_tokens_extended..) rr;rrtrrrVupdatemapr)rI all_tokensr tokens_to_addr;r=r<r ws  z.SpecialTokensMixin.all_special_tokens_extendedcCsdd|jD}|S)z `List[str]`: A list of the unique special tokens (`''`, `''`, ..., etc.). Convert tokens of `tokenizers.AddedToken` type to string. cSrr;r0)rsr;r;r<rrz9SpecialTokensMixin.all_special_tokens..)r )rIall_toksr;r;r<all_special_tokenssz%SpecialTokensMixin.all_special_tokenscCs|j}||}|S)zy `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. )rEr4)rIrDall_idsr;r;r<all_special_idss z"SpecialTokensMixin.all_special_idscCsZ|jt||_|D]\}}t|ttfr||j|<qtd|dt |dS)a Adds new special tokens to the "SPECIAL_TOKENS_ATTRIBUTES" list which will be part of "self.special_tokens" and saved as a special token in tokenizer's config. This allows us to dynamically add new model-type specific tokens after initializing the tokenizer. For example: if the model tokenizers is multimodal, we can support special image or audio tokens. rrN) rrrrrtrVr>rrr)rIr rrr;r;r<"_set_model_specific_special_tokenss  z5SpecialTokensMixin._set_model_specific_special_tokensF)T)rRrSrTrUrrJrhr r rVrr>rr rr rrrr-rr:r;r rErGrHrr;r;ryr<r=sF   g (1  $. ra  add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to add special tokens when encoding the sequences. This will use the underlying `PretrainedTokenizerBase.build_inputs_with_special_tokens` function, which defines which tokens are automatically added to the input ids. This is useful if you want to add `bos` or `eos` tokens automatically. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. stride (`int`, *optional*, defaults to 0): If set to a number along with `max_length`, the overflowing tokens returned when `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences. The value of this argument defines the number of overlapping tokens. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. a return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) a Class attributes (overridden by derived classes) - **vocab_files_names** (`Dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_vocab_files_map** (`Dict[str, Dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **model_input_names** (`List[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Args: model_max_length (`int`, *optional*): The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`], this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. truncation_side (`str`, *optional*): The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. chat_template (`str`, *optional*): A Jinja template string that will be used to format lists of chat messages. See https://huggingface.co/docs/transformers/chat_templating for a full description. model_input_names (`List[string]`, *optional*): The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name. bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. Will be associated to `self.bos_token` and `self.bos_token_id`. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. Will be associated to `self.eos_token` and `self.eos_token_id`. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. Will be associated to `self.unk_token` and `self.unk_token_id`. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). Will be associated to `self.sep_token` and `self.sep_token_id`. pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. Will be associated to `self.pad_token` and `self.pad_token_id`. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). Will be associated to `self.cls_token` and `self.cls_token_id`. mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`. additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. Add them here to ensure they are skipped when decoding with `skip_special_tokens` is set to True. If they are not part of the vocabulary, they will be added at the end of the vocabulary. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. split_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the special tokens should be split during the tokenization process. Passing will affect the internal state of the tokenizer. The default behavior is to not split special tokens. This means that if `` is the `bos_token`, then `tokenizer.tokenize("") = ['`]. Otherwise, if `split_special_tokens=True`, then `tokenizer.tokenize("")` will be give `['<','s', '>']`. c,s eZdZUdZiZeeefed<iZeeeeeffed<dZ e eed<gdZ e eed<dZ eed <dZeed <dZfd d Zed efddZed efddZejd efddZejd efddZdefddZed eeeffddZd efddZd efddZd eeeffddZ            dd!ee eeefe e eeeffd"e e eeefd#e e eeefd$e ed%ed&ed'ed(eeeefd)ed*e ed+e eee fd,ed-ed.e eee!fd eee ee ee e ee"ffd/d0Z#dd$e ed"e e ed efd1d2Z$e%ddddd3dd4d5eee&j'fd6e eee&j'fd7ed8ed9e eeefd:ef d;d<Z(e%ddddddd=d>d?Z)e*d@dAZ+e%ddBeee!ffdCdDZ,   ddEeee&j'fdFe edGe edHed e-ef dIdJZ.  ddEeee&j'fdKe-edFe edGe ed e-ef dLdMZ/ddEedGe ed e-efdNdOZ0ddPedQe edRed e efdSdTZ1e2e3dUdV     W  ddPee4e5e6fdXe ee4e5e6fdRed(eeeefd)eeee7dfd*e edYed e ed+e eee fd e efdZd[Z8ddQed efd\d]Z9 dd^d_Z:e2e3e;        W           ddPee4e5e e4e e5dfdXe ee4e5e e4e e5fd`ee4e5e e4e e5dfdae ee4e5e e4e e5fdRed(eeeefd)eeee7dfd*e edYedbedce ed e ed+e eee fdde edee edfedgedhediedjed e"f*dkdlZ<     W           ddPee4e5e e4e e5fdXe ee4e5e e4e e5fdRed(eeeefd)eeee7dfd*e edYedbedce ed e ed+e eee fdde edee edfedgedhediedjedmed e"f(dndoZ=e2e3e;     W           ddPee4e5e6fdXe ee4e5e6fdRed(eeeefd)eeee7dfd*e edYedbedce ed e ed+e eee fdde edee edfedgedhediedjed e"f&dpdqZ>dd ej?e7j@ddWddddddddddd dfdPee4e5e6fdXe ee4e5e6fdRedredse7d*e edYedbedce ed e ed+e eee fdde edee edfedgedhediedjedmed e"f(dtduZAe2e3e;    W           ddvee e4e eBe e5e eCe e6e eDfdRed(eeeefd)eeee7dfd*e edYedbedce ed e ed+e eee fdde edee edfedgedhediedjedmed e"f&dwdxZEd ej?e7j@ddWddddddddddd dfdvee e4e eBe e5e eCe e6e eDfdRedredse7d*e edYedbedce ed e ed+e eee fdde edee edfedgedhediedjedmed e"f&dydzZF      dd{ee"e e"eee6feee e6fe eee6ffd(eeeefd*e edce ed e edee ed+e eee fdjed e"fd|d}ZG dd~e ede e ed e efddZH dd~e ede e ed e efddZIe2e3e;     W          dde ede e edRed(eeeefd)eeee7dfd*e edYedce ed e ed+e eee fdde edee edfedgedhediedjeded e"f&ddZJ  W Wdde ede e ededseee7fdYed e-e ee ee eff ddZKdej?dddfd{eeee6fe"fd*e edredce ed e edee ed eLfddZMde ed efddZN  ddee ee e edddfdede ed e efddZO  ddeee edddfdede ed efddZP  ddeee efdede ed efddZQ dd~e ede e eded e efddZRe*ded efddZSde ed*e edjefddZTddZUddZVeWddZXe%dddZY     dde ede e ed*e ede ed(ed+e ed)ed e"fddZZZ[S)PreTrainedTokenizerBasez Base class for [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`]. Handles shared (mostly boiler plate) methods for those two classes. vocab_files_namespretrained_vocab_files_mapN _auto_class) input_idstoken_type_idsattention_maskmodel_input_namesright padding_sidetruncation_sidec sd|_|D]}t||r!tt||r!t|d|d|jjqt||_ | dd|_ | dd|_ | d| dd}|durF|nt |_| d |j|_|jd vr^td |j| d |j|_|jd vrstd |j| d|j|_| dd|_| dd|_i|_d|_| dd|_t|jttfrdd|jD|_tjdi|| di|_|j|jddS)Nr;z conflicts with the method z in name_or_pathr5processor_classmodel_max_lengthmax_lenrS)rRleftzKPadding side should be selected between 'right' and 'left', current value: rTzNTruncation side should be selected between 'right' and 'left', current value: rQclean_up_tokenization_spacesFsplit_special_tokens chat_templatecSsi|] }|d|dqSr'templater;)rr^r;r;r<rz4PreTrainedTokenizerBase.__init__..extra_special_tokensr) init_inputshasattrcallabler3rrzrRcopydeepcopy init_kwargspoprU_processor_classVERY_LARGE_INTEGERrWrSrrTrQrZr[deprecation_warnings_in_target_context_managerr\rtrrrsrJr`rH)rIrrrWryr;r<rJzs>     z PreTrainedTokenizerBase.__init__r{cC|j|jddS)zW `int`: The maximum length of a sentence that can be fed to the model. FpairrWnum_special_tokens_to_addrNr;r;r<max_len_single_sentencez/PreTrainedTokenizerBase.max_len_single_sentencecCrl)zi `int`: The maximum combined length of a pair of sentences that can be fed to the model. TrmrorNr;r;r<max_len_sentences_pairrrz.PreTrainedTokenizerBase.max_len_sentences_paircCsJ||j|jddkr!|jr!|jddstdd|jd<dStd)NFrmrqzXSetting 'max_len_single_sentence' is now deprecated. This value is automatically set up.TrWrprrjr+rrrrIrr;r;r<rqscCsJ||j|jddkr!|jr!|jddstdd|jd<dStd)NTrmrsFzWSetting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.rtrur;r;r<rssrVcCs ||_dS)z%Sets processor class as an attribute.N)rh)rIrVr;r;r<_set_processor_classs z,PreTrainedTokenizerBase._set_processor_classcCtrCrrNr;r;r<added_tokens_decodersz,PreTrainedTokenizerBase.added_tokens_decodercCspddd|jD}|jjd|jd|jd|jd|jd|j d |j d |j d |j d |d S)Nz cSs$g|]\}}|d|dqS)z: ,)__repr__rr;r;r<rs$z4PreTrainedTokenizerBase.__repr__..z(name_or_path='z', vocab_size=z, model_max_length=z , is_fast=z, padding_side='z', truncation_side='z', special_tokens=z, clean_up_tokenization_spaces=z, added_tokens_decoder={ z } )) joinrxrrzrRrU vocab_sizerWr}rSrTr:rZ)rIadded_tokens_decoder_repr;r;r<rzs*z PreTrainedTokenizerBase.__repr__cCrwrCrrNr;r;r<__len__rPzPreTrainedTokenizerBase.__len__cCrw)a Returns the vocabulary as a dictionary of token to index. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. Returns: `Dict[str, int]`: The vocabulary. rrNr;r;r< get_vocabs z!PreTrainedTokenizerBase.get_vocabFT conversationtools documentsr\add_generation_promptcontinue_final_messagetokenizepadding truncation max_lengthreturn_tensors return_dictreturn_assistant_tokens_masktokenizer_kwargsc) Ks| r|std| r| std|duri}|||}| r)td|s)tdt|}t|tt frIt|dtt fsDt |ddrI|}d}n|g}d }|r\|rVtd | r\td |durg}|D]}t|t rq| |qdt |r}| t|qdtd nd}|dur|D] }t|t std qg}g}i|j|}|D]}t |dr|j}| rtd|||||d|\}}| |n |jd||||d|}|r0|dd}t|tt frt|D] }d|vr|d}nqtd||vrtd||}|||t||kr$|d|t|}n |d|t|}| |q|s=|d}|r||f|| | d | d|}| r| rg} |s\| ra|d}!n|dg}!tt|!D]H}"dgt|!|"}#||"D]2\}$}%||"|$}&||"|%d}'|&durnt|&|'r|'dnt|!|"D]}(d|#|(<qq{| |#ql|s| s| d} | |d<| r|j| d|S|dS|S)a[ Converts a list of dictionaries with `"role"` and `"content"` keys to a list of token ids. This method is intended for use with chat models, and will read the tokenizer's chat_template attribute to determine the format and control tokens to use when converting. Args: conversation (Union[List[Dict[str, str]], List[List[Dict[str, str]]]]): A list of dicts with "role" and "content" keys, representing the chat history so far. tools (`List[Dict]`, *optional*): A list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#automated-function-conversion-for-tool-use) for more information. documents (`List[Dict[str, str]]`, *optional*): A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys. Please see the RAG section of the [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#arguments-for-RAG) for examples of passing documents with chat templates. chat_template (`str`, *optional*): A Jinja template to use for this conversion. It is usually not necessary to pass anything to this argument, as the model's template will be used by default. add_generation_prompt (bool, *optional*): If this is set, a prompt with the token(s) that indicate the start of an assistant message will be appended to the formatted output. This is useful when you want to generate a response from the model. Note that this argument will be passed to the chat template, and so it must be supported in the template for this argument to have any effect. continue_final_message (bool, *optional*): If this is set, the chat will be formatted so that the final message in the chat is open-ended, without any EOS tokens. The model will continue this message rather than starting a new one. This allows you to "prefill" part of the model's response for it. Cannot be used at the same time as `add_generation_prompt`. tokenize (`bool`, defaults to `True`): Whether to tokenize the output. If `False`, the output will be a string. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, defaults to `False`): Whether to truncate sequences at the maximum length. Has no effect if tokenize is `False`. max_length (`int`, *optional*): Maximum length (in tokens) to use for padding or truncation. Has no effect if tokenize is `False`. If not specified, the tokenizer's `max_length` attribute will be used as a default. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Has no effect if tokenize is `False`. Acceptable values are: - `'tf'`: Return TensorFlow `tf.Tensor` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. return_dict (`bool`, defaults to `False`): Whether to return a dictionary with named outputs. Has no effect if tokenize is `False`. tokenizer_kwargs (`Dict[str: Any]`, *optional*): Additional kwargs to pass to the tokenizer. return_assistant_tokens_mask (`bool`, defaults to `False`): Whether to return a mask of the assistant generated tokens. For tokens generated by the assistant, the mask will contain 1. For user and system tokens, the mask will contain 0. This functionality is only available for chat templates that support it via the `{% generation %}` keyword. **kwargs: Additional kwargs to pass to the template renderer. Will be accessible by the chat template. Returns: `Union[List[int], Dict]`: A list of token ids representing the tokenized chat so far, including control tokens. This output is ready to pass to the model, either directly or via methods like `generate()`. If `return_dict` is set, will return a dict of tokenizer outputs instead. zr`return_dict=True` is incompatible with `tokenize=False`, because there is no dict of tokenizer outputs to return.zL`return_assistant_tokens_mask=True` is incompatible with `return_dict=False`Nz\{\%-?\s*generation\s*-?\%\}zareturn_assistant_tokens_mask==True but chat template does not contain `{% generation %}` keyword.rmessagesTFa continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead.zKcontinue_final_message is not compatible with return_assistant_tokens_mask.zTools should either be a JSON schema, or a callable function with type hints and a docstring suitable for auto-conversion to a schema.zADocuments should be a list of dicts with 'title' and 'text' keys!)compiled_templaterrrr)rrrrrBtextz]continue_final_message is set but we could not find any text to continuein the final message!acontinue_final_message is set but the final message does not appear in the chat after applying the chat template! This can happen if the chat template deletes portions of the final message. Please verify the chat template and final message in your chat to ensure they are compatible.)rrrrrrNrassistant_masksror;)rget_chat_templateresearchrr r2rtrrrbrrrr!rr:rr3renderreversedstriprindexrvrErangerrx))rIrrrr\rrrrrrrrrrrr conversations is_batched tool_schemastooldocumentrenderedall_generation_indicestemplate_kwargschat rendered_chatgeneration_indices final_message content_block final_msg_locoutrrNi current_maskassistant_start_charassistant_end_char start_token end_tokentoken_idr;r;r<apply_chat_templatesZ                 $   z+PreTrainedTokenizerBase.apply_chat_templatecCst|jtrA|j}|dur||vr||}|S|dur?|dur)d|vr)|d}|Sd|vr3|d}|Stdt|d|S|durS|jdurO|j}|Std|S)a Retrieve the chat template string used for tokenizing chat messages. This template is used internally by the `apply_chat_template` method and can also be used externally to retrieve the model's chat template for better generation tracking. Args: chat_template (`str`, *optional*): A Jinja template or the name of a template to use for this conversion. It is usually not necessary to pass anything to this argument, as the model's template will be used by default. tools (`List[Dict]`, *optional*): A list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#automated-function-conversion-for-tool-use) for more information. Returns: `str`: The chat template string. Ntool_usedefaultzThis model has multiple chat templates with no default specified! Please either pass a chat template or the name of the template you wish to use to the `chat_template` argument. Available template names are .a.Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating)rtr\rrsortedr)rIr\r template_dictr;r;r<rs4    z)PreTrainedTokenizerBase.get_chat_templatemain) cache_dirforce_downloadlocal_files_onlyrrevisiontrust_remote_codepretrained_model_name_or_pathrrrrrcOs|| dd} | dd} | dd} | dd} | dd}| dd}| d d}| d d}| durDtd t|durBtd | }d |d|jvd}|durU||d<tra|sat dd}t |}i}i}t j |}d}t j |s{t|rt|jdkr|std|jdtd|jdtt|jd}|||<|}nd|r||d<n]tttttd}i|j|}d|vr t}t|t||| | |||| |ddd|d}t||}|durt|dd}t|}d|vrt|d}Wdn 1swY||d<i}|D]M\}}|dur!d||<q||krBt j |r3|||<qt|r@t || d||<qt||||| | ||||| ddd|d ||<t|||}qt!d!d"|"Dr~|s~t#d#|d$|d%|jd&|D]&\}}||vrq|rt d'|qt d'|d(||q|j$|||g|R||||||d)| S)*a Instantiate a [`~tokenization_utils_base.PreTrainedTokenizerBase`] (or a derived class) from a predefined tokenizer. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`] method, e.g., `./my_model_directory/`. - (**Deprecated**, not applicable to all derived classes) A path or url to a single saved vocabulary file (if and only if the tokenizer only requires a single vocabulary file like Bert or XLNet), e.g., `./my_model_directory/vocab.txt`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded predefined tokenizer vocabulary files should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the vocabulary files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). local_files_only (`bool`, *optional*, defaults to `False`): Whether or not to only rely on local files and not to attempt to download any files. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__` method. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__` for more details. Passing `token=True` is required when you want to use a private model. Examples: ```python # We can't instantiate directly the base class *PreTrainedTokenizerBase* so let's show our examples on a derived class: BertTokenizer # Download vocabulary from huggingface.co and cache. tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased") # Download vocabulary from huggingface.co (user-uploaded) and cache. tokenizer = BertTokenizer.from_pretrained("dbmdz/bert-base-german-cased") # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) tokenizer = BertTokenizer.from_pretrained("./test/saved_model/") # If the tokenizer uses a single vocabulary file, you can point directly to this file tokenizer = BertTokenizer.from_pretrained("./test/saved_model/my_vocab.txt") # You can link tokens to special vocabulary when instantiating tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased", unk_token="") # You should be sure '' is in the vocabulary when doing that. # Otherwise use tokenizer.add_special_tokens({'unk_token': ''}) instead) assert tokenizer.unk_token == "" ```resume_downloadNproxiesuse_auth_token subfolder_from_pipeline _from_autoF _commit_hash gguf_filerThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`. tokenizerFast) file_typefrom_auto_classr}using_pipelinez+Offline mode: forcing local_files_only=TrueTrzCalling z.from_pretrained() with the path to a single file or url is not supported for this tokenizer. Use a model identifier or the path to a directory instead.z.from_pretrained() with the path to a single file or url is deprecated and won't be possible anymore in v5. Use a model identifier or the path to a directory instead.r vocab_file)added_tokens_filespecial_tokens_map_filetokenizer_config_filetokenizer_filechat_template_filer) rrrrrrrr user_agent _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsrutf-8rnfast_tokenizer_files)r) rrrrrrrrrrrrrcss|]}|duVqdSrCr;)rfull_file_namer;r;r<rz:PreTrainedTokenizerBase.from_pretrained..zCan't load tokenizer for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'zI' is the correct path to a directory containing all relevant files for a z tokenizer.z loading file z from cache at rrrr _is_localr)%rgr+rrrrrRr&rrrVospathisdirisfiler(rvrKrrADDED_TOKENS_FILESPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILEFULL_TOKENIZER_FILECHAT_TEMPLATE_FILErr openjsonloadget_fast_tokenizer_filerrrrEnvironmentError_from_pretrained)clsrrrrrrrrarrrrr from_pipeliner commit_hashrr vocab_filesinit_configurationis_localsingle_file_idfile_idadditional_files_namesfast_tokenizer_fileresolved_config_filereadertokenizer_configresolved_vocab_files file_pathr;r;r<from_pretrained's [                          z'PreTrainedTokenizerBase.from_pretrainedrc5 OsJ| dd} | dd} |dddu}| s|s=|jdur=| s=|jjt||t|g| R||||dt| }nd}|dd}|durt|dd }t|}Wdn1s_wY|d }|d d|sw|dd|d d }| s|} nd}|}|d d}|durt|}| |d <Wdn1swY|sd|vrt |dt t frd|di|d<t |d||d<d|vrt|d||d<|dur@ddlm}z|j||||| |d}|j}Wntttfyd}Ynw|dur@ddlm}t|dr|j}nd}|D]}|t|vr*|}nq|dur@||d\}}|dur@|}|dur`|jdd|ddkr`td|d|jd| | |dd}|dd}|!D]\}} ||vr| ||<qu|dd}!|dur||d<||d<i}"i}#d |vr|d !D]-\}$}t |t"rt#d i|}t |t#r||"t$|$<||#t|<qtd!|j%d"n:|durZt|ddp}%t|%}&|&!D]^\}'}(|'| vr| |'rqt |(t"r d#|(d$<t#d i|(}(n7|'d%krDt |(t rD|d%gp g})|(D]}t |t"r6d#|d$<t#d i|}||)vr@|)&|q#|)}(|(||'<qWdn 1sUwY|durg}*|j'|@D]%}'||'dur|'d%kr|*d&d'||'D7}*qh|*&t||'qht|dd }+t|+},Wdn 1swY|,!D]\}-}.|-|*v}/t#|-dd|/ |/d(|"|.<|"|.|#t|<q|!durt|!dd}0t|0}0|0d)}1Wdn 1swY|1D]}2|2d*}$t#d i|2|"|$<|"|$|#t|"|$<q|"|d <|j(|dd+}|j'|@D]"}'|#ikrC||'durC|'d%krC|#t||'||'||'<q"z || i|}3Wn;t)y_t*d,YdSt+y~}4zd-t|4vrst*d.WYd}4~4dSd}4~4wtytd/w|"ikrt,t |"d0d1|3j-krt*d2|3S)3N from_slowFrr)rrrrrrrtokenizer_classrar;rr\auto_map AutoTokenizercustom_pipelinesr) AutoConfig)rrrrr)TOKENIZER_MAPPING_NAMES model_typeNNrr5zThe tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'z/'. The class this function is called from is 'z'.rr__slow_tokenizerrUrxzFound a zV in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instanceTrGrcSrr;r0r<r;r;r<rrz.rrid)savezUnable to load tokenizer model from SPM, loading from TikToken will be attempted instead.(Google protobuf error: Tried to load SPM model with non-SPM vocab file).zsentencepiece_processor.cczUnable to load tokenizer model from SPM, loading from TikToken will be attempted instead.(SentencePiece RuntimeError: Tried to load SPM model with non-SPM vocab file).zoUnable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.rrzuSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.).r+slow_tokenizer_classrrdrergrrrreadrtrrrrmodels.auto.configuration_autorrrOSErrorrrmodels.auto.tokenization_autorrbrrrVrRreplacerrr?rrr>rhrzrrconvert_added_tokensr=r RuntimeErrormaxr|)5rrrrrrrrrrrarrrhas_tokenizer_fileslow_tokenizerrtokenizer_config_handlerfconfig_tokenizer_classsaved_init_inputsrchat_template_handlerconfigrrpatternconfig_tokenizer_class_fastrr args_namerrrxadded_tokens_mapidxspecial_tokens_map_handler:rrrr added_tokens_handleadded_tok_encoder str_tokenindexrGtokenizer_file_handlerserialized_tokensrrr;r;r<rs                                                 (z(PreTrainedTokenizerBase._from_pretrainedcCs|SrCr;)rmax_model_lengthinit_max_model_lengthr;r;r<!_eventually_correct_t5_max_length sz9PreTrainedTokenizerBase._eventually_correct_t5_max_lengthrcst|trd|vr|ddkr|dtdi|St|tr5r5|}r.d|d<|S|d|St|ttfrGfdd|DSt|trYfdd|DS|S) N__typer>rGcsg|] }j|dqSradd_type_fieldr)ror+rrr;r<r/ rz@PreTrainedTokenizerBase.convert_added_tokens..cs"i|] \}}|j|dqSr)r,rr.r;r<r1 s"z@PreTrainedTokenizerBase.convert_added_tokens..r;)rtrrgr>rOrrr)rrrr+r;r.r<r! s   z,PreTrainedTokenizerBase.convert_added_tokenssave_directory legacy_formatfilename_prefix push_to_hubcKs|dd}|dur tdt|dddurtd||d<tj|r1t d|ddStj |dd |r[|d d}|d | tjj d }|j|fi|}||} tj||re|d ndt} tj||rt|d ndt} tj||r|d ndt} t|j} t|j}|ddg|D]}t||rt||| |<q| |jd| vr|j| d<| |jd}|jdurt|jt rdd|j!D| d<n>|ddrt"| ddd}|#|jWdn1swYd}t $d| d| vr| dn|j| d<t%|j&dkr+t|j&| d<|j'D] }| |dq0|j(| ddd} i}|j)!D] \}}|*||<qI|| d<|j+j,}|-drpt|d drp|dd!}|| d"<t|d#ddur|j.| d$<t|d%ddur|j/| d&<|j0durt1||| d'd(| vr| d(| d)d| d*dd+| vr| d+t"| ddd}t2j3| d,ddd-d.}|#|Wdn 1swYt $d/| |j(|j4ddd0}t"| ddd}t2j3|d,ddd-d.}|#|Wdn 1swYt $d1| | | f}|r2|| f7}|j5||||d2}|rK|j6||| ||dd3|S)4a Save the full tokenizer state. This method make sure the full tokenizer can then be re-loaded using the [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] class method.. Warning,None This won't save modifications you may have applied to the tokenizer after the instantiation (for instance, modifying `tokenizer.do_lower_case` after creation). Args: save_directory (`str` or `os.PathLike`): The path to a directory where the tokenizer will be saved. legacy_format (`bool`, *optional*): Only applicable for a fast tokenizer. If unset (default), will save the tokenizer in the unified JSON format as well as in legacy format if it exists, i.e. with tokenizer specific vocabulary and a separate added_tokens files. If `False`, will only save the tokenizer in the unified JSON format. This format is incompatible with "slow" tokenizers (not powered by the *tokenizers* library), so the tokenizer will not be able to be loaded in the corresponding "slow" tokenizer. If `True`, will save the tokenizer in legacy format. If the "slow" tokenizer doesn't exits, a value error is raised. filename_prefix (`str`, *optional*): A prefix to add to the names of the files saved by the tokenizer. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. Returns: A tuple of `str`: The files saved. rNrrrzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_idr-r5rWrZr`FcSsg|] \}}||dqS)r]r;rr;r;r<r r_z;PreTrainedTokenizerBase.save_pretrained..r\save_raw_chat_templatewrrzchat template saved in rra)r+rrxrcan_save_slow_tokenizerr#r _auto_maprrhrV)rrUrr device_mapindent sort_keys ensure_ascii ztokenizer config file saved in r*zSpecial tokens file saved in )r/ file_namesr0r1)r4r)7rgrrrr+rrrrrr2makedirssplitsep _create_repo_get_files_timestampsr{rrrrdrerfrrr?rbr3r:r`r\rtrrrwriterrvrarKrrxrOrzrRr*r:rhrMrrdumpsr;_save_pretrained_upload_modified_files)rIr/r0r1r2rrr4r5files_timestampsrrrr target_keysrsaved_raw_chat_templatefrrrrrout_str write_dictrB save_filesr;r;r<save_pretrained4 s +                          z'PreTrainedTokenizerBase.save_pretrainedrBc s|durtdt|}tj||r|dndt}fddjD}|rXt|ddd  }t j |d d dd d }| |t d|Wdn1sSwYj||d} || |fS)a2 Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens. Fast tokenizers can also be saved in a unique JSON file containing {config + vocab + added-tokens} using the specific [`~tokenization_utils_fast.PreTrainedTokenizerFast._save_pretrained`] Fz^Only fast tokenizers (instances of PreTrainedTokenizerFast) can be saved in non legacy format.r6r5cs i|] \}}|jkr||qSr;)r|)rr1r"rNr;r<r  z.r8rrr<Tr=rAzadded tokens file saved in N)r1)rrVrrr{radded_tokens_encoderrrrrIrHrrsave_vocabulary) rIr/rBr0r1r added_vocabrOrPrr;rNr<rJ s"  z(PreTrainedTokenizerBase._save_pretrainedcCr)aO Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token mappings of the tokenizer. Use [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. r)rIr/r1r;r;r<rV sz'PreTrainedTokenizerBase.save_vocabularyrrnrcKr)a Converts a string into a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. pair (`str`, *optional*): A second sequence to be encoded with the first. add_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add the special tokens associated with the corresponding model. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `List[str]`: The list of tokens. r)rIrrnrrr;r;r<r$ sz PreTrainedTokenizerBase.tokenizezI **kwargs: Passed along to the `.tokenize()` method. z Returns: `List[int]`, `torch.Tensor`, `tf.Tensor` or `np.ndarray`: The tokenized ids of the text. r text_pairstridec Ks,|j|f|||||||| d| } | dS)aC Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`. Args: text (`str`, `List[str]` or `List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). )rXrrrrrYrSrrN) encode_plus) rIrrXrrrrrYrSrrencoded_inputsr;r;r<encode8 s&  zPreTrainedTokenizerBase.encodecCrrCr)rIrnr;r;r<rpm rz1PreTrainedTokenizerBase.num_special_tokens_to_addc Ks|dd}|dd}|dur-|dur-|dur-|r+|jdds&tdd|jd<d }|durG|rG|r;td t|durCtj } nEtj } nA|dur|durs|ro|durf|dusa|dusa|dkrftd |durotd tj } nt |ts}t|} n t |tr|} ntj } |dur|dkr|rtd tt |} n%|dur|dur|durt j} nt |t st |} n t |t r|} nt j} |dur| tj kr|jtkr|r|jddstdd|jd<tj } n|j}| t jkr|jtkr|r |jddstdd|jd<t j} n|j}| tj kr)|jdus%|jdkr)td| t jkrQ| tj krQ|durQ|durQ||dkrQtd|d|d| | ||fS)z Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) and behaviors. truncation_strategyr^pad_to_max_lengthFNz#Truncation-not-explicitly-activatedatTruncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.Tr]aThe `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).z`max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.zLThough `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.aThe `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert). If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).zAsking-to-pad-to-max_lengthzAsking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.z Asking-to-truncate-to-max_lengthzAsking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.rzAsking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.zATruncation and padding are both activated but truncation length (z+) is not a multiple of pad_to_multiple_of (z).)rgrjr+rrrrrrLONGEST MAX_LENGTHrt DO_NOT_PADrZrarbrW LARGE_INTEGERr pad_token_idr) rIrrrpad_to_multiple_ofrrold_truncation_strategyold_pad_to_max_lengthpadding_strategyr]r;r;r<"_get_padding_truncation_strategiesp s                   $     z:PreTrainedTokenizerBase._get_padding_truncation_strategies text_targettext_pair_targetis_split_into_wordsrdreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthrcKsid|d|d|d|d| d| d| d| d | d |d |d |d |d|d|d|d|jd|}|||durK|durKtd|dura|jsV||jd||d|}|durt||jd||d|}||dur~|S|dur|S|d|d<|S)a  Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: text (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). rrrrrYrkrdrSrrlrmrnrorprqr[rNz3You need to specify either `text` or `text_target`.)rrXrNlabelsr;)rgr[r?rrk_switch_to_input_mode _call_one_switch_to_target_mode)rIrrXrirjrrrrrYrkrdrSrrlrmrnrorprqrr all_kwargsrtarget_encodingsr;r;r<__call__ sh2       z PreTrainedTokenizerBase.__call__r[cKsdd}||s td|dur||std|r-t|ttfo+|o+t|dttf}nt|ttf}|rt|tr?td|durZt|t|krZtdt|dt|d|durett||n|}|jdid |d |d |d |d |d|d|d| d| d| d| d| d|d|d|d|d|d||S|j did|d|d |d |d |d |d|d|d| d| d| d| d| d|d|d|d|d|d||S)NcSszt|trdSt|ttfr;t|dkrdSt|dtrdSt|dttfr9t|ddkp8t|ddtSdSdS)NTrF)rtrVrrrv)rr;r;r<_is_valid_text_inputn s  "z?PreTrainedTokenizerBase._call_one.._is_valid_text_inputztext input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: rbatch_text_or_text_pairsrrrrrYrkrdrSrrlrmrnrorprqrr[rrXr;) rrtrrrVrrvzipbatch_encode_plusrZ)rIrrXrrrrrYrkrdrSrrlrmrnrorprqrr[rryrrzr;r;r<rtV s&              z!PreTrainedTokenizerBase._call_onecKs|jd|||| |d|\}}}}|jdid|d|d|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d|d|d|j|S)ax Tokenize and prepare for the model a sequence or a pair of sequences. This method is deprecated, `__call__` should be used instead. Args: text (`str`, `List[str]` or (for non-fast tokenizers) `List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). rrrrdrrrXrrgr]rrYrkrdrSrrlrmrnrorprqrr[Nr;)rh _encode_plusrgr[)rIrrXrrrrrYrkrdrSrrlrmrnrorprqrrrgr]r;r;r<rZ sb,      z#PreTrainedTokenizerBase.encode_plusrgr]cKrrCr)rIrrXrrgr]rrYrkrdrSrrlrmrnrorprqrr[rr;r;r<r~ sz$PreTrainedTokenizerBase._encode_plusrzcKs|jd|||||d|\}}}}|jdid|d|d|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d||S)a Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. This method is deprecated, `__call__` should be used instead. Args: batch_text_or_text_pairs (`List[str]`, `List[Tuple[str, str]]`, `List[List[str]]`, `List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also `List[List[int]]`, `List[Tuple[List[int], List[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in `encode_plus`). r}rzrrgr]rrYrkrdrSrrlrmrnrorprqrr[Nr;)rh_batch_encode_plus)rIrzrrrrrYrkrdrSrrlrmrnrorprqrr[rrgr]r;r;r<r|1 s^/      z)PreTrainedTokenizerBase.batch_encode_pluscKrrCr)rIrzrrgr]rrYrkrdrSrrlrmrnrorprqrr[rr;r;r<r sz*PreTrainedTokenizerBase._batch_encode_plusr[c  s|jjdr|jddstd|jjdd|jd<ttt fr9tdt r9fdd d D|j dvrQt d |j dd t |j d} | d usgt| trot| dkro|rmgd <S| d} t| tt fr| D]} t| dkr| d} nq|t| ttt fst| r|d urdn|}n)t| r|d urdn|}nt| tjr|d urdn|}n t d| dt| dD] \} } t| | <q|j|||d\}}}}|j d} | r t| dtt fs |j|||||dt|dSt| tfddDsJd|tjkr1tdd| D}tj }i}t!D]3fdd D}|j||||||d}|D]\} } | |vr`g|| <|| "| qSq7t||dS)a Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`). Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. If the `encoded_inputs` passed are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however. Args: encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`): Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str, List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader collate function. Instead of `List[int]` you can have tensors (numpy arrays, PyTorch tensors or TensorFlow tensors), see the note above for the return type. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. rzAsking-to-pad-a-fast-tokenizerFzYou're using a z tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.Trcs i|] fddDqS)csg|]}|qSr;r;)rexamplerr;r<r rz:PreTrainedTokenizerBase.pad...r;)r)r[rr<r rTz/PreTrainedTokenizerBase.pad..zRYou should supply an encoding or a list of encodings to this method that includes z, but you provided NrPrptrztype of z unknown: zA. Should be one of a python, numpy, pytorch or tensorflow object.)rrr)rrgrdrSrmrc3s|] }t|kVqdSrCr)rr) batch_sizer;r<r6 sz.PreTrainedTokenizerBase.pad..zLSome items in the output dictionary have a different batch size than others.css|]}t|VqdSrCr)rinputsr;r;r<r; rcsi|] \}}||qSr;r;r)rr;r<r@ r_)#rzrRr*rjr+rwarning_advicertrrrrrQrrrvrhr*r.rrrrr1rh_padrlrrrr_rr`rr)rIr[rrrdrSrmrrrequired_input first_elementr~rrrg_ batch_outputsroutputsr;)rr[rr<pad sN         zPreTrainedTokenizerBase.pad token_ids_0 token_ids_1cCs2|dur t|dgSdgt|dgt|S)a Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The token type ids. NrrrrIrrr;r;r<$create_token_type_ids_from_sequencesQ sz= 7.5` (Volta). padding_side: The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) NrPrrrRrOrrYzInvalid padding strategy:) rQrr_rvrarSrrcr) rIr[rrgrdrSrmrneeds_to_be_padded differencer;r;r<rsH$      zPreTrainedTokenizerBase._padrcCr)aT Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time. Args: tokens (`List[str]`): The token to join in a string. Returns: `str`: The joined tokens. r)rIrr;r;r<convert_tokens_to_strings z0PreTrainedTokenizerBase.convert_tokens_to_string sequencesrXrYz tf.Tensorskip_special_tokensrZc sfdd|DS)aj Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. cs$g|]}j|fdqS))rrZ)decode)rseqrZrrIrr;r<rsz8PreTrainedTokenizerBase.batch_decode..r;)rIrrrZrr;rr< batch_decodesz$PreTrainedTokenizerBase.batch_decode token_idscKs t|}|jd|||d|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )rrrZNr;)r1_decoderIrrrZrr;r;r<rszPreTrainedTokenizerBase.decodecKrrCrrr;r;r<r%szPreTrainedTokenizerBase._decodealready_has_special_tokenscs0|r|dus Jd|jfdd|D}|S)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. Na You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument. Or set `return_special_tokens_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer. csg|] }|vr dndqS)rrr;r<rGr;r<rIrzCPreTrainedTokenizerBase.get_special_tokens_mask..r)rIrrrrr;rr<r.s z/PreTrainedTokenizerBase.get_special_tokens_mask out_stringcCsX|ddddddddd d d d d ddddddd}|S)a Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms. Args: out_string (`str`): The text to clean up. Returns: `str`: The cleaned-up string. z .rz ??z !!z ,ryz ' 'z n'tzn'tz 'mz'mz 'sz'sz 'vez'vez 'rez're)r)rr;r;r<clean_up_tokenizationMs  z-PreTrainedTokenizerBase.clean_up_tokenizationcCsn|dur/t||jkr1|r3|jdkr5|jdds(tdt|d|jdd|jd<dSdSdSdSdS) a Depending on the input and internal state we might trigger a warning about a sequence that is too long for its corresponding model Args: ids (`List[str]`): The ids produced by the tokenization max_length (`int`, *optional*): The max_length desired (does not trigger a warning if it is set) verbose (`bool`): Whether or not to print more information and warnings. Nrz4sequence-length-is-longer-than-the-specified-maximumFzcToken indices sequence length is longer than the specified maximum sequence length for this model (z > zI). Running this sequence through the model will result in indexing errorsT)rvrWrjr+rr)rIrrrr;r;r<rfs$ z>PreTrainedTokenizerBase._eventual_warn_about_too_long_sequencecCdS)zs Private method to put the tokenizer in input mode (when it has different modes for input/outputs) Nr;rNr;r;r<rszz-PreTrainedTokenizerBase._switch_to_input_modecCr)zt Private method to put the tokenizer in target mode (when it has different modes for input/outputs) Nr;rNr;r;r<rurz.PreTrainedTokenizerBase._switch_to_target_modeccs2td|d|_dVd|_|dS)z Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels. a`as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your labels by using the argument `text_target` of the regular `__call__` method (either in the same call as your input texts if you use the same keyword arguments, or in a separate call.TNF)rrrurkrsrNr;r;r<as_target_tokenizers z+PreTrainedTokenizerBase.as_target_tokenizerrcCsDt|ts|j}ddlmm}t||st|d||_dS)a Register this class with a given auto class. This should only be used for custom tokenizers as the ones in the library are already mapped with `AutoTokenizer`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoTokenizer"`): The auto class to register this new tokenizer with. rNz is not a valid auto class.) rtrVrRtransformers.models.automodelsautorbrrM)r auto_class auto_moduler;r;r<register_for_auto_classs   z/PreTrainedTokenizerBase.register_for_auto_classlongest src_texts tgt_textsmax_target_lengthc Ksd} t| t|dd|dd|dur|j}||fd||||d|} |dur/| S|dur5|}|||fd||||d|} Wdn1sRwY| d| d <| S) a Prepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts (`List[str]`): List of documents to summarize or source language texts. tgt_texts (`list`, *optional*): List of summaries or target language texts. max_length (`int`, *optional*): Controls the maximum length for encoder inputs (documents to summarize or source language texts) If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. max_target_length (`int`, *optional*): Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set to `None`, this will use the max_length value. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). **kwargs: Additional keyword arguments passed along to `self.__call__`. Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to the encoder. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model. - **labels** -- List of token ids for tgt_texts. The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed. Otherwise, input_ids, attention_mask will be the only keys. a `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular `__call__` method to prepare your inputs and targets. Here is a short example: model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...) If you either need to use different keyword arguments for the source and target texts, you should do two calls like this: model_inputs = tokenizer(src_texts, ...) labels = tokenizer(text_target=tgt_texts, ...) model_inputs["labels"] = labels["input_ids"] See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice. For a more complete example, see the implementation of `prepare_seq2seq_batch`. src_langNtgt_langT)rrrrr)rrrrrrNrr)rrrrgrWr) rIrrrrrrrrformatted_warning model_inputsrrr;r;r<prepare_seq2seq_batchsHG      z-PreTrainedTokenizerBase.prepare_seq2seq_batch) NNNFFTFFNNFFNr)FT)NNFrCr)NTFNNrNNrI)FNNNT)NNNNTFNNrFNNNNNFFFFT)NTFNNrFNNNNNFFFFTF)NTFNNrFNNNNNFFFFT)TFNNrFNNNNNFFFFTF)TNNNNNT)NTFNNrNNNNNFFFFTF)Nrr]r)FN)r)NNNrNT)\rRrSrTrUrKr rVrirLrMrrQr rSrTr rJrrhrqrssetterrvr>rxrzr~rrr rrrr rlrr classmethodrPathLikerr staticmethodr'rrrSrJrVrrENCODE_KWARGS_DOCSTRING TextInputPreTokenizedInput EncodedInputrZr\rprh'ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGrxrtrZrarbr~ TextInputPairPreTokenizedInputPairEncodedInputPairr|rrrrrrrrrrrrrrrrsrurrrrrr;r;ryr<rJgs    4   &       ${6  u}   @  #$      +        ^       v        N             W     (       5                 R % (        rJtokenization_filesr{cCsxi}|D]}t|}|dur|d}|||<qt|}t}tt}|D]}t||kr7||}q)|S|S)z Get the tokenization file to use for this version of transformers. Args: tokenization_files (`List[str]`): The list of available configuration files. Returns: `str`: The tokenization file to use. Nr) _re_tokenizer_filergroupsrrrrparser)rtokenizer_files_map file_namerravailable_versionsrtransformers_versionr;r;r<r,s      rrrztokenizer files)r object_class object_files)r5)qrUrdrrrr collectionsrcollections.abcrr contextlibr dataclassesrinspectrtypingrr r r r r rrrrrr packagingrr5rdynamic_module_utilsrutilsrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1utils.chat_template_utilsr2r3utils.import_utilsr4rrrrrr= tokenizersr>r?rW get_loggerrRrrhrirbrVrrrrrr AudioInputrrrrrcompilerrZrdrkrlrrrINIT_TOKENIZER_DOCSTRINGrJrr2r9r;r;r;r<s    0   x           s<7FY