o  h@s^dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m ZddlmZddlmZddlmZdd lmZmZmZmZd d lmZd d lmZd d lm Z d dl!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1e12e3Z4dZ5dZ6dZ7dZ8dZ9e$d7Z$eeeedZ:e5e8dZ;e0e$Gddde)Zd?Z+d@e,dAe-dBedCedDeedEeef dFdGZ.de,j/e-j0ddHddddddddddddfdIe#e!e1e!e2e!e3e!e4fd=e d@e,dAe-dBeedCedJe dDeedEeedKeed ee d!ee d"e d#e d$e d%e d&e dLe de5f&dMdNZ6dde,j/e-j0ddHddddddddddddfdS)fPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classcs|dd}|dd}|dd}|dd}|dd}|di}|dd|_|r:|dur:|jdur:td |durDt|} nw|durP|sPt|} nk|rWt |} nd|durt |d } | d d } | d } | d} t | | \} }| | t |dkr| |n3|jdur|dur|j|i|}t |} n|s|d d|_|dg|_t |dd} d}ntd| |_|dur| |jd|_|jj}|dur|jjd+i||d|d|d|d|d|d|d|dn|j|jj}|dur<|jjd+i||d|d|d|d|d|d|d|d|d|dtjd+i||j|j_d d!|jDfd"d#t|d$d%d&Dt |j!"d'd#Dfd(d#|j#D7t dkrg}|j$}D].}t%|t&r|j'pt(||vnt(||v}t%|t(rt&||d)}n||_'|)|q|r|*|z0t+,|j-j./}|d|j|jkrt0t1|d*}|j|d<|d+i||j-_.WdSWdSt2yYdSw),Ntokenizer_object__slow_tokenizer gguf_filer$ from_slowFadded_tokens_decoderadd_prefix_spacezCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r%config model_type tokenizertokenizer_configradditional_special_tokensT) from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofcSsh|]}tt|qShashrepr.0tokenr@r@x/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py z3PreTrainedTokenizerFast.__init__..cs$g|]\}}tt|vr|qSr@rA)rEindexrF)added_tokens_decoder_hashr@rG s z4PreTrainedTokenizerFast.__init__..cS|dSNrr@)xr@r@rGz2PreTrainedTokenizerFast.__init__..keycSsg|]}t|qSr@)strrDr@r@rGrLscs g|] }|vr|vr|qSr@r@rD)encoder tokens_to_addr@rGrLs)specialtyper@)3popgetr-r' ValueErrorcopydeepcopy TokenizerFast from_filerrrupdatelenr%r2 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr,sorteditemslistadded_tokens_encoderkeysall_special_tokens_extendedall_special_tokens isinstancerrWrTappend add_tokensjsonloadsbackend_tokenizer pre_tokenizer __getstate__getattrpre_tokenizers_fast Exception)selfargskwargsr(slow_tokenizerr*fast_tokenizer_filer+r,fast_tokenizer gguf_param architecturetokenizer_dictr1additional_kwargs _truncation_paddingtokensspecial_tokensrF is_special pre_tok_state pre_tok_class __class__)rKrUrVrGrlbs                         z PreTrainedTokenizerFast.__init__returncCdS)NTr@rr@r@rGis_fastszPreTrainedTokenizerFast.is_fastcCr)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. Tr@rr@r@rGcan_save_slow_tokenizersz/PreTrainedTokenizerFast.can_save_slow_tokenizercC|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrbget_vocab_sizerr@r@rG vocab_sizesz"PreTrainedTokenizerFast.vocab_sizecCr)NTr)rb get_vocabrr@r@rGrsz!PreTrainedTokenizerFast.get_vocabcCs|SN)rrr@r@rGvocabszPreTrainedTokenizerFast.vocabcC ddt|jdddDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cSi|]\}}|j|qSr@contentrEvkr@r@rG rIz@PreTrainedTokenizerFast.added_tokens_encoder..cSrMrNr@itemr@r@rGrPrQz>PreTrainedTokenizerFast.added_tokens_encoder..rRror,rprr@r@rGrrs z,PreTrainedTokenizerFast.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )rbget_added_tokens_decoderrr@r@rGr,s z,PreTrainedTokenizerFast.added_tokens_decodercCr)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. cSrr@rrr@r@rGrrIz;PreTrainedTokenizerFast.get_added_vocab..cSrMrNr@rr@r@rGrPrQz9PreTrainedTokenizerFast.get_added_vocab..rRrrr@r@rGget_added_vocab s z'PreTrainedTokenizerFast.get_added_vocabcCr)zD Size of the full vocabulary with the added tokens. Trrrr@r@rG__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rbrr@r@rGr{sz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rbdecoderrr@r@rGr szPreTrainedTokenizerFast.decoderFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec Cs|dur d|jv}|durd|jv}|r |jdur |g|j} n|g} tt} | D]>} | d| j|r=| d| j|rG| d| j|rQ| d| j|r[| d| j |rg| dt | jq)| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr>) model_input_names overflowingrrqrwidstype_idsrroffsetsra) rrrrrrrrr encodings encoding_dicter@r@rG_convert_encoding's,  z)PreTrainedTokenizerFast._convert_encodingrcs&t|tr |Sfdd|DS)aX Converts a token string (or a sequence of tokens) in a single integer id (or a Iterable of ids), using the vocabulary. Args: tokens (`str` or `Iterable[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. cg|]}|qSr@)#_convert_token_to_id_with_added_vocrDrr@rGrLdzAPreTrainedTokenizerFast.convert_tokens_to_ids..)rvrTrrrr@rrGconvert_tokens_to_idsVs z-PreTrainedTokenizerFast.convert_tokens_to_idsrFcCs|j|}|dur |jS|Sr)rb token_to_id unk_token_id)rrFrJr@r@rGrfs z;PreTrainedTokenizerFast._convert_token_to_id_with_added_vocrJcCs|jt|Sr)rb id_to_tokenint)rrJr@r@rG_convert_id_to_tokenlsz,PreTrainedTokenizerFast._convert_id_to_token new_tokenscCs|r|j|S|j|Sr)rbadd_special_tokensrx)rrrr@r@rG _add_tokensos  z#PreTrainedTokenizerFast._add_tokenspaircCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rbnum_special_tokens_to_add)rrr@r@rGrus z1PreTrainedTokenizerFast.num_special_tokens_to_addrskip_special_tokenscCsRt|tr |j|Sg}|D]}t|}|r||jvrq||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rvrrbrall_special_idsrw)rrrrrJr@r@rGconvert_ids_to_tokenss  z-PreTrainedTokenizerFast.convert_ids_to_tokenstextrcKs|jd|||d|S)N)r text_pairrr@) encode_plusr)rrrrrr@r@rGtokenizesz PreTrainedTokenizerFast.tokenizepadding_strategyr8r4r7r?r=c s|jj|jj}|tjkrdur|jn&|||j|jd}dur'd} n fdd|D} | |kr=|jjdi||t j krO|durM|j dSdS|t j krV|nd} | |dur_|n|j |j|j|j|d}||krz|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r4r7r9r6csi|] }||dqSrrZ)rErrr@rGrszFPreTrainedTokenizerFast.set_truncation_and_padding..)r>r6pad_idr:r<r?r@)rbrerirDO_NOT_TRUNCATErhvaluer5rfr DO_NOT_PAD no_padding MAX_LENGTHr= pad_token_idr:r;rj) rrr8r4r7r?r=rtargetcurrentr>r@rrGset_truncation_and_paddings>!    z2PreTrainedTokenizerFast.set_truncation_and_paddingrbatch_text_or_text_pairsis_split_into_wordsreturn_tensorsrmcs&t|ttfstdt|dj|||||| djj|kr&|j_jj|||d}fdd|D}i}|dd D]fdd|D}||<qId d|D}r~g}t |D]\}\}}||gt |d 7}qh||d <|d D] } ||qt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr8r4r7r?r=)ris_pretokenizedc s&g|]}j|dqS))rrrrrrrr)r)rEr)rrrrrrrrr@rGrL's z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|] \}}|D]}|q qSr@r@)rEr_rrRr@rGrL=s"cSsg|] \}}|D]}|qqSr@r@)rErrrr@r@rGrL?sroverflow_to_sample_mapping) tensor_type)rvtuplerq TypeErrorrXrrbrn encode_batchrs enumeratera&_eventual_warn_about_too_long_sequencer)rrrrr8r4r7rr?r=rrrrrrrrrmrtokens_and_encodingssanitized_tokensstacksanitized_encodingsritoksrrr@) rSrrrrrrrrrG_batch_encode_plussF     z*PreTrainedTokenizerFast._batch_encode_plusrcKs|r||fgn|g}|j|fid|d|d|d|d|d|d| d| d | d | d | d |d |d|d|d|d||}| durY|sYtdd|D|j}||d|||S)Nrrrr8r4r7r?r=rrrrrrrrrmcSs8i|]\}}|t|dkrt|dtr|dn|qS)r)rarvrq)rErSrr@r@rGrs&z8PreTrainedTokenizerFast._encode_plus..r)rrrprr)rrrrrr8r4r7rr?r=rrrrrrrrrmr batched_inputbatched_outputr@r@rG _encode_plusMs`      z$PreTrainedTokenizerFast._encode_pluscCs$|jjdur |jj|Sd|S)N )r{rdecodejoinrr@r@rGconvert_tokens_to_strings  z0PreTrainedTokenizerFast.convert_tokens_to_string token_idsclean_up_tokenization_spacescKsV|dd|_t|tr|g}|jj||d}|dur|n|j}|r)||}|S|S)Nuse_source_tokenizerF)r)rYrdrvrrbrrclean_up_tokenization)rrrrrr clean_textr@r@rG_decodes  zPreTrainedTokenizerFast._decodesave_directory file_names legacy_formatfilename_prefixc s6t|}jdur|durtd|dus|duo jduo j}|dup(|du}|r}tj||r5|dndt}fddj D}|rot |d d d } t j |d ddd d} | | Wdn1sjwYj||d} || |f}|rtj||r|dndt} j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-cs i|] \}}|jkr||qSr@)r)rEtokrJrr@rGrs z.wzutf-8)r)indent sort_keys ensure_ascii )r)rTr'r[rospathrADDED_TOKENS_FILErrrpopenrydumpswritesave_vocabularyTOKENIZER_FILEr{save) rr r r r save_slow save_fastadded_tokens_file added_vocabfout_str vocab_filesr$r@rrG_save_pretraineds<    z(PreTrainedTokenizerFast._save_pretrainedc s,t|j}|d}|d} d} |dddkr)i|dd<g|dd<nW|ddd kre|dd durd|dd } |dd| d } durU| vrU| } d |dd <| d gg|dd<n|ddd vrti|dd<n td|ddddurd|dvr|ddvr|dd|dd<tt|g} |D]5} | dd}| dd}|ddd kr|sqdur| dvrՈ| d| d<| t d+i| q|dur| ||dddkrd|vr|dddur|dd|d<|dddkr'd|vr'|dddur'|dd|d<|ddd kr9| dur9| |d<|ddurn|dddksg|dddkrnd|dvrnt dd|ddDrnt j|d<t|dd}|d+|| d|}j|||d| dur!t}d| vr| dD]D}| d|d }durfd!d"|D}|| d|d <|D]}|}|durtd#qfd$d"|D| d|d%<qd&D]0}|| vr| |\}}dur|vr|}|}|dur td#||g| |<q| |d<tt||j}tj}|d'|D]A}t||durrt||}durO|vrO|}|j|d}t|t rnt ||j|j|j|jd(d)||<q2|||<q2|j }|dur| |t!|d kr||d'<|j"d+d*i|S),uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrXr rmergesr!unk_idrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrWidrcontinuing_subword_prefixend_of_word_suffixr| ByteLevelSequence pretokenizerscss|] }|ddkVqdS)rXr2Nr@)rE pretokenizerr@r@rG Bs  zBPreTrainedTokenizerFast.train_new_from_iterator..initial_alphabet)rr)r>trainerrrcsg|]}||qSr@rrD)special_tokens_mapr@rGrLTrIzCPreTrainedTokenizerFast.train_new_from_iterator..zQAttempted to set a token in the post processor that does not exist in the mappingcrr@)rrD)r0r@rGrL]rr)clssepr2T) single_wordlstriprstrip normalizedrWr(r@)#ryrzrbto_strrYr[r^from_strrrwrextendanyrr2alphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrrcr\rSPECIAL_TOKENS_ATTRIBUTESremover~_special_tokens_maprZrvr<r=r>r?r2rar)r text_iteratorrr>new_special_tokensr9rtokenizer_jsonr)r*r.r-r added_tokenrWr trainer_classr8trained_tokenizer_jsonrSrrFtoken_id special_tokenspecial_tokens_listspecial_token_fullr2r@)r9r0rGtrain_new_from_iterators"                  "             z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)NF)FN)NN)NNN)?__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesr'r__annotations__rlpropertyboolrrrrdictrTrrrrrr,rrr^r{ DecoderFastr EncodingFastrrrrqrrrrrrrrrrrrrrrrrrrrrrrr rPathLiker(rT __classcell__r@r@rrGr&Qs   |     */  $ P      ^       =    5r&)=rXr\ryr collectionsrcollections.abcrtypingrrrtokenizers.pre_tokenizerspre_tokenizersr tokenizersrr`rr^tokenizers.decodersr r_tokenizers.trainersr r r r rintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utilsrtokenization_utils_baserrrrrrrrrrutilsrrr get_loggerrUloggerrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILErrErYr&r@r@r@rGsB           0