o h@s|ddlZddlZddlmZmZmZmZmZddlm Z m Z m Z m Z m Z mZmZddlmZddlmZGdddeZdS) N)IteratorListOptionalUnionTuple) AddedTokenRegex Tokenizerdecoders normalizerspre_tokenizerstrainers)Unigram) BaseTokenizercseZdZdZ   ddeeeeefdede ffdd Z    dd e eeefd e d e deee ee fdeeedeef ddZ     dde eeeeefd e d e deee ee fdeeedeedee fddZedefddZZS)SentencePieceUnigramTokenizerzzSentencePiece Unigram Tokenizer Represents the Unigram algorithm, with the pretokenization used by SentencePiece N▁Tvocab replacementadd_prefix_spacecs|dur tt|}ntt}tttttddg|_|r'dnd}t j ||d|_ t j ||d|_ d||d}t||dS)N {2,} alwaysneverrprepend_schemeSentencePieceUnigram)modelrr)r rr SequenceNmtNFKCReplacer normalizerr Metaspace pre_tokenizerr decodersuper__init__)selfrrr tokenizerr parameters __class__/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/tokenizers/implementations/sentencepiece_unigram.pyr's  z&SentencePieceUnigramTokenizer.__init__@files vocab_size show_progressspecial_tokensinitial_alphabet unk_tokencCsP|durg}|dur g}tj|||||d}t|tr|g}|jj||ddS)a Train the model using the given files Args: files (:obj:`List[str]`): A list of path to the files that we should use for training vocab_size (:obj:`int`): The size of the final vocabulary, including all tokens and alphabet. show_progress (:obj:`bool`): Whether to show progress bars while training. special_tokens (:obj:`List[Union[str, AddedToken]]`, `optional`): A list of special tokens the model should know of. initial_alphabet (:obj:`List[str]`, `optional`): A list of characters to include in the initial alphabet, even if not seen in the training dataset. If the strings contain more than one character, only the first one is kept. unk_token (:obj:`str`, `optional`): The unknown token to be used by the model. Nr1r3r2r4r5)trainer)r UnigramTrainer isinstancestr _tokenizertrain)r(r0r1r2r3r4r5r7r-r-r.r<,s z#SentencePieceUnigramTokenizer.trainiteratorlengthc CsB|durg}|dur g}tj|||||d}|jj|||ddS)a Train the model using the given iterator Args: iterator (:obj:`Union[Iterator[str], Iterator[Iterator[str]]]`): Any iterator over strings or list of strings vocab_size (:obj:`int`): The size of the final vocabulary, including all tokens and alphabet. show_progress (:obj:`bool`): Whether to show progress bars while training. special_tokens (:obj:`List[Union[str, AddedToken]]`, `optional`): A list of special tokens the model should know of. initial_alphabet (:obj:`List[str]`, `optional`): A list of characters to include in the initial alphabet, even if not seen in the training dataset. If the strings contain more than one character, only the first one is kept. unk_token (:obj:`str`, `optional`): The unknown token to be used by the model. length (:obj:`int`, `optional`): The total number of sequences in the iterator. This is used to provide meaningful progress tracking Nr6)r7r>)r r8r;train_from_iterator) r(r=r1r2r3r4r5r>r7r-r-r.r?\s " z1SentencePieceUnigramTokenizer.train_from_iteratorfilenamecCsBzddl}|jdddl}Wn tytdw|}|t|d|j j }dd|j D}|j j }|j j}|j j}|dkrJtdd } d } tt|||} |rjtt|ttd d g| _n tttd d g| _| r{d nd} tj| | d| _tj| | d| _ddi} tt| | }t || | |S)Nr.a\You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.rbcSsg|]}|j|jfqSr-)piecescore).0rCr-r-r. sz:SentencePieceUnigramTokenizer.from_spm..rz]You're trying to run a `Unigram` model but you're file was trained with a different algorithmrTrrrrrrr)!syspathappendsentencepiece_model_pb2 Exception ModelProtoParseFromStringopenreadnormalizer_specprecompiled_charsmappieces trainer_specunk_id model_type byte_fallbackr rr r Precompiledr!rr"r r#r$r r%r__new__rr')r@rGrmrQrrTrUrVrrr)rr*objr-r-r.from_spmsL    z&SentencePieceUnigramTokenizer.from_spm)NrT)r/TNNN)r/TNNNN)__name__ __module__ __qualname____doc__rrrr:floatboolr'rintrr<rr? staticmethodr[ __classcell__r-r-r+r.r sj  3  6r)jsonostypingrrrrr tokenizersrrr r r r r tokenizers.modelsrbase_tokenizerrrr-r-r-r.s$