o  h_@s4ddlZddlmZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl mZddlmZddlmZdd lmZmZmZd d lmZmZerYddl Z d d lmZmZeeZeGd ddZ GdddZ!GdddeZ"eGdddeZ#Gdddej$Z%GdddeZ&GdddZ'dS)N) dataclass) lru_cache)AnyDictOptionalTupleUnion)nn)BCELoss)PreTrainedModel) ModelOutputis_torch_availablelogging)PretrainedConfigWatermarkingConfig)#SynthIDTextWatermarkLogitsProcessorWatermarkLogitsProcessorc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZeejed <dS) WatermarkDetectorOutputa Outputs of a watermark detector. Args: num_tokens_scored (np.array of shape (batch_size)): Array containing the number of tokens scored for each element in the batch. num_green_tokens (np.array of shape (batch_size)): Array containing the number of green tokens for each element in the batch. green_fraction (np.array of shape (batch_size)): Array containing the fraction of green tokens for each element in the batch. z_score (np.array of shape (batch_size)): Array containing the z-score for each element in the batch. Z-score here shows how many standard deviations away is the green token count in the input text from the expected green token count for machine-generated text. p_value (np.array of shape (batch_size)): Array containing the p-value for each batch obtained from z-scores. prediction (np.array of shape (batch_size)), *optional*: Array containing boolean predictions whether a text is machine-generated for each element in the batch. confidence (np.array of shape (batch_size)), *optional*: Array containing confidence scores of a text being machine-generated for each element in the batch. Nnum_tokens_scorednum_green_tokensgreen_fractionz_scorep_value prediction confidence)__name__ __module__ __qualname____doc__rrnparray__annotations__rrrrrrr$r$x/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/generation/watermarking.pyr(s rc @seZdZdZ  d!dededeeefde de f d d Z d e j d e fd dZde j fddZdejdejdejfddZd"ddZ  d#de j dede deeejffddZd S)$WatermarkDetectoraS Detector for detection of watermark generated text. The detector needs to be given the exact same settings that were given during text generation to replicate the watermark greenlist generation and so detect the watermark. This includes the correct device that was used during text generation, the correct watermarking arguments and the correct tokenizer vocab size. The code was based on the [original repo](https://github.com/jwkirchenbauer/lm-watermarking/tree/main). See [the paper](https://arxiv.org/abs/2306.04634) for more information. Args: model_config (`PretrainedConfig`): The model config that will be used to get model specific arguments used when generating. device (`str`): The device which was used during watermarked text generation. watermarking_config (Union[`WatermarkingConfig`, `Dict`]): The exact same watermarking config and arguments used when generating text. ignore_repeated_ngrams (`bool`, *optional*, defaults to `False`): Whether to count every unique ngram only once or not. max_cache_size (`int`, *optional*, defaults to 128): The max size to be used for LRU caching of seeding/sampling algorithms called for every token. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, WatermarkDetector, WatermarkingConfig >>> model_id = "openai-community/gpt2" >>> model = AutoModelForCausalLM.from_pretrained(model_id) >>> tok = AutoTokenizer.from_pretrained(model_id) >>> tok.pad_token_id = tok.eos_token_id >>> tok.padding_side = "left" >>> inputs = tok(["This is the beginning of a long story", "Alice and Bob are"], padding=True, return_tensors="pt") >>> input_len = inputs["input_ids"].shape[-1] >>> # first generate text with watermark and without >>> watermarking_config = WatermarkingConfig(bias=2.5, seeding_scheme="selfhash") >>> out_watermarked = model.generate(**inputs, watermarking_config=watermarking_config, do_sample=False, max_length=20) >>> out = model.generate(**inputs, do_sample=False, max_length=20) >>> # now we can instantiate the detector and check the generated text >>> detector = WatermarkDetector(model_config=model.config, device="cpu", watermarking_config=watermarking_config) >>> detection_out_watermarked = detector(out_watermarked, return_dict=True) >>> detection_out = detector(out, return_dict=True) >>> detection_out_watermarked.prediction array([ True, True]) >>> detection_out.prediction array([False, False]) ``` F model_configdevicewatermarking_configignore_repeated_ngramsmax_cache_sizecCsdt|tr |}|js|jn|j|_|d|_||_td|j |d||_ t |d|j |_ dS)Ngreenlist_ratio) vocab_sizer))maxsizer$) isinstancerto_dictis_encoder_decoder bos_token_iddecoder_start_token_idr-r+rr. processorr_get_ngram_score_get_ngram_score_cached)selfr(r)r*r+r,r$r$r%__init__}s  zWatermarkDetector.__init__prefixtargetcCs|j|}||vSN)r5_get_greenlist_ids)r8r:r; greenlist_idsr$r$r%r6s z"WatermarkDetector._get_ngram_score input_idscCs:|j\}}t|jjdk}|jjd|}t|dt||dd}|dd|f}t |}t |} t |jdD]V} t || } i} | D]} |rW| n| dd}| d}|||| | <qQ|jr~t| || <t| | | <qBt| || <tddt| | D| | <qB|| fS)Nselfhashrrcss|] \}}||VqdSr<r$).0freqoutcomer$r$r% s  z=WatermarkDetector._score_ngrams_in_passage..)shapeintr5seeding_scheme context_widthtorcharange unsqueezer!zerosrange collectionsCounterkeysr7r+lensumvalueszip)r8r? batch_size seq_lengthr@nindices ngram_tensorsnum_tokens_scored_batchgreen_token_count_batch batch_idxfrequencies_tablengram_to_watermark_lookup ngram_exampler:r;r$r$r%_score_ngrams_in_passages, (   z*WatermarkDetector._score_ngrams_in_passagegreen_token_counttotal_num_tokensreturncCs4|j}|||}t||d|}||}|S)Nr)r-r!sqrt)r8rbrcexpected_countnumerdenomzr$r$r%_compute_z_scores  z"WatermarkDetector._compute_z_scorerrc Cs>|||}dddt|dtd|dtjS)Nr?r )r!signexppi)r8xlocscalerir$r$r% _compute_pvals 2zWatermarkDetector._compute_pval@ z_threshold return_dictc Cs|d|jkr|ddddf}|jd|jjdkr&td|jjd||\}}|||}||k}|rO||}d|} t|||||||| dS|S)ai Args: input_ids (`torch.LongTensor`): The watermark generated text. It is advised to remove the prompt, which can affect the detection. z_threshold (`Dict`, *optional*, defaults to `3.0`): Changing this threshold will change the sensitivity of the detector. Higher z threshold gives less sensitivity and vice versa for lower z threshold. return_dict (`bool`, *optional*, defaults to `False`): Whether to return `~generation.WatermarkDetectorOutput` or not. If not it will return boolean predictions, ma Return: [`~generation.WatermarkDetectorOutput`] or `np.array`: A [`~generation.WatermarkDetectorOutput`] if `return_dict=True` otherwise a `np.array`. )rrNrrAzEMust have at least `1` token to score after the first min_prefix_len=z' tokens required by the seeding scheme.)rrrrrrr) r3rFr5rI ValueErrorrarjrsr) r8r?rurvrrbrrrrr$r$r%__call__s0   zWatermarkDetector.__call__N)Fr')rr)rtF)rrrr rstrrrrboolrGr9rJ LongTensorr6rar!r"rjrsfloatrrxr$r$r$r%r&Is<8    r&cs8eZdZdZd deedeffdd Zdd ZZ S) BayesianDetectorConfigaZ This is the configuration class to store the configuration of a [`BayesianDetectorModel`]. It is used to instantiate a Bayesian Detector model according to the specified arguments. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: watermarking_depth (`int`, *optional*): The number of tournament layers. base_rate (`float1`, *optional*, defaults to 0.5): Prior probability P(w) that a text is watermarked. Nrkwatermarking_depth base_ratec s.||_||_d|_d|_tjdi|dS)Nr$)r~r model_namer*superr9)r8r~rkwargs __class__r$r%r9s zBayesianDetectorConfig.__init__cCs||_||_dSr<)rr*)r8rr*r$r$r%set_detector_information s z/BayesianDetectorConfig.set_detector_information)Nrk) rrrr rrGr|r9r __classcell__r$r$rr%r}s r}c@s6eZdZUdZdZeejed<dZ eejed<dS)$BayesianWatermarkDetectorModelOutputa\ Base class for outputs of models predicting if the text is watermarked. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. posterior_probabilities (`torch.FloatTensor` of shape `(1,)`): Multiple choice classification loss. Nlossposterior_probabilities) rrrr rrrJ FloatTensorr#rr$r$r$r%rs  rcs\eZdZdZdeffdd ZdejdeejejffddZ dejdejfd d Z Z S) %BayesianDetectorWatermarkedLikelihoodz~Watermarked likelihood model for binary-valued g-values. This takes in g-values and returns p(g_values|watermarked). r~c sTt||_tjddtdd||_tjdtdd|j||_dS)z!Initializes the model parameters.ggMbP?rN) rr9r~rJr Parameterrandnbetadelta)r8r~rr$r%r9(s  $z.BayesianDetectorWatermarkedLikelihood.__init__g_valuesrdcCsttjtj|dd|jdd}tj|dd}|jddddf||jjd|j }t |}d |}||fS) aSComputes the unique token probability distribution given g-values. Args: g_values (`torch.Tensor` of shape `(batch_size, seq_len, watermarking_depth)`): PRF values. Returns: p_one_unique_token and p_two_unique_tokens, both of shape [batch_size, seq_len, watermarking_depth]. p_one_unique_token[i,t,l] gives the probability of there being one unique token in a tournament match on layer l, on timestep t, for batch item i. p_one_unique_token[i,t,l] + p_two_unique_token[i,t,l] = 1. rldim)axisrA)diagonal.N).Nr) rJrepeat_interleaverLr~trilrtypedtypesqueezersigmoid)r8rrplogitsp_two_unique_tokensp_one_unique_tokenr$r$r%_compute_latents/s 0 z6BayesianDetectorWatermarkedLikelihood._compute_latentscCs"||\}}d|d||S)a8Computes the likelihoods P(g_values|watermarked). Args: g_values (`torch.Tensor` of shape `(batch_size, seq_len, watermarking_depth)`): g-values (values 0 or 1) Returns: p(g_values|watermarked) of shape [batch_size, seq_len, watermarking_depth]. rk)r)r8rrrr$r$r%forwardOs z-BayesianDetectorWatermarkedLikelihood.forward) rrrr rGr9rJTensorrrrrr$r$rr%r"s   rc seZdZdZeZdZfddZddZde j de j d e j d e d e j f d d Z   dde j d e j de e j d efddZZS)BayesianDetectorModelag Bayesian classifier for watermark detection. This detector uses Bayes' rule to compute a watermarking score, which is the sigmoid of the log of ratio of the posterior probabilities P(watermarked|g_values) and P(unwatermarked|g_values). Please see the section on BayesianScore in the paper for further details. Paper URL: https://www.nature.com/articles/s41586-024-08025-4 Note that this detector only works with non-distortionary Tournament-based watermarking using the Bernoulli(0.5) g-value distribution. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BayesianDetectorConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. modelcsFt||j|_|j|_t|jd|_tjt |jg|_ dS)N)r~) rr9r~rrlikelihood_model_watermarkedrJr rtensorprior)r8configrr$r%r9}s zBayesianDetectorModel.__init__cCs&t|tjr|jjjddddSdS)zInitialize the weights.gg{Gz?)meanstdN)r0r rweightdatanormal_)r8moduler$r$r% _init_weightss z#BayesianDetectorModel._init_weightslikelihoods_watermarkedlikelihoods_unwatermarkedmaskrrdc Cstj|dd}tj|ddd}ttj|dtdd}ttj|dtdd}||}td||}t|td |} | |} t| S) a7 Compute posterior P(w|g) given likelihoods, mask and prior. Args: likelihoods_watermarked (`torch.Tensor` of shape `(batch, length, depth)`): Likelihoods P(g_values|watermarked) of g-values under watermarked model. likelihoods_unwatermarked (`torch.Tensor` of shape `(batch, length, depth)`): Likelihoods P(g_values|unwatermarked) of g-values under unwatermarked model. mask (`torch.Tensor` of shape `(batch, length)`): A binary array indicating which g-values should be used. g-values with mask value 0 are discarded. prior (`float`): the prior probability P(w) that the text is watermarked. Returns: Posterior probability P(watermarked|g_values), shape [batch]. rArh㈵>wJ?)minmaxgKH9infzi...->ir)rJrLclamplogr|einsumr) r8rrrrlog_likelihoods_watermarkedlog_likelihoods_unwatermarkedlog_oddsrelative_surprisal_likelihoodrelative_surprisal_priorrelative_surprisalr$r$r%_compute_posteriors z(BayesianDetectorModel._compute_posteriorNrFrlabelsc Cs||}dt|}|j||||jd}d} |dur8t} t|jjd} | |} | t|dd|| } |sE| durA|fS|| fSt | |dS)a Computes the watermarked posterior P(watermarked|g_values). Args: g_values (`torch.Tensor` of shape `(batch_size, seq_len, watermarking_depth, ...)`): g-values (with values 0 or 1) mask: A binary array shape [batch_size, seq_len] indicating which g-values should be used. g-values with mask value 0 are discarded. Returns: p(watermarked | g_values), of shape [batch_size]. rk)rrrrNr rr)rr) rrJ ones_likerrr rSrrr) r8rrrloss_batch_weightrvrroutrloss_fctloss_unwweight loss_weightr$r$r%rs"  zBayesianDetectorModel.forward)NrF)rrrr r} config_classbase_model_prefixr9rrJrr|rrrrrr$r$rr%r`s:   .rc@s6eZdZdZdededefddZdej fdd Z d S) SynthIDTextWatermarkDetectora SynthID text watermark detector class. This class has to be initialized with the trained bayesian detector module check script in examples/synthid_text/detector_training.py for example in training/saving/loading this detector module. The folder also showcases example use case of this detector. Parameters: detector_module ([`BayesianDetectorModel`]): Bayesian detector module object initialized with parameters. Check https://github.com/huggingface/transformers-research-projects/tree/main/synthid_text for usage. logits_processor (`SynthIDTextWatermarkLogitsProcessor`): The logits processor used for watermarking. tokenizer (`Any`): The tokenizer used for the model. Examples: ```python >>> from transformers import ( ... AutoTokenizer, BayesianDetectorModel, SynthIDTextWatermarkLogitsProcessor, SynthIDTextWatermarkDetector ... ) >>> # Load the detector. See https://github.com/huggingface/transformers-research-projects/tree/main/synthid_text for training a detector. >>> detector_model = BayesianDetectorModel.from_pretrained("joaogante/dummy_synthid_detector") >>> logits_processor = SynthIDTextWatermarkLogitsProcessor( ... **detector_model.config.watermarking_config, device="cpu" ... ) >>> tokenizer = AutoTokenizer.from_pretrained(detector_model.config.model_name) >>> detector = SynthIDTextWatermarkDetector(detector_model, logits_processor, tokenizer) >>> # Test whether a certain string is watermarked >>> test_input = tokenizer(["This is a test input"], return_tensors="pt") >>> is_watermarked = detector(test_input.input_ids) ``` detector_modulelogits_processor tokenizercCs||_||_||_dSr<)rrr)r8rrrr$r$r%r9s z%SynthIDTextWatermarkDetector.__init__tokenized_outputscCs\|jj||jjddd|jjddf}|jj|d}||}|jj|d}|||S)N)r? eos_token_idr)r?)rcompute_eos_token_maskrr ngram_lencompute_context_repetition_maskcompute_g_valuesr)r8reos_token_maskcontext_repetition_mask combined_maskrr$r$r%rxs z%SynthIDTextWatermarkDetector.__call__N) rrrr rrrr9rJrrxr$r$r$r%rs$  r)(rO dataclassesr functoolsrtypingrrrrrnumpyr!rJr torch.nnr modeling_utilsr utilsr rrconfiguration_utilsrrlogits_processrr get_loggerrloggerrr&r}rModulerrrr$r$r$r%s4       ->