o  hW@sdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZdd lmZmZd d lmZeeZeGd d d eZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZdS)zTPyTorch IdeficsVision model: a copy of CLIPVisionModel using a simpler config objectN) dataclass)OptionalTupleUnion)nn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling) ModelOutputlogging)IdeficsVisionConfigc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)IdeficsVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrrv/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/models/idefics/vision.pyr"s rcs\eZdZdeffdd Zdejdededejfdd Zdd ej d e dejfd dZ Z S)IdeficsVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr position_ids)r ) persistent)super__init__r hidden_size embed_dim image_size patch_sizer Parameterrrandnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferarangeexpandselfr __class__rrr)As"  "z IdeficsVisionEmbeddings.__init__ embeddingsheightwidthreturncCs|jdd}||j}|jdd}||kr||kr|S|dddf}|ddddf}|jd} ||jj} ||jj} | d| d} } t|} |dt| t| | }| dddd}|j t j k} | rvt d|t j}tjj|| | | | fd d d }| r|t j }t| |jd kst| |jdkrtd t| t| fd|jd |jdfd| dddddd| }t j|d|fddS)a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 r Nrr&g?rr$zUpcasting patch_pos_embed to fp32 for interpolation since `upsample_bicubic2d_out_frame` in nn.functional.interpolate is not implemented for 'torch.bfloat16' dtype. This will result in a slight overhead.bicubicF) scale_factormode align_cornerszNumber of patches for images (z/) don't match the shape of position embedding ()dim)shaper7r%rr-mathsqrtreshapeintpermutedtyperbfloat16logger warning_oncetofloatr functional interpolate ValueErrorviewcat unsqueeze)r<r?r@rAr4 pos_embedr5class_pos_embedpatch_pos_embedr+ num_h_patches num_w_patchessqrt_num_positionsfp32_upcastingrrrinterpolate_pos_encodingXsH        $z0IdeficsVisionEmbeddings.interpolate_pos_encodingF pixel_valuesrdc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model (z8). You should try to set `interpolate_pos_encoding=True`)rQr$r r&rI)rKr,rYr3weightrQrUflatten transposer0r:rr[rdr7r%) r<rerd batch_sizer2r@rA target_dtype patch_embeds class_embedsr?rrrforwards( zIdeficsVisionEmbeddings.forwardF) rrrrr)rTensorrOrdrboolrn __classcell__rrr=rr@s$1rcseZdZdZfddZdejdedefddZ dd ejd e ejd e ejde e de eje ejff ddZ Z S)IdeficsVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)r(r)rr*r+num_attention_heads num_headshead_dimrYscaleattention_dropoutdropoutrLineark_projv_projq_projout_projr;r=rrr)s"   zIdeficsVisionAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nr r$)rZrurvri contiguous)r<rrrrrr_shapes zIdeficsVisionAttention._shapeNFrattention_maskcausal_attention_maskoutput_attentionsrBcCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrmt d||j|| fd| |dur||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dur||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }t || }|||j||jfkr$t d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x Channelr&r r$z$Attention weights should be of size z , but is Nz!Attention mask should be of size rI)ptrainingz `attn_output` should be of size )sizer}rwrr{r|rurvrZrbmmrirYrrWsoftmaxryrrNr~)r<rrrrrtgt_lenr+ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputrrrrnsd       zIdeficsVisionAttention.forward)NNF)rrrrr)rrprOrrrqrrnrrrrr=rrss$ rscs2eZdZfddZdejdejfddZZS)IdeficsVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r(r)rr hidden_act activation_fnrrzr*intermediate_sizefc1fc2r;r=rrr)s  zIdeficsVisionMLP.__init__rrBcCs"||}||}||}|Sr)rrr)r<rrrrrns   zIdeficsVisionMLP.forward)rrrr)rrprnrrrrr=rr s rc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) IdeficsVisionEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)eps) r(r)r*r+rs self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r;r=rrr)s   z"IdeficsVisionEncoderLayer.__init__FrrrrrBcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr)rrrr)r<rrrrresidualroutputsrrrrn&s"     z!IdeficsVisionEncoderLayer.forwardro) rrrrr)rrprrqrrrnrrrrr=rrs rcsteZdZdZdeffdd Z     ddeejdeejdee d ee d ee d e e e ff d d Z ZS)IdeficsVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`IdeficsVisionEncoderLayer`]. Args: config: IdeficsVisionConfig rcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr)r).0_rrr \sz1IdeficsVisionEncoder.__init__..F) r(r)rr ModuleListrangenum_hidden_layerslayersgradient_checkpointingr;r=rrr)Ys   zIdeficsVisionEncoder.__init__Nrrroutput_hidden_states return_dictrBc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD]1\} } |r<|| f}|jrM|jrM|| j | |||} n| | |||d} | d} |rb|| df}q1|rj|| f}|sxt dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr)rrr css|] }|dur|VqdSrr)rvrrr sz/IdeficsVisionEncoder.forward..)rrr) rrruse_return_dict enumeraterrr_gradient_checkpointing_func__call__tupler ) r< inputs_embedsrrrrrencoder_statesall_attentionsridx encoder_layer layer_outputsrrrrn_sF&     zIdeficsVisionEncoder.forward)NNNNN)rrrrrr)rrrprqrrr rnrrrrr=rrPs*  rcsneZdZdeffdd Z     ddeejdeedeed eed eed e e e ff d d Z Z S)IdeficsVisionTransformerrcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r(r)rr*rr?rrr pre_layrnormrencoderpost_layernorm)r<rr+r=rrr)s   z!IdeficsVisionTransformer.__init__NFrerrrdrrBc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j||||d}|d}|dddddf} || } |s[|| f|ddSt || |j |j dS)z Returns: Nz You have to specify pixel_values)rd)rrrrrr )r pooler_outputrr) rrrrrYr?rrrr rr) r<rerrrdrrencoder_outputsr pooled_outputrrrrns2   z IdeficsVisionTransformer.forward)NNNFN)rrrrr)rrrrqrrr rnrrrrr=rrs(  r) rrL dataclassesrtypingrrrrtorch.utils.checkpointr activationsrmodeling_outputsr r utilsr r configuration_ideficsr get_loggerrrSrModulerrsrrrrrrrrs(     di3b