o  h$@sndZddlmZmZddlZddlmZddlmZGdddej Z Gdd d ej Z Gd d d ej Z dS) a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) IdeficsConfigcsPeZdZdededededededdffd d Zd ejdejfd d ZZ S)IdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNcst||||f\____jj_t j t jjdd_ tjds1jdnjjd_t fddt|D_t j_dS)ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). T) requires_gradrc s4g|]}ttjjjjtjgqS) nn ModuleListIdeficsPerceiverAttentionrr r qk_layer_norms IdeficsMLPintermediate_dim).0_rselfry/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/models/idefics/perceiver.py Ss z6IdeficsPerceiverResampler.__init__..N)super__init__rr r r perceiver_configqk_layer_norms_perceiverrr Parametertorchrandnlatentshasattr vision_configrrrangeblocks LayerNorm layer_norm)rrrr r r r  __class__rrr1s      z"IdeficsPerceiverResampler.__init__contextcCsJ|j|jddd}|jD]\}}||||}|||}q||S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsrr)r$repeatshaper(r*)rr-r$attnffrrrforward_s  z!IdeficsPerceiverResampler.forward) __name__ __module__ __qualname__rintrr"Tensorr2 __classcell__rrr+rr0s".rc sNeZdZdededededdf fdd Zd ejd ejdejfd d ZZ S) rrr r rr Ncst||||_|_|_||_t|j|_t|j|_ |jr2t|j|_ t|j|_ |jd|_ tj |j|j|jdd|_tj |j|j|jdd|_tj |j|j|jdd|_tj |j|j|dd|_dS)ziPerceiver Cross-Attention Module --> let long-form inputs be `context`, resampled embeddings be `latents`gFbiasN)rrrr r rrr)context_layer_normlatents_layer_norm q_layer_norm k_layer_normqk_scaleLinearq_projk_projv_proj output_proj)rrr r rr+rrrms  z"IdeficsPerceiverAttention.__init__r-r$c s|}|}|jdd\}}|}tj||gdd}tj||gdd}fdd|||fD\}}}jrN |} |}t d|j |}||j dd d } | jdd} t d | |} | d d dS)aF Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`torch.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`torch.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `torch.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. N)dimcs.g|]}||jdjjddqS)r)reshaper/r r transpose)rx batch_sizerrrrs.z5IdeficsPerceiverAttention.forward..z... i d, ... j d -> ... i jT)rGkeepdimz... i j, ... j d -> ... i drrH)r;r<r/rArBr"catrCrr=r>einsumr?amaxdetachsoftmaxrDrJflatten) rr-r$ seq_lengthrqkvscoresstabilized_scoresr0 resampledrrLrr2s       z!IdeficsPerceiverAttention.forward) r3r4r5r6boolrr"r7r2r8rrr+rrls"$rcs@eZdZdeffdd ZdeeejdejfddZ Z S)rrcsXt|jj|_t|j|_tj|j|dd|_t |_ tj||jdd|_ dS)z:Simple MLP block with intermediate_size and embedding sizeFr9N) rrr&rrr)lnr@fcReLUactc_proj)rintermediate_sizerr+rrrs   zIdeficsMLP.__init__ hidden_statesr cCs,||}||}||}||}|S)N)r^r_rarb)rrdrrrr2s    zIdeficsMLP.forward) r3r4r5rrrrr" FloatTensorr2r8rrr+rrs& r) __doc__typingrrr"torch.nnrconfiguration_ideficsrModulerrrrrrrs  <A