o
h" @ s d dl Z d dlmZ ddlmZ ddlmZ ddlmZm Z ddl
m Z m Z ddl
mZmZ d d
lmZ e rAd dlZd dlmZ d Zd Zd
ZdZdZdddZG dd dZdS ) N)Image )_get_model_class)
AutoConfig) MODEL_FOR_PRETRAINING_MAPPING
MODEL_MAPPING)PROCESSOR_MAPPING_NAMES
AutoProcessor)TOKENIZER_MAPPING_NAMES
AutoTokenizer )is_torch_availablez[92mz[93mz[0mu ■u ⬚![]()
c
s jdkrdddddf jdkr%ddddddf ttdd D }d}g }tD ]6\ }|krL|sL }d f< |dkrp|ksZ d krp d krd d7 d| | f< d}q:ddurfd d
tD d fd dtD } t t t d
t
t t d}
| d |
d |d d t| d } dur| d7 } | | g } tD ]2\}
}|
|
f dkr| dd
t
t|
ttD q| t
t|
tt qt
tt
t| } | D ] }| dur#|d d d | d d | nd qtD ]Y\ }t| |}|v rDt
| t n|}d fddtD }d}durod fddtD }| | dt d d | d| q,d|S )z
Generates an attention matrix from a given attention mask.
Optionally applies a sliding window mask (e.g., for Gemma2/3) and
marks regions where image tokens occur based on the specified `img_token`.
r N c s s | ] }t t|V qd S N)lenrepr).0word r {/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/utils/attention_visualizer.py 5 s z6generate_attention_matrix_from_mask..r r c s$ g | ] fd dt D qS )c s0 g | ]}d | krk rn ndnd qS )r r r r j)isliding_windowr r
F s 0 zBgenerate_attention_matrix_from_mask...)range)r )nr )r r r F s $ z7generate_attention_matrix_from_mask.. c 3 sX | ]'} d |f rt t t nd |kr t t t n d |f r&tntV qdS )r NYELLOW BLACK_SQUARERESETGREEN WHITE_SQUAREr )maskr r r H s
z: i == j (diagonal) z: token_type_ids zAttention MatrixzSliding Window Maskc S s g | ]
}t | t qS r )r" r$ )r kr r r r ^ s z | c 3 sp | ]3}| v r |f r v r t t t n |kr*t t t n |f r2tntV qd S r r! r )r img_tokenr' wordsr r r n s $
c 3 sd | ]-}| v r v rt t t n |kr$t t t n | r,tntV qd S r r! r )r r+ sliding_window_maskr, r r r z s
z:
)intndimr max enumerater joinr% r# r$ r" appendljustliststrrjustmapzipr )r, r' r+ r token_type_idsmax_word_length
first_img_idxoutputr) row_dummylegendf_stringvertical_headeridxr row word_repr colored_word row_displaysliding_window_rowr )r r+ r' r r r- r, r #generate_attention_matrix_from_mask' sp
,"
$
.
rI c @ s: e Zd ZdefddZd defddZd defdd Zd
S ) AttentionMaskVisualizer
model_namec C s t |}d| _t| drd|_zt|t}W n ty' t|t }Y nw |d u r4t
d| d|| _ G dd d|t j
}|||| _| j|j || _|| _d S )Nr r r( z Model name z- is not supported for attention visualizationc @ s e Zd Zdd ZdS )z7AttentionMaskVisualizer.__init__.._ModelWrapperc S s$ t j| t dd| _|| _d S )Nr )nnModule__init__Linear dummy_moduleconfig)selfrQ rK r r r rN s
z@AttentionMaskVisualizer.__init__.._ModelWrapper.__init__N)__name__
__module__ __qualname__rN r r r r
_ModelWrapper s rV )r from_pretrained image_tokenhasattrget_text_configr r r Exceptionr
ValueError
mapped_clsrL rM modelto torch_dtyperepo_idrQ )rR rK rQ r] rV r r r rN s"
z AttentionMaskVisualizer.__init__r* input_sentencec C s | j ||d d S )N)suffix)visualize_attention_mask)rR rb rc r r r __call__ s z AttentionMaskVisualizer.__call__c C s | j }i }| jjtv rgd}ttj|ddj}t j
| j dd}t |dr)|j
}n
|j|jgd }|r;|d|}||||d d
}|j|jgd | _
|d } d |v r\|d |d < |j|d
d }
n$| jjtv rt
| j } | |}
| |d dd } n
td|jj dd|j_| |jd | | | j jt | j d d d| } dt d| jj d| j d } d}
t!d| t!dd| jj d| j d| j j" #t | d |
t!| t$|
| | j
t%| jd d |d d d }t!| t!| d S ) Nzchttps://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=trueT)streamr( )image_seq_lengthrX r r pt)rc return_tensorsattention_maskr; input_ids)ri z Model type z) does not support attention visualizationeagerr )rj input_tensorcache_positionpast_key_valuesz##z Attention visualization for z | r r. z" Attention visualization for [1m:z[0m z r )r+ r r; r )&r^ rQ
model_typer r openrequestsgetrawr rW ra rY rX tokenizerconvert_ids_to_tokensimage_token_idreplacer
r tokenizer\ _attn_implementationtrain_update_causal_maskr_ dtypetorcharangeshapeboolr r] printrS centerrI getattr)rR rb rc r^ kwargsimg processorrX inputsrj tokensrv top_bottom_border side_borderrA r r r rd sx
z0AttentionMaskVisualizer.visualize_attention_maskN)r* )rS rT rU r7 rN re rd r r r r rJ s rJ )r NN) rs PILr models.auto.auto_factoryr models.auto.configuration_autor models.auto.modeling_autor r models.auto.processing_autor r models.auto.tokenization_autor
r import_utilsr
r torch.nnrL r% r" r$ r# r& rI rJ r r r r s$
c