o h 9@sddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z ddl mZmZejeZejedZee ZeZWdn1sLwYGdddZe eZe e eZGdd d e ZGd d d ZGd d d ZGdddZ efde e!de!fddZ"dS)N)Template)AnyCallableDictList NamedTupleOptionalTuple)Encoding Tokenizerzvisualizer-styles.cssc@s<eZdZUeed<eed<eed<dededefddZdS) AnnotationstartendlabelcCs||_||_||_dSN)r rr)selfr rrro/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/tokenizers/tools/visualizer.py__init__s zAnnotation.__init__N)__name__ __module__ __qualname__int__annotations__strrrrrrr s r c@s&eZdZUeeed<eeed<dS) CharStateKeytoken_ixanno_ixN)rrrrrrrrrrrs rc@sHeZdZUeeed<ddZeddZeddZ de fd d Z d S) CharStatechar_ixcCs||_d|_g|_dSr)rrtokens)rrrrrr's zCharState.__init__cCst|jdkr |jdSdS)Nrlenr rrrrr-szCharState.token_ixcCst|jdkS)zJ BPE tokenizers can output more than one token for a char r!r#rrr is_multitoken1szCharState.is_multitokenreturncCst|j|jdS)N)rr)rrrr#rrr partition_key8szCharState.partition_keyN) rrrrrrrpropertyrr%rr'rrrrr$s   rc@s eZdZdS)AlignedN)rrrrrrrr)?sr)c @seZdZdZejdejdZ  ddede de e e ge ffd d Zgdfd ed ede e d e efddZed ed eeeffddZedeed edefddZed eded ed efddZed ed ed efddZed eded ed eefddZdS)EncodingVisualizera Build an EncodingVisualizer Args: tokenizer (:class:`~tokenizers.Tokenizer`): A tokenizer instance default_to_notebook (:obj:`bool`): Whether to render html output in a notebook by default annotation_converter (:obj:`Callable`, `optional`): An optional (lambda) function that takes an annotation in any format and returns an Annotation object z(.{1})?(unk|oov)(.{1})?)flagsTN tokenizerdefault_to_notebookannotation_convertercCsF|rz ddlm}m}Wn tytdw||_||_||_dS)NrHTMLdisplayzWe couldn't import IPython utils for html display. Are you running in a notebook? You can also pass `default_to_notebook=False` to get back raw HTML )IPython.core.displayr0r1 ImportError Exceptionr,r-annotation_coverter)rr,r-r.r0r1rrrrVs zEncodingVisualizer.__init__text annotationsr&c Cs|j}|dur |}|r!z ddlm}m}Wn ty tdw|jdur.tt|j|}|j |}t |||}|rE|||dS|S)a Build a visualization of the given text Args: text (:obj:`str`): The text to tokenize annotations (:obj:`List[Annotation]`, `optional`): An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function default_to_notebook (:obj:`bool`, `optional`, defaults to `False`): If True, will render the html in a notebook. Otherwise returns an html string. Returns: The HTML string if default_to_notebook is False, otherwise (default) returns None and renders the HTML in the notebook Nrr/zeWe couldn't import IPython utils for html display. Are you running in a notebook?) r-r2r0r1r3r4r5listmapr,encoder*_EncodingVisualizer__make_html) rr6r7r-final_default_to_notebookr0r1encodinghtmlrrr__call__ls$   zEncodingVisualizer.__call__c Cst|dkriSttdd|}t|}td|}|dkr!d}d}d}d}i}t|D]}d |d |d |d ||<||7}q-|S) a Generates a color palette for all the labels in a given set of annotations Args: annotations (:obj:`Annotation`): A list of annotations Returns: :obj:`dict`: A dictionary mapping labels to colors in HSL format rcSs|jSr)r)xrrrsz;EncodingVisualizer.calculate_label_colors.. @ zhsl(,z%,%)r"setr9rsorted) r7labels num_labelsh_stepslhcolorsrrrrcalculate_label_colorss    z)EncodingVisualizer.calculate_label_colorsconsecutive_chars_listr=cCs.|d}|jdur|j|j}d|dS|d}|j}|jd}|||}g} i} |jdurg| d|jr=| d|jd rH| d n| d tj|j|jdurf| d |j|j| d <n| ddd| d} d} | D]\} }| d| d|d7} q{d| d| d|dS)a Converts a list of "consecutive chars" into a single HTML element. Chars are consecutive if they fall under the same word, token and annotation. The CharState class is a named tuple with a "partition_key" method that makes it easy to compare if two chars are consecutive. Args: consecutive_chars_list (:obj:`List[CharState]`): A list of CharStates that have been grouped together text (:obj:`str`): The original text being processed encoding (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`str`: The HTML span for a set of consecutive chars rNz(r$tokenz multi-tokenz odd-tokenz even-tokenz special-tokenstokz non-tokenzclass=" "z data-z="z) rr rappendr%r*unk_token_regexsearchjoinitems)rSr6r=firststokenlastr r span_text css_classes data_itemscssdatakeyvalrrrconsecutive_chars_to_htmls6             z,EncodingVisualizer.consecutive_chars_to_htmlcCsVt|||}|dg}|dj}g}t|}|dj}|dur8||} | j} || } |d| d| d|ddD][} | j}||kry|tj|||d| g}|dur^|d|dury||} | j} || } |d| d| d|}| |dkr|| q>|tj|||d| g}q>|tj|||dt|} | S)Nrz&r$)r6r=r[) r*%_EncodingVisualizer__make_char_statesrrRrr\rkr'HTMLBody)r6r=r7 char_statescurrent_consecutive_chars prev_anno_ixspanslabel_colors_dict cur_anno_ixannorcolorcsresrrr __make_htmlsb      zEncodingVisualizer.__make_htmlcCs@dgt|}t|D]\}}t|j|jD]}|||<qq |S)a Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`AnnotationList`): A (possibly empty) list of annotations Returns: A list of length len(text) whose entry at index i is None if there is no annotation on character i or k, the index of the annotation that covers index i where k is with respect to the list of annotations N)r" enumerateranger r)r6r7annotation_maprairrr__make_anno_map<s  z"EncodingVisualizer.__make_anno_mapc Cst||}ddtt|D}t|jD]!\}}||}|dur7|\}} t|| D] } || j|q,qt|D] \} } | || _q<|S)a For each character in the original text, we emit a tuple representing it's "state": * which token_ix it corresponds to * which word_ix it corresponds to * which annotation_ix it corresponds to Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`List[Annotation]`): A (possibly empty) list of annotations encoding: (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`List[CharState]`: A list of CharStates, indicating for each char in the text what it's state is cSsg|]}t|qSr)r).0rrrr jsz9EncodingVisualizer.__make_char_states..N) r*"_EncodingVisualizer__make_anno_maprzr"ryr token_to_charsr\r) r6r=r7r{rnrrUoffsetsr rr}rrrrr__make_char_statesQs   z%EncodingVisualizer.__make_char_states)TN)rrr__doc__recompile IGNORECASEr]r boolrrrr rrAnnotationListr? staticmethodrrRrrr rkr;PartialIntListrrlrrrrr*CsN  - CA$r*childrenr&cCsd|}d|d|dS)a[ Generates the full html with css from a list of html spans Args: children (:obj:`List[str]`): A list of strings, assumed to be html elements css_styles (:obj:`str`, `optional`): Optional alternative implementation of the css Returns: :obj:`str`: An HTML string with style markup rZz?
z4
)r_)r css_styles children_textrrrrmws  rm)# itertoolsosrstringrtypingrrrrrrr tokenizersr r pathdirname__file__r_ css_filenameopenfreadrgr rrrrrr)r*rrmrrrrs* $    6