o  hz @sdZddlmZddlmZmZmZerddlZddlmZer0ddl m Z ddl m Z m Z er9dd lmZdd d Zd ed efddZd ded ededefddZGdddejjZ   d!ddZd"ddZdS)#zZHIGGS through FLUTE (Flexible Lookup Table Engine for LUT-quantized LLMs) integration file)sqrt)is_flute_availableis_hadamard_availableis_torch_availableN)nn)prepare_data_transposed) TuneMetaDataqgemm_v2)hadamard_transformc Csnddtdt|jD}|D]}|j|}|d|d|}||}||d|d<qtj||d|S)NcSsg|]}dqSr).0_r r s/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/integrations/higgs.py 'sz pad_to_block..rconstant)rangelenshaper functionalpad) tensordimshad_block_sizevaluepad_dimsdimsizenext_multiple_of_1024deltar r r pad_to_block&s r#pncCs||fdkrtgddgddgddgdd gd d gd d gddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gdd?gd@dAgdBdCgdDdEgdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdfdggdhdigdjdkgdldmgdndogdpdqgdrdsgdtdugdvdwgdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgdd gd d gd d gddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gd<d=gd>d?gd@dAgdBdCgdDdEgdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdfdggdhdigdjdkgdldmgdndogdpdqgdrdsgdtdugdvdwgdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgdĐdgdƐdgdȐdgdʐdgd̐dgdΐdgdАdgdҐdgdԐdgd֐dgdؐdgdڐdgdܐdgdސdgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgS||fdkrtgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgdd gd!d"gd#d$gd%d&gd'd(gd)d*gd+d,gd-d.gd/d0gd1d2gd3d4gd5d6gd7d8gd9d:gd;dgd?d@gdAdBgdCdDgdEdFgdGdHgdIdJgdKdLgdMdNgdOdPgdQdRgdSdTgdUdVgdWdXgdYdZgd[d\gd]d^gd_d`gdadbgdcddgdedfgdgdhgdidjgdkdlgdmdngdodpgdqdrgdsdtgdudvgdwdxgdydzgd{d|gd}d~gddgddgS||fdkrtddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddggS||fdkr5tdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdggS||fdkrZtdgdgdgdgdgdgdgdggS||fdkrstdgdgdgdggStd|d|(N)rgg f?gBg]?g_g\?g9g@g?gg@y?gЂ @g@?g?g`Qcg2g:7gdig c?g`ֿg`ؿg?g+@g?g`(f?g@8ig=g򩣿g g+gfg gLgx?g`*(g?g/տg@/aſgEggg ?gGg{?gh?g_a?g?g?g3@g#?gygѱܿg @g׿g d,gg&gٿg`{X?gYb?g@"gيg3gTg@@gkgg`{ڿg?g>g ?gg`5g ?g?g5?g?gIؿg A?g?gѿg9?g@@gzg j g g@g@N?g}g?gZg`@gcg ҿg@=Dgg?g`gb@g{@gh@gg?g]ƿgK?ggw?g`?g 4?g b?gg`ſgp?g@?g@Q=s?g@տgg@@g1u?g͔ƿg@?g䉫?g`@g_g@g,?g`dg`T?gL?gb?g3?gȃgxg v?g`?g섥gK3g/ۿg`Eg@#?g3?g g c?g1g?g`%?g%?g@w@gI?gkg g@J߿g?gǿgFֿg9?g`g8?g` @gxg@wg x?gg@ҿg͆gvg?gn@g`9ǿgY?g g@?g[m?g F?g@ ?g`Ьg`Ggc?g@gPug@?gg@Hg4g@gwg#?gd?gMg g ?g?g@vPgſgr?gLg ?g@?g:ڿg ?g?g` ?g@_g$Rg Z?g3g`g ?g@Ogew@g LgV?gߩҿgV?g?ggŠ?g`;?g Կg`@g-g?g`gZY?gg ?g8;?g?gaοg@Rh?g`F ?gA?g@?gg`g8?g`>?g?g@pgi?g ?g`пg@@gB?gbg@@gg`\gg)W?g`?g`g`@g 0$ӿg >gi?gcg?gH?g T?g ]g.ܿgyؿg*g`F?g@g`Is?g_;?gE?g g`޿g.g@\?g`[?g #?g ?g?ggN?gjg=?gR޿g耎?g`gLg@g=g@g@?gR@grgg?g`v_?g@T?g?g@?g8?g{}gi?gSg@k?gs?g?g?g _g@3@g?g@ ?g`Xrg[gg?g`?g@?g5g?)r@ggx?g gk?g gl?gy?g6g?gпg`5?gk@g$gz?g 1J@g ?go?g.ghgӿg_?g?g$?gg6@g@T?g-?g g@ggù?g.?g?gfg?g,'?g?g^@g@Lg>ۿg4bg?g Ng޿g ?gnZgtgi~@g Jf?g^Gg`mg`6@g/?g ?gOܿg?gggg^DgFg|g1g`8g@*?gֿg?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~                                 ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~                                    ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @D                 "r-r*r&bits group_size hadamard_sizec Cst|jdks Jdt|d|||j}tjj|ddd}|j}|j}|jdtj d}t |dg|}|jd|} | d| |}tjj|dd} t |d| dddddf}t |dg| |jd| d|}tj |jdd|tjd } td|jdd D] } tjd|| | d |j|dd tj| | | d <q~| | jdd} | t|} t| tj| |||dd |||||||d d \}} } }}|| | |jtjd|dS)Nrz%Only 2D weights are supported for now)axisT)copydtyperrdevicer5r()rF)num_bitsr0 vector_sizer5r7check_correctnessr5)weightscalestablestables2 tune_metadata)rrr-tor7r+linalgnormr5float32r#reshaper emptyuint8rargmaxTrrrepeat_interleaveviewfloat16)r<r/r$r0r1grid grid_norm_2r7r5multr=codesir>r?r@r r rquantize_with_higgssF  >   rRcsReZdZ     ddedededejd ejd ed effd d ZddZZ S) HiggsLinearTNr&r. in_features out_featuresr8r5r7r0r1c st||_||_||_||_||_||dksJ|dvs"Jtjt j ||d|ft j |ddd|_ tjt j |||f||ddd|_ tjt j d|f||ddd|_tjt j d|d|df||ddd|_|rtjt j |||ddd|_n|d dd|_d|_dS) Nr)rr*r()r5r7F) requires_gradrr6bias)super__init__rTrUr8r0r1r Parameterr+rFint16r<r=r>r?rXregister_parameter workspacer@) selfrTrUr8rXr5r7r0r1 __class__r rrZs0  "  zHiggsLinear.__init__c CsRt|dg|j}|jdurtdt||j|j|j|jj t j d|j|j |jdS)Nr2z,Workspace must be set before calling forwardr;)r1) r#r1r^ Exceptionr r<r=r>r?rKr+rDr@)r_xr r rforwards zHiggsLinear.forward)TNNr&r.) __name__ __module__ __qualname__intr+r5r7rZrd __classcell__r r r`rrSs, )rSFc sddlm}|D]}\}}|durg}||t|tjrnd|tfdd|j Dsn|3|j }|j }t |||j du|j|j|jd|j|<d}t||j|_|j|d Wdn1siwYtt|dkrt||||d \} }|d q ||fS) a Public method that recursively replaces the Linear layers of the given model with HIGGS quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`HiggsConfig`): The quantization config object that contains the quantization parameters. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. r)init_empty_weightsN.c3s|]}|VqdSN)endswith)rkeycurrent_key_name_strr r Isz,replace_with_higgs_linear..)rXr8r1r0TF)quantization_configcurrent_key_namehas_been_replacedr2) acceleraterjnamed_childrenappend isinstancerLinearjoinanymodules_to_not_convertrTrUrSrXr/r1r0_modulestype source_clsrequires_grad_rlistchildrenreplace_with_higgs_linearpop) modelrrrsrtrjnamemodulerTrUrr rorr(s@      rc Csti|D]Z\}}|durg}||t|trN|j}|j}tjj |||j du|j j |j j d|j|<|tj||j j |j j dj|j|j_tt|dkr^t||d}|dq |WdS1spwYdS)a Dequantizes the HiggsLinear layers in the given model by replacing them with standard torch.nn.Linear layers. Args: model (torch.nn.Module): The model containing HiggsLinear layers to be dequantized. current_key_name (list, optional): A list to keep track of the current module names during recursion. Defaults to None. Returns: torch.nn.Module: The model with HiggsLinear layers replaced by torch.nn.Linear layers. N)rXr7r5r6r)rsr2)r+no_gradrvrwrxrSrTrUrryrXr=r7r5r}eyerI contiguousr<datarrrdequantize_higgsr)rrsrrrTrUrr r rrhs8      $rr )r*rr&r.)NNFrl)__doc__mathrutilsrrrr+rflute.integrations.higgsr flute.tuner r fast_hadamard_transformr r#rhr-rRModulerSrrr r r rs.       3> @