o  hdP @sdZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z ddl mZmZmZmZe r=ddlZddlmZe eZgd gd gd d d gd gdgd d ddgd gd gd d d gd gd gd d d dZdddddddddddddddddddddddddZddZ    d-defddZdd Zd!d"Zd#d$Zd%d&Z d'd(Z!d)d*Z"d+d,Z#dS).z;AWQ (Activation aware Weight Quantization) integration fileN)version)ACT2FN)PreTrainedModel)is_auto_awq_availableis_ipex_availableis_torch_availablelogging)AwqBackendPackingMethod AwqConfigAWQLinearVersionExllamaVersion)q_projk_projv_projo_proj) gate_projup_proj down_proj)input_layernormpost_attention_layernormnormF) attentionmlp layernorm use_alibi)w1w3w2g.A)rrrr rope_theta)mistralmixtralllamallavaactc_fc)r$layer_before_act dense_h_to_4hrfc_in gelu_impl) starcoder2RefinedWebModelfalconmptgptjgpt_neox gpt_bigcodebloomc Csddlm}|tvr |S|D]7\}}t|d}t|d}||krBt||rBt|t|d}|j}t|} ||| |j |<t ||} q|S)Nr)ScaledActivationr$r&) awq.modules.actr2AWQ_SCALES_MAPPINGSnamed_childrenhasattrgetattr out_featurestorchones_modulesreplace_quantization_scales) model model_typer2namemoduleact_namelayer_before_act_namer&size scale_like_rFq/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/integrations/awq.pyr<Ms     r<returnc s|durg}|j}tstd|tjkr||jtjkr$ddlm }|}n`|jtj kr3ddl m }|}nQ|jtj kre|jdtjkrJddlm} | }n:|jdtjkr[ddlm} | }n)td |jd|jtjkrtdd lm} | }ntd |jdd lm} | }|D]b\} }durg| t|tjr| |vrt fd d|Ds|j!}|j"}||j#|j$|||j%du|j&j'd|j(| <d}|j(| )dt*t+|,dkrt-||||d\}}.dq||fS)a Public method that recursively replaces the Linear layers of the given model with AWQ quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. During the module replacement, we also infer the backend to use through the `quantization_config` object. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`AwqConfig`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. NzAWQ (either `autoawq` or `llmawq`) is not available. Please install it with `pip install autoawq` or check out the installation guide in https://github.com/mit-han-lab/llm-awqr) WQLinear_GEMM) WQLinear_GEMVr)WQLinear_Exllama)WQLinear_ExllamaV2Unrecognized Exllama version:  WQLinear_IPEXzUnrecognized AWQ version: )WQLinearc3s|] }|dvVqdS).N)join).0keycurrent_key_namerFrG sz*replace_with_awq_linear..)w_bit group_size in_featuresr8biasdevTF)modules_to_not_convertrVquantization_confighas_been_replaced)/backendr ValueErrorr AUTOAWQrr GEMMawq.modules.linear.gemmrIGEMVawq.modules.linear.gemvrJEXLLAMAexllama_configr ONEawq.modules.linear.exllamarKTWOawq.modules.linear.exllamav2rLIPEXawq.modules.linear.gemm_ipexrOawq.quantize.qmodulerPr5append isinstancennLinearanyrZr8bitsrYr[weightdevicer;requires_grad_lenlistchildrenreplace_with_awq_linearpop)r=r]r^rVr_rarI target_clsrJrKrLrOrPr?r@rZr8rErFrUrGr}^sp              r}cCst|tstd|jj|jdur|j}|j|d<|S|jjt vrOt |jj}|jj dd}|j }|j }t |d|}||d<||d<||d<|j|d<|Std ) af Returns the fusing mapping given the quantization config and the model Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`~transformers.quantization_config.AWQConfig`): The quantization configuration to use. z:The model should be an instance of `PreTrainedModel`, got N max_seq_lenTdecodernum_key_value_heads hidden_sizenum_attention_headsaFusing mapping not found either on the quantization config or the supported `AWQ_FUSED_MAPPINGS`. Please pass a `fused_mapping` argument in the `quantization_config` or raise an issue on transformers https://github.com/huggingface/transformers to add its support.)rrr TypeError __class____name__modules_to_fusefuse_max_seq_lenconfigr>AWQ_FUSED_MAPPINGSget_text_configrrr7rb)r=r^current_fused_mappingrrrrrFrFrGget_modules_to_fuses(      rc sXt|tr t|}|j}t||}t|dd}|tjkr0ddl m }ddl m }ddl m}ntdg|D]E\}|durNtfdd |DrNq:t|d |||jd krft||d ||ntd t||||} | rddq:tdkr|D]\}tfdd Drt|drt|jdrd|j_q|S)aJ Optionally fuse some modules in the model to speedup inference. Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`Union[AwqConfig, dict]`): The quantization configuration to use. r]Nr)QuantAttentionFused) QuantFusedMLP)FasterTransformerRMSNormz0Fusing is only supported for the AutoAWQ backendc3s|]}|vVqdSNrF)rSmodule_name_to_not_convert)r?rFrGrWsz#fuse_awq_modules..ripexrz7The IPEX version AWQ does not support fuse mlp for now.rQc3s|]}vVqdSrrF)rSfused_attention_parent_module)fused_attention_modules module_namerFrGrW(s r_attn_implementationcustom)rrdictr from_dictrarr7r rcawq.modules.fused.attnrawq.modules.fused.mlprawq.modules.fused.normrrb named_modulesru_fuse_awq_layernormr _fuse_awq_mlploggerinfo_fuse_awq_attention_layersrqsplitrzr6rr) r=r^rarr]rrrr@attention_has_been_fusedrF)rrr?rGfuse_awq_modulessD          rcCsB|D]}t||rt||}||j|j|jj|j|<~qdS)a Fuse the LayerNorm layers into a target class using autoawq Args: fuse_module_names (`List[str]`): The list of module names to fuse module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.FasterTransformerRMSNorm`): The `FasterTransformerRMSNorm` class as it only supports that class for now. N)r6r7rwvariance_epsilontorxr;)fuse_module_namesr@rr old_modulerFrFrGr0s   rcCst|dkrdSt||drXt||d}t||d}t||d}|jj}|jjdd} | j} t| } ||||| } | dd\} }| | }t ||| |~~~dSdS)a Fuse the MLP layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model current_module_name (`str`): The current submodule name fuse_module_names (`List[str]`): The list of module names to fuse. For the MLP layers it has to be an array of length 3 that consists of the 3 MLP layers in the order (gate (dense layer post-attention) / up / down layers) module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.QuantFusedMLP`): The `QuantFusedMLP` class as it only supports that class for now. rNrTrrQ) rzr6r7qweightrxrr hidden_actrrsplit get_submodulesetattrr)r=current_module_namerr@rrrrprevious_devicerr activation_fn new_module parent_name child_nameparentrFrFrGrGs   rc CsHddlm}m}d}t|ddkr|St||ddr"t||dd}t||r1|} d} n/t||r;|} d} n%tr\t t j dt dkr\ddlm } t|| r[| } d} nt d |jj} t||dd} t||dd }t||dd }|jd urtj|j| j|jgdd nd }| |j|j|j|j| j|j|jd utt|j}tj|j| j|jg| d |_tj|j| j|jg| d |_tj|j| j|jg| d |_t||r|j|_||_||d|d|d||| |d|d|ddd }d|_| dd\}}|!|}t"|||#| ~~ ~~d}|S)a Fuse the Attention layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse modules_to_fuse (`List[str]`): The module fusing mapping. The dictionary has to contain a field `attention` with attention module names in the correct order: q, k, v, o layer current_module_name (`str`): The current submodule name target_cls (`~autoawq.QuantAttentionFused`): The `QuantAttentionFused` class as it only supports that class for now. r)rIrJFrrautoawqz0.2.6rNz'Unsupported q_proj type: {type(q_proj)}rN)dimrrrrrrg@)rrTrQ)$awq.modules.linearrIrJrzr6r7rrrrparse importlibmetadatarOrbrrxr[r9catrXrYrZr8nextiter state_dictvaluesqzerosscales split_k_itersgetis_hf_transformersrrrr)r=r@rrrrIrJmodule_has_been_fusedrlinear_target_clscat_dimrOrrrrr[ qkv_layerfused_attention_layerrrrrFrFrGrpsp  "  (     rcCsl|dtjkrddlm}||}|S|dtjkr-ddlm}|||d|dd}|Std|d) z Runs post init for Exllama layers which performs: - Weights unpacking, reordering and repacking - Devices scratch space allocation rr)exllama_post_init)exllamav2_post_init max_input_lenmax_batch_size)rrrM)r rjrkrrlrmrrb)r=rirrrFrFrGpost_init_awq_exllama_moduless   rcCsddlm}||}|S)zl Runs post init for IPEX layers which performs: - Weights packing, reordering and repacking r)ipex_post_init)ror)r=rrFrFrGpost_init_awq_ipex_moduless r)NNNF)$__doc__r packagingr activationsrmodeling_utilsrutilsrrrr utils.quantization_configr r r r r9torch.nnrs get_loggerrrrr4r<boolr}rrrrrrrrFrFrFrGsr        i)@)_