o  h8@sddlmZmZddlmZmZmZmZmZm Z ddl m Z ddl m Z mZddlmZer4ddlmZe rBdd lZdd lmZneZGd d d eZGd ddeZdeejejgdiZd S))ABCabstractmethod) TYPE_CHECKINGAnyDictListOptionalUnion)is_torch_available)QuantizationConfigMixinQuantizationMethod)get_module_from_name)PreTrainedModelN) ModuleListc @s&eZdZdZdZdZdZdefddZdLd d Z d e e e e fd e e e e ffd dZdLddZdee de d ee fddZdee de d ee fddZdee de d ee fddZdee dee d ee fddZddd e e dffddZd e e eee ffd e e eee fffd!d"Zd#d$d%d&d'e d(e e e fd ef d)d*ZdMd,d-Zd.d/Zd0d1ZdNd2d3ZdNd4d5Zd6d7Z d8d9Z!e"  dOd#d$d:e ee d;e ee fdd?Z%e$d efd@dAZ&e'dBdCZ(e'dDdEZ)e'dPdFdGZ*e$e'dHdIZ+dJdKZ,dS)Q HfQuantizera Abstract class of the HuggingFace quantizer. Supports for now quantizing HF transformers models for inference and/or quantization. This class is used only for transformers.PreTrainedModel.from_pretrained and cannot be easily used outside the scope of that method yet. Attributes quantization_config (`transformers.utils.quantization_config.QuantizationConfigMixin`): The quantization config that defines the quantization parameters of your model that you want to quantize. modules_to_not_convert (`List[str]`, *optional*): The list of module names to not convert when quantizing the model. required_packages (`List[str]`, *optional*): The list of required pip packages to install prior to using the quantizer requires_calibration (`bool`): Whether the quantization method requires to calibrate the model before using it. requires_parameters_quantization (`bool`): Whether the quantization method requires to create a new Parameter. For example, for bitsandbytes, it is required to create a new xxxParameter in order to properly quantize the model. FNquantization_configcKsH||_|dg|_|dd|_|js |jr"td|jddSdS)Nmodules_to_not_convert pre_quantizedTzThe quantization method z does require the model to be pre-quantized. You explicitly passed `pre_quantized=False` meaning your model weights are not quantized. Make sure to pass `pre_quantized=True` while knowing what you are doing.)rpoprrrequires_calibration ValueError quant_method)selfrkwargsrp/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/quantizers/base.py__init__8s  zHfQuantizer.__init__ torch_dtype torch.dtypereturncC|S)aU Some quantization methods require to explicitly set the dtype of the model to a target dtype. You need to override this method in case you want to make sure that behavior is preserved Args: torch_dtype (`torch.dtype`): The input dtype that is passed in `from_pretrained` rrrrrrupdate_torch_dtypeF zHfQuantizer.update_torch_dtype device_mapcCr")a Override this method if you want to pass a override the existing device map with a new one. E.g. for bitsandbytes, since `accelerate` is a hard requirement, if no device_map is passed, the device_map is set to `"auto"`` Args: device_map (`Union[dict, str]`, *optional*): The device_map that is passed through the `from_pretrained` method. r)rr&rrrupdate_device_mapRr%zHfQuantizer.update_device_mapcCr")a Override this method if you want to adjust the `target_dtype` variable used in `from_pretrained` to compute the device_map in case the device_map is a `str`. E.g. for bitsandbytes we force-set `target_dtype` to `torch.int8` and for 4-bit we pass a custom enum `accelerate.CustomDtype.int4`. Args: torch_dtype (`torch.dtype`, *optional*): The torch_dtype that is used to compute the device_map. rr#rrradjust_target_dtype^r%zHfQuantizer.adjust_target_dtype missing_keysprefixcC|S)z Override this method if you want to adjust the `missing_keys`. Args: missing_keys (`List[str]`, *optional*): The list of missing keys in the checkpoint compared to the state dict of the model rrmodelr)r*rrrupdate_missing_keysjzHfQuantizer.update_missing_keysunexpected_keyscCr+)z Override this method if you want to adjust the `unexpected_keys`. Args: unexpected_keys (`List[str]`, *optional*): The list of unexpected keys in the checkpoint compared to the state dict of the model r)rr-r0r*rrrupdate_unexpected_keystr/z"HfQuantizer.update_unexpected_keyscCr+)aE Override this method if you want to adjust the `missing_keys` after loading the model params, but before the model is post-processed. Args: missing_keys (`List[str]`, *optional*): The list of missing keys in the checkpoint compared to the state dict of the model rr,rrr!update_missing_keys_after_loading~s z-HfQuantizer.update_missing_keys_after_loading expected_keys loaded_keyscCr+)aV Override this method if you want to adjust the `update_expected_keys`. Args: expected_keys (`List[str]`, *optional*): The list of the expected keys in the initialized model. loaded_keys (`List[str]`, *optional*): The list of the loaded keys in the checkpoint. r)rr-r3r4rrrupdate_expected_keysr%z HfQuantizer.update_expected_keyscsfdd|DS)a returns dtypes for modules that are not quantized - used for the computation of the device_map in case one passes a str as a device_map. The method will use the `modules_to_not_convert` that is modified in `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize torch_dtype (`torch.dtype`): The dtype passed in `from_pretrained` method. cs.i|]\}tfddjDrqS)c3s|]}|vVqdSNr).0mnamerr szCHfQuantizer.get_special_dtypes_update...)anyrr7_r#r9r s z9HfQuantizer.get_special_dtypes_update..)named_parameters)rr-rrr#rget_special_dtypes_updates z%HfQuantizer.get_special_dtypes_update max_memorycCr")zaadjust max_memory argument for infer_auto_device_map() if extra memory is needed for quantizationr)rrBrrradjust_max_memoryzHfQuantizer.adjust_max_memoryr-r param_value torch.Tensor param_name state_dictcKdS)a checks if a loaded state_dict component is part of quantized param + some validation; only defined if requires_parameters_quantization == True for quantization methods that require to create a new parameters for quantization. Fr)rr-rErGrHrrrrcheck_quantized_params z!HfQuantizer.check_quantized_paramtorch.nn.ParametercOs|js td|jjddS)z takes needed components from state_dict and creates quantized param; only applicable if requires_parameters_quantization == True zG`.create_quantized_param()` method is not supported by quantizer class .N) requires_parameters_quantizationAttributeError __class____name__rargsrrrrcreate_quantized_params z"HfQuantizer.create_quantized_paramcOrI)a& This method is used to potentially check for potential conflicts with arguments that are passed in `from_pretrained`. You need to define it for all future quantizers that are integrated with transformers. If no explicit check are needed, simply return nothing. NrrQrrrvalidate_environmentsz HfQuantizer.validate_environmentcCr")z"updates the tp plan for the scalesrrconfigrrrupdate_tp_planrDzHfQuantizer.update_tp_plancKs2d|_|jj|_|jr|||j|fi|S)aQ Setting model attributes and/or converting model before weights loading. At this point the model should be initialized on the meta device so you can freely manipulate the skeleton of the model in order to replace modules in-place. Make sure to override the abstract method `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_before_weight_loading`. T) is_quantizedrrquantization_methodr_convert_model_for_quantization$_process_model_before_weight_loadingrr-rrrrpreprocess_models   zHfQuantizer.preprocess_modelcKs|j|fi|S)a Post-process the model post weights loading. Make sure to override the abstract method `_process_model_after_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_after_weight_loading`. )#_process_model_after_weight_loadingr\rrrpostprocess_models zHfQuantizer.postprocess_modelcCs$||}|`|j`|j`d|_|S)z Potentially dequantize the model to retrive the original model, with some loss in accuracy / performance. Note not all quantization schemes support this. F) _dequantize hf_quantizerrVr_pre_quantization_dtyperXrr-rrr dequantizes zHfQuantizer.dequantizecCst|jjd)NzH has no implementation of `dequantize`, please raise an issue on GitHub.)NotImplementedErrorrrrcrrrr`s zHfQuantizer._dequantize skip_moduleskeep_in_fp32_modulescCs<ddlm}g}|dur||}n|}|dur|||S)Nr )get_keys_to_not_convert) integrationsrhextend)r-rfrgrhrrrrget_modules_to_not_converts   z&HfQuantizer.get_modules_to_not_convertcCrI)zUFlag indicating whether the quantized model can carry out quantization aware trainingFrrrrris_qat_trainablezHfQuantizer.is_qat_trainablecCrI)z;Flag indicating whether the quantized model can be compiledFrrlrrris_compileablernzHfQuantizer.is_compileablecKdSr6rr\rrrr[!z0HfQuantizer._process_model_before_weight_loadingcKrpr6rr\rrrr^$rqz/HfQuantizer._process_model_after_weight_loadingcCrpr6r)rsafe_serializationrrris_serializable'rqzHfQuantizer.is_serializablecCrpr6rrlrrr is_trainable*rDzHfQuantizer.is_trainablec Csddlm}|D];\}}|jj}|tvrE|jjt j krE|t ||\}}t||j |j|<Wdn1s@wYq dS)Nr)init_empty_weights) accelerateru named_modulesrOrP!MODULES_TO_PATCH_FOR_QUANTIZATIONkeysrrr COMPRESSED_TENSORSrrVget_text_config_modules)rr-rur:modulemodule_class_name parent_modulerrrrZ.s   z+HfQuantizer._convert_model_for_quantization)rr r!r )r!rK)r-r)NNr6)-rP __module__ __qualname____doc__rrequired_packagesrMr rr$rrstrrr'r(rr.r1r2r5rAr intrCboolrJrSrTrWr]r_rdr` staticmethodrkpropertyrmrorr[r^rsrtrZrrrrr sr *    "  2              rcs2eZdZdZfddZ    d ddZZS) SequentialLlama4TextExpertsz A module that implements a compressed version of a list of expert modules. This is specifically designed to work with Llama4TextExperts in MoE layers. cs:ddlmtfddtjDj|_dS)Nr) Llama4TextMLPcsg|]}qSrrr=rrVrr Gsz8SequentialLlama4TextExperts.__init__..)*transformers.models.llama4.modeling_llama4rsuperrrangenum_local_experts num_expertsrUrOrrrDs " z$SequentialLlama4TextExperts.__init__ hidden_statesrFr!cCsH||jd|jd}t|}t|jD] }||||||<q|S)N)reshapershapetorch zeros_liker)rr routed_out expert_idxrrrforwardJs  z#SequentialLlama4TextExperts.forward)rrFr!rF)rPrrrrr __classcell__rrrrr>s rLlama4TextExperts) module_namequantization_methods)abcrrtypingrrrrrr utilsr utils.quantization_configr r quantizers_utilsrmodeling_utilsrrtorch.nnrrrrrzBITS_AND_BYTESrxrrrrs,