o  h2a@s^ddlZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZmZmZe rGddlZddlZddlmZddlmZe rbddlZddlmZdd lmZmZdd lmZee Z!d*d d Z"    d+ddZ#d,ddZ$ddZ%ddZ&ddZ'd-d.ddZ(ddZ)    d+d d!Z*  d*d"d#Z+d$d%Z,d&d'Z-d/d(d)Z.dS)0N)deepcopy) signature)version)get_available_devicesis_accelerate_availableis_bitsandbytes_available'is_bitsandbytes_multi_backend_availableis_ipex_availableis_torch_availablelogging)Conv1D)init_empty_weights)add_hook_to_moduleremove_hook_from_module)find_tied_parameterscCsJd|vr+|d}|ddD]}t||}|dur$t|d|d|}q|d}||jvr?||jvr?t|d|d||jv}t||} | jtdkrh|dtdfvrh|durht|d|d|du} |sqtsvd} d} ntt j d ot |j|t j j } t |j|t j j } | s| rm|j|} | jjd krk|dur| |}nt |tjr|d }ntj|d d }t|jtr| s|j}| j}| |jtjtjfvkrtd |jd| r ttjdtdk}|jtjtjfvr|stdt j j |fddi||}| rt|d|d|nD| rd| rUttjdtdk}|jtjtjfvrE|sEtdt j j jd||d|d|}nt j j |fddi||}||j|<dSdS|durx| |}nt |tjr||}ntj||d }|r||j|<dSt j || j!d}||j|<dS)a A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing `param.to(device)` creates a new tensor not linked to the parameter, which is why we need this function). The function is adapted from `set_module_tensor_to_device` function from accelerate that is adapted to support the class `Int8Params` from `bitsandbytes`. Args: module (`torch.nn.Module`): The module in which the tensor we want to move lives. tensor_name (`str`): The full name of the parameter/buffer. device (`int`, `str` or `torch.device`): The device on which to set the tensor. value (`torch.Tensor`, *optional*): The value of the tensor (useful when going from the meta device to any other device). quantized_stats (`dict[str, Any]`, *optional*): Dict with items for either 4-bit or 8-bit serialization .Nz has no attribute z- does not have a parameter or a buffer named metaz7 is on the meta device, we need a `value` to put in on F Params4bitcudacpu)devicez Value dtype `z7` is not compatible with parameter quantization status. bitsandbytesz0.37.2zDetected int8 weights but the version of bitsandbytes is not compatible with int8 serialization. Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`. requires_gradSCBz0.41.3zDetected 4-bit weights but the version of bitsandbytes is not compatible with 4-bit serialization. Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`.)dataquantized_statsrr)r)"splitgetattr ValueError _parameters_buffersrtorchrhasattrbnbnn isinstancer Int8ParamstypetoTensortensor issubclass source_clsr T__dict__dtypeint8uint8rparse importlibmetadatasetattrfrom_prequantized Parameterr)module tensor_namervaluersplitsr new_module is_buffer old_valueprequantized_loadingis_8bitis_4bitparam new_valuekwargsis_8bit_serializableis_4bit_serializablerrz/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/integrations/bitsandbytes.py%set_module_quantized_tensor_to_device$s    *          6  rKFc s|D]\}}|durg}||t|tjst|tr||vrd|tfdd|Dst~t|trB|j j \}}n|j }|j }| dkrdtjj|||jdu|j|jd|j|<d}n8|jduro||jvron-dtttjjjvrd|jini} tjj|||jdu|jf|j|jd | |j|<d}t||j|_|j|d Wdn1swYt t|!d krt"|||||d \} }|#d q||fS)z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. Nrc3$|] }|dvp |kVqdSrNr.0keycurrent_key_name_strrrJ  z+_replace_with_bnb_linear..llm_int8)has_fp16_weights thresholdT quant_storage)compress_statistics quant_typeFrhas_been_replacedr)$named_childrenappendr(r'Linearr joinanyrweightshape in_features out_featuresquantization_methodr& Linear8bitLtbiasllm_int8_has_fp16_weightllm_int8_threshold_modulesllm_int8_skip_moduleslistr Linear4bit parametersbnb_4bit_quant_storagebnb_4bit_compute_dtypebnb_4bit_use_double_quantbnb_4bit_quant_typer*r/requires_grad_lenchildren_replace_with_bnb_linearpop) modelmodules_to_not_convertcurrent_key_namequantization_configr\namer;rdre extra_kwargs_rrQrJrwsn            * rwcCs6|durdgn|}t||||\}}|std|S)a A helper function to replace all `torch.nn.Linear` modules by `bnb.nn.Linear8bit` modules from the `bitsandbytes` library. This will enable running your models using mixed int8 precision as described by the paper `LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale`. Make sure `bitsandbytes` compiled with the correct CUDA version of your hardware is installed before running this function. `pip install -i https://test.pypi.org/simple/ bitsandbytes` The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no CPU/GPU memory is required to run this function. Int8 mixed-precision matrix decomposition works by separating a matrix multiplication into two streams: (1) and systematic feature outlier stream matrix multiplied in fp16 (0.01%), (2) a regular stream of int8 matrix multiplication (99.9%). With this method, int8 inference with no predictive degradation is possible for very large models (>=176B parameters). Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`): Names of the modules to not convert in `Linear8bitLt`. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). quantization_config ('transformers.utils.quantization_config.BitsAndBytesConfig'): To configure and manage settings related to quantization, a technique used to compress neural network models by reducing the precision of the weights and activations, thus making models more efficient in terms of both storage and computation. Nlm_headzYou are loading your model in 8bit or 4bit but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)rwloggerwarning)ryrzr{r|r\rrrJreplace_with_bnb_linearsrcOtdtt|i|S)Nzj`replace_8bit_linear` will be deprecated in a future version, please use `replace_with_bnb_linear` instead)warningswarn FutureWarningrargsrGrrrJreplace_8bit_linear rcOr)Nz`set_module_8bit_tensor_to_device` will be deprecated in a future version, please use `set_module_quantized_tensor_to_device` instead)rrrrKrrrrJ set_module_8bit_tensor_to_devicerrc st|}|t|}t|tr!tt|gt|}nt|g}t |dk}|sC| durCfdd| D}|St| }|ddg}t |t |}tt |t|}ddg} g} |D]} | D] } | | vrz| | d} qn| | qj| S) a An utility function to get the key of the module to keep in full precision if any For example for CausalLM modules we may want to keep the lm_head in full precision for numerical stability reasons. For other architectures, we want to keep the tied weights of the model. The function will return a list of the keys of the modules to not convert in int8. Parameters: model (`torch.nn.Module`): Input model rNcs$g|]\}}t|tkr|qSr)id)rOr}r; output_embrrJ @s$z+get_keys_to_not_convert..rz.weightz.bias)r tie_weightsrr(dictsumrmvalueskeysruget_output_embeddings named_modulesnamed_parameterssetreplacer^) ry tied_model tied_params tied_keyshas_tied_paramslist_last_module list_modules intersectionlist_untouchednames_to_removefiltered_module_namesr}name_to_removerrrJget_keys_to_not_convert$s2        rrbtorch.nn.Parameterr2 torch.dtypecCst|tjjstdt|d|jj}|dvr|S|dkr7tj |j |j }t d|jd||S|jdur@|j|_ttj drPtj |j |j}n |j |jd d d }||S) z Helper function to dequantize 4bit or 8bit bnb weights. If the weight is not a bnb quantized weight, it will be returned as is. z1Input weight should be of type nn.Parameter, got z instead)rr)rz(The model is going to be dequantized in z - if you want to upcast it to another dtype, make sure to pass the desired dtype when quantizing the model through `bnb_4bit_quant_type` argument of `BitsAndBytesConfig`Nint8_vectorwise_dequantrg@ ?)r(r$r'r: TypeErrorr* __class____name__r& functionaldequantize_4bitr quant_stater warning_oncer2r+rr%rview)rbr2statecls_name output_tensor dequantizedrrrJdequantize_bnb_weightWs"     rcCs\ttj|jj}|j}i}t|j}| D] }||j vr$||||<q|di|}|S)a Creates a new hook based on the old hook. Use it only if you know what you are doing ! This method is a copy of: https://github.com/huggingface/peft/blob/748f7968f3a31ec06a1c2b0328993319ad9a150a/src/peft/utils/other.py#L245 with some changes Nr) r acceleratehooksrrr1inspectr__init__rro)old_hook old_hook_cls old_hook_attrfiltered_old_hook_attrold_hook_init_signatureknew_hookrrrJ_create_accelerate_new_hookxs    rc s|}|dkr tjjntjj}|D]\}} |durg}||t| |r||vrd|t fdd|Dst | dd} | j j } t tjj| j| j| dud} Wdn1sawY|dkrn| j} nd} tjt| j || | _ | dur| | _t| dr| j}t|}t| t| || | | |j|<d }tt| d krt | |||||d \}}|!d q||fS) aq Converts a quantized model into its dequantized original version. The newly converted model will have some performance drop compared to the original model before quantization - use it only for specific usecases such as QLoRA adapters merging. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. rUNrc3rLrMrrNrQrrJrSrTz*_dequantize_and_replace..rh)rh_hf_hookTrr[r)"rfr&r'rgrnr]r^r(r`rar rbrrr$r_rdrerr:rrhr%rrrrr+rkrurmrv_dequantize_and_replacerx)ryr2rzr{r|r\ quant_method target_clsr}r;rhrr?rrrrrrQrJrsR         rcCs(t||j||d\}}|std|S)N)rzr|z_For some reason the model has not been properly dequantized. You might see unexpected behavior.)rr2rr)ryrzr|r\rrrJdequantize_and_replaces rcCsddl}t|dt}tt}|dhkr0ts0ddlm}|dr'tdt dd|D}| |s[|rTtd d |D}d |pDdd |d }t |t |tddSt ddS)Nrsupported_torch_devicesr) find_specintel_extension_for_pytorchzYou have Intel IPEX installed but if you're intending to use it for CPU, it might not have the right version. Be sure to double check that your PyTorch and IPEX installs are compatible.cSsg|]}|dkr|qS)rrrOrrrrJrsz<_validate_bnb_multi_backend_availability..css |] }|dkr dn|VqdS)rzl"cpu" (needs an Intel CPU and intel_extension_for_pytorch installed and compatible with the PyTorch version)NrrrrrJrSs z;_validate_bnb_multi_backend_availability..z3None of the available devices `available_devices = zY` are supported by the bitsandbytes version you have installed: `bnb_supported_devices = z`. Please check the docs to see if the backend you intend to use is available and how to install it: https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backendz:No supported devices found for bitsandbytes multi-backend.Fz$Multi-backend validation successful.T)rr rrr importlib.utilrrr frozensetrerror RuntimeErrordebug)raise_exceptionr&bnb_supported_devicesavailable_devicesrbnb_supported_devices_with_infoerr_msgrrrJ(_validate_bnb_multi_backend_availabilitys2       rcCsRtsdSddl}|js"d}|rt|t|t|dStddS)NFraBCUDA is required but not available for bitsandbytes. Please consider installing the multi-platform enabled version of bitsandbytes, which is currently a work in progress. Please check currently supported platforms and installation instructions at https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backendz#CUDA backend validation successful.T) r r$r is_availablerrrrr)rr$log_msgrrrJ'_validate_bnb_cuda_backend_availabilitys    rcCsPtstjdrttjdtdkrt|SdStr$t |St|S)zs Validates if the available devices are supported by bitsandbytes, optionally raising an exception if not. rz0.43.1F) rr6utilrrr5r7rr r)rrrrJ!validate_bnb_backend_availability$s r)NN)NNNF)NNN)N)rbrr2r)F)/importlib.metadatar6rrcopyrr packagingrutilsrrrr r r r rr&r$torch.nnr' pytorch_utilsr rraccelerate.hooksrraccelerate.utilsr get_loggerrrrKrwrrrrrrrrrrrrrrrJsR   $       u N.  3! J )