o  h@snddlmZmZmZerddlZddlmZer ddlmZe e Z     d ddZ d dd Z dS) )is_accelerate_availableis_eetq_availableloggingN)init_empty_weightsFc s|durg}|D]}\}}||t|tjrl||vrld|tfdd|Dslt4|j}|j } t || |j du|j j|j|<|rS|j||j jd}|j|dWdn1sgwYtt|dkrt||||||d\} }|d q ||fS) z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. N.c3s$|] }|dvp |kVqdS)rN).0keycurrent_key_name_strrr/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/integrations/eetq.py 2s z,_replace_with_eetq_linear..TFr)has_been_replaced pre_quantized)named_childrenappend isinstancennLinearjoinanyr in_features out_featureseetq EetqLinearbiasweightdevice_modulesregister_scalerequires_grad_lenlistchildren_replace_with_eetq_linearpop) modelmodules_to_not_convertcurrent_key_namequantization_configrrnamemodulerr_rr r r&s>       r&cCs\|durdgn|}|jdur||jtt|}t|||||d\}}|s,td|S)a A helper function to replace all `torch.nn.Linear` modules by `eetq.EetqLinear` modules from the `eetq` library. This will enable running your models using high performance int8 weight-only gemm kerner from FasterTransformer and TensorRT-LLM. Make sure `eetq` compiled with the correct CUDA version of your hardware is installed before running this function. EETQ shall be installed via the source 'https://github.com/NetEase-FuXi/EETQ' The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no CPU/GPU memory is required to run this function. Each weight will be quantized along the channel. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`): Names of the modules to not convert in `EetqLinear`. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). Nlm_head)rzYou are loading your model using eetq but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)r)extendr$setr&loggerwarning)r(r)r*r+rrrrr replace_with_eetq_linearOs     r4)NNNFF)NNNF)utilsrrrrtorch.nnr accelerater get_logger__name__r2r&r4rrrr s    4