o
hl @ s d dl Z d dlZd dlZd dlmZmZmZmZmZ d dl m
Z
ddl m Z m
Z
mZmZmZmZ e r7d dlZe rHd dlmZ d dlmZmZ dZeeZG d d
d
Z dS ) N)AnyDictListOptionalUnion)version )check_peft_versionfind_adapter_config_fileis_accelerate_availableis_peft_availableis_torch_availablelogging)dispatch_model)get_balanced_memoryinfer_auto_device_mapz0.5.0c @ s e Zd ZdZdZ d*dee dee dee dee d ee d
ee d ee d ee d
eee f deeedf de
de
deeee f ddf ddZ d+dee ddfddZ de
ee ef ddfddZd,ddZd,d d Zdee fd d Zdefd d!Zd-dee d"ee defd#d$Z d.d ed
ee d ee d ee ddf
d%d&Zd'e
ee ef ddfd(d)ZdS )/PeftAdapterMixina<
A class containing all functions for loading and using adapters weights that are supported in PEFT library. For
more details about adapters and injecting them on a transformer-based model, check out the documentation of PEFT
library: https://huggingface.co/docs/peft/index
Currently supported PEFT methods are all non-prefix tuning methods. Below is the list of supported PEFT methods
that anyone can load, train and run with this mixin class:
- Low Rank Adapters (LoRA): https://huggingface.co/docs/peft/conceptual_guides/lora
- IA3: https://huggingface.co/docs/peft/conceptual_guides/ia3
- AdaLora: https://arxiv.org/abs/2303.10512
Other PEFT models such as prompt tuning, prompt learning are out of scope as these adapters are not "injectable"
into a torch module. For using these methods, please refer to the usage guide of PEFT library.
With this mixin, if the correct PEFT version is installed, it is possible to:
- Load an adapter stored on a local path or in a remote Hub repository, and inject it in the model
- Attach new adapters in the model and train them with Trainer or by your own.
- Attach multiple adapters and iteratively activate / deactivate them
- Activate / deactivate all adapters from the model.
- Get the `state_dict` of the active adapter.
FNauto
peft_model_id adapter_namerevisiontoken
device_map
max_memoryoffload_folder
offload_index peft_configadapter_state_dictz torch.Tensorlow_cpu_mem_usage is_trainableadapter_kwargsreturnc sn t td i }| r&d}ttjdt|kr | |d< ntd| d dur, nd |
du r4i }
d d
lm}m }m
} d d l m } | j
rT | jv rTtd d
|du rd|
du rd| du rdtdd|
vrzt| dsp| jnt| j d }n|
d}t|tjrt|}|durd|
vr||
d< n|durd|
v r||
d krtd d|
v r|
d}| du rt|fd|i|
}|du rtd| d|j|fd|i|
} | | _ || | fi | | j
sd| _
|dur||f||d|
}
i }d}|
D ]\}}| |r|t |d }n|}|||< q|| | fi |}|dur|d} |dur5|nd} t|drRt |j d krRd | d d !|j d } t"|d d} | rt fd!d"| D } | rt| d | d#d !| 7 } | r|t#| | j r| $ t"| dddurt t%| j &d$d%hd krt | jd&kr| j'||||d' dS dS dS dS )(a
Load adapter weights from file or remote Hub folder. If you are not familiar with adapters and PEFT methods, we
invite you to read more about them on PEFT official documentation: https://huggingface.co/docs/peft
Requires peft as a backend to load the adapter weights.
Args:
peft_model_id (`str`, *optional*):
The identifier of the model to look for on the Hub, or a local path to the saved adapter config file
and adapter weights.
adapter_name (`str`, *optional*):
The adapter name to use. If not set, will use the default adapter.
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`.
token (`str`, `optional`):
Whether to use authentication token to load the remote folder. Useful to load private repositories
that are on HuggingFace Hub. You might need to call `huggingface-cli login` and paste your tokens to
cache it.
device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*):
A map that specifies where each submodule should go. It doesn't need to be refined to each
parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank
like `1`) on which the model will be allocated, the device map will map the entire model to this
device. Passing `device_map = 0` means put the whole model on GPU 0.
To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
more information about each option see [designing a device
map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
max_memory (`Dict`, *optional*):
A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
GPU and the available CPU RAM if unset.
offload_folder (`str` or `os.PathLike`, `optional`):
If the `device_map` contains any value `"disk"`, the folder where we will offload weights.
offload_index (`int`, `optional`):
`offload_index` argument to be passed to `accelerate.dispatch_model` method.
peft_config (`Dict[str, Any]`, *optional*):
The configuration of the adapter to add, supported adapters are non-prefix tuning and adaption prompts
methods. This argument is used in case users directly pass PEFT state dicts
adapter_state_dict (`Dict[str, torch.Tensor]`, *optional*):
The state dict of the adapter to load. This argument is used in case users directly pass PEFT state
dicts
low_cpu_mem_usage (`bool`, *optional*, defaults to `False`):
Reduce memory usage while loading the PEFT adapter. This should also speed up the loading process.
Requires PEFT version 0.13.0 or higher.
is_trainable (`bool`, *optional*, defaults to `False`):
Whether the adapter should be trainable or not. If `False`, the adapter will be frozen and can only be
used for inference.
adapter_kwargs (`Dict[str, Any]`, *optional*):
Additional keyword arguments passed along to the `from_pretrained` method of the adapter config and
`find_adapter_config_file` method.
min_versionz0.13.0peftr zcThe version of PEFT you are using does not support `low_cpu_mem_usage` yet, please install PEFT >= .Ndefaultr )
PeftConfiginject_adapter_in_modelload_peft_weights)set_peft_model_state_dictAdapter with name - already exists. Please use a different name.zhYou should either pass a `peft_model_id` or a `peft_config` and `adapter_state_dict` to load an adapter.device
hf_device_mapr zYou passed a `revision` argument both in `adapter_kwargs` and as a standalone argument. The one in `adapter_kwargs` will be used.r z adapter model file not found in zB. Make sure you are passing the correct path to the adapter model.T)r r- zbase_model.model.
state_dictunexpected_keysz Loading adapter weights from z0 led to unexpected keys not found in the model: , z. missing_keysc s g | ] }d |v r |v r|qS )lora_ ).0kr r5 r/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/integrations/peft.py
s z1PeftAdapterMixin.load_adapter..z# led to missing keys in the model: cpudisk )r r r r )(r MIN_PEFT_VERSIONr parse importlibmetadata
ValueErrorr$ r' r( r)
peft.utilsr* _hf_peft_config_loadedr hasattrr- listr. valuespop
isinstancetorchstrloggererrorr
from_pretrainedinference_modeitems
startswithlenr1 joingetattrwarningevalset intersection_dispatch_accelerate_model) selfr r r r r r r r r r r r r peft_load_kwargsmin_version_lcmur' r( r) r* r- adapter_config_file processed_adapter_state_dictprefixkeyvaluenew_keyincompatible_keyserr_msg origin_namer3 lora_missing_keysr5 r8 r9 load_adapterH s
L
$
"
z PeftAdapterMixin.load_adapterc C s t td ddlm}m} |pd}| jsd| _n
|| jv r%td| dt||s4t dt
| d | j d
d |_
||| | | | d S ) aI
If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
official documentation: https://huggingface.co/docs/peft
Adds a fresh new adapter to the current model for training purpose. If no adapter name is passed, a default
name is assigned to the adapter to follow the convention of PEFT library (in PEFT we use "default" as the
default adapter name).
Args:
adapter_config (`~peft.PeftConfig`):
The configuration of the adapter to add, supported adapters are non-prefix tuning and adaption prompts
methods
adapter_name (`str`, *optional*, defaults to `"default"`):
The name of the adapter to add. If no name is passed, a default name is assigned to the adapter.
r" r )r' r( r& Tr+ r, z8adapter_config should be an instance of PeftConfig. Got z instead. name_or_pathN)r r> r$ r' r( rD r rB rI TypeErrortype__dict__getbase_model_name_or_path set_adapter)rZ adapter_configr r' r( r5 r5 r9 add_adapter s
z PeftAdapterMixin.add_adapterc C s t td | js tdt|tr3t|t| j }t|dkr2tdd | dt| j
n|| jvrGtd| dt| j
dd l m } dd
l
m} d }| D ]\}}t|||frtt|d ro|| n||_d
}qY|s{tddS )a
If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
official documentation: https://huggingface.co/docs/peft
Sets a specific adapter by forcing the model to use a that adapter and disable the other adapters.
Args:
adapter_name (`Union[List[str], str]`):
The name of the adapter to set. Can be also a list of strings to set multiple adapters.
r" 0No adapter loaded. Please load an adapter first.r z)Following adapter(s) could not be found: r2 zV. Make sure you are passing the correct adapter name(s). current loaded adapters are: r+ z7 not found. Please pass the correct adapter name among BaseTunerLayerModulesToSaveWrapperFrn TzhDid not succeeded in setting the adapter. Please make sure you are using a model that supports adapters.N)r r> rD rB rI rF rW r rR rS keyspeft.tuners.tuners_utilsrs rC ru
named_modulesrE rn active_adapter)rZ r missingrs ru _adapters_has_been_set_moduler5 r5 r9 rn @ s>
z PeftAdapterMixin.set_adapterc C sr t td | js tdddlm} ddlm} | D ]\}}t |||fr6t
|dr3|j dd q d |_ q d
S ) a,
If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
official documentation: https://huggingface.co/docs/peft
Disable all adapters that are attached to the model. This leads to inferring with the base model only.
r" rq r rr rt enable_adaptersFenabledTN)
r r> rD rB rw rs rC ru rx rI rE r~ disable_adapters)rZ rs ru r| r} r5 r5 r9 r m s
z!PeftAdapterMixin.disable_adaptersc C sb t td | js tdddlm} | D ]\}}t||r.t|dr+|j dd qd|_
qd S )
z
If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
official documentation: https://huggingface.co/docs/peft
Enable adapters that are attached to the model.
r" rq r rr r~ Tr FN) r r> rD rB rw rs rx rI rE r~ r )rZ rs r| r} r5 r5 r9 r~ s
z PeftAdapterMixin.enable_adaptersc C sl t td t s td| jstdddlm} | D ]\}}t ||r+|j
} nq t |t r4|g}|S )a3
If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
official documentation: https://huggingface.co/docs/peft
Gets the current active adapters of the model. In case of multi-adapter inference (combining multiple adapters
for inference) returns the list of all active adapters so that users can deal with them accordingly.
For previous PEFT versions (that does not support multi-adapter inference), `module.active_adapter` will return
a single string.
r" zTPEFT is not available. Please install PEFT to use this function: `pip install peft`.rq r rr ) r r> r ImportErrorrD rB rw rs rx rI ry rK )rZ rs r| r} active_adaptersr5 r5 r9 r s
z PeftAdapterMixin.active_adaptersc C s t dt | d S )NzRThe `active_adapter` method is deprecated and will be removed in a future version.r )warningswarn
FutureWarningr rZ r5 r5 r9 ry s z PeftAdapterMixin.active_adapterr0 c C sJ t td | js tdddlm} |du r | d }|| ||d}|S )ax
If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
official documentation: https://huggingface.co/docs/peft
Gets the adapter state dict that should only contain the weights tensors of the specified adapter_name adapter.
If no adapter_name is passed, the active adapter is used.
Args:
adapter_name (`str`, *optional*):
The name of the adapter to get the state dict from. If no name is passed, the active adapter is used.
state_dict (nested dictionary of `torch.Tensor`, *optional*)
The state dictionary of the model. Will default to `self.state_dict()`, but can be used if special
precautions need to be taken when recovering the state dictionary of a model (like when using model
parallelism).
r" rq r )get_peft_model_state_dictN)r0 r )r r> rD rB r$ r r )rZ r r0 r r r5 r5 r9 get_adapter_state_dict s
z'PeftAdapterMixin.get_adapter_state_dictc C sp i }dt tjv r||d< | j}|dkr t| |||dkd}t|tr+t| ||d}t| f||d| dS )a
Optional re-dispatch the model and attach new hooks to the model in case the model has been loaded with
accelerate (i.e. with `device_map=xxx`)
Args:
device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*):
A map that specifies where each submodule should go. It doesn't need to be refined to each
parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank
like `1`) on which the model will be allocated, the device map will map the entire model to this
device. Passing `device_map = 0` means put the whole model on GPU 0.
To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
more information about each option see [designing a device
map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
max_memory (`Dict`, *optional*):
A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
GPU and the available CPU RAM if unset.
offload_folder (`str` or `os.PathLike`, *optional*):
If the `device_map` contains any value `"disk"`, the folder where we will offload weights.
offload_index (`int`, *optional*):
The offload_index argument to be passed to `accelerate.dispatch_model` method.
r
sequentialbalanced_low_0)r no_split_module_classeslow_zero)r r )r offload_dirN) inspect signaturer
parameters_no_split_modulesr rI rK r )rZ r r r r dispatch_model_kwargsr r5 r5 r9 rY s.
z+PeftAdapterMixin._dispatch_accelerate_model
adapter_namesc s t td js tdddlm} t|tr|g} fdd|D }|r/tdd| |D ]/} D ]}t||rMt
|d rI| | q7td
q7t d d r`t
d
r` j
|d q1t j
dkro `
d _dS dS )a
Delete an adapter's LoRA layers from the underlying model.
Args:
adapter_names (`Union[List[str], str]`):
The name(s) of the adapter(s) to delete.
Example:
```py
from diffusers import AutoPipelineForText2Image
import torch
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
).to("cuda")
pipeline.load_lora_weights(
"jbilcke-hf/sdxl-cinematic-1", weight_name="pytorch_lora_weights.safetensors", adapter_names="cinematic"
)
pipeline.delete_adapters("cinematic")
```
r" rq r rr c s g | ] }| j vr|qS r5 )r )r6 namer r5 r9 r: 9 s z3PeftAdapterMixin.delete_adapter..z@The following adapter(s) are not present and cannot be deleted: r2 delete_adapterzdThe version of PEFT you are using is not compatible, please use a version that is greater than 0.6.1rD Fr N)r r> rD rB rw rs rI rK rS modulesrE r rT r rH rR )rZ r rs missing_adaptersr r} r5 r r9 r s6
z PeftAdapterMixin.delete_adapter)
NNNNr NNNNNFFN)N)r! N)NN)NNN)__name__
__module__ __qualname____doc__rD r rK intr r boolrg rp r r rn r r~ r ry dictr rY r r5 r5 r5 r9 r . s
T %
-
"8r ) r@ r r typingr r r r r packagingr utilsr r
r r r
r rJ
accelerater accelerate.utilsr r r>
get_loggerr rL r r5 r5 r5 r9 s