o  hõ@s@ddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZmZmZmZerSdd lmZeeZ Gd d d Z!eGd d d Z"eGddde"Z#eGddde"Z$Gddde!Z%de%fddZ&de%fddZ'dejj(j)fddZ*ddZ+edrejj(j,e%e&e*e%j-de%je'd ej.j(/e%e+Gd!d"d"e%Z0Gd#d$d$e%Z1Gd%d&d&e1Z2Gd'd(d(e1Z3Gd)d*d*e!Z4Gd+d,d,e!Z5Gd-d.d.e5Z6Gd/d0d0e!Z7Gd1d2d2e!Z8Gd3d4d4e!Z9Gd5d6d6Z:Gd7d8d8e5Z;dS)9N) dataclass)AnyDictIterableListOptionalTupleUnion)version)"is_torch_greater_or_equal_than_2_6)PretrainedConfig)is_hqq_availableis_optimum_quanto_availableis_torch_greater_or_equallogging) Quantizerc seZdZdZdZfddZ ddejdejded e e e e fd e ejejff d d Zdde ed efddZd e efddZddede ed efddZdejfddZeddZZS)Cachezf Base, abstract class for all caches. The actual data structure is specific to each subclass. FcstdSN)super__init__self __class__l/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/cache_utils.pyrszCache.__init__N key_states value_states layer_idx cache_kwargsreturncCtd)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. These are specific to each subclass and allow new types of cache to be created. Return: A tuple containing the updated key and value states. z.Make sure to implement `update` in a subclass.NotImplementedError)rrrrr rrrupdate!sz Cache.updatercCr")YReturns the sequence length of the cached states. A layer index can be optionally passed.z6Make sure to implement `get_seq_length` in a subclass.r#rrrrrget_seq_length;szCache.get_seq_lengthcCr")zKReturns the maximum sequence length (i.e. max capacity) of the cache objectz;Make sure to implement `get_max_cache_shape` in a subclass.r#rrrrget_max_cache_shape@zCache.get_max_cache_shapenew_seq_lengthcCs2|}||}|dur|||kr||S|S)zTGiven the sequence length of the new inputs, returns the usable length of the cache.N)r)r()rr+r max_lengthprevious_seq_lengthrrrget_usable_lengthDs  zCache.get_usable_lengthbeam_idxcCstt|jD]:}|j|r%|j|j}|j|d|||j|<|j|rA|j|j}|j|d|||j|<qdS)DReorders the cache for beam search, given the selected beam indices.rN)rangelen key_cachenumeldevice index_selectto value_cache)rr/rr5rrr reorder_cacheOs  zCache.reorder_cachecCstdt|dr |jSdS)NzuThe `seen_tokens` attribute is deprecated and will be removed in v4.41. Use the `cache_position` model input instead. _seen_tokens)logger warning_oncehasattrr:rrrr seen_tokensYs  zCache.seen_tokensrr)__name__ __module__ __qualname____doc__is_compileablertorchTensorintrrstrrrr%r(r)r. LongTensorr9propertyr> __classcell__rrrrrs,    rc@steZdZUdZded<eddZdeee j ffddZ d e ee ffd d Zd d ZddZddZddZdS) CacheConfigz& Base class for cache configs Ncache_implementationcKs^|di|}g}|D]\}}t||r!t|||||q |D]}||dq$|S)ar Constructs a CacheConfig instance from a dictionary of parameters. Args: config_dict (Dict[str, Any]): Dictionary containing configuration parameters. **kwargs: Additional keyword arguments to override dictionary values. Returns: CacheConfig: Instance of CacheConfig constructed from the dictionary. Nr)itemsr=setattrappendpop)cls config_dictkwargsconfig to_removekeyvaluerrr from_dictms    zCacheConfig.from_dictjson_file_pathcCsZt|ddd}|}tj|dddd}||WddS1s&wYdS) a Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this configuration instance's parameters will be saved. use_diff (`bool`, *optional*, defaults to `True`): If set to `True`, only the difference between the config instance and the default `QuantizationConfig()` is serialized to JSON file. wzutf-8)encodingT)indent sort_keys N)opento_dictjsondumpswrite)rrZwriterrS json_stringrrr to_json_files   "zCacheConfig.to_json_filer!cCs t|jS)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. )copydeepcopy__dict__rrrrrbs zCacheConfig.to_dictccs*t|jD] \}}||fVq dS)zTallows `dict(obj)` for situations where obj may be a dict or QuantizationConfigMixinN)rirjrkrN)rattrrXrrr__iter__s zCacheConfig.__iter__cCs|jjd|S)N )rr@to_json_stringrrrr__repr__szCacheConfig.__repr__cCstj|jdddS)z Serializes this instance to a JSON formatted string. Returns: str: JSON formatted string representing the configuration instance. r])r^r`)rcrdrkrrrrroszCacheConfig.to_json_stringc sPg|D]\}}t||rt||||qfdd|D}|S)a Updates attributes of this class instance with attributes from `kwargs` if they match existing attributes, returning all the unused kwargs. Args: kwargs (`Dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `Dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. csi|] \}}|vr||qSrr).0rWrXrVrr sz&CacheConfig.update..)rNr=rOrP)rrTrWrX unused_kwargsrrrrr%s    zCacheConfig.update)r@rArBrC__annotations__ classmethodrYr rHosPathLikerhrrrbrmrpror%rrrrrLes   rLc@steZdZdZddddddejdfded eed eed eed eed eedeej deefddZ ddZ dS)QuantizedCacheConfiga Configuration class for quantized cache settings. Attributes: backend (`str`, *optional*, defaults to `"quanto"`): Backend to use when performing quantization, Can be one of [`quanto`, `HQQ`] nbits (`Optional[int]`, *optional*, defaults to 4): Number of bits, can be 2 or 4 for the `quanto` backend and one of [1, 2, 3, 4, 8] for the `HQQ` backend. Defaults to 2. axis_key (`int`, *optional*, defaults to 0): Axis over which to perform grouping for the key tensors. Can be [0, -1] for `quanto` backend and [0, 1] for `HQQ` backend. axis_value (`int`, *optional*, defaults to 0): Axis over which to perform grouping for the value tensors. Can be [0, -1] for `quanto` backend and [0, 1] for `HQQ` backend. q_group_size (`Optional[int]`, *optional*, defaults to 64): Size of the quantization group, should be a divisor of the model's hidden dimension. Defaults to 64. residual_length (`Optional[int]`, *optional*, defaults to 128): Length of the residual cache which will always be stored in original precision. Defaults to 128. compute_dtype (`torch.dtype`, *optional*, defaults to `torch.float16`): The default dtype used for computations in the model. Keys and Values will be cast to this dtype after dequantization. device (`str`, *optional*, defaults to `"cpu"`): Device on which to perform computations, should be same as the model's device. quantor@cpubackendnbitsaxis_key axis_value q_group_sizeresidual_length compute_dtyper5c Cs4||_||_||_||_||_||_||_||_dSr)rrrrrrrr5) rrrrrrrrr5rrrrs  zQuantizedCacheConfig.__init__cCsd}|jdvrt|jdd|jd|jdkr"t|jdd|jd|jdkr2t|jd d|jd|jd vrBt|jd d |jd|jd vrRt|jd d|jddS)-Validates if the arguments passed are correctvSome of the keys in `cache_config` are defined incorrectly. `{key}` should be {correct_value}` but found {found_value}r r]r{rz 2 or 4 or 8rW correct_value found_valuerrza positive integerr)rr rz`1` or `0`, `-1`rz`1` or `0` or `-1`N)r ValueErrorformatrrrrrincorrect_arg_msgrrrvalidatesV     zQuantizedCacheConfig.validateN) r@rArBrCrEfloat16rHrrGdtyperrrrrrrys8 ryc@s0eZdZdZdZd dedefddZdd Zd S) StaticCacheConfigz8 Configuration class for static cache settings. staticr~ batch_size max_cache_lencCs||_||_||_dSr)rrr5)rrrr5rrrr1s zStaticCacheConfig.__init__cCsHd}|jdkrt|jdd|jd|jdkr"t|jdd|jddS)rrrrz> 0rrN)rrrrrrrrr6s&  zStaticCacheConfig.validateN)r~)r@rArBrCrMrGrrrrrrr)s  rc sheZdZdZd-deddffdd Zdedeee j fdd Z d d Z d d Z d-de j de j dedeeeefdee j e j ff ddZd.deedefddZdeefddZdeee j ee j ffddZed-deeee jddfddZdefddZd ed!ededfd"d#Zed$edddfd%d&Zd'efd(d)Zd*e j fd+d,ZZS)/ DynamicCachea A cache that grows dynamically as more tokens are generated. This is the default for generative models. It stores the Key and Value states as a list of tensors, one for each layer. The expected shape for each tensor is `[batch_size, num_heads, seq_len, head_dim]`. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> past_key_values = DynamicCache() >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation DynamicCache() ``` N_distributed_cache_datar!csRtd|_g|_g|_|dur%|D]\}}|j||j|qdSdSNr)rrr:r3r8rP)rrrrrrrrjs   zDynamicCache.__init__rcCs8|t|kr|j||j|fStdt|d|z Support for backwards-compatible `past_key_value` indexing, e.g. `past_key_value[0][0].shape[2]` to get the sequence length. Cache only has . layers, attempted to access layer with index )r2r3r8KeyErrorr'rrr __getitem__{s zDynamicCache.__getitem__ccs.tt|D] }|j||j|fVqdS)z Support for backwards-compatible `past_key_value` iteration, e.g. `for x in past_key_value:` to iterate over keys and values Nr1r2r3r8r'rrrrmszDynamicCache.__iter__cC t|jSz Support for backwards-compatible `past_key_value` length, e.g. `len(past_key_value)`. This value corresponds to the number of layers in the model. )r2r3rrrr__len__ zDynamicCache.__len__rrr cCs|dkr|j|jd7_|durst|j|krCtt|j|D]}|jtg|jtgq!|j||j|n0|j| sU||j|<||j|<ntj |j||gdd|j|<tj |j||gdd|j|<|j||j|fS)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. No additional arguments are used in `DynamicCache`. Return: A tuple containing the updated key and value states. rNdim) r:shaper2r3r1rPrEtensorr8r4cat)rrrrr _rrrr%s     zDynamicCache.updatercCsLt|jdkpt|j|kp|j| }|s"|j|jd}|Sd}|Sr&rr)r2r3r4r)rris_empty_layerlayer_seq_lengthrrrr(s zDynamicCache.get_seq_lengthcCsdS)zeReturns the maximum sequence length of the cache object. DynamicCache does not have a maximum length.Nrrrrrr)sz DynamicCache.get_max_cache_shapecCs4d}tt|D]}||j||j|ff7}q|S)zConverts the `DynamicCache` instance into the its equivalent in the legacy cache format. Used for backward compatibility.rr)r legacy_cacherrrrto_legacy_cacheszDynamicCache.to_legacy_cachepast_key_valuescCs>|}|durtt|D]}||\}}||||q |S)zwConverts a cache in the legacy cache format into an equivalent `DynamicCache`. Used for backward compatibility.N)r1r2r%rRrcacherrrrrrfrom_legacy_caches  zDynamicCache.from_legacy_cacher,cCs|dkr |t|}||krdS||_tt|jD]+}|j|rI|j|dd|ddf|j|<|j|dd|ddf|j|<qdS)zCrop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative to remove `max_length` tokens. This is used in assisted decoding and contrastive search.rN.)r(absr:r1r2r3r4r8)rr,idxrrrcrops ""zDynamicCache.cropfull_batch_size split_sizecsbg}td|D]&t}|j|_fdd|jD|_fdd|jD|_||q|S)Split the current instance into a list of `DynamicCache` by the batch size. This will be used by `_split_model_inputs()` in `generation.utils`rcg|] }|qSrrrqrirrr z,DynamicCache.batch_split..crrrrrrrrr)r1rr:r3r8rP)rrrout current_splitrrr batch_splits zDynamicCache.batch_splitsplitscsv|}tt|dD]-fdd|D}fdd|D}|gkr8tj|dd}tj|dd}|||q |S)This is the opposite of the above `batch_split()` method. This will be used by `stack_model_outputs` in `generation.utils`rc$g|]}|jr|jqSr)r3r4rqcurrentrrrr$z2DynamicCache.from_batch_splits..crr)r8r4rrrrrrr)r1r2rErr%)rRrrr3r8 layer_keys layer_valuesrrrfrom_batch_splitsszDynamicCache.from_batch_splitsrepeatscCsJtt|D]}|j|j|dd|j|<|j|j|dd|j|<qdS)TRepeat the cache `repeats` times in the batch dimension. Used in contrastive search.rrN)r1r2r3repeat_interleaver8)rrrrrrbatch_repeat_interleavesz$DynamicCache.batch_repeat_interleaveindicescCsFtt|D]}|j||df|j|<|j||df|j|<qdS)XOnly keep the `indices` in the batch dimension of the cache. Used in contrastive search..Nr)rrrrrrbatch_select_indicessz!DynamicCache.batch_select_indicesrr?)r@rArBrCrrrGrrrErFrrmrrrrHrr%r(r)rrv FloatTensorrrrrrrrKrrrrrQs:   / "$   r dynamic_cachecCsDt|ts tdtstdt|dt|dd}tjj |S)zTFlattens DynamicCache into flat list of tensors for `torch.export.export` to consumezFThis pytree flattening function should only be applied to DynamicCachez[DynamicCache + torch.export is tested on torch 2.6.0+ and may not work on earlier versions.r3r8r3r8) isinstancer RuntimeErrorr r;r<getattrrEutils_pytree _dict_flattenr dictionaryrrr_flatten_dynamic_caches rcCs$t|dt|dd}tjj|S)Nr3r8r)rrErr_dict_flatten_with_keysrrrr _flatten_with_keys_dynamic_cache1srcontextcCs8tjj||}t}|D] \}}t|||q|Sr)rErr_dict_unflattenrrNrO)valuesrrrkvrrr_unflatten_dynamic_cache9s rcCs(t|dt|dd}tjj|dS)Nr3r8rr)rrErr tree_flatten)rspecrrrr_flatten_dynamic_cache_for_fxDsrz2.3.)serialized_type_nameflatten_with_keys_fnc seZdZdZdfdd ZdefddZdefd d Zdedee e j fd d Z d e j fddZ dde j de j dedeeeefde e j e j ff ddZdZdZZS)OffloadedCachea A drop-in replacement for DynamicCache that conserves accelerator(GPU, XPU) memory at the expense of more CPU memory. Useful for generating from models with very long context. In addition to the default accelerator stream, where all forward() computations happen, this class uses another stream, the prefetch stream, which it creates itself. Since scheduling of operations on separate streams happens independently, this class uses the prefetch stream to asynchronously prefetch the KV cache of layer k+1 when layer k is executing. The movement of the layer k-1 cache to the CPU is handled by the default stream as a simple way to ensure the eviction is scheduled after all computations on that cache are finished. r!Ncstjs tdddrtjs tdtdddrddtg|_d|_ tdddr5t ntj |_ d|_ dS)N2.7T accept_devz*OffloadedCache can only be used with a GPUz or XPU) rEcuda is_availablerxpurrroriginal_deviceprefetch_streamStreamr/rrrrres(   zOffloadedCache.__init__rcCs|t|krHtdddr|jntj|j(|j|}|j|j|dd|j|<|j |j|dd|j |<WddS1sAwYdSdS)z'Starts prefetching the next layer cacherTr non_blockingN) r2rrrErstreamrr3r7r8)rrr5rrrprefetch_layerws    "zOffloadedCache.prefetch_layercCsXt|dkr*|dt|}|j|jddd|j|<|j|jddd|j|<dSdS)z)Moves the previous layer cache to the CPUr]r r~TrN)r2r3r7r8)rrprev_layer_idxrrrevict_previous_layers z#OffloadedCache.evict_previous_layercCs|t|kr]tdddrtjntj|||j|}|j |j |}|j |}|j durN|j ||_ |d|j }|d|j }||dt|||fStdt|d|) z_Gets the cache for this layer to the device. Prefetches the next and evicts the previous layer.rTrNrr rr)r2rrE acceleratorcurrent_stream synchronizerrrrr3r8r/r7r6rr)rrr key_tensor value_tensorrrrrs        zOffloadedCache.__getitem__r/cCs|`||_dS)zTSaves the beam indices and reorders the cache when the tensor is back to its device.N)r/clonerr/rrrr9szOffloadedCache.reorder_cacherrr cCs|dkr|j|jd7_t|j|krtdt|j|kr9|j||j||j|j| |n||\}}t j ||gdd|j|<t j ||gdd|j|<|j||j|fS)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. No additional arguments are used in `OffloadedCache`. Return: A tuple containing the updated key and value states. rrzWOffloadedCache does not support model usage where layers are skipped. Use DynamicCache.r) r:rr2r3rrPr8rr5rrEr)rrrrr rrrrrr%s    zOffloadedCache.updater!Nr)r@rArBrCrrGrrrrrErFrrIr9rrrHrr%rrrKrrrrrXs*    * rc seZdZdZdeddffdd Z ddejdejd ed e e e e fde ejejff d d Zdd e edefddZddZddZZS)QuantizedCacheaK A quantizer cache similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://arxiv.org/abs/2402.02750). It allows the model to generate longer sequence length without allocating too much memory for Key and Value cache by applying quantization. The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size` for both Keys and Values, in contrast to what was described in the paper. It stores Keys and Values a list of quantized tensors (tuples in case we need to store metadata), one for each layer. Additionally, it stores the Key and Value in original precision states as a list of tensors, one for each layer. The size of each tensor is `[batch_size, num_heads, seq_len - residual_length, head_dim]` cache_configr!Ncs\tg|_g|_|j|_|j|_|j|_|j|_|j|_|j |_ |j |_ tdSr) rr_quantized_key_cache_quantized_value_cacherrrrrrr5rr rrrrs zQuantizedCache.__init__rrrr c Cs|dkr|j|jd7_t|j|krtdt|j|kra|j|j||j d|j |j||j d|jt j d|j|jd|jt j d|j|jd||}}||fS||j|}||j |}||j||g}||j||g}t j|dd}t j|dd}|j|dkr|j|jdd|jkr|j||j d|j|<|j||j d|j |<t j d|j|jd|j|<t j d|j|jd|j|<||fSt j|j||gdd|j|<t j|j||gdd|j|<||fS) NrrzWQuantizedCache does not support model usage where layers are skipped. Use DynamicCache.)axisrr5rr{r )r:rr2r3rr rP _quantize contiguousrr rrEzerosrr5r8 _dequantizerrr) rrrrr keys_to_returnvalues_to_return dequant_key dequant_valuerrrr%s:   zQuantizedCache.updatercCs*t|j|kr dS|dkr|jS|jdS)r&rr )r2r3r:r'rrrr(#szQuantizedCache.get_seq_lengthcCr")z:Quantizes a key/value using a defined quantization method.z1Make sure to implement `_quantize` in a subclass.r#)rrrrrrr,r*zQuantizedCache._quantizecCr")zDDequantizes back the tensor that was quantized by `self._quantize()`z3Make sure to implement `_dequantize` in a subclass.r#)rq_tensorrrrr0r*zQuantizedCache._dequantizerr?)r@rArBrCryrrErFrGrrrHrrr%r(rrrKrrrrr s$  + r c:eZdZdZdeddffdd ZddZd d ZZS) QuantoQuantizedCachea Quantized Cache class that uses `quanto` as a backend to perform quantization. Current implementation supports `int2` and `int4` dtypes only. Parameters: cache_config (`QuantizedCacheConfig`): A configuration containing all the arguments to be used by the quantizer, including axis, qtype and group size. Example: ```python >>> # Run pip install quanto first if you don't have it yet >>> from transformers import AutoTokenizer, AutoModelForCausalLM, QuantoQuantizedCache, QuantizedCacheConfig >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> cache_config = QuantizedCacheConfig(nbits=4) >>> past_key_values = QuantoQuantizedCache(cache_config=cache_config) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation QuantoQuantizedCache() ``` r r!Ncst|tr+ttjd}|tdkr!td|dddlm }m }m }|j dvr8t d|j |jd vrEt d |j|jd vrRt d |j|j d krY|n||_||_dS) Nzoptimum-quantoz0.2.5zzYou need optimum-quanto package version to be greater or equal than 0.2.5 to use `QuantoQuantizedCache`. Detected version rr) MaxOptimizerqint2qint4)r]r{zA`nbits` for `quanto` backend has to be one of [`2`, `4`] but got )rrzE`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got zG`axis_value` for `quanto` backend has to be one of [`0`, `-1`] but got r{)rrrr parse importlibmetadata ImportErroroptimum.quantorrrrrrrqtype optimizer)rr optimum_quanto_versionrrrrrrrQs$       zQuantoQuantizedCache.__init__cCsHtr"ddlm}|||j||j\}}|||j||||j}|SdS)Nr)quantize_weight)rr#r'r%r$r)rrrr'scale zeropointqtensorrrrrjs  zQuantoQuantizedCache._quantizecCs|Sr) dequantize)rr*rrrrssz QuantoQuantizedCache._dequantize r@rArBrCrLrrrrKrrrrr5s  rcr) HQQQuantizedCachea Quantized Cache class that uses `HQQ` as a backend to perform quantization. Current implementation supports `int2`, `int4`, `int8` dtypes. Parameters: cache_config (`QuantizedCacheConfig`): A configuration containing all the arguments to be used by the quantizer, including axis, qtype and group size. Example: ```python >>> # Run pip install hqq first if you don't have it yet >>> from transformers import AutoTokenizer, AutoModelForCausalLM, HQQQuantizedCache, QuantizedCacheConfig >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> cache_config = QuantizedCacheConfig(nbits=4, axis_key=1, axis_value=1) >>> past_key_values = HQQQuantizedCache(cache_config=cache_config) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation HQQQuantizedCache() ``` r r!Ncsdt||jdvrtd|j|jdvr td|j|jdvr-td|jt|_dS)NrzM`nbits` for `HQQ` backend has to be one of [`1`, `2`, `3`, `4`, `8`] but got )rr zA`axis_key` for `HQQ` backend has to be one of [`0`, `1`] but got zC`axis_value` for `HQQ` backend has to be one of [`0`, `1`] but got )rrrrrr HQQQuantizer quantizerrrrrrs      zHQQQuantizedCache.__init__cCsJ|jj|||j|j|j|jd\}}|j|d<|jj|||jd||fS)N)rr5rr group_sizer)metar5)r/quantizer5rrrr)rrrr*r1rrrrs zHQQQuantizedCache._quantizecCs|\}}|j||}|Sr)r/r+)rr* quant_tensorr1rrrrrszHQQQuantizedCache._dequantizer,rrrrr-ws  r-c seZdZdZdZdededdffdd Zed d Zd e j d e j d e j de j fddZ d e j d e j d e j de e j e j ffddZ ddeedefddZdeefddZ dd e j de j dedeeeefde e j e j ff ddZZS) SinkCachea A cache that as described in the [Attention Sinks paper](https://arxiv.org/abs/2309.17453). It allows the model to generate beyond the length of its context window, without losing fluency in the conversation. As it discards past tokens, the model will lose the ability to generate tokens that depend on the context that was discarded. It stores the Key and Value states as a list of tensors, one for each layer. The expected shape for each tensor is `[batch_size, num_heads, seq_len, head_dim]`. Parameters: window_length (`int`): The length of the context window. num_sink_tokens (`int`): The number of sink tokens. See the original paper for more information. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, SinkCache >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> past_key_values = SinkCache(window_length=256, num_sink_tokens=4) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation SinkCache() ``` T window_lengthnum_sink_tokensr!Ncs>tg|_g|_||_||_i|_d|_d|_d|_ dSr) rrr3r8r5r6cos_sin_rerotation_cache _cos_cache _sin_cacher:)rr5r6rrrrs  zSinkCache.__init__cCsH|dd|jddf}|d|jdddf}tj| |fddS)N.rr]r)rrEr)xx1x2rrr _rotate_halfszSinkCache._rotate_halfrcossincCs|||||}|Sr)r=)rrr>r?rotated_key_statesrrr_apply_key_rotary_pos_embsz#SinkCache._apply_key_rotary_pos_embc Cs|jd|jvrk|tj}|tj}||j|jdd}||j|jd }||j|jdd}||j|jd }||||}| |||} ||jd| |jdf|j|jd<|j|jdS)Nrr)rr7r7rEfloat32r6r unsqueeze) rrr>r? original_cos shifted_cos original_sin shifted_sinrerotation_cosrerotation_sinrrr_get_rerotation_cos_sins  z!SinkCache._get_rerotation_cos_sinrrcCs"t|j|kr dS|j|jdSr)r2r3rr'rrrr(szSinkCache.get_seq_lengthcC|jS)zfReturns the maximum sequence length of the cache object, in case of SinkCache it is the window length.)r5rrrrr) szSinkCache.get_max_cache_shaperr cCs|duri}|d}|d}|d}|duo|du}|dkr+|j|jd7_|rq|dkrq|dkr>||_||_n3|jdurN|d|_|d|_n#|jjd|jkrqtj|j|dgdd |_tj|j|dgdd |_t |j |kr|j ||j |n|jd| ||jkrtj|j ||gdd |j |<tj|j ||gdd |j |<n|j |dddd|j |j|jddf} |r |||jd|j|jd|j\} } |dur| d d|f| d |df} } || | | } |dur tj| | fd d } |j |ddddd|jf} tj| | |gdd |j |<|j |ddddd|jf}|j |dddd|j |j|jddf}tj|||gdd |j |<|j ||j |fS) a; Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. The following arguments can be used in `SinkCache`: `sin`, `cos` and `partial_rotation_size`. These arguments are used with models using RoPE, to recompute the rotation as the tokens are shifted. Return: A tuple containing the updated key and value states. Nr?r>partial_rotation_sizerrr])r.r.r)getr:rrr8r9r5rErr2r3rPr8r(r6rJrA)rrrrr r?r>rL using_rope keys_to_keeprHrI keys_pass sink_keys sink_valuesvalues_to_keeprrrr%s\          ( ""(zSinkCache.updater?r)r@rArBrC is_slidingrGr staticmethodr=rErFrArrJrr(r)rrHrr%rKrrrrr4sL     r4cseZdZdZdZddejdfdedede ede ej e dfdej d e eee e ej effd dffd d Z dd ejdejdede ee efd eejejff ddZdde ed efddZd e efddZddZZS) StaticCachea Static Cache class to be used with `torch.compile(model)` and `torch.export()`. Parameters: config (`PretrainedConfig`): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. Note that a new instance must be instantiated if a smaller batch size is used. If you are manually setting the batch size, make sure to take into account the number of beams if you are running beam search max_cache_len (`int`, *optional*): The maximum sequence length with which the model will be used. device (`torch.device` or `str`, *optional*): The device on which the cache should be initialized. If you're using more than 1 computation device, you should pass the `layer_device_map` argument instead. dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The default `dtype` to use when initializing the layer. layer_device_map (`Optional[Dict[int, Union[str, torch.device, int]]]]`, *optional*): Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is split between different gpus. You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, StaticCache >>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") >>> inputs = tokenizer(text="My name is Llama", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = StaticCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation StaticCache() ``` TNrUmax_batch_sizerr5rlayer_device_mapr!c st||_|dur|jn||_t|dr|jn|j|j|_||_ t |dddur/|jn|j |_ g|_ g|_ |j|j |j|jf}|durLt|nd}t|jD]7}|dur^||} n|} tj||j | d} tj||j | d} tj| tj| |j | |j | qSdS)Nhead_dimnum_key_value_headsr)rrrWmax_position_embeddingsrr=rY hidden_sizenum_attention_heads_dtyperrZr3r8rEr5r1num_hidden_layersr_dynamomark_static_addressrP) rrUrWrr5rrX cache_shaper layer_devicenew_layer_key_cachenew_layer_value_cacherrrrs2     zStaticCache.__init__rrrr cCs|duri}|d}|j|}|j|}||j}||j}|dur3||||||fSz|d|||d||W||fStyh||dddd|f<||dddd|f<Y||fSw)a' Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. It is VERY important to index using a tensor, otherwise you introduce a copy to the device. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. The `StaticCache` needs the `cache_position` input to know how where to write in the cache. Return: A tuple containing the updated key and value states. Ncache_positionr])rMr3r8r7rcopy_ index_copy_r$)rrrrr rfk_outv_outrrrr%s(         zStaticCache.updatercCs|j|djddS)MReturns the sequence length of the cached states that were seen by the model.rrrr)r3anysumr'rrrr(szStaticCache.get_seq_lengthcCrKrrrrrrr)zStaticCache.get_max_cache_shapecC4tt|jD]}|j||j|qdS4Resets the cache values while preserving the objectsNr1r2r3zero_r8r'rrrresetzStaticCache.resetrr?)r@rArBrCrDrErBr rGrr r5rHrrrrFrrr%r(r)rvrKrrrrrVrsJ+1 2rVcseZdZdZdZdZddejdfdede de e de ej e dfdejd e ee e e ej e ffd dffd d Z dd ejdejde de ee efd eejejff ddZd e e fddZddZZS)SlidingWindowCachea; Sliding Window Cache class to be used with `torch.compile` for models like Mistral that support sliding window attention. Every time when we try to update the cache, we compute the `indices` based on `cache_position >= self.config.sliding_window - 1`, if true(which means the cache can not hold all the old key value states and new states together because of the sliding window constraint), we need to do a cycle shift based on `indices` to replace the oldest states by the new key value states passed in. The `to_shift` is only true once we are above sliding_window. Thus with `sliding_window==64`: indices = (slicing + to_shift[-1].int()-1) % self.config.sliding_window tensor([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 0]) We overwrite the cache using these, then we always write at cache_position (clamped to `sliding_window`) Parameters: config (`PretrainedConfig`): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. Note that a new instance must be instantiated if a smaller batch size is used. max_cache_len (`int`, *optional*): The maximum sequence length with which the model will be used. device (`torch.device` or `str`, *optional*): The device on which the cache should be initialized. If you're using more than 1 computation device, you should pass the `layer_device_map` argument instead. dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The default `dtype` to use when initializing the layer. layer_device_map (`Optional[Dict[int, Union[str, torch.device, int]]]]`, *optional*): Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is split between different gpus. You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, SlidingWindowCache >>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3") >>> inputs = tokenizer(text="My name is Mistral", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = SlidingWindowCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation SlidingWindowCache() ``` TNrUrWrr5rrXr!csDt|dr |jdurtdt|j|}tj||||||ddS)Nsliding_windowSetting `cache_implementation` to 'sliding_window' requires the model config supporting sliding window attention, please check if there is a `sliding_window` field in the model config and it's not set to None.)rUrWrr5rrX)r=ryrminrrrrUrWrr5rrXrrrrJs   zSlidingWindowCache.__init__rrrr c Cs|duri}|d}|j|}|j|}||j}||j}|jd|jkrc|dddd|j dddf}|dddd|j dddf}|j||7<|j||7<||fStj|jtj |j d d}| d|jd}||jdk} || d d|j} |dddd| f}|dddd| f}z|d|||d||Wnty||dddd|f<||dddd|f<Ynw|j||j||j||7<|j||7<||fS)Nrfrrr rr])rMr3r8r7rrrrEoneslongr5cumsumclamprGrhr$ru) rrrrr rfrirjslicingto_shiftrrrrr%cs>     $$ zSlidingWindowCache.updatecCrKrrorrrrr)rpz&SlidingWindowCache.get_max_cache_shapecC4tt|jD]}|j||j|qdSrrtr'rrrrvzSlidingWindowCache.resetr)r@rArBrCrTrDrErBr rGrr r5rHrrrrFrrr%r)rvrKrrrrrxsJ6 3rxcs2eZdZdZdedeffdd Zdedeee j fdd Z d d Z deee j ee j ffd d Z e d.deeee jddfddZd/deedefddZddZde jfddZdefddZdefddZd ed!edd"fd#d$Zed%edddfd&d'Zd(efd)d*Zd+e j fd,d-ZZS)0EncoderDecoderCachea Base, abstract class for all encoder-decoder caches. Can be used to hold combinations of self-attention and cross-attention caches. Example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM, DynamicCache, EncoderDecoderCache >>> model = AutoModelForCausalLM.from_pretrained("openai/whisper-small") >>> processor = AutoProcessor.from_pretrained("openai/whisper-small") >>> inputs = processor(audio=YOUR-AUDIO, return_tensors="pt") >>> # Prepare cache classes for encoder and decoder and pass it to model's forward >>> self_attention_cache = DynamicCache() >>> cross_attention_cache = DynamicCache() >>> past_key_values = EncoderDecoderCache(self_attention_cache, cross_attention_cache) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation EncoderDecoderCache() ``` self_attention_cachecross_attention_cachecs\t||_||_t|jdd|_i|_tt|j D]}t | |dk|j|<qdS)NrDFr) rrrrrrD is_updatedr1r2r3boolr()rrrrrrrrs zEncoderDecoderCache.__init__rr!cCsP|t|kr|jj||jj||jj||jj|fStdt|d|r)r2rr3r8rrr'rrrrs     zEncoderDecoderCache.__getitem__cCrr)r2rrrrrrrzEncoderDecoderCache.__len__cCsRd}t|jdkr"t|j|jD] \}}|||f7}q|S|j}|S)z\Converts the `EncoderDecoderCache` instance into its equivalent in the legacy cache format.rr)r2rziprr)rr self_attn cross_attnrrrrs  z#EncoderDecoderCache.to_legacy_cacheNrcCs|ttd}|durFtt|D]3}||dd\}}|j|||t||dkrE||dd\}}|j|||d|j|<q|S)zUConverts a cache in the legacy cache format into an equivalent `EncoderDecoderCache`.)rrNr]T)rr1r2rr%rrrrrrrs z%EncoderDecoderCache.from_legacy_cachercCs |j|S)r&)rr(r'rrrr(s z"EncoderDecoderCache.get_seq_lengthcCst|jdr |jt|jdr|jnt|jds4t|jds4td|jd|jd|jD]}d|j|<q7dS)NrvzNeither self nor cross-attention cache have valid `.reset()` methods. `.reset()` should only be called on compatible cache classes, such as `StaticCache` or `SlidingWindowCache`. Got " for the self attention cache and  for the cross attention cache.F)r=rrvrr__str__rr'rrrrvs      zEncoderDecoderCache.resetr/cCs|j||j|dS)r0N)rr9rrrrrr9 s z!EncoderDecoderCache.reorder_cachemethodcCsDt|jtr t|jts td|d|jd|jddS)N`z)` is only defined for dynamic cache, got rr)rrrrrr)rrrrrcheck_dynamic_caches  z'EncoderDecoderCache.check_dynamic_cachemaximum_lengthcCs||jj|j|dS)zCrop the past key values up to a new `maximum_length` in terms of tokens. `maximum_length` can also be negative to remove `maximum_length` tokens. This is used in assisted decoding and contrastive search.N)rrr@r)rrrrrrszEncoderDecoderCache.croprrzList[EncoderDecoderCache]cCsV||jj|j||}|j||}g}t||D] \}}|t||q|S)r)rrr@rrrrPr)rrrrrrrrrrrr#szEncoderDecoderCache.batch_splitrcst}t}tt|dD]Htjfdd|Ddd}tjfdd|Ddd}|||tjfdd|Ddd}tjfdd|Ddd}|||q|||S)rrcg|]}|jjqSr)rr3rrrrr6z9EncoderDecoderCache.from_batch_splits..rcrr)rr8rrrrr7rcrr)rr3rrrrr:rcrr)rr8rrrrr;r)rr1r2rErr%)rRrrrrrrrrr/s z%EncoderDecoderCache.from_batch_splitsrcC*||jj|j||j|dS)rN)rrr@rr)rrrrrr? z+EncoderDecoderCache.batch_repeat_interleavercCr)rN)rrr@rr)rrrrrrErz(EncoderDecoderCache.batch_select_indicesrr?)r@rArBrCrrrGrrrErFrrrrvrrrr(rvrIr9rHrrrrrrrKrrrrrs. "    rcseZdZdZddejdfdededeede ej e dfdej dee ee e ej effd dffd d Zd d ZddZ ddejdejdedee e efd eejejff ddZd eefddZddeefddZddZZS) HybridCachea: Hybrid Cache class to be used with `torch.compile` for Gemma2 models that alternate between a local sliding window attention and global attention in every other layer. Under the hood, Hybrid Cache leverages ["SlidingWindowCache"] for sliding window attention and ["StaticCache"] for global attention. For more information, see the documentation of each subcomponeent cache class. Parameters: config (`PretrainedConfig): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. Note that a new instance must be instantiated if a smaller batch size is used. max_cache_len (`int`, *optional*): The maximum sequence length with which the model will be used. device (`torch.device` or `str`, *optional*): The device on which the cache should be initialized. If you're using more than 1 computation device, you should pass the `layer_device_map` argument instead. dtype (torch.dtype, *optional*, defaults to `torch.float32`): The default `dtype` to use when initializing the layer. layer_device_map (`Optional[Dict[int, Union[str, torch.device, int]]]]`, *optional*): Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is split between different gpus. You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, HybridCache >>> model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b") >>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b") >>> inputs = tokenizer(text="My name is Gemma", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = HybridCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation HybridCache() ``` NrUrWrr5rrXr!cstt|dr|jdurtd||_||_t|dr!|jn|j|j |_||_ |j dur3|j n|j |_ t|dr?|j ndt jfddt|jDt jd|_g|_g|_|j|j ||jf}|j|j t|j||jf}|dur~t|tr~t |nd}t|jD]@} |dur|| } n|} |j| s|n|} t j| |j | d } t j| |j | d } t j| t j| |j| |j| qdS) NryrzrYsliding_window_patternr]cg|] }t|dqSr rrqr layer_switchrrrrz(HybridCache.__init__..)rr)rrr=ryrrrWrYr\r]r^rZrrErr1r_rrTr3r8r{rrHr5rr`rarP)rrUrWrr5rrXglobal_cache_shapesliding_cache_shaperrcrbrdrerrrr|sJ       zHybridCache.__init__c Csl|jd|kr?|dddd| dddf}|dddd| dddf}|j||7<|j||7<||fStj|tj|jdd}|d|d}||dk} || d d|} |dddd| f}|dddd| f}||dddd|f<||dddd|f<|j| |j| |j||7<|j||7<||fS)Nrrr r) rr3r8rEr}r~r5rrrGru) rrfrrrrirjrrrrrrr_sliding_updates&"" zHybridCache._sliding_updatecCH||dddd|f<||dddd|f<||j|<||j|<||fSrrrrfrrrrirjrrrr_static_update   zHybridCache._static_updaterrrr c Cs|duri}|d}|d}|j|j|jkr%|j||j|j|<|j|j|jkr:|j||j|j|<|j|}|j|}||j}||j}|rV|j} n|j} | |||||||jdS)Nrfryr]) rMr3r5r7r8rrrr) rrrrr rfryrirj update_fnrrrr%s0      zHybridCache.updatecCrKrrorrrrr)rpzHybridCache.get_max_cache_shapercCs*|dkrtd|j|djddSNrz`get_seq_length` on `HybridCache` may get inconsistent results depending on the layer index. Using the `layer_idx` argument is not supported.rlrr)rr3rmrnr'rrrr(s zHybridCache.get_seq_lengthcCrqrrrtr'rrrrvrwzHybridCache.resetrr?)r@rArBrCrErBr rGrr r5rHrrrrrrFrrr%r)r(rvrKrrrrrLsL39  ' rcseZdZdZdZddejdfdedede ede ej e dfdej d e eee e ej effd dffd d Zd dZddZddZ d dejdejdede ee efd eejejff ddZd e efddZd!de efddZddZZS)"HybridChunkedCachea; Hybrid Cache class to be used with `torch.compile` for Gemma2 models that alternate between a local sliding window attention and global attention in every other layer. Under the hood, Hybrid Cache leverages ["SlidingWindowCache"] for sliding window attention and ["StaticCache"] for global attention. For more information, see the documentation of each subcomponeent cache class. Parameters: config (`PretrainedConfig): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. Note that a new instance must be instantiated if a smaller batch size is used. max_cache_len (`int`, *optional*): The maximum sequence length with which the model will be used. device (`torch.device` or `str`, *optional*): The device on which the cache should be initialized. If you're using more than 1 computation device, you should pass the `layer_device_map` argument instead. dtype (torch.dtype, *optional*, defaults to `torch.bfloat16`): The default `dtype` to use when initializing the layer. layer_device_map (`Optional[Dict[int, Union[str, torch.device, int]]]]`, *optional*): Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is split between different gpus. You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, HybridCache >>> model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b") >>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b") >>> inputs = tokenizer(text="My name is Gemma", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = HybridCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation HybridCache() ``` TNrUrWrr5rrXr!cstt|dr|jdurt|dd|_n|j|_||_||_t|d|j|j |_ ||_ t|dr=|j |_ nt|ddfdd t|jD|_ g|_g|_d d t|jD|_dS) Nryattention_chunk_sizei rYno_rope_layersrr]crrrrrrrr]rz/HybridChunkedCache.__init__..cSg|]}dqSr?rrqrrrrra)rrr=ryrget_text_configrrWr\r]rYr^rrTr1r_r3r8cumulative_lengthr|rrrrFs   zHybridChunkedCache.__init__c Cst|j|kr dS|jd}|j}|j||j|jf}|j||j|jf}|j|r*|n|}t j ||j |d}t j ||j |d} t j |t j | |j||j| dS)Nr r)r2r3rr5rWrrYryrTrErr^r`rarPr8) rrrrZr5rrrbrdrerrrinitialise_cache_layercs"    z)HybridChunkedCache.initialise_cache_layerc Cs|j|}|j||jd7<||k} | rgtj|ddddddddf|fdd} tj|ddddddddf|fdd} |jddkrf|j|| |j|| |j||j|fSna| s||jd|kr|dkr{|} |} nMtj|ddddd|ddf|fdd} tj|ddddd|ddf|fdd} n|j|d|||j|d|||j||j|fS|j|| dddd| dddf|j|| dddd| dddf| | fS)Nrr rr]r)rrrErr3rgr8rh) rrfrrrrirjrris_fullfull_key_statesfull_value_statesrrrrzs. ...0..z"HybridChunkedCache._sliding_updatecCrrrrrrrrrz!HybridChunkedCache._static_updaterrrr c Cs|duri}|d}||||j|}|j|}||j}||j}|j|r0|j}n|j}||||||||j dS)Nrfr]) rMrr3r8r7rrTrrr) rrrrr rfrirjrrrrr%s(       zHybridChunkedCache.updatecCrKrrorrrrr)rpz&HybridChunkedCache.get_max_cache_shapercCs<|dkrtdt|jdkrdS|j|djddSr)rr2r3rmrnr'rrrr(sz!HybridChunkedCache.get_seq_lengthcCsNtt|jD]}|j||j|qddtt|jD|_dS)rscSrr?rrrrrrrz,HybridChunkedCache.reset..N)r1r2r3rur8rr'rrrrvszHybridChunkedCache.resetrr?)r@rArBrCrDrEbfloat16r rGrr r5rHrrrrrrrFrrr%r)r(rvrKrrrrrsP-"    rc @seZdZdZdZejdfdededej de ej e dffdd Z d ed ejd ejd ejfddZd edejfddZddZdS) MambaCachea Cache for mamba model which does not have attention mechanism and key value states. Arguments: config (`PretrainedConfig): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. Note that a new instance must be instantiated if a smaller batch size is used. dtype (`torch.dtype`, *optional*, defaults to `torch.float16`): The default `dtype` to use when initializing the layer. device (`torch.device` or `str`, *optional*): The device on which the cache should be initialized. Should be the same as the layer. Example: ```python >>> from transformers import AutoTokenizer, MambaForCausalLM, MambaCache >>> model = MambaForCausalLM.from_pretrained("state-spaces/mamba-130m-hf") >>> tokenizer = AutoTokenizer.from_pretrained("state-spaces/mamba-130m-hf") >>> inputs = tokenizer(text="My name is Mamba", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> past_key_values = MambaCache(config=model.config, max_batch_size=1, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values MambaCache() ``` TNrUrWrr5cCs||_||_|j|_|j|_|j|_g|_g|_|dur!t |nd}t |j D]6}t j |j|j|j||jd}t j |j|j|j||jd}t j|t j||j||j|q(dS)Nr5r)rWr^intermediate_size state_sizessm_state_size conv_kernelconv_kernel_size conv_states ssm_statesrEr5r1r_rr`rarP)rrUrWrr5r conv_state ssm_staterrrrs8   zMambaCache.__init__rnew_conv_staterfr!cCs|j|j|jkr|j||j|j|<|j|}|d|jd}|jddd}|j|j|jd|dddd|f<|j||j||7<|j|S)Nrr r)shiftsdimsr)rr5r7rrrollrru)rrrrfrrrrupdate_conv_state%s $ zMambaCache.update_conv_state new_ssm_statecCs"||j|j|j|<|j|Sr)r7rr5)rrrrrrupdate_ssm_state6s zMambaCache.update_ssm_statecCrr)r1r2rrurr'rrrrv:rzMambaCache.reset)r@rArBrCrDrErr rGrr r5rHrrFrIrrrvrrrrrs2 %  rcs^eZdZdZdZdeddfdedede ede e ejfd e ej d e e ejfd e e ee e ejeffd dffd d Z d&dejdejdede e e efd eejejff ddZd'de ed efddZd e efddZd(ddZed efddZdeedfdejd eejejffd d!Zded dfd"d#Zded dfd$d%ZZS))OffloadedStaticCachea3 Static cache class to be used with `torch.compile(model)` that offloads to the CPU or another device. Args: config (`PretrainedConfig): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. max_cache_len (`int`): The maximum sequence length with which the model will be used. device (`Union[str, torch.device]`): The device on which the cache should be initialized. If you're using more than 1 computation device, you should pass the `layer_device_map` argument instead. dtype (`torch.dtype`, *optional*): The default `dtype` to use when initializing the cache. offload_device (`Union[str, torch.device]`, *optional*, defaults to `cpu`): The device to offload to. Defaults to CPU. layer_device_map (`Dict[int, Union[str, torch.device, int]]`, *optional*): Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between differents gpus. You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, OffloadedStaticCache >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> inputs = tokenizer(text="My name is GPT2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = OffloadedStaticCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> past_kv_length = outputs.past_key_values # access cache filled with key/values from generation ``` TNr~rUrWrr5roffload_devicerXr!csxtt|||_|dur|jn||_|durt|nt|d|_t||_|dur1|ntj |_ t |dr=|j n|j |j}t|dddurN|jn|j} || |j|f} g|_g|_t|jD] } | dkrl|jn|j}|| |\} } |j| |j| qcg|_g|_tdD]} || |j\} } |j| |j| qd|_|jjdkrtj|_dSd|_dS)NrrYrZr]r)rrrrWr[rrEr5rrBr^r=rYr\r]rrZr3r8r1r__create_key_value_cache_tensorsrP_device_key_cache_device_value_cacher:typerr_prefetch_stream)rrUrWrr5rrrXrYrZrbrr3r8rrrros6 "    &zOffloadedStaticCache.__init__rrrr cCs|dkr|j|jd7_|jd}|jd}n|jdur)tj|j |j|j |d@}|j |d@}| |d|durG| dnd}|durw|||||dkrs|j|||j|j|||j||fSz|d|||d||Wnty||dddd|f<||dddd|f<Ynw|dkr||j}||j}||j}z|j|d|||j|d||W||fSty||j|dddd|f<||j|dddd|f<Y||fSw||fS)a0 Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. It is VERY important to index using a tensor, otherwise you introduce a copy to the device. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, *optional*): Additional arguments for the cache subclass. The `OffloadedStaticCache` needs the `cache_position` input to know how where to write in the cache. Return: A tuple containing the updated key and value states. rrNr rfr])r:rr3r8rrErdefault_streamr5 wait_streamrr_prefetch_layerrMrgr7rrhr$)rrrrr rirjrfrrrr%sL          zOffloadedStaticCache.updatercCrK)rkr:r'rrrr(z#OffloadedStaticCache.get_seq_lengthcCrK)z9Returns the maximum sequence length of the cached states.rorrrrr) sz(OffloadedStaticCache.get_max_cache_shapecCs:d|_tt|jD]}|j||j|q dS)z5Resets the cache values while preserving the objects.rN)r:r1r2r3rur8r'rrrrv s zOffloadedStaticCache.resetcCrKrrrrrrr> rz OffloadedStaticCache.seen_tokensr.cCsV|tdk}tj||j||d}tj||j||d}tj|tj|||fS)a8Creates K/V cache tensors on a device. Pins memory for CPU tensors. Marks them as static addresses for non-CPU tensors. Args: shape (`Tuple[int, ...]`): Shape. device (`torch.device`): Device. Returns: Key and value cache tensors as a tuple. r~)rr5 pin_memory)rEr5rr^r`ra)rrr5 is_cpu_devicer3r8rrrr s   z4OffloadedStaticCache._create_key_value_cache_tensorscCsh|t|jkr dS|jdur-tj|j||WddS1s&wYdS||dS)zMPrefetch a layer to the device. Needs to be called in order of layer indices.N)r2r3rrErr_prefetch_layer_in_contextr'rrrr4 s  "z$OffloadedStaticCache._prefetch_layercCs@|j|d@j|j|dd|j|d@j|j|dddS)z6Performs the actual copy of the layer to device cache.r TrN)rrgr3rr8r'rrrrB s"z/OffloadedStaticCache._prefetch_layer_in_contextrr?r )r@rArBrCrDrEr5r rGrr rHrrrrFrrr%r(r)rvrJr>rrrrKrrrrrAsb+   ? S    r)sv $     N`b'M   ZB>>-KHd