o  hKC@sddlmZddlZddlmZddlmZer)ddlmZm Z ddl m Z m Z Gdd d ej jZ  dd ed eejd eejfd dZGdddej jZGdddej jZGdddej jZdS))OptionalN)GenerationConfig)is_torch_available)PreTrainedModel StaticCache)is_torch_greater_or_equal"is_torch_greater_or_equal_than_2_3c sbeZdZdZdeffdd ZdejdejfddZe d ej j d ejd e d ejfd dZ ZS)$TorchExportableModuleWithStaticCachea A wrapper module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for use with static caching. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Note: This class is specifically designed to support export process using `torch.export` in a way that ensures the model can be further lowered and run efficiently in `ExecuTorch`. modelcs$t|jdurtd|jjstd|jjdkr td||_t|jj|jjj j |jjj j |jjj j |jj d|_tt|jjD] }|jd||jj|dd |jd ||jj|dd qEtd d |jjjD|_|jrttj|jj |jj tjd }|jd|dd dSdS)a Initializes the wrapper module with the pretrained model. Args: model (`PreTrainedModel`): The pretrained model to wrap. The model must have caching enabled and use a 'static' caching implementation. Raises: AssertionError: If the pretrained model does not have caching enabled or if it does not use a 'static' caching implementation in `model.generation_config`. NzkThe model must have a generation config to be exported with static caching. Please set `generation_config`.zvThe model must have caching enabled to be exported with static caching. Please set `generation_config.use_cache=True`.staticzThe model must use a 'static' caching implementation to be exported with static caching. Please set `generation_config.cache_implementation='static'`.configmax_batch_size max_cache_lendevicedtype key_cache_F persistent value_cache_css|]}d|vVqdS)CausalLMN).0archrrx/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/integrations/executorch.py Ssz@TorchExportableModuleWithStaticCache.__init__..rmask)super__init__generation_configAssertionError use_cachecache_implementationr rr cache_config batch_sizerrr static_cacherangelen key_cacheregister_buffer value_cacheany architectures is_causaltorchtrilonesbool)selfr i causal_mask __class__rrr &sF      z-TorchExportableModuleWithStaticCache.__init__ input_idscache_positionc CsR|j\}}|jr|j|d|fnd}|d}|j}|j|||||dd}|jS)a Forward pass of the module, which is compatible with the ExecuTorch runtime. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. This forward adapter serves two primary purposes: 1. **Making the Model `torch.export`-Compatible**: The adapter hides unsupported objects, such as the `Cache`, from the graph inputs and outputs, enabling the model to be exportable using `torch.export` without encountering issues. 2. **Ensuring Compatibility with `ExecuTorch` runtime**: The adapter matches the model's forward signature with that in `executorch/extension/llm/runner`, ensuring that the exported model can be executed in `ExecuTorch` out-of-the-box. NrT)r9attention_mask position_idsr:past_key_valuesr#)shaper/r unsqueezer'r logits) r4r9r:_seqlen attn_maskr<r=outsrrrforward^s  z,TorchExportableModuleWithStaticCache.forwardexported_programprompt_token_idsmax_new_tokensreturnc Csb|jd}||}|D]\}}|dr"|jd}t||}nq g}tt||D]'} |j|dd| | dftj| gtj dd} | |d|  q,tj | dddddfdd  } | | t ||kr|jtj| ggtj dtjt |gtj dd} tj | dddddfdd  } | | t ||ksqtj|gtj dS) a Generate a sequence of tokens using an exported program. This util function is designed to test exported models by simulating the generation process. It processes the input prompt tokens sequentially (no parallel prefill). This generate function is not intended to replace the original `generate` method, and the support for leveraging the original `generate` is potentially planed! Args: exported_program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. prompt_token_ids (`torch.Tensor`): Tensor representing the input prompt token IDs. max_new_tokens (`int`): Maximum number of new tokens to generate. Note that the total generation length is limited by both `max_new_tokens` and the model's cache size. Returns: torch.Tensor: A tensor containing the generated sequence of token IDs, including the original prompt tokens. r*rNr)r9r:rdim)r> named_buffers startswithminr(modulerEr0tensorlongappenditemargmaxr)) rFrGrHprompt_token_lenmax_generation_length buffer_namebufferrresponse_tokens input_posresult current_tokenrrrgenerates4    $  $  z-TorchExportableModuleWithStaticCache.generate)__name__ __module__ __qualname____doc__rr r0TensorrE staticmethodexportExportedProgramintr_ __classcell__rrr7rr s 8$r r example_input_idsexample_cache_positioncCststdddl}|N|dur|n |jdgg|jd}|dur%|n|jdg|jd}tdrB|jjt||fd|idd }n|jj j t||fd|id dd }|WdS1s_wYdS) a6 Convert a `PreTrainedModel` into an exportable module and export it using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`torch.Tensor`): Example input token id used by `torch.export`. example_cache_position (`torch.Tensor`): Example current cache position used by `torch.export`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. ztorch >= 2.3 is required.rNrKrz2.5.0r:T)argskwargsstrictF)rlrm pre_dispatchrn) r ImportErrortorch.export._traceno_gradrRrSrrfr _trace_export)r rjrkr0rFrrrconvert_and_export_with_caches0  $ruc(eZdZdZfddZddZZS) Seq2SeqLMEncoderExportableModulez A wrapper module designed to make a Seq2Seq LM encoder exportable with `torch.export`. This module ensures that the exported encoder model is compatible with ExecuTorch. cst||_dS)N)rr encoder)r4 encoder_modelr7rrr s  z)Seq2SeqLMEncoderExportableModule.__init__cCs|j|djS)N)r9)rxlast_hidden_state)r4r9rrrrEsz(Seq2SeqLMEncoderExportableModule.forwardr`rarbrcr rErirrr7rrws rwcrv)/Seq2SeqLMDecoderExportableModuleWithStaticCachez A wrapper module designed to make a Seq2Seq LM decoder exportable with `torch.export`, specifically for use with static caching. This module ensures the exported decoder is compatible with ExecuTorch. cst||_|j|_|j|_t|j||dtjd|_ t t |j j D] }|j d||j j |dd|j d||j j|ddq&dS)Ncpur rFrr)rr get_decoderdecoderlm_headrrr0float32r'r(r)r*r+r,)r4r max_static_cache_lengthr&r5r7rrr s    z8Seq2SeqLMDecoderExportableModuleWithStaticCache.__init__cCs(|j|||jd|d}||d}|S)NT)r9encoder_hidden_statesr=r#r:r)rr'r)r4decoder_input_idsrr:outputs lm_logitsrrrrEs z7Seq2SeqLMDecoderExportableModuleWithStaticCache.forwardr{rrr7rr|s r|csBeZdZ dfdd ZddZd d Zdd d ZddZZS)Seq2SeqLMExportableModulerKr csPt||_||_|j|_||_td||||dd|_d|_ d|_ dS)NT)r&r)r# max_lengthr$r%) rr full_model get_encoderrxrmax_hidden_seq_lengthrr!exported_encoderexported_decoder)r4r r&rr$max_cache_lengthr7rrr &s   z"Seq2SeqLMExportableModule.__init__cCsrt|jd}tjjd|jd}ttjj||fdd|iidd}Wd|S1s2wY|S)Nr}encoder_seq_lengthmaxr9rKTdynamic_shapesrn) rwrxtoevalr0rfDimrrr)r4encoder_input_idswrapped_encoder seq_len_dimrrrr_export_encoder;s  z)Seq2SeqLMExportableModule._export_encodercCst|j|jjj|jjjdd}tj j d|j d}t tj j ||||fdd|idddd}Wd|S1s?wY|S) N)r rr&r}encoder_hidden_seq_lengthrrK)rrr:Tr) r|rr!r%rr&rrr0rfrrrr)r4rrr:wrapped_decoderencoder_seq_len_dimrrrr_export_decoderIs.    z)Seq2SeqLMExportableModule._export_decoderNc Cs|dur|ntjdtjd}|dur|n tjdggtjd}|dur$|ntjdgtjd}|dur3|ntj|jjjd|jj ftj d}| ||_ | ||||_|S)N)rK rrr)r0r2rSrRzerosr!r%r&rd_modelrrrrr) r4rrrr:example_encoder_input_idsexample_decoder_input_idsrkexample_encoder_hidden_statesrrrrffs  z Seq2SeqLMExportableModule.exportc Cst`|j|}tjdggtjd}dg}t|dD];}|j||tj|gtjd}tj|dddddfdd }| |tj|ggtjd}||j j krZnq|WdS1sgwYdS)NrrrKrJrL) r0rrrrQrRrSr(rrVrUrTr eos_token_id) r4rGrHencoder_outputr generated_idsr5r@ next_tokenrrrr_}s $  $z"Seq2SeqLMExportableModule.generate)rKrr r)NNNN) r`rarbr rrrfr_rirrr7rr%s r)NN)typingrr0+transformers.generation.configuration_utilsrutils.import_utilsr transformersrrtransformers.pytorch_utilsrr nnModuler rdrurwr|rrrrrs*    5-