o hgp @sTUddlZddlmZmZmZddlmZddlmZm Z m Z m Z m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZddlZd d gZe dZedZejjZddZiZ e!eefe"d<ddZ#dUde e eefge eefffdd Z$e$ej%ddde&fddZ'e$ej(dVde&fddZ)e$ej*dVde&fddZ+e$ej,dVde&fdd Z-e$ej.     dWde&fd!d"Z/ dUd#e0e&d$e0e&d%e0e&d&e1de&f d'd(Z2e$ej3ej4gddde&fd)d*Z5e$ej6de&fd+d,Z7d-d.Z8e$ej9ej:ej;gddde&fd/d0Ze>e&d4fe>e&d4fe>e&d4fe e>e&d4fffd5d6Z?dd3dee>e>e&d4fe>e&d4fe>e&d4fe e>e&d4fffd7d8Z@e$ejAd9d:ddde&fd;d<ZBe$ejCd9d:de&fd=d>ZDd?d@ZEe$ejFejGejHgddde&fdAdBZIe$ejJd9d:de&fdCdDZKe$ejLd9d:de&fdEdFZMiej%e'ej(e)ej*e+ej,e-ej.e/ej3e5ej4e5ej6e7ej9e<ej:e<ej;e<ejFeIejGeIejHeIejAeBejCeDejJeKejLeMiZ dGdHZNgdIZOdJdKZPdLdMZQdNdOZRdPdQZSGdRd d ZTGdSdTdTeZUdS)XN)tree_map tree_flattentree_unflatten) ModuleTracker)AnyOptionalUnionTypeVarCallable)Iterator) ParamSpec) defaultdict)TorchDispatchModeprodwrapsFlopCounterModeregister_flop_formula_T_PcCst|tjr |jS|SN) isinstancetorchTensorshape)irl/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/torch/utils/flop_counter.py get_shapes r flop_registrycstddfdd }|S)N)out_valcs(tt|||f\}}}|d|i|S)N out_shape)rr )r"argskwargsr#frrnfszshape_wrapper..nfrr'r(rr&r shape_wrappersr*Freturncs,dtttfdtttfffdd }|S)N flop_formular+cs,stfdd}tjj|S)NcsHt|tjjstd|dt||tvrtd|t|<dS)Nzlregister_flop_formula(targets): expected each target to be OpOverloadPacket (i.e. torch.ops.mylib.foo), got z which is of type zduplicate registrations for )rr_opsOpOverloadPacket ValueErrortyper! RuntimeError)targetr,rrregister(s z=register_flop_formula..register_fun..register)r*rutils_pytree tree_map_)r,r4get_rawtargetsr3r register_fun$s   z+register_flop_formula..register_fun)r rr)r:r9r;rr8rr#s()r#c Os,|\}}|\}}||ksJ||d|S)zCount flops for matmul.r) a_shapeb_shaper#r$r%mkk2nrrrmm_flop9s rCcK t||S)zCount flops for addmm.rC self_shaper=r>r#r%rrr addmm_flopDs rHc KsD|\}}}|\}}} ||ksJ||ksJ||| d|} | S)z"Count flops for the bmm operation.r<r) r=r>r#r%br?r@b2rArBfloprrrbmm_flopIs    rLcKrD)z&Count flops for the baddbmm operation.rLrFrrr baddbmm_flopVs rNc Ks t||S)zCount flops for _scaled_mm.rE) r=r> scale_a_shape scale_b_shape bias_shapescale_result_shape out_dtypeuse_fast_accumr#r%rrr_scaled_mm_flop]s rUx_shapew_shaper# transposedc CsL|d}|r|n|dd}|^}}} t|t||||d} | S)aCount flops for convolution. Note only multiplication is counted. Computation for bias are ignored. Flops for a transposed convolution are calculated as flops = (x_shape[2:] * prod(w_shape) * batch_size). Args: x_shape (list(int)): The input shape before convolution. w_shape (list(int)): The filter shape. out_shape (list(int)): The output shape after convolution. transposed (bool): is the convolution transposed Returns: int: the number of flops rr<Nr) rVrWr#rX batch_size conv_shapec_outc_in filter_sizerKrrrconv_flop_countns   r^c Ost||||dS)zCount flops for convolution.rX)r^) rVrW_bias_stride_padding _dilationrXr#r$r%rrr conv_flopsrdc Csdd} d} | drt| d}| t|||| 7} | drIt| d}|r9| t| || || |dd7} | S| t| || || |dd7} | S)NcSs |d|dgt|ddS)Nrrr<)list)rrrrts zconv_backward_flop..trrFr_)r r^)grad_out_shaperVrWr`rarbrcrX_output_padding_groups output_maskr#rf flop_countgrad_input_shapegrad_weight_shaperrrconv_backward_flopsF    rncCs|\}}}}|\}}} } |\} } } }||kr| kr8nJ||kr)| kr8nJ|| kr8| | kr8|| ks:Jd}|t||||f|||| f7}|t|||| f||| |f7}|S)z^ Count flops for self-attention. NB: We can assume that value_shape == key_shape rrM) query_shape key_shape value_shaperIhs_qd_q_b2_h2s_k_d2_b3_h3_s3d_v total_flopsrrrsdpa_flop_counts   P""r~cOs t|||S)Count flops for self-attention.r~)rorprqr#r$r%rrr sdpa_flops rcCsRddlm}ddlm}t|||fs|jjdkr|S|g| ddS)z If the offsets tensor is fake, then we don't know the actual lengths. In that case, we can just assume the worst case; each batch has max length. r) FakeTensor)FunctionalTensormetar) torch._subclasses.fake_tensorr#torch._subclasses.functional_tensorrrdevicer0difftolistsize)offsetsmax_lenrrrrr_offsets_to_lengthss   r)grad_out.ccs&|durt|jdksJt|jdksJ|dus#|j|jks#J|j\}} } |j\}} } |j\}} }|dus;J|dusAJ|j|jksIJt||}t||}t||D]%\}}d| || f}d| || f}d| ||f}|durt|nd}||||fVqXdS|j|j|j|dur|jndfVdS)a; Given inputs to a flash_attention_(forward|backward) kernel, this will handle behavior for NestedTensor inputs by effectively unbinding the NestedTensor and yielding the shapes for each batch element. In the case that this isn't a NestedTensor kernel, then it just yields the original shapes. Nrlenrrzip)querykeyvaluer cum_seq_q cum_seq_kmax_qmax_k_h_qrth_kd_kh_vr| seq_q_lengths seq_k_lengths seq_q_len seq_k_lennew_query_shape new_key_shapenew_value_shapenew_grad_out_shaperrr%_unpack_flash_attention_nested_shapes*s*          &rccs,|durt|jdksJt|jdksJ|dus#|j|jks#J|j\}}} } |j\}}} } |j\}}} }|dus>J|dusDJ|j|jksLJt||}t||}t||D]%\}}d| || f}d| || f}d| ||f}|durw|nd}||||fVq[dS|j|j|j|dur|jndfVdS)a? Given inputs to a efficient_attention_(forward|backward) kernel, this will handle behavior for NestedTensor inputs by effectively unbinding the NestedTensor and yielding the shapes for each batch element. In the case that this isn't a NestedTensor kernel, then it just yields the original shapes. Nrr)rrrr cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_krrrtrrrr| seqlens_q seqlens_klen_qlen_krrrrrrr)_unpack_efficient_attention_nested_shapesXs*       &rT)r9c Os(t|||||||d} tdd| DS)r)rrrrrrrcs$|] \}}}}t|||VqdSrr.0rorprqrrrr    z0_flash_attention_forward_flop..rsum) rrrrrrrr#r$r%sizesrrr_flash_attention_forward_flop rc Os(t|||||||d} tdd| DS)r)rrrrrrrcsrrrrrrrrrz4_efficient_attention_forward_flop..rr) rrrbiasrrrrr$r%rrrr!_efficient_attention_forward_floprrcCsVd}|\}}}}|\} } } } |\} }}}|\}}}}|| kr)| kr)|krBnJ|| kr;|kr;|krBnJ|| ksDJ||krP| |krP||ksRJd}|t||||f|||| f7}|t||||f|||| f7}|t||| |f||||f7}|t|||| f||| |f7}|t||||f|||| f7}|S)NrrM)rgrorprqr}rIrrrsrtrurvrwrxryrzr{r|_b4_h4_s4_d4rrrsdpa_backward_flop_counts    T"""""rcOst||||S)z(Count flops for self-attention backward.r)rgrorprqr#r$r%rrrsdpa_backward_flopsrc O*t|||||||| d} tdd| DS)N)rrrrrrrrcs&|]\}}}}t||||VqdSrrrrorprqrgrrrr   z1_flash_attention_backward_flop..r) rrrrout logsumexprrrrr$r%shapesrrr_flash_attention_backward_flop rc Or)N)rrrrrrrrcsrrrrrrrr&rz5_efficient_attention_backward_flop..r) rrrrrrrrrrr$r%rrrr"_efficient_attention_backward_flop rrcCst|ts|fS|Sr)rtuple)xrrrnormalize_tupleAs r)KMBTcCs0tdtttdtt|dd}t|S)Nrrr<r)maxminrsuffixesstr)numberindexrrrget_suffix_strJs(rcCs&t|}|d|d}|t|S)Niz.3f)rr)rsuffixrrrrrconvert_num_with_suffixQs  rcCs|dkrdS||dS)Nr0%z.2%r)numdenomrrrconvert_to_percent_strXs rcstfdd}|S)Ncst|\}}|}t||Sr)rr)r$ flat_argsspecrr&rrr(^s  z)_pytreeify_preserve_structure..nfrr)rr&r_pytreeify_preserve_structure]src seZdZdZ    ddeeejje ejjfde de dee e e fffd d Zd e fd d Zd e ee e e fffddZdddZddZddZddZZS)ra ``FlopCounterMode`` is a context manager that counts the number of flops within its context. It does this using a ``TorchDispatchMode``. It also supports hierarchical output by passing a module (or list of modules) to FlopCounterMode on construction. If you do not need hierarchical output, you do not need to use it with a module. Example usage .. code-block:: python mod = ... with FlopCounterMode(mod) as flop_counter: mod.sum().backward() Nr<Tmodsdepthdisplaycustom_mappingcstttdd|_||_||_d|_|duri}|dur&tjdddit dd| D|_ t |_ dS)NcSsttSr)rintrrrrsz*FlopCounterMode.__init__..zs*z,FlopCounterMode.__init__..) super__init__r flop_countsrrmodewarningswarnr!itemsr mod_tracker)selfrrrr __class__rrr{s  zFlopCounterMode.__init__r+cCst|jdS)NGlobal)rrvaluesrrrrget_total_flopsszFlopCounterMode.get_total_flopscCsdd|jDS)aReturn the flop counts as a dictionary of dictionaries. The outer dictionary is keyed by module name, and the inner dictionary is keyed by operation name. Returns: Dict[str, Dict[Any, int]]: The flop counts as a dictionary. cSsi|] \}}|t|qSr)dictrrrrrsz3FlopCounterMode.get_flop_counts..)rrrrrrget_flop_countss zFlopCounterMode.get_flop_countsc s|durj}|dur d}ddl}d|_gd}g}tdfdd}tjD]}|dkr;q4|d d }||krGq4|||d }| |q4djvroso|D] } d | d| d<q]|dd|}t |dkrzgd g}|j||d dS)Ni?BrT)ModuleFLOPz% TotalFcstj|}|kOd|}g}|||t|t|gj|D]\}}||dt|t|t|gq,|S)N z - )rrrappendrrrr)mod_namerr}paddingrr@r global_flops global_suffixis_global_subsumedrrr process_mods  z.FlopCounterMode.get_table..process_modr.rr)r0r)leftrightr)headerscolalign) rtabulatePRESERVE_WHITESPACErrsortedrkeyscountextendr) rrrheaderrrmod mod_depth cur_valuesrrr r get_tables6   zFlopCounterMode.get_tablecCs,|j|jt||_|j|Sr)rclearr __enter___FlopCounterModerrrrrr"s    zFlopCounterMode.__enter__cGsD|jdusJ|jj|}d|_|j|jr t||j|Sr)r__exit__rrprintr r)rr$rIrrrr$s  zFlopCounterMode.__exit__cCsV||jvr)|j|}||i|d|i}t|jjD] }|j|||7<q|S)Nr")r!setrparentsr)r func_packetrr$r%flop_count_funcrkparrrr _count_flopss  zFlopCounterMode._count_flops)Nr<TNr)__name__ __module__ __qualname____doc__rr rnnrrerboolrrrrrrr r"r$r+ __classcell__rrrrrgs* = c@s$eZdZdefddZdddZdS) r#countercCs ||_dSr)r3)rr3rrrrs z_FlopCounterMode.__init__rNcCs0|r|ni}|tjjjjtjjjjtjjjjtjjjjtjjjjtjjj jtjjj jtjjj jtjjj jtjjj jtjjjjtjjjjtjjjjtjjjjhvrRtS||jjvr|tjjjjur||j|i|}|turx|WdSWdn1swY||i|}|j|j|||Sr)ropsaten is_contiguousdefault memory_formatis_strides_like_formatis_non_overlapping_and_densersym_sizestride sym_stridestorage_offsetsym_storage_offsetnumel sym_numeldimprimlayoutNotImplementedr3r!r decomposer+_overloadpacket)rfunctypesr$r%rrrrr__torch_dispatch__s6               z#_FlopCounterMode.__torch_dispatch__)rN)r,r-r.rrrKrrrrr#sr#)Fr)NNNFN)Vrtorch.utils._pytreerrrmodule_trackerrtypingrrr r r collections.abcr typing_extensionsr collectionsrtorch.utils._python_dispatchrmathr functoolsrr__all__rrr4r5r r!r__annotations__r*rmmrrCaddmmrHbmmrLbaddbmmrN _scaled_mmrUrer1r^ convolution _convolutionrdconvolution_backwardrnr~'_scaled_dot_product_efficient_attention#_scaled_dot_product_flash_attention#_scaled_dot_product_cudnn_attentionrrrrr_flash_attention_forwardr_efficient_attention_forwardrr0_scaled_dot_product_efficient_attention_backward,_scaled_dot_product_flash_attention_backward,_scaled_dot_product_cudnn_attention_backwardr_flash_attention_backwardr_efficient_attention_backwardrrrrrrrrr#rrrrs        *    ' g6 36 0