o h@s2dZddlZddlmZmZddlmZddlZGdddeZGdddeZ Gd d d eZ Gd d d eZ Gd ddeZ GdddeZ GdddeZGdddeZddZddZd3ddZddZd3ddZd3dd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2ZdS)4zPruning methods.N)ABCabstractmethod)Iterablec@sZeZdZUdZeed<ddZeddZddZ e d d d d Z dd dZ ddZ d S)BasePruningMethodzAbstract base class for creation of new pruning techniques. Provides a skeleton for customization requiring the overriding of methods such as :meth:`compute_mask` and :meth:`apply`. _tensor_namecCst||j||dS)aMultiply the mask into original tensor and store the result. Multiplies the mask (stored in ``module[name + '_mask']``) into the original tensor (stored in ``module[name + '_orig']``) and stores the result into ``module[name]`` by using :meth:`apply_mask`. Args: module (nn.Module): module containing the tensor to prune inputs: not used. N)setattrr apply_mask)selfmoduleinputsr h/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/torch/nn/utils/prune.py__call__s zBasePruningMethod.__call__cCsdS)aCompute and returns a mask for the input tensor ``t``. Starting from a base ``default_mask`` (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the ``default_mask`` according to the specific pruning method recipe. Args: t (torch.Tensor): tensor representing the importance scores of the parameter to prune. default_mask (torch.Tensor): Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as ``t``. Returns: mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t`` Nr )r t default_maskr r r compute_mask szBasePruningMethod.compute_maskcCsP|jdus Jd|dt||jd}t||jd}|j|jd|}|S)aSimply handles the multiplication between the parameter being pruned and the generated mask. Fetches the mask and the original tensor from the module and returns the pruned version of the tensor. Args: module (nn.Module): module containing the tensor to prune Returns: pruned_tensor (torch.Tensor): pruned version of the input tensor NModule z has to be pruned_mask_origdtype)rgetattrtor)r r maskorig pruned_tensorr r r r4s  zBasePruningMethod.apply_maskNimportance_scoresc Os2dd}||||g|Ri|}t||}|dur+|j|jks*Jd|d|n|}t|tsD||d||j|=t|} nt||dj tj d} z |j || d } | |d| t ||||||W|Sty} zt|tst||d}||||j|d=| d} ~ ww) aAdd pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. args: arguments passed on to a subclass of :class:`BasePruningMethod` importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the parameter will be used in its place. kwargs: keyword arguments passed on to a subclass of a :class:`BasePruningMethod` c _sd}d}g}|jD]\}} t| tr$| j|kr$| }|||d7}q |dks4Jd|d|d|D]}|j|=q6||i|} || _|durit|trY|| |} | St|trit|} | | | } | S)NrzFAvoid adding multiple pruning hooks to the same tensor z of module z. Use a PruningContainer.)_forward_pre_hooksitems isinstancerrappendPruningContaineradd_pruning_method) clsr nameargskwargs old_methodfoundhooks_to_removekhookmethod containerr r r _get_composite_methodas<       z6BasePruningMethod.apply.._get_composite_methodNzJimportance_scores should have the same shape as parameter z of rr memory_formatr)rshaper!r#register_parameter _parameterstorch ones_likedetachclonecontiguous_formatrregister_bufferrrregister_forward_pre_hook Exception) r%r r&rr'r(r0r.rrrer r r applyJs>/         zBasePruningMethod.applycCsJ|dur|j|jksJdn|}|dur|nt|}||j||dS)aCompute and returns a pruned version of input tensor ``t``. According to the pruning rule specified in :meth:`compute_mask`. Args: t (torch.Tensor): tensor to prune (of same dimensions as ``default_mask``). importance_scores (torch.Tensor): tensor of importance scores (of same shape as ``t``) used to compute mask for pruning ``t``. The values in this tensor indicate the importance of the corresponding elements in the ``t`` that is being pruned. If unspecified or None, the tensor ``t`` will be used in its place. default_mask (torch.Tensor, optional): mask from previous pruning iteration, if any. To be considered when determining what portion of the tensor that pruning should act on. If None, default to a mask of ones. Returns: pruned version of tensor ``t``. Nz8importance_scores should have the same shape as tensor tr3)r4r7r8r)r rrrr r r pruneszBasePruningMethod.prunecCs|jdus Jd|d||}t||jrt||j|j|jd}|j|_|j|jd=|j|jd=t||j|dS)arRemove the pruning reparameterization from a module. The pruned parameter named ``name`` remains permanently pruned, and the parameter named ``name+'_orig'`` is removed from the parameter list. Similarly, the buffer named ``name+'_mask'`` is removed from the buffers. Note: Pruning itself is NOT undone or reversed! Nrz: has to be pruned before pruning can be removedrr)rrhasattrdelattrr6data_buffersr)r r weightrr r r removes    zBasePruningMethod.remove)NN)__name__ __module__ __qualname____doc__str__annotations__rrrr classmethodr@rArGr r r r r s    rc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)r#aContainer holding a sequence of pruning methods for iterative pruning. Keeps track of the order in which pruning methods are applied and handles combining successive pruning calls. Accepts as argument an instance of a BasePruningMethod or an iterable of them. cGshd|_t|ts|j|_||dSt|dkr(|dj|_||ddS|D]}||q*dS)Nr rr)_pruning_methodsr!rrr$len)r r'r.r r r __init__s    zPruningContainer.__init__cCsrt|ts|durtt|d|dur/|j|jkr/td|jd|dd|jd|j|f7_dS)zAdd a child pruning ``method`` to the container. Args: method (subclass of BasePruningMethod): child pruning method to be added to the container. Nz$ is not a BasePruningMethod subclasszszIPruningContainer.compute_mask.._combine_masks..rfNglobalzUnrecognized PRUNING_TYPE r3)rr PRUNING_TYPErBAttributeErrorrc IndexErrorsumrangeslicerPr4rVr)r.rrnew_maskslcn_dims keep_channel partial_maskr rfr _combine_masksSs6    "   z5PruningContainer.compute_mask.._combine_masksr^)r rrrtr.rr r r r7s ? zPruningContainer.compute_maskN) rHrIrJrKrQr$rZr]r`rr r r r r#s   r#cs0eZdZdZdZddZefddZZS)IdentityzsUtility pruning method that does not prune any units but generates the pruning parametrization with a mask of ones.racCs|}|SrXr r rrrr r r rszIdentity.compute_maskcst||S)Add pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. superr@)r%r r& __class__r r r@s zIdentity.apply) rHrIrJrKrirrNr@ __classcell__r r r{r rvs rvcs8eZdZdZdZddZddZefddZZ S) RandomUnstructuredaPrune (currently unpruned) units in a tensor at random. Args: name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. racCt|||_dSrX_validate_pruning_amount_initamountr rr r r rQ zRandomUnstructured.__init__cCsf|}t|j|}t|||jtjd}|dkr1t|}tj| d|d}d| d|j <|S)Nr1rrur,) nelement_compute_nparams_topruner_validate_pruning_amountr:r7r; rand_liketopkviewindices)r rr tensor_sizenparams_toprunerprobrr r r rs   zRandomUnstructured.compute_maskctj|||dS)aAdd pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. rry)r%r r&rr{r r r@szRandomUnstructured.apply rHrIrJrKrirQrrNr@r}r r r{r r~s r~cs:eZdZdZdZddZddZed fdd ZZ S) L1UnstructuredavPrune (currently unpruned) units in a tensor by zeroing out the ones with the lowest L1-norm. Args: amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. racCrrXrrr r r rQrzL1Unstructured.__init__cCsd|}t|j|}t|||jtjd}|dkr0tjt| d|dd}d| d|j <|S)Nr1rruFr,largest) rrrrr:r7r;rabsrr)r rrrrrrr r r rs  zL1Unstructured.compute_maskNctj||||dS)aJAdd pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. rrry)r%r r&rrr{r r r@szL1Unstructured.applyrXrr r r{r rs rcs<eZdZdZdZd ddZddZed fdd ZZ S) RandomStructuredaPrune entire (currently unpruned) channels in a tensor at random. Args: amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. dim (int, optional): index of the dim along which we define channels to prune. Default: -1. rbrucCst|||_||_dSrX)rrrc)r rrcr r r rQ;s zRandomStructured.__init__cCstt|t||j|j|j}t|j|}t||dd}|dkr'|}|S|||j||}||j|jd9}|S)aCompute and returns a mask for the input tensor ``t``. Starting from a base ``default_mask`` (which should be a mask of ones if the tensor has not been pruned yet), generate a random mask to apply on top of the ``default_mask`` by randomly zeroing out channels along the specified dim of the tensor. Args: t (torch.Tensor): tensor representing the parameter to prune default_mask (torch.Tensor): Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as ``t``. Returns: mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t`` Raises: IndexError: if ``self.dim >= len(t.shape)`` c SsTt|}tj||dj}||k}t|}tdgt|j}|||<d||<|S)Nrr)r7randkthvaluevalues zeros_likernrPr4) rrc nchannelsnchannels_topruner threshold channel_maskrrpr r r make_maskis  z0RandomStructured.compute_mask..make_maskrr) _validate_structured_pruning_validate_pruning_dimrcr4rrrrr)r rrrrrrr r r rAs    zRandomStructured.compute_maskcr)a;Add pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. dim (int, optional): index of the dim along which we define channels to prune. Default: -1. )rrcry)r%r r&rrcr{r r r@szRandomStructured.applyrurr r r{r r-s ?rcs<eZdZdZdZd ddZddZed fd d ZZ S) LnStructuredaePrune entire (currently unpruned) channels in a tensor based on their L\ ``n``-norm. Args: amount (int or float): quantity of channels to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument ``p`` in :func:`torch.norm`. dim (int, optional): index of the dim along which we define channels to prune. Default: -1. rbrucCst|||_||_||_dSrX)rrnrc)r rrrcr r r rQs zLnStructured.__init__c Cst|t||j|j|j}t|j|}||}t||t||j|j}t j ||dd}dd}|dkr;|} | S|||j|j } | |j | j d9} | S)aCompute and returns a mask for the input tensor ``t``. Starting from a base ``default_mask`` (which should be a mask of ones if the tensor has not been pruned yet), generate a mask to apply on top of the ``default_mask`` by zeroing out the channels along the specified dim with the lowest L\ ``n``-norm. Args: t (torch.Tensor): tensor representing the parameter to prune default_mask (torch.Tensor): Base mask from previous pruning iterations, that need to be respected after the new mask is applied. Same dims as ``t``. Returns: mask (torch.Tensor): mask to apply to ``t``, of same dims as ``t`` Raises: IndexError: if ``self.dim >= len(t.shape)`` TrcSs2t|}tdgt|j}|||<d||<|S)Nr)r7rrnrPr4)rrcrrrpr r r rs z,LnStructured.compute_mask..make_maskrr)rrrcr4rrr _compute_normrr7rrrr) r rrrrnparams_tokeepnormrrrr r r rs     zLnStructured.compute_maskNcstj||||||dS)a=Add pruning on the fly and reparametrization of a tensor. Adds the forward pre-hook that enables pruning on the fly and the reparametrization of a tensor in terms of the original tensor and the pruning mask. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument ``p`` in :func:`torch.norm`. dim (int): index of the dim along which we define channels to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. )rrrcrry)r%r r&rrrcrr{r r r@szLnStructured.applyrrXrr r r{r rs Frcs4eZdZdZddZddZefddZZS)CustomFromMaskrhcCs ||_dSrXr)r rr r r rQr[zCustomFromMask.__init__cCs*|j|jjks J||jj|jd}|S)Nr)r4rrrrwr r r rszCustomFromMask.compute_maskcr)rxrry)r%r r&rr{r r r@"s zCustomFromMask.apply) rHrIrJrirQrrNr@r}r r r{r rs rcCst|||S)aBApply pruning reparametrization without pruning any units. Applies pruning reparametrization to the tensor corresponding to the parameter called ``name`` in ``module`` without actually pruning any units. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Note: The mask is a tensor of ones. Args: module (nn.Module): module containing the tensor to prune. name (str): parameter name within ``module`` on which pruning will act. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.identity(nn.Linear(2, 3), 'bias') >>> print(m.bias_mask) tensor([1., 1., 1.]) )rvr@)r r&r r r identity2s rcCt||||S)a?Prune tensor by removing random (currently unpruned) units. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified ``amount`` of (currently unpruned) units selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.random_unstructured(nn.Linear(2, 3), 'weight', amount=1) >>> torch.sum(m.weight_mask == 0) tensor(1) )r~r@)r r&rr r r random_unstructuredUs!rcCstj||||d|S)aPrune tensor by removing units with the lowest L1-norm. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified `amount` of (currently unpruned) units with the lowest L1-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.l1_unstructured(nn.Linear(2, 3), 'weight', amount=0.2) >>> m.state_dict().keys() odict_keys(['bias', 'weight_orig', 'weight_mask']) r)rr@)r r&rrr r r l1_unstructuredzs&rcCst|||||S)aPrune tensor by removing random channels along the specified dimension. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified ``amount`` of (currently unpruned) channels along the specified ``dim`` selected at random. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. dim (int): index of the dim along which we define channels to prune. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> # xdoctest: +SKIP >>> m = prune.random_structured( ... nn.Linear(5, 3), 'weight', amount=3, dim=1 ... ) >>> columns_pruned = int(sum(torch.sum(m.weight, dim=0) == 0)) >>> print(columns_pruned) 3 )rr@)r r&rrcr r r random_structureds%rcCstj||||||d|S)aPrune tensor by removing channels with the lowest L\ ``n``-norm along the specified dimension. Prunes tensor corresponding to parameter called ``name`` in ``module`` by removing the specified ``amount`` of (currently unpruned) channels along the specified ``dim`` with the lowest L\ ``n``-norm. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. amount (int or float): quantity of parameters to prune. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument ``p`` in :func:`torch.norm`. dim (int): index of the dim along which we define channels to prune. importance_scores (torch.Tensor): tensor of importance scores (of same shape as module parameter) used to compute mask for pruning. The values in this tensor indicate the importance of the corresponding elements in the parameter being pruned. If unspecified or None, the module parameter will be used in its place. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> from torch.nn.utils import prune >>> m = prune.ln_structured( ... nn.Conv2d(5, 3, 2), 'weight', amount=0.3, dim=1, n=float('-inf') ... ) r)rr@)r r&rrrcrr r r ln_structureds) rc st|ts tddurnittstdtjjfdd|D}tjjdd|D}t}d|_ |d i|}d|_ |j dkrTtd |d |j | || ||}d } |D]#\} } t | | } | } || | | | }t| | |d | | 7} qcdS)a Globally prunes tensors corresponding to all parameters in ``parameters`` by applying the specified ``pruning_method``. Modifies modules in place by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: parameters (Iterable of (module, name) tuples): parameters of the model to prune in a global fashion, i.e. by aggregating all weights prior to deciding which ones to prune. module must be of type :class:`nn.Module`, and name must be a string. pruning_method (function): a valid pruning function from this module, or a custom one implemented by the user that satisfies the implementation guidelines and has ``PRUNING_TYPE='unstructured'``. importance_scores (dict): a dictionary mapping (module, name) tuples to the corresponding parameter's importance scores tensor. The tensor should be the same shape as the parameter, and is used for computing mask for pruning. If unspecified or None, the parameter will be used in place of its importance scores. kwargs: other keyword arguments such as: amount (int or float): quantity of parameters to prune across the specified parameters. If ``float``, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If ``int``, it represents the absolute number of parameters to prune. Raises: TypeError: if ``PRUNING_TYPE != 'unstructured'`` Note: Since global structured pruning doesn't make much sense unless the norm is normalized by the size of the parameter, we now limit the scope of global pruning to unstructured methods. Examples: >>> from torch.nn.utils import prune >>> from collections import OrderedDict >>> net = nn.Sequential(OrderedDict([ ... ('first', nn.Linear(10, 4)), ... ('second', nn.Linear(4, 1)), ... ])) >>> parameters_to_prune = ( ... (net.first, 'weight'), ... (net.second, 'weight'), ... ) >>> prune.global_unstructured( ... parameters_to_prune, ... pruning_method=prune.L1Unstructured, ... amount=10, ... ) >>> print(sum(torch.nn.utils.parameters_to_vector(net.buffers()) == 0)) tensor(10) z4global_unstructured(): parameters is not an IterableNz=global_unstructured(): importance_scores must be of type dictcs&g|]\}}||ft||qSr )getrrdr r&rr r rgEsz'global_unstructured..c Ss,g|]\}}t||dtt||qS)r)rr7r8rr r r rgMstemprazROnly "unstructured" PRUNING_TYPE supported for the `pruning_method`. Found method z of type rrr )r!rrTdictr7nnutilsparameters_to_vectorr#rrir$rrnumelview_ascustom_from_mask) parameterspruning_methodrr(relevant_importance_scoresrr/r. final_maskpointerr r&param num_param param_maskr rr global_unstructuredsJ >         rcCr)aEPrune tensor corresponding to parameter called ``name`` in ``module`` by applying the pre-computed mask in ``mask``. Modifies module in place (and also return the modified module) by: 1) adding a named buffer called ``name+'_mask'`` corresponding to the binary mask applied to the parameter ``name`` by the pruning method. 2) replacing the parameter ``name`` by its pruned version, while the original (unpruned) parameter is stored in a new parameter named ``name+'_orig'``. Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. mask (Tensor): binary mask to be applied to the parameter. Returns: module (nn.Module): modified (i.e. pruned) version of the input module Examples: >>> from torch.nn.utils import prune >>> m = prune.custom_from_mask( ... nn.Linear(5, 3), name='bias', mask=torch.tensor([0, 1, 0]) ... ) >>> print(m.bias_mask) tensor([0., 1., 0.]) )rr@)r r&rr r r rusrcCsX|jD]\}}t|tr |j|kr |||j|=|Sqtd|d|d)aRemove the pruning reparameterization from a module and the pruning method from the forward hook. The pruned parameter named ``name`` remains permanently pruned, and the parameter named ``name+'_orig'`` is removed from the parameter list. Similarly, the buffer named ``name+'_mask'`` is removed from the buffers. Note: Pruning itself is NOT undone or reversed! Args: module (nn.Module): module containing the tensor to prune name (str): parameter name within ``module`` on which pruning will act. Examples: >>> m = random_unstructured(nn.Linear(5, 7), name='weight', amount=0.2) >>> m = remove(m, name='weight') z Parameter 'z ' of module z/ has to be pruned before pruning can be removed)rr r!rrrGrV)r r&r,r-r r r rGs rGcCs8|D]\}}|jD] }t|trdSq qdS)atCheck if a module is pruned by looking for pruning pre-hooks. Check whether ``module`` is pruned by looking for ``forward_pre_hooks`` in its modules that inherit from the :class:`BasePruningMethod`. Args: module (nn.Module): object that is either pruned or unpruned Returns: binary answer to whether ``module`` is pruned. Examples: >>> from torch.nn.utils import prune >>> m = nn.Linear(5, 7) >>> print(prune.is_pruned(m)) False >>> prune.random_unstructured(m, name='weight', amount=0.2) >>> print(prune.is_pruned(m)) True TF) named_modulesrrr!r)r _ submoduler-r r r is_pruneds rcCslt|tjstd|dt|tjr|dks*t|tjs2t|dks*t|dkr4td|ddSdS) aValidate helper to check the range of amount at init. Args: amount (int or float): quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune. Raises: ValueError: if amount is a float not in [0, 1], or if it's a negative integer. TypeError: if amount is neither a float nor an integer. Note: This does not take into account the number of parameters in the tensor to be pruned, which is known only at prune. zInvalid type for amount: z. Must be int or float.rg?gamount=zG should either be a float in the range [0, 1] or a non-negative integerN)r!numbersRealrTIntegralfloatrVrr r r rs   rcCs0t|tjr||krtd|d|dSdS)a?Validate that the pruning amount is meaningful wrt to the size of the data. Validation helper to check that the amount of parameters to prune is meaningful wrt to the size of the data (`tensor_size`). Args: amount (int or float): quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune. tensor_size (int): absolute number of parameters in the tensor to prune. rz: should be smaller than the number of parameters to prune=N)r!rrrVrrr r r rs rcCs0|j}t|dkrtd|dt|ddS)aiValidate that the tensor to be pruned is at least 2-Dimensional. Validation helper to check that the tensor to be pruned is multi- dimensional, such that the concept of "channels" is well-defined. Args: t (torch.Tensor): tensor representing the parameter to prune Raises: ValueError: if the tensor `t` is not at least 2D. rzZStructured pruning can only be applied to multidimensional tensors. Found tensor of shape z with z dimsN)r4rPrV)rr4r r r rs rcCst|tjr|St||S)aConvert the pruning amount from a percentage to absolute value. Since amount can be expressed either in absolute value or as a percentage of the number of units/channels in a tensor, this utility function converts the percentage to absolute value to standardize the handling of pruning. Args: amount (int or float): quantity of parameters to prune. If float, should be between 0.0 and 1.0 and represent the fraction of parameters to prune. If int, it represents the absolute number of parameters to prune. tensor_size (int): absolute number of parameters in the tensor to prune. Returns: int: the number of units to prune in the tensor )r!rrroundrr r r rs  rcCs&||krtd|d|jdS)zValidate that the pruning dimension is within the bounds of the tensor dimension. Args: t (torch.Tensor): tensor representing the parameter to prune dim (int): index of the dim along which we define channels to prune zInvalid index z for tensor of size N)rcrkr4)rrcr r r r6s rcCs>tt|}|dkr||}||tj|||d}|S)aZCompute the L_n-norm of a tensor along all dimensions except for the specified dimension. The L_n-norm will be computed across all entries in tensor `t` along all dimension except for the one identified by dim. Example: if `t` is of shape, say, 3x2x4 and dim=2 (the last dim), then norm will have Size [4], and each entry will represent the `L_n`-norm computed using the 3x2=6 entries for each of the 4 channels. Args: t (torch.Tensor): tensor representing the parameter to prune n (int, float, inf, -inf, 'fro', 'nuc'): See documentation of valid entries for argument p in torch.norm dim (int): dim identifying the channels to prune Returns: norm (torch.Tensor): L_n norm computed across all dimensions except for `dim`. By construction, `norm.shape = t.shape[-1]`. r)prc)listrmrcrGr7r)rrrcdimsrr r r rAs  rrX)rKrabcrrcollections.abcrr7rr#rvr~rrrrrrrrrrrrGrrrrrrrr r r r s> |;Bi# %, ) /w!