o h1M@sdZddlmZmZmZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZddgZGd ddeZd d ed e d e d e d e d de_     d(deedeedeeededeedeedeedeededededededefddZdeed eedeeedeedeedededededededefd!d"Zdeed eedeeedeedeedededededededefd#d$Zdeed eedeeedeedeedededededededed%dfd&d'ZdS))z9Implementation for Stochastic Gradient Descent optimizer.)castOptionalUnionN)Tensor) _default_to_fused_or_foreach_device_dtype_check_for_fused_differentiable_doc _foreach_doc _fused_doc _maximize_doc _params_doc_use_grad_for_differentiable DeviceDict OptimizerParamsTSGDsgdcseZdZ     dddddddedeeefded ed eeefd ed ed eededeeffddZ fddZ ddZ e dddZ ZS)rMbP?rFN)maximizeforeachdifferentiablefusedparamslrmomentum dampening weight_decaynesterovrrrrc st|tr|dkrtd|dkrtd||dkr%td||dkr0td|t|||||||| | d } |rK|dksG|dkrKtd t|| | rfd |_d |_| r`t d |rht d dSdS) NrzTensor lr must be 1-elementgzInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: ) rrrrrrrrrrz8Nesterov momentum requires a momentum and zero dampeningTz)`fused` does not support `differentiable`z0`fused` and `foreach` cannot be `True` together.) isinstancernumel ValueErrordictsuper__init___step_supports_amp_scaling"_need_device_dtype_check_for_fused RuntimeError) selfrrrrrrrrrrdefaults __class__c/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/torch/optim/sgd.pyr$s> z SGD.__init__csXt||jD] }|dd|dd|dd|dd|ddq dS)NrFrrrr)r# __setstate__ param_groups setdefault)r(stategroupr*r,r-r.Is      zSGD.__setstate__cCsd}|dD]<}|jdurB|drt|ddrt|d|_||||j|jjr/d}|ddkrB|j|}||dq|S) NFrrr&Trrmomentum_buffer)gradgetattrrr&append is_sparser1get)r(r2rgradsmomentum_buffer_listhas_sparse_gradpr1r,r,r- _init_groupRs"       zSGD._init_groupc Csd}|durt |}Wdn1swY|jD]S}g}g}g}|||||}t||||d|d|d|d|d|d||d|d t|d dt|d dd |dd krst||D] \}} |j|} | | d<qeq |S)zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrrr grad_scale found_inf) rrrrrrr;rrr>r?rr3)torch enable_gradr/r=rr5zipr1) r(closurelossr2rr9r:r;r<r3r1r,r,r-stepgsD       zSGD.step)rrrrFN)__name__ __module__ __qualname__rrfloatrboolrr$r.r=rrE __classcell__r,r,r*r-rsH        . a Implements stochastic gradient descent (optionally with momentum). .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{13mm} \:\mu \text{ (momentum)}, \:\tau \text{ (dampening)}, \:\textit{ nesterov,}\:\textit{ maximize} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}\textbf{if} \: \mu \neq 0 \\ &\hspace{10mm}\textbf{if} \: t > 1 \\ &\hspace{15mm} \textbf{b}_t \leftarrow \mu \textbf{b}_{t-1} + (1-\tau) g_t \\ &\hspace{10mm}\textbf{else} \\ &\hspace{15mm} \textbf{b}_t \leftarrow g_t \\ &\hspace{10mm}\textbf{if} \: \textit{nesterov} \\ &\hspace{15mm} g_t \leftarrow g_{t} + \mu \textbf{b}_t \\ &\hspace{10mm}\textbf{else} \\[-1.ex] &\hspace{15mm} g_t \leftarrow \textbf{b}_t \\ &\hspace{5mm}\textbf{if} \: \textit{maximize} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} + \gamma g_t \\[-1.ex] &\hspace{5mm}\textbf{else} \\[-1.ex] &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__. z Args: a lr (float, Tensor, optional): learning rate (default: 1e-3) momentum (float, optional): momentum factor (default: 0) dampening (float, optional): dampening for momentum (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) nesterov (bool, optional): enables Nesterov momentum. Only applicable when momentum is non-zero. (default: False) z z a Example: >>> # xdoctest: +SKIP >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ http://www.cs.toronto.edu/%7Ehinton/absps/momentum.pdf .. note:: The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et al. and implementations in some other frameworks. Considering the specific case of Momentum, the update can be written as .. math:: \begin{aligned} v_{t+1} & = \mu * v_{t} + g_{t+1}, \\ p_{t+1} & = p_{t} - \text{lr} * v_{t+1}, \end{aligned} where :math:`p`, :math:`g`, :math:`v` and :math:`\mu` denote the parameters, gradient, velocity, and momentum respectively. This is in contrast to Sutskever et al. and other frameworks which employ an update of the form .. math:: \begin{aligned} v_{t+1} & = \mu * v_{t} + \text{lr} * g_{t+1}, \\ p_{t+1} & = p_{t} - v_{t+1}. \end{aligned} The Nesterov version is analogously modified. Moreover, the initial value of the momentum buffer is set to the gradient value at the first step. This is in contrast to some other frameworks that initialize it to all zeros. Frd_p_listr:r;rrr>r?rrrrrrcCs|dur|durtjst|ddd\}}nd}d}|dur!d}|dur'd}|r2tjr2td|r=tjr=td|rGtjsGt}n |rQtjsQt}nt}|||||| | | | || ||d dS)zlFunctional API that performs SGD algorithm computation. See :class:`~torch.optim.SGD` for details. NF)r use_fusedz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers) rrrrrr;rr>r?)r@jit is_scriptingrr'_multi_tensor_sgd _fused_sgd_single_tensor_sgd)rrMr:r;rrr>r?rrrrrrfuncr,r,r-rsD   r9cCs,|dur|dus Jt|D]\} } | s|| n||  }|dkrAt|tr:|jr2|| |}n|j| |d}n|j| |d}|dkrq|| }|durYt|}||| <n | |j |d|d| ro|j||d}n|}t|tr|jr| j||ddq| j || dq| j || dqdS)Nralphar)value) enumeraterr requires_gradaddcmul_cloneaddr@detachmul_add_)rr9r:r>r?rrrrrrr;iparamr4bufr,r,r-rS<s0   rScCsB|dur|dus Jt|dkrdStj|||gdd} | D]\\} }}}ttt| }ttt|}| o?tdd|D}| rGt |}|dkr^| rVtj |||dntj |||d}|dkrg}d}t t|D]}||durxd}n | tt||ql|rt||tj ||d|dn?g}t t|D]6}||durt||}||<|||<ntt||}||j||d|d| |q| rtj |||dn|}|st|tjrtjrt|| }t ||q tj ||| dq t t|D]}||j||| dqq dS) NrT with_indicescss|]}|jVqdSrF)r7).0r4r,r,r- s z$_multi_tensor_sgd..rUFr)lenr"_group_tensors_by_device_and_dtypevaluesrlistranyr@ _foreach_neg _foreach_add_ _foreach_addranger6 _foreach_mul_r\r^r_r`rcompiler is_compiling _foreach_mul)rr9r:r>r?rrrrrrr;grouped_tensorsdevice_params_ device_grads_device_momentum_buffer_listindices device_params device_gradsdevice_has_sparse_gradbufsall_states_with_momentum_bufferrarc grads_x_lrr,r,r-rQsst        rQreturncCsP|sdS| r td|dur|j|ini} |dur|j|ini} |dk}tdd|Do/| }|rBt|D] \}}t|||<q6tj|||gdd}|D]U\\}}\\}}}}t t t |}t t t |}d\}}|durz| || |}| dur|dur| || |}tj|||rgnt t t |||||| | |||d qPdS) Nz.`_fused_sgd` does not support sparse gradientsrcss|]}|duVqdSrFr,)rftr,r,r-rgsz_fused_sgd..Frd)NN) rrrrrr is_first_stepr>r?)r'deviceallrYr@ empty_likerriitemsrrkrr0to _fused_sgd_)rr9r:r>r?rrrrrrr;grad_scale_dictfound_inf_dictno_momentum_bufferrragrur_rvrwrxrzr{device_grad_scaledevice_found_infr,r,r-rRs^   rR)FNNNN)__doc__typingrrrr@r optimizerrrr r r r r rrrr__all__rrkrKrJrrSrQrRr,r,r,r-s  4~#    1e      F      7      ]