o  hY@shdZddlZddlZddlmZddlmZmZddlZddl m Z ddl m Z m Z ddlmZmZdd lmZdd lmZdd lmZeeZdUd d ZdVde defddZde fddZdedefddZdVde dedefddZdededefddZ dVddZ!dededed e"fd!d"Z# dWde deded e"def d$d%Z$dededed efd&d'Z% dXde deded edef d(d)Z&dededed*e"d+e"d,ef d-d.Z' dYd1d2Z(dd3deded4eefd5d6Z) dZde ded4eedefd7d8Z*d9d:dededed e"d;e"f dd?Z+deded@edAedBe,dCe,dDe"d e"fdEdFZ-   G H  # d\de dedAedeed@eedBe,dCe,dDe"d e"defdIdJZ.ej/e!ej0e$ej1e&ej2e(ej3eej4eej5e*ej6eej7e+ej8e.i Z9   d]dKee,efde deedeedLee:f dMdNZ;GdOdPdPe ZdS)_z$PyTorch optimization for BERT model.N)partial)OptionalUnion) Optimizer)LambdaLRReduceLROnPlateau)LayerWiseDummyOptimizerLayerWiseDummyScheduler) SchedulerType)logging)require_versioncCsdSNr_rrm/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/optimization.py_get_constant_lambda"r optimizer last_epochcCst|t|dS)a Create a schedule with a constant learning rate, using the learning rate set in optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r)rr)rrrrrget_constant_schedule&srcKst|fi|S)a Create a schedule with a constant learning rate that decreases when a metric has stopped improving. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. kwargs (`dict`, *optional*): Extra parameters to be passed to the scheduler. See `torch.optim.lr_scheduler.ReduceLROnPlateau` for possible parameters. Return: `torch.optim.lr_scheduler.ReduceLROnPlateau` with the appropriate schedule. )r)rkwargsrrrget_reduce_on_plateau_schedule7sr current_stepnum_warmup_stepscCs"||krt|ttd|SdS)N?floatmax)rrrrr,_get_constant_schedule_with_warmup_lr_lambdaIsr"cCstt|d}t|||dS)ad Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. rr)rr"r)rrr lr_lambdarrr!get_constant_schedule_with_warmupOs r%num_training_stepscCsB||krt|ttd|Stdt||ttd||S)Nrr)rrr&rrr*_get_linear_schedule_with_warmup_lr_lambdads$r(cCstt||d}t|||S)a Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. rr&)rr(r)rrr&rr$rrrget_linear_schedule_with_warmupjs  r* num_cyclesc Csf||krt|ttd|St||ttd||}tdddttjt|d|S)Nrr'?r@r r!mathcospirrr&r+progressrrr*_get_cosine_schedule_with_warmup_lr_lambdas*r4r,cCtt|||d}t|||S)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. rr&r+)rr4rrrr&r+rr$rrrget_cosine_schedule_with_warmups r8c Csr||krt|ttd|St||ttd||}|dkr$dStdddttjt||dS)Nrrr'r,r.r2rrr=_get_cosine_with_hard_restarts_schedule_with_warmup_lr_lambdas *r9cCr5)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`int`, *optional*, defaults to 1): The number of hard restarts to use. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r6)rr9rr7rrr2get_cosine_with_hard_restarts_schedule_with_warmups r:lr_endpowerlr_initc Csf||krt|ttd|S||kr||S||}||}d|||}||||} | |Srr) rrr&r;r<r=lr_range decay_steps pct_remainingdecayrrr4_get_polynomial_decay_schedule_with_warmup_lr_lambdas rBHz>rcCsH|jd}||kstd|d|dtt|||||d}t|||S)a Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the optimizer to end lr defined by *lr_end*, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. lr_end (`float`, *optional*, defaults to 1e-7): The end LR. power (`float`, *optional*, defaults to 1.0): Power factor. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Note: *power* defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT implementation at https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37 Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. lrzlr_end (z#) must be smaller than initial lr ())rr&r;r<r=)defaults ValueErrorrrBr)rrr&r;r<rr=r$rrr)get_polynomial_decay_schedule_with_warmups  rH) timescalerIcCs@||krt|ttd|S||}dt|||}|S)Nrr)r r!r/sqrt)rrrIshiftrArrr$_get_inverse_sqrt_schedule_lr_lambdas rLcCs,|dur|pd}tt||d}t|||dS)a Create a schedule with an inverse square-root learning rate, from the initial lr set in the optimizer, after a warmup period which increases lr linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. timescale (`int`, *optional*, defaults to `num_warmup_steps`): Time scale. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. Ni')rrIr)rrLr)rrrIrr$rrrget_inverse_sqrt_schedule'srMr') min_lr_raterNcCsz||krt|ttd|St||ttd||}ddttjt|d|}|d||}td|S)Nrr,rr-rr.)rrr&r+rNr3factorrrrr4Es $ min_lrcCs^|dur |dur td|dur||jd}n|dur tdtt||||d}t|||S)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to min_lr, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. min_lr (`float`, *optional*): The minimum learning rate to reach after the cosine schedule. min_lr_rate (`float`, *optional*): The minimum learning rate as a ratio of the initial learning rate. If set, `min_lr` should not be set. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. Nz/Only one of min_lr or min_lr_rate should be setrDzLOne of min_lr or min_lr_rate should be set through the `lr_scheduler_kwargs`)rr&r+rN)rGrFrr4r)rrr&r+rrPrNr$rrr+get_cosine_with_min_lr_schedule_with_warmupPs# rQnum_stable_stepsnum_decay_steps warmup_type decay_type min_lr_ratioc Cs4||krAt|ttd|}|dkr|} n|dkr'ddttj|} n |dkr4dtd|} | d||} td| S|||krIdS||||krt|||ttd|}|dkrid|} n"|dkrddttjt|d|} n |dkrdt|} | d||} td| S|S) Nrlinearcosiner,r1-sqrtr'r-)r r!r/r0r1rJ) rrrRrSrTrUrVr+r3rOrrr_get_wsd_scheduler_lambdas.    & rZrWrXc Cs|dur |dur td|dur|durtd|dvr%td|d|dvr1td|d|dur;|||}tt|||||||d} t|| | S) a Create a schedule with a learning rate that has three stages: 1. warmup: increase from min_lr_ratio times the initial learning rate to the initial learning rate following a warmup_type. 2. stable: constant learning rate. 3. decay: decrease from the initial learning rate to min_lr_ratio times the initial learning rate following a decay_type. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_decay_steps (`int`): The number of steps for the decay phase. num_training_steps (`int`, *optional*): The total number of training steps. This is the sum of the warmup, stable and decay steps. If `num_stable_steps` is not provided, the stable phase will be `num_training_steps - num_warmup_steps - num_decay_steps`. num_stable_steps (`int`, *optional*): The number of steps for the stable phase. Please ensure that `num_warmup_steps + num_stable_steps + num_decay_steps` equals `num_training_steps`, otherwise the other steps will default to the minimum learning rate. warmup_type (`str`, *optional*, defaults to "linear"): The type of warmup to use. Can be 'linear', 'cosine' or '1-sqrt'. decay_type (`str`, *optional*, defaults to "cosine"): The type of decay to use. Can be 'linear', 'cosine' or '1-sqrt'. min_lr_ratio (`float`, *optional*, defaults to 0): The minimum learning rate as a ratio of the initial learning rate. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. Nz@Either num_training_steps or num_stable_steps must be specified.zZBoth num_training_steps and num_stable_steps are specified. num_stable_steps will be used.)rWrXrYzUnknown warmup type: z), expected 'linear', 'cosine' or '1-sqrt'zUnknown decay type: )rrRrSrTrUrVr+)rGwarningswarnrrZr) rrrSr&rRrTrUrVr+rr$rrrget_wsd_schedules*-   r]namescheduler_specific_kwargsc sTt|}t|}|durGt|trG|j}i|D]}t|||||d|<qfdd}|D] }|jr=||q3t ||j ddS|tj krP||S|durVi}|tj krc||fi|S|durnt |d|tjkry|||dS|tjkr|||dS|tjkr||f||d |S|durt |d ||f||d |S) a Unified API to get any scheduler from its name. Args: name (`str` or `SchedulerType`): The name of the scheduler to use. optimizer (`torch.optim.Optimizer`): The optimizer that will be used during training. num_warmup_steps (`int`, *optional*): The number of warmup steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. num_training_steps (`int``, *optional*): The number of training steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. scheduler_specific_kwargs (`dict`, *optional*): Extra parameters for schedulers such as cosine with restarts. Mismatched scheduler types and scheduler parameters will cause the scheduler function to raise a TypeError. N)rrr&cs|dSN)step)paramscheduler_dictrrscheduler_hook+sz%get_scheduler..scheduler_hookrD)optimizer_dictrDz; requires `num_warmup_steps`, please provide that argument.r#r)z= requires `num_training_steps`, please provide that argument.)r TYPE_TO_SCHEDULER_FUNCTION isinstancer rfkeys get_scheduler requires_grad"register_post_accumulate_grad_hookr rFCONSTANTREDUCE_ON_PLATEAUrGCONSTANT_WITH_WARMUP INVERSE_SQRTWARMUP_STABLE_DECAY) r^rrr&r_ schedule_funcrfrbrerrcrrjsb           rjcsveZdZdZ         dfd d Zed d Zed dZeddZeddZ e dddZ Z S) Adafactora) AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py Paper: *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost* https://arxiv.org/abs/1804.04235 Note that this optimizer internally adjusts the learning rate depending on the `scale_parameter`, `relative_step` and `warmup_init` options. To use a manual (external) learning rate schedule you should set `scale_parameter=False` and `relative_step=False`. Arguments: params (`Iterable[nn.parameter.Parameter]`): Iterable of parameters to optimize or dictionaries defining parameter groups. lr (`float`, *optional*): The external learning rate. eps (`Tuple[float, float]`, *optional*, defaults to `(1e-30, 0.001)`): Regularization constants for square gradient and parameter scale respectively clip_threshold (`float`, *optional*, defaults to 1.0): Threshold of root mean square of final gradient update decay_rate (`float`, *optional*, defaults to -0.8): Coefficient used to compute running averages of square beta1 (`float`, *optional*): Coefficient used for computing running averages of gradient weight_decay (`float`, *optional*, defaults to 0.0): Weight decay (L2 penalty) scale_parameter (`bool`, *optional*, defaults to `True`): If True, learning rate is scaled by root mean square relative_step (`bool`, *optional*, defaults to `True`): If True, time-dependent learning rate is computed instead of external learning rate warmup_init (`bool`, *optional*, defaults to `False`): Time-dependent learning rate computation depends on whether warm-up initialization is being used This implementation handles low-precision (FP16, bfloat) values, but we have not thoroughly tested. Recommended T5 finetuning settings (https://discuss.huggingface.co/t/t5-finetuning-tips/684/3): - Training without LR warmup or clip_threshold is not recommended. - use scheduled LR warm-up to fixed LR - use clip_threshold=1.0 (https://arxiv.org/abs/1804.04235) - Disable relative updates - Use scale_parameter=False - Additional optimizer operations like gradient clipping should not be used alongside Adafactor Example: ```python Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=1e-3) ``` Others reported the following combination to work well: ```python Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) ``` When using `lr=None` with [`Trainer`] you will most likely need to use [`~optimization.AdafactorSchedule`] scheduler as following: ```python from transformers.optimization import Adafactor, AdafactorSchedule optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) lr_scheduler = AdafactorSchedule(optimizer) trainer = Trainer(..., optimizers=(optimizer, lr_scheduler)) ``` Usage: ```python # replace AdamW with Adafactor optimizer = Adafactor( model.parameters(), lr=1e-3, eps=(1e-30, 1e-3), clip_threshold=1.0, decay_rate=-0.8, beta1=None, weight_decay=0.0, relative_step=False, scale_parameter=False, warmup_init=False, ) ```NgKH9gMbP?r皙r'TFc sVtd|dur| rtd| r| std|||||||| | d } t|| dS)Nz torch>=1.5.0z;Cannot combine manual `lr` and `relative_step=True` optionsz0`warmup_init=True` requires `relative_step=True`) rDepsclip_threshold decay_ratebeta1 weight_decayscale_parameter relative_step warmup_init)r rGsuper__init__) selfparamsrDrvrwrxryrzr{r|r}rF __class__rrrs   zAdafactor.__init__cCsj|d}|dr |drd|dnd}t|dt|d}d}|dr1t|d d |d }||S) NrDr|r}gư>rag{Gz?rr{rvrRMS)minr/rJr!) param_group param_state rel_step_szmin_step param_scalerrr_get_lrszAdafactor._get_lrcCs t|dk}|ddu}||fS)Nry)len)r param_shapefactoreduse_first_momentrrr _get_optionss  zAdafactor._get_optionscCs|d|dS)Nrr,)normnumel)tensorrrr_rmsszAdafactor._rmscCs6||jdddd}|d}t||S)NrT)dimkeepdim)meanrsqrt_ unsqueezersqrttorchmul)exp_avg_sq_rowexp_avg_sq_colr_factorc_factorrrr_approx_sq_grads zAdafactor._approx_sq_gradcCs$d}|dur |}|jD]}|dD]z}|jdurq|j}|jtjtjhvr,|}|jr3td|j |}|j }| ||\}} t |dkrd|d<| rVt ||d<|r{t|dd||d<t|dd |dd||d <nt ||d <d|d <n)| r|d||d<|r|d||d<|d ||d <n |d ||d <|} |jtjtjhvr| } |dd 7<|| |d <|||} dt|d|d} |d|dd} |r!|d}|d }|| j| jddd| d|| j| jd dd| d|||} | |n|d }|| j| d| d||} | || |djdd| | | rf|d}||dj| d |dd|} |ddkry| j| |d | d| | |jtjtjhvr|| qq |S)z Performs a single optimization step Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz,Adafactor does not support sparse gradients.rraexp_avgrrrr exp_avg_sqrrrrxrrv)r)alpharw)rryrz) param_groupsgraddtyperfloat16bfloat16r is_sparse RuntimeErrorstateshaperr zeros_likezerostorrr/powmul_add_rrrdiv_clamp_copy_)rclosurelossgroupprr grad_shaperr p_data_fp32rDbeta2tupdaterrrrrrrras|     *           NzAdafactor.step) NrtrruNr'TTFr`)__name__ __module__ __qualname____doc__r staticmethodrrrrrno_gradra __classcell__rrrrrs^s,W    rscs*eZdZdZdfdd ZddZZS)AdafactorSchedulea8 Since [`~optimization.Adafactor`] performs its own scheduling, if the training loop relies on a scheduler (e.g., for logging), this class creates a proxy object that retrieves the current lr values from the optimizer. It returns `initial_lr` during startup and the actual `lr` during stepping. r'csDfdd}|jD]}|d<q t|||jD]}|d=qdS)NcsSr`rr initial_lrrrr$Xrz-AdafactorSchedule.__init__..lr_lambdar)rr~r)rrrr$rrrrrWs    zAdafactorSchedule.__init__cs0|jfddjD}t|dkr|j}|S)Ncs8g|]}|ddjdur|j|ddqS)rrN)rrr).0roptrr cs z,AdafactorSchedule.get_lr..r)rrrbase_lrs)rlrsrrrget_lras  zAdafactorSchedule.get_lrr')rrrrrrrrrrrrOs rcCs t||S)aX Get a proxy schedule for [`~optimization.Adafactor`] Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. initial_lr (`float`, *optional*, defaults to 0.0): Initial lr Return: [`~optimization.Adafactor`] proxy schedule object. )r)rrrrrget_adafactor_schedulems rr`)r)r,r)rr)rCrr)Nr)r,rNN)NNrWrXrr,r)NNNr)?rr/r[ functoolsrtypingrrr torch.optimrtorch.optim.lr_schedulerrrtrainer_pt_utilsr r trainer_utilsr utilsr utils.versionsr get_loggerrloggerrintrrr"r%r(r*r r4r8r9r:rBrHrLrMrQstrrZr]LINEARCOSINECOSINE_WITH_RESTARTS POLYNOMIALrmrorprnCOSINE_WITH_MIN_LRrqrgdictrjrsrrrrrrs            "   !  .    4 *  J  ]r