o  hބ@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z m Z mZddlZddlmZmZmZmZmZmZmZmZmZmZmZmZmZer[ddlZddZ dUd e!d e"fd d Z#dUd e!d e"fddZ$ddZ%GdddZ&Gddde Z'Gddde Z(Gddde Z)dZ*e+de*dZ,ddZ-Gdd d eZ.Gd!d"d"eZ/Gd#d$d$eZ0Gd%d&d&eZ1Gd'd(d(e Z2d)e3e4e5fd*e5fd+d,Z6d*e3e4e5ffd-d.Z7d*e3e4e5ffd/d0Z8d1d2Z9d*e3e4e5ffd3d4Z:Gd5d6d6eZ;d7d8ZGd=d>d>eZ?Gd?d@d@Z@dAdBZAdCdDZBdEdFZC dWdHeDdIe!dJe"fdKdLZEGdMdNdNeZFGdOdPdPZGdUdQe4dRe"fdSdTZHdS)Xz6 PyTorch-independent utilities for the Trainer class. N)Any NamedTupleOptionalUnion) ExplicitEnumis_psutil_availableis_tf_availableis_torch_availableis_torch_cuda_availableis_torch_hpu_availableis_torch_mlu_availableis_torch_mps_availableis_torch_musa_availableis_torch_npu_availableis_torch_xla_availableis_torch_xpu_availablerequires_backendscCstd}t|dS)zN Helper function to set worker seed during Dataloader initialization. lN)torch initial_seedset_seed)_ worker_seedrn/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/trainer_utils.py seed_worker4s  rFseed warn_onlycCst|tr1dtjd<dtjd<dtjd<dtjd<dtjd<tjd|d dtjj_d tjj_ t r@d d l }|j j d Sd S) a  Helper function for reproducible behavior during distributed training. See - https://pytorch.org/docs/stable/notes/randomness.html for pytorch - https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow 1CUDA_LAUNCH_BLOCKINGz:16:8CUBLAS_WORKSPACE_CONFIGASCEND_LAUNCH_BLOCKINGHCCL_DETERMINISTICFLASH_ATTENTION_DETERMINISTICT)rFrN)rr osenvironruse_deterministic_algorithmsbackendscudnn deterministic benchmarkr tensorflowconfig experimentalenable_op_determinism)rrtfrrrenable_full_determinism<s       r0r)cCst|tj|tr t|tj||r tdt r)tj |t r2tj |t r;tj|trDtj|trMtj|trdddl}|j||rf|jjdSdSdS)aY Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed). Args: seed (`int`): The seed to set. deterministic (`bool`, *optional*, defaults to `False`): Whether to use deterministic algorithms where available. Can slow down training. TrN)randomrnpr r manual_seedcudamanual_seed_allr&r mlurmusarnpur hpurxpur r+rr,r-r.)rr)r/rrrr\s0          rcCsL|jr$t|d|d}|jt|}|t|| |}|S)aL Implements the NEFTune forward pass for the model using forward hooks. Note this works only for torch.nn.Embedding layers. This method is slightly adapted from the original source code that can be found here: https://github.com/neelsjain/NEFTune Simply add it to your model as follows: ```python model = ... model.embed_tokens.neftune_noise_alpha = 0.1 model.embed_tokens.register_forward_hook(neftune_post_forward_hook) ``` Args: module (`torch.nn.Module`): The embedding module where the hook is attached. Note that you need to set `module.neftune_noise_alpha` to the desired noise alpha value. input (`torch.Tensor`): The input tensor to the model. output (`torch.Tensor`): The output tensor of the model (i.e. the embeddings). r)trainingrtensorsizeneftune_noise_alphasqrt zeros_likeuniform_)moduleinputoutputdimsmag_normrrrneftune_post_forward_hooks rHc @seZdZdZ  d deejeejfdeejeejfdeeejeejfdeeejeejffddZ d d Z d d Z dS)EvalPredictionaz Evaluation output (always contains labels), to be used to compute metrics. Parameters: predictions (`np.ndarray`): Predictions of the model. label_ids (`np.ndarray`): Targets to be matched. inputs (`np.ndarray`, *optional*): Input data passed to the model. losses (`np.ndarray`, *optional*): Loss values computed during evaluation. N predictions label_idsinputslossescCsf||_||_||_||_|j|jf|_|jdur!|j|jf7_|jdur1|j|jf7_dSdSN)rJrKrLrMelements)selfrJrKrLrMrrr__init__s  zEvalPrediction.__init__cCs t|jSrN)iterrOrPrrr__iter__s zEvalPrediction.__iter__cCs(|dks |t|jkrtd|j|S)Nrztuple index out of range)lenrO IndexError)rPidxrrr __getitem__s zEvalPrediction.__getitem__)NN) __name__ __module__ __qualname____doc__rr2ndarraytuplerrQrTrXrrrrrIs  rIc@sbeZdZUeejeejfed<eeejeejfed<ee e e fed<ee ed<dS)EvalLoopOutputrJrKmetrics num_samplesN) rYrZr[rr2r]r^__annotations__rdictstrfloatintrrrrr_s r_c@sVeZdZUeejeejfed<eeejeejfed<ee e e fed<dS)PredictionOutputrJrKr`N) rYrZr[rr2r]r^rbrrcrdrerrrrrgs rgc@s.eZdZUeed<eed<eeefed<dS) TrainOutput global_step training_lossr`N)rYrZr[rfrbrercrdrrrrrhs rh checkpoint^z\-(\d+)$csFt}fdd|D}t|dkrdStjt|dddS)Ncs4g|]}t|durtjtj|r|qSrN)_re_checkpointsearchr$pathisdirjoin).0rofolderrr s $z'get_last_checkpoint..rcSstt|dS)Nr)rfrmrngroups)xrrrz%get_last_checkpoint..key)r$listdirrUrorqmax)rtcontent checkpointsrrsrget_last_checkpoints   rc@eZdZdZdZdZdS)IntervalStrategynostepsepochNrYrZr[NOSTEPSEPOCHrrrrrrc@eZdZdZdZdZdZdS) SaveStrategyrrrbestN)rYrZr[rrrBESTrrrrr rc@r)EvaluationStrategyrrrNrrrrrrrrc@r) HubStrategyend every_saverkall_checkpointsN)rYrZr[END EVERY_SAVE CHECKPOINTALL_CHECKPOINTSrrrrrrrc@sNeZdZUdZeed<eeeefed<e ee fed<dZ e e ed<dS)BestRunac The best run found by a hyperparameter search (see [`~Trainer.hyperparameter_search`]). Parameters: run_id (`str`): The id of the best run (if models were saved, the corresponding checkpoint will be in the folder ending with run-{run_id}). objective (`float`): The objective that was obtained for this run. hyperparameters (`Dict[str, Any]`): The hyperparameters picked to get this run. run_summary (`Optional[Any]`): A summary of tuning experiments. `ray.tune.ExperimentAnalysis` object for Ray backend. run_id objectivehyperparametersN run_summary) rYrZr[r\rdrbrrelistrcrrrrrrrrs rr`returncCsft|}|dd}|dd}dd|D}|D]}||d}qt|dkr-|St|S)aj The default objective to maximize/minimize when doing an hyperparameter search. It is the evaluation loss if no metrics are provided to the [`Trainer`], the sum of all metrics otherwise. Args: metrics (`Dict[str, float]`): The metrics returned by the evaluate method. Return: `float`: The objective to minimize or maximize eval_lossNrcSs.g|]}|ds|ds|dr|qS)_runtime _per_second_compilation_time)endswithrrmrrrru!sz-default_compute_objective..r)copydeepcopypopkeysrUsumvalues)r`lossr speed_metricssmrrrdefault_compute_objectives  rcCsVddlm}|s Jd|jddddd|d dd |d dd |d gddS)Nr)is_optuna_availablez:This function needs Optuna installed: `pip install optuna` learning_rateư>-C6?T)lognum_train_epochsr(per_device_train_batch_size @rrrr) integrationsr suggest_float suggest_intsuggest_categorical)trialrrrrdefault_hp_space_optuna+s   rcCs^ddlm}|s Jdddlm}|dd|ttdd|dd |gd d S) Nr)is_ray_tune_availablez:This function needs ray installed: `pip install ray[tune]`r)tunerrrrr) rrrayr loguniformchoicerrangeuniform)rrrrrrdefault_hp_space_ray7s     rcCsDddddddddd dd d d dd ddd d gddddgS)Nrr)minr}rdoubler)boundsnametypetransformationrrrrf)rrrrr)48163264r categorical)categorical_valuesrrr)rrrrdefault_hp_space_sigoptEsrcCsVddlm}|s tddddddd d d d dd d d ddd dgdiddS)Nr)is_wandb_availablez8This function needs wandb installed: `pip install wandb`r1rminimize)rgoalrrr) distributionrr} int_uniformrrrrr)methodmetric parameters)rr ImportError)rrrrrdefault_hp_space_wandbRs     rc@r)HPSearchBackendoptunarsigoptwandbN)rYrZr[OPTUNARAYSIGOPTWANDBrrrrrdrrcCs,trddlmm}|dkS|dvS)z Whether or not the current process is the local process, based on `xm.get_ordinal()` (for TPUs) first, then on `local_rank`. rN)r)rtorch_xla.core.xla_modelcore xla_model get_ordinal) local_rankxmrrris_main_processks rcCsDtrddlmm}|S|dkr tr ddl}|jSdS)zg Return the number of processes launched in parallel. Works with `torch.distributed` and TPUs. rNrr) rrrrxrt_world_sizer r distributedget_world_size)rrrrrrtotal_processes_numberws rc Cst|}|dt|di}|dkr|S|dur(||}t|d||d<|dur:||}t|d||d<|durL||} t| d||d<|S) a Measure and return speed performance metrics. This function requires a time snapshot `start_time` before the operation to be measured starts and this function should be run immediately after the operation to be measured has completed. Args: - split: name to prefix metric (like train, eval, test...) - start_time: operation start time - num_samples: number of samples processed - num_steps: number of steps processed - num_tokens: number of tokens processed rrrN_samples_per_second_steps_per_second_tokens_per_second)timeround) split start_timera num_steps num_tokensruntimeresultsamples_per_secondsteps_per_secondtokens_per_secondrrrrs rc@s8eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d S) SchedulerTypeaL Scheduler names for the parameter `lr_scheduler_type` in [`TrainingArguments`]. By default, it uses "linear". Internally, this retrieves `get_linear_schedule_with_warmup` scheduler from [`Trainer`]. Scheduler types: - "linear" = get_linear_schedule_with_warmup - "cosine" = get_cosine_schedule_with_warmup - "cosine_with_restarts" = get_cosine_with_hard_restarts_schedule_with_warmup - "polynomial" = get_polynomial_decay_schedule_with_warmup - "constant" = get_constant_schedule - "constant_with_warmup" = get_constant_schedule_with_warmup - "inverse_sqrt" = get_inverse_sqrt_schedule - "reduce_lr_on_plateau" = get_reduce_on_plateau_schedule - "cosine_with_min_lr" = get_cosine_with_min_lr_schedule_with_warmup - "warmup_stable_decay" = get_wsd_schedule linearcosinecosine_with_restarts polynomialconstantconstant_with_warmup inverse_sqrtreduce_lr_on_plateaucosine_with_min_lrwarmup_stable_decayN)rYrZr[r\LINEARCOSINECOSINE_WITH_RESTARTS POLYNOMIALCONSTANTCONSTANT_WITH_WARMUP INVERSE_SQRTREDUCE_ON_PLATEAUCOSINE_WITH_MIN_LRWARMUP_STABLE_DECAYrrrrr sr c@sdeZdZdZddddddZddd Zd d Zd d ZddZddZ ddZ ddZ dddZ dS)TrainerMemoryTrackera A helper class that tracks cpu and gpu memory. This class will silently skip unless `psutil` is available. Install with `pip install psutil`. When a stage completes, it can pass metrics dict to update with the memory metrics gathered during this stage. Example : ```python self._memory_tracker = TrainerMemoryTracker(self.args.skip_memory_metrics) self._memory_tracker.start() # code ... metrics = {"train_runtime": 10.5} self._memory_tracker.stop_and_update_metrics(metrics) ``` At the moment GPU tracking is only for `pytorch`, but can be extended to support `tensorflow`. To understand this class' intricacies please read the documentation of [`~Trainer.log_metrics`]. inittrainevaltest)rQr"_inner_training_loopevaluatepredictFcCs||_ts d|_|jrdSddl}tststr&ddl}||_i|_n;tr4ddl}||_i|_n-t rBddl}||_i|_nt rPddl}||_i|_nt r^ddl}||_i|_nd|_| |_ d|_i|_d|_dS)NTrF)skip_memory_metricsrpsutilr r rrgpurrrr Processprocess cur_stagecpu init_reported)rPr(r)rrrrrQs>  zTrainerMemoryTracker.__init__cCs>tjjjj}||jvr|j|Std|d|j)z+derives the stage/caller name automaticallyzwas called from z+, but only expect to be called from one of )inspect currentframef_backf_codeco_namestages ValueErrorr)rPcallerrrr derive_stages   z!TrainerMemoryTracker.derive_stagecCs |jjS)z4get resident set size memory for the current process)r, memory_inforssrSrrr cpu_mem_useds z!TrainerMemoryTracker.cpu_mem_usedcCs&d|_ t||j|_|jsdSq)Nr)cpu_mem_used_peakr}r;peak_monitoringrSrrrpeak_monitor_funcs z&TrainerMemoryTracker.peak_monitor_funccCs|jrdS|}|jdur|j|krdS||_t|jdurtjr3|jj|jj nSt rC|jj |jj nCt rS|jj |jj n3trc|jj|jj n#trs|jj|jj ntr}|jjn tr|jj |jdurtjr|jj|_nAt r|jj |_n6t r|jj |_n+tr|jj|_n tr|jj|_ntr|jj|_n tr|jj|_||_d|_tj|jd}d|_| dS)z%start tracking for the caller's stageNT)target)!r(r8r-gccollectrr4 is_availablereset_peak_memory_stats empty_cacher r6rr7rr:rr8r r9rmpsmemory_allocatedgpu_mem_used_at_startcurrent_allocated_memoryr;cpu_mem_used_at_startr= threadingThreadr>daemonstart)rPstagepeak_monitor_threadrrrrM)s^            zTrainerMemoryTracker.startcCs^|jdur |j|kr dSd|_t|jdurYtjr$|jjn5tr.|jj n+t r8|jj n!t rB|jj ntrL|jjn trPn trY|jj|jdur tjrs|jj|_|jj|_nltr|jj |_|jj |_nZt r|jj |_|jj |_nHt r|jj |_|jj |_n6tr|jj|_|jj|_n$tr|jj|_|jj|_ntr|jj|_d|_ntd|j|j|j|jd|j|j<|jdurtd|j|j|j|jd<nd|j|jd<||_|j|j|j|jtd|j |jd|j!|j<d|_dS) z"stop tracking for the passed stageNFzNo available GPU device found!)beginrallocrpeakedz Not available)rPrrQrR)"r-r=r@rArr4rBrDr r6rr7rr:rr8r rrErFgpu_mem_used_nowmax_memory_allocatedgpu_mem_used_peakr9rHr6rGr*r}r;cpu_mem_used_nowrIr<r.)rPrNrrrstopgsp           zTrainerMemoryTracker.stopcCs|jrdS|jdur|j|krdS|g}|js |ddd|_|D]B}dD]=}||jvrC||j|vrC|j||||d|d<|jdurc||jvrc||j|vrc|j||||d|d<q&q"|ddkr|jdd |d <|jdur|jdd |d <dSdSdS) zupdates the metricsNrr!T)rQrR _mem_cpu__delta _mem_gpu_rPbefore_init_mem_cpubefore_init_mem_gpu)r(r-r/insertr.rr*)rPrNr`r5trrrupdate_metricss, "  z#TrainerMemoryTracker.update_metricsNcCs8|jrdS|}|||dur|||dSdS)zrMrWr_r`rrrrr s  .  >V)r cCs6zt|duWStyYdStyYdSw)zR Checks if the dataset implements __len__() and it doesn't raise an error NF)rU TypeErrorAttributeError)datasetrrr has_lengths  recCst|ttfrt|dd|DSt|tr$t|dd|DSt|tjr.|St rAt|t j rA| dkrA|S|S)zM Recursively calls `.item()` on the element of the dictionary passed css|]}t|VqdSrNdenumpify_detensorizerrrr z(denumpify_detensorize..cSsi|] \}}|t|qSrrfrrkvrrr sz)denumpify_detensorize..r) isinstancerr^rrcitemsr2genericitemr rTensornumel)r`rrrrgs  rgcCsFt|tjrtt|jj}|t|jt|j Stt|jS)za Return the number of arguments of the passed function, even if it's a partial function. ) rn functoolspartialrUr0 signaturefuncrargskeywords)rw total_argsrrrnumber_of_argumentss r{functionstarting_batch_sizeauto_find_batch_sizecCsL|dur tjt||dS|rttdddlm}|||dStj||dS)a% Args: A basic decorator that will try to execute `function`. If it fails from exceptions related to out-of-memory or CUDNN, the batch size is cut in half and passed to `function`. `function` must take in a `batch_size` parameter as its first argument. function (`callable`, *optional*) A function to wrap starting_batch_size (`int`, *optional*) The batch size to try and fit into memory auto_find_batch_size (`bool`, *optional*) If False, will just execute `function` N)r~r accelerater)find_executable_batch_size)r}r~) batch_size)rtrurraccelerate.utils)r}r~r%accelerate_find_executable_batch_sizerrrrs   rc@s(eZdZdZdZdZdZdZdZdZ dS) FSDPOption full_shard shard_grad_opno_shard hybrid_shardhybrid_shard_zero2offload auto_wrapN) rYrZr[ FULL_SHARD SHARD_GRAD_OPNO_SHARD HYBRID_SHARDHYBRID_SHARD_ZERO2OFFLOAD AUTO_WRAPrrrrr9src@sVeZdZdZ   ddeedeefddZdedefd d Zd e efd d Z dS)RemoveColumnsCollatorzWWrap the data collator to remove unused columns before they are passed to the collator.N model_name descriptioncCs(||_||_||_||_||_d|_dS)NF) data_collatorsignature_columnsloggerrrmessage_logged)rPrrrrrrrrrQFs  zRemoveColumnsCollator.__init__featurerc st|ts|SjsRjrRjrRtt|tj}t |dkrRj dur*dndj d}j d|djdd |d d |d jd d _fd d| DS)Nrzin the z setzThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: z, z. If z are not expected by `z/.forward`, you can safely ignore this message.Tcs i|] \}}|jvr||qSr)rrjrSrrrmcs z9RemoveColumnsCollator._remove_columns..)rnrcrrrrsetrrrUrinforqro)rPrignored_columnsdset_descriptionrrSr_remove_columnsUs&  z%RemoveColumnsCollator._remove_columnsfeaturescsfdd|D}|S)Ncsg|]}|qSr)r)rrrrSrrrufryz2RemoveColumnsCollator.__call__..)r)rPrrrSr__call__es zRemoveColumnsCollator.__call__NNN) rYrZr[r\rrdrQrcrrrrrrrrCs rr{return_is_regexcsd}d}t|trtt|}|ksdnd}n$|vr!d}ntfdd|Dr/d}ntfdd|Dr>d}d}|rD||fS|S)aKA helper method to check if the passed module's key name matches any of the target modules in the optim_target_modules. Args: optim_target_modules (`Union[str, List[str]]`): A list of strings to try to match. Can be also a full string. key (`str`): A key to search any matches in optim_target_modules return_is_regex (`bool`): If set to `True`, the method will return whether the passed `optim_target_modules` is a regex or not. Returns: `bool` : True of match object if key matches any target modules from config, False or None if no match found `bool` : If the matched target module is a regex to silence out the warnings in Trainer for extra modules being found (only if `target_module_found=True` for an array of regex). FTc3s|]}|vVqdSrNr)rr target_keyrzrrrhriz-check_target_module_exists..c3s |] }tt|VqdSrN)boolre fullmatch)rroptim_target_modulerzrrrhs)rnrdrrrany)optim_target_modulesr{rtarget_module_foundis_regexrrzrcheck_target_module_existsjs rrar)Nr|F)Ir\rrtr@r0r$r1rrJrtypingrrrrnumpyr2utilsrrr r r r r rrrrrrrrrfrr0rrHrIr_rgrhPREFIX_CHECKPOINT_DIRcompilermrrrrrrrcrdrerrrrrrrrrr r rergr{callablerrrrrrrrst < $%    4   '