o  h @sdZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z ddlmZmZddlmZddlmZddlmZmZmZddlZddlZddlmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$dd l%m&Z&d d l'm(Z(d dl)m*Z*d dl+m,Z,m-Z-m.Z.m/Z/mZe/re0eej1e.rddl2m3m4Z5e-rddl6m7Z7e8e9Z:ddZ;deejdyddZ?dyddZ@dyddZAddZBdd ZCd!d"ZDd#d$ZEdzd%ed&eeFd'efd(d)ZGdeHd*fd+eIeeFeJfd&eeFd,eejHd'ejd?ZSdyd@dAZTdBdCZUGdDdEdEZVeGdFdGdGZWd{dHdIZXGdJdKdKe$ZYGdLdMdMe&ZZGdNdOdOe$Z[GdPdQdQe"Z\dRdSZ]dTdUZ^dVe_e`eJfd'e_e`eJffdWdXZadYdZZbd|d\d]Zcd^d_Zdd}dadbZedzdcddZfdedfZgdgdhZhe,rddlimjmZkekld~didjZmekldkdlZndmdnZododpZpeGdqdrdrZqGdsdtdtejrjsZtGdudvdve7ZudwdxZvdS)z( Torch utilities for the Trainer class. N)IteratorMapping)contextmanager) dataclassfield)chain) StreamHandler)AnyOptionalUnion)nn)DatasetIterableDataset RandomSamplerSampler)DistributedSampler)is_deepspeed_zero3_enabled) BatchEncoding)is_sagemaker_mp_enabledis_torch_availableis_torch_xla_availableis_training_run_on_sagemakerlogging) LRSchedulercCs2t|dr|jdurt|jSt|dr|jSdS)N batch_samplersampler)hasattrrget_dataloader_samplerr) dataloaderr q/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/trainer_pt_utils.pyr@s   rtensor_or_arraycCsHt|tjrttdrt|}|S|jdkr|d}|St|}|S)N atleast_1dr) isinstancetorchTensorrr#ndimnp)r"r r r!r#Gs     r#cCst|}t|}t|jdks|jd|jdkr"tj||fddS|jd|jdt|jd|jdf|jdd}|||}||d|jdd|jdf<|||jddd|jdf<|S)z`Concatenates `tensor1` and `tensor2` on first axis, applying padding on the second if necessary.rrdimN)r#lenshaper%catmaxnew_full)tensor1tensor2 padding_index new_shaperesultr r r!torch_pad_and_concatenateRs"8   r7cCst|}t|}t|jdks|jd|jdkr"tj||fddS|jd|jdt|jd|jdf|jdd}tj|||d}||d|jdd|jdf<|||jddd|jdf<|S)z^Concatenates `array1` and `array2` on first axis, applying padding on the second if necessary.rr)axisr,Nr.)r#r-r.r( concatenater0 full_like)array1array2r4r5r6r r r!numpy_pad_and_concatenateds"8  r>cst|tjr ttjs#t|tus#Jdt|dtdt|ttfr:t|fddt|DSt|tjrGt|dSt|tr\t|fdd| DSt|t j rit |dSt d t|) z Concat the `new_tensors` to `tensors` on the first dim and pad them on the second if needed. Works for tensors or nested list/tuples/dict of tensors. zEExpected `tensors` and `new_tensors` to have the same type but found z and .c3s"|] \}}t||dVqdS)r4N nested_concat).0tnr@r r!  z nested_concat..r@cs$i|]\}}|t||dqS)r@rArCkrD new_tensorsr4r r! s$z!nested_concat..z(Unsupported type for concatenation: got )r$r%r&typelisttuplezipr7ritemsr(ndarrayr> TypeError)tensorsrKr4r rJr!rBvs    rBcCst|ttfr|D]}t|}|dur|Sq dSt|tr6|D]\}}t|}|dur3|Sq#dSt|tjrJt|j dkrH|j dSdSt|t j r^t|j dkr\|j dSdSdS)zV Find the first dimension of a tensor in a nested list/tuple/dict of tensors. Nrr) r$rNrOfind_batch_sizerrQr%r&r-r.r(rR)rTrDr6keyvaluer r r!rUs&   rUcCspt|ttfrt|dd|DSt|tr$t|dd|DS|}|jtj kr4| tj }| S)zENumpify `tensors` (even if it's a nested list/tuple/dict of tensors).cs|]}t|VqdSNnested_numpifyrCrDr r r!rFz!nested_numpify..cSi|] \}}|t|qSr rZrHr r r!rLz"nested_numpify..) r$rNrOrMrrQcpudtyper%bfloat16tofloat32numpy)rTrDr r r!r[s   r[cCs`t|ttfrt|dd|DSt|tr$t|dd|DSt|tjr.|S|S)zDDetach `tensors` (even if it's a nested list/tuple/dict of tensors).csrXrY nested_detachr\r r r!rFr]z nested_detach..cSr^r rfrHr r r!rLr_z!nested_detach..) r$rNrOrMrrQr%r&detach)rTr r r!rgs  rgcstrDddlmm}t|ttfr"t|fddt|DSt|t r8t|fddt| DSt |}| |t jStd)Nrc3s(|]\}}t|d|VqdS)_Nnested_xla_mesh_reduce)rCirDnamer r!rFs&z)nested_xla_mesh_reduce..cs*i|]\}\}}|t|d|qS)rirj)rCrlrIrDrmr r!rLs*z*nested_xla_mesh_reduce..z;Torch xla must be installed to use `nested_xla_mesh_reduce`)rtorch_xla.core.xla_modelcore xla_modelr$rNrOrM enumeraterrQr# mesh_reducer%r/ ImportError)rTrnxmr rmr!rks rktensornum_total_examplesreturncszWtttfrtfddDWSttr+tfddDWStfddtt D}t |t j |dd}durU|d}|WStybtd w) Nc3|]}t|VqdSrYdistributed_concatr\rwr r!rFz%distributed_concat..ci|] \}}|t|qSr rzrHr|r r!rLz&distributed_concat..cg|]}qSr clonerCrirvr r! z&distributed_concat..rr*(Not currently using distributed training)r$rOrNrMrrQr# contiguousrangedistget_world_size all_gatherr%r/AssertionError)rvrwoutput_tensorsconcatr )rwrvr!r{s      r{cudascalarsdevicecstz.tj||dfddttD}t|tj|dd}|dur,|d|}|WSty9tdw)N)rcrr rrtensorized_scalarr r!rrz1distributed_broadcast_scalars..rr*r)r%rvrrrrr/r)rrwrrrr rr!distributed_broadcast_scalarss   rcCs8t|dkr|D]}|jturt|j|jqdSdS)Nr)r-category UserWarningwarningswarnmessage)caught_warningswr r r!reissue_pt_warningss  r local_rankccs0|dvr tdV|dkrtdSdS)z Decorator to make all processes in distributed training wait for each local_master to do something. Args: local_rank (`int`): The rank of the local process. )rNr)rbarrier)rr r r!torch_distributed_zero_firsts rcs,eZdZdZfddZfddZZS)DistributedSamplerWithLoopa Like a torch.utils.data.distributed.DistributedSampler` but loops at the end back to the beginning of the shuffled samples to make each process have a round multiple of batch_size samples. Args: dataset (`torch.utils.data.Dataset`): Dataset used for sampling. batch_size (`int`): The batch size used with this sampler kwargs (`Dict[str, Any]`, *optional*): All other keyword arguments passed to `DistributedSampler`. c stj|fi|||_dSrY)super__init__ batch_size)selfdatasetrkwargs __class__r r!rs z#DistributedSamplerWithLoop.__init__csrtt}t||jdkrdn |jt||j}|jt|j|jkr)dnd}|||||7}t|S)Nrr) rNr__iter__r-rrankr num_replicasiter)rindices remainderstart_remainderrr r!rs *z#DistributedSamplerWithLoop.__iter__)__name__ __module__ __qualname____doc__rr __classcell__r r rr!rs rc@s@eZdZdZddedefddZdd d Zdd d ZddZ d S)EvalLoopContaineraa Container to store intermediate results of evaluation loop. Args: do_nested_concat (`bool`, *optional*, defaults to `True`): If set to `True`, each iteration will recursively concatenate a new object containing tensors to the existing stored tensors, provided that the structure of the existing object and the new one are identical. If set to `False`, all newly added tensors will be stored in a list. padding_index (`int`, *optional*, defaults to -100): Value used to pad tensors of different shapes when `do_nested_concat=True`. Tr)do_nested_concatr4cCs||_||_d|_d|_dSrY)rr4rTarrays)rrr4r r r!r1s zEvalLoopContainer.__init__rxNcCsT|jdur|jr ||_dS|g|_dS|jr"t|j||jd|_dS|j|dS)zlAdd tensors to the stored objects. If `do_nested_concat=True`, the tensors will be concatenated recursively.Nr@)rTrrBr4append)rrTr r r!add7s zEvalLoopContainer.addcCs\|jdurdSt|j}|jdur||_n|jr#t|j||jd|_n|j|d|_dS)zGMove tensors in stored objects to CPU and convert them to numpy arrays.Nr@)rTr[rrrBr4extend)r new_arraysr r r!to_cpu_and_numpy@s     z"EvalLoopContainer.to_cpu_and_numpycCs||jS)z6Returns the numpified and moved to CPU stored objects.)rrrr r r! get_arraysRszEvalLoopContainer.get_arrays)Tr))rxN) rrrrboolintrrrrr r r r!r$s   rc@s*eZdZdZd ddZddZddZdS) SequentialDistributedSamplera Distributed Sampler that subsamples indices sequentially, making it easier to collate all results at the end. Even though we only use this sampler for eval and predict (no training), which means that the model params won't have to be synced (i.e. will not hang for synchronization even if varied number of forward passes), we still add extra samples to the sampler to make it evenly divisible (like in `DistributedSampler`) to make it easy to `gather` or `reduce` resulting tensors at the end of the loop. NcCstdt|durtstdt}|dur&ts"tdt}||_||_ ||_ t |j}|durGt t |||||_n t t |||_|j|j |_||_dS)NzUSequentialDistributedSampler is deprecated and will be removed in v5 of Transformers.,Requires distributed package to be available)rr FutureWarningr is_available RuntimeErrorrget_rankrrrr-rmathceil num_samples total_sizer)rrrrrrr r r!rbs*  z%SequentialDistributedSampler.__init__cCsttt|j}||d|jt|7}t||jks+Jdt|d|jd||j|j|jd|j}t||jksPJdt|d|jdt|S)NzIndices length z and total size z mismatchedrz and sample number )rNrr-rrrrrrrr r r!r{s z%SequentialDistributedSampler.__iter__cC|jSrYrrr r r!__len__z$SequentialDistributedSampler.__len__)NNN)rrrrrrrr r r r!rXs   rrrcCs*tdkr t|St|ttdS)Nr)rr)ruxrt_world_sizerr get_ordinal)rrr r r!get_tpu_samplers rcsJt|ttfrt|fdd|DStj||g|jddRdS)z\Create the same nested structure as `arrays` with a first dimension always at `num_samples`.c3ryrY)nested_new_likerCxrr r!rFr}z"nested_new_like..rNr9)r$rNrOrMr(r;r.)rrr4r rr!rs"rcCsFtj|||jd|f|jddd}||ddd|jdf<|S)zmExpand the `arrays` so that the second dimension grows to `new_seq_length`. Uses `padding_index` for padding.rr,Nr9r)r(r;r.)rnew_seq_lengthr4r6r r r! expand_likes(rcs\t|ttfrt|fdd|DSt|tr(t|fdd|DS|dS)zQTruncate `tensors` at `limit` (even if it's a nested list/tuple/dict of tensors).c3ryrYnested_truncater\limitr r!rFr}z"nested_truncate..cr~r rrHrr r!rLrz#nested_truncate..N)r$rNrOrMrrQ)rTrr rr!rs   rc@s2eZdZdZd ddZddZdd Zd d ZdS) DistributedTensorGathereraS A class responsible for properly gathering tensors (or nested list/tuple of tensors) on the CPU by chunks. If our dataset has 16 samples with a batch size of 2 on 3 processes and we gather then transfer on CPU at every step, our sampler will generate the following indices: `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1]` to get something of size a multiple of 3 (so that each process gets the same dataset length). Then process 0, 1 and 2 will be responsible of making predictions for the following samples: - P0: `[0, 1, 2, 3, 4, 5]` - P1: `[6, 7, 8, 9, 10, 11]` - P2: `[12, 13, 14, 15, 0, 1]` The first batch treated on each process will be: - P0: `[0, 1]` - P1: `[6, 7]` - P2: `[12, 13]` So if we gather at the end of the first batch, we will get a tensor (nested list/tuple of tensor) corresponding to the following indices: `[0, 1, 6, 7, 12, 13]` If we directly concatenate our results without taking any precautions, the user will then get the predictions for the indices in this order at the end of the prediction loop: `[0, 1, 6, 7, 12, 13, 2, 3, 8, 9, 14, 15, 4, 5, 10, 11, 0, 1]` For some reason, that's not going to roll their boat. This class is there to solve that problem. Args: world_size (`int`): The number of processes used in the distributed training. num_samples (`int`): The number of samples in our dataset. make_multiple_of (`int`, *optional*): If passed, the class assumes the datasets passed to each process are made to be a multiple of this argument (by adding samples). padding_index (`int`, *optional*, defaults to -100): The padding index to use if the arrays don't all have the same sequence length. Nr)cCsftdt||_||_|dur|n||}tt||||_|j||_ d|_ d|_ ||_ dS)NzRDistributedTensorGatherer is deprecated and will be removed in v5 of Transformers.) rrr world_sizerrr(r total_samplesprocess_length_storage_offsetsr4)rrrmake_multiple_ofr4rr r r!rs  z"DistributedTensorGatherer.__init__cCsz|durdS|jdur t||j|jd|_ttd|j|j|_||j|\}|_t|j D] }|j||7<q/dS)z Add `arrays` to the internal storage, Will initialize the storage to the full size at the first arrays passed so that if we're bound to get an OOM, it happens at the beginning. Nr@r) rrrr4rNrrr_nested_set_tensorsr)rr slice_lenrlr r r! add_arrayss z$DistributedTensorGatherer.add_arrayscsXt|ttfr$fddt||D}|ddt|dd|DfS|jdjdks=Jdjd|jdd|jdj}tjD]]}t|jd krl||||d ||j |j ||<qJt|jd kr|jd |jd krt ||jd j d }||||d ||j |j ||d|jd f<qJ||fS) Ncsg|] \}}||qSr )r)rCryrr r!rrzADistributedTensorGatherer._nested_set_tensors..rcss|]}|dVqdS)rNr )rCrr r r!rFr]z@DistributedTensorGatherer._nested_set_tensors..zNot all data has been set. Are you sure you passed all values?)rrrloggerwarningrrrr r r!finalizes  z"DistributedTensorGatherer.finalize)Nr))rrrrrrrrr r r r!rs  - rc@s4eZdZUdZdZeed<dZeed<d ddZ d S) LabelSmoothera@ Adds label-smoothing on a pre-computed output from a Transformers model. Args: epsilon (`float`, *optional*, defaults to 0.1): The label smoothing factor. ignore_index (`int`, *optional*, defaults to -100): The index in the labels to ignore when computing the loss. g?epsilonr) ignore_indexFc Cst|tr |dn|d}|r&|dddddf}|dddf}tjj|dd }||dkr>|d}||j }t j |dd}|j d|d}|j dd t jd }||d ||d || } | | }| | |jd}d|j||j|S) Nlogitsr.rrr*)min)r+indexT)r+keepdimrag)r$dictrr functional log_softmaxr+ unsqueezeeqrr%clampgathersumrd masked_fill_numellongr.r) r model_outputlabels shift_labelsr log_probs padding_masknll_loss smoothed_lossnum_active_elementsr r r!__call__)s"     zLabelSmoother.__call__NF) rrrrrfloat__annotations__rrrr r r r!rs  rcs|durtt|dd}|dkrd}tjt|d||fddtdtD}fd d|D}fd d|D}tt|}||d|dd|dd<||d<d d|DS) a Return a list of indices so that each slice of `batch_size` consecutive indices correspond to elements of similar lengths. To do this, the indices are: - randomly permuted - grouped in mega-batches of size `mega_batch_mult * batch_size` - sorted by length in each mega-batch The result is the concatenation of all mega-batches, with the batch of `batch_size` containing the element of maximum length placed first, so that an OOM happens sooner rather than later. N2rr generatorcs g|] }||qSr )tolist)rCrl)rmegabatch_sizer r!r[s z.get_length_grouped_indices..cs"g|] }t|fddddqS)cs|SrYr )rllengthsr r!\sz7get_length_grouped_indices...T)rVreverse)sortedrC megabatchrr r!r\s"csg|]}|dqS)rr rrr r!r`cSsg|] }|D]}|qqSr r )rCrrlr r r!rer_)rr-r%randpermrargmaxrvitem)rrmega_batch_multr megabatchesmegabatch_maximumsmax_idxr )rrrr!get_length_grouped_indicesEs  *r!c @sTeZdZdZ    d dedeedeeedeefddZ d d Z d d Z dS)LengthGroupedSamplerz Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while keeping a bit of randomness. Nrrrmodel_input_namecs|dur |dur td||_|durAdurndt|dts)t|dtr/|dvr7tddfdd|D}nt|tjrPtd| }||_ ||_ dS) N,One of dataset and lengths must be provided. input_idsrXCan only automatically infer lengths for datasets whose items are dictionaries with an '' key.cg|]}t|qSr r-rCfeaturer#r r!rrz1LengthGroupedSampler.__init__..zcIf lengths is a torch.Tensor, LengthGroupedSampler will be slow. Converting lengths to List[int]...) ValueErrorrr$rrr%r&rinforrr)rrrrr#rr r,r!rns.     zLengthGroupedSampler.__init__cCs t|jSrY)r-rrr r r!rs zLengthGroupedSampler.__len__cCst|j|j|jd}t|SNr)r!rrrrrr r r!rszLengthGroupedSampler.__iter__)NNNN) rrrrrr r rNstrrrrr r r r!r"hs"   r"c@speZdZdZ       ddedeedeedeed ed ed eeed ee fd dZ de fddZ dS)DistributedLengthGroupedSamplerz Distributed Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while keeping a bit of randomness. NrFrrrrseed drop_lastrr#c sh|dur |dur td|durtstdt}|dur,ts(tdt}||_||_||_d|_ ||_ |durmdurEndt |dt sUt |dt r[|dvrctddfdd|D}nt |tjr|td |}||_|j rt|j|jdkrtt|j|j|j|_n tt|j|j|_|j|j|_||_dS) Nr$rrr%r&r'cr(r r)r*r,r r!rrz.znIf lengths is a torch.Tensor, DistributedLengthGroupedSampler will be slow. Converting lengths to List[int]...)r-rrrrrrrrepochr3r$rrr%r&rr.rrr-rrrrr2) rrrrrr2r3rr#r r,r!rsN       z(DistributedLengthGroupedSampler.__init__rxcCst}||j|jt|j|j|d}|js'||d|j t |7}n|d|j }t ||j ks7J||j |j |j }t ||j ksJJt|Sr/)r% Generator manual_seedr2r4r!rrr3rr-rrrr)rgrr r r!rsz(DistributedLengthGroupedSampler.__iter__)NNNrFNN) rrrrrr r rrNr0rrrr r r r!r1s6    If your IterableDataset implements some randomization that needs to be applied the same way on all processes (for instance, a shuffling), you should use a `torch.Generator` in a `generator` attribute of the `dataset` to generate your random numbers and call the [`~trainer_pt_utils.IterableDatasetShard.set_epoch`] method of this object. It will set the seed of this `generator` to `seed + epoch` on all processes before starting the iteration. Alternatively, you can also implement a `set_epoch()` method in your iterable dataset to deal with this. Args: dataset (`torch.utils.data.IterableDataset`): The batch sampler to split in several shards. batch_size (`int`, *optional*, defaults to 1): The size of the batches per shard. drop_last (`bool`, *optional*, defaults to `False`): Whether or not to drop the last incomplete batch or complete the last batches by using the samples from the beginning. num_processes (`int`, *optional*, defaults to 1): The number of processes running concurrently. process_index (`int`, *optional*, defaults to 0): The index of the current process. seed (`int`, *optional*, defaults to 0): A random seed that will be used for the random number generation in [`~trainer_pt_utils.IterableDatasetShard.set_epoch`]. rFrrrr3r9r:r2cCs4||_||_||_||_||_||_d|_d|_dS)Nr)rrr3r9r:r2r4 num_examples)rrrr3r9r:r2r r r!rFs  zIterableDatasetShard.__init__cCs&||_t|jdr|j|dSdS)N set_epoch)r4rrrA)rr4r r r!rAXs zIterableDatasetShard.set_epochccs2d|_t|jds#t|jdr#t|jjtjr#|jj|j|j |j |j }t |j |j |j d|j }d}g}|jD](}|jd7_||t||krg|D]}||VqU|dure|}g}q?|jst|dkr|dury|}t||kr||7}t||ks|D] }||VqdSdSdS)NrrArr)r@rrr$rr%r5r6r2r4rr9rr:rr-copyr3)rreal_batch_size process_slice first_batch current_batchelementrlr r r!r]sB          zIterableDatasetShard.__iter__cCsD|jrt|j|j|j|jStt|j|j|j|jSrY)r3r-rrr9rrrr r r!r~s"zIterableDatasetShard.__len__N)rFrrr) rrrrrrrrrArrr r r r!r?s.+  !r?c Cs|jr0z |jd}W|Sty/}zdt|vr#tdd}nWYd}~|Sd}~wwt|jtj jj rB|j j dd}n|jd}t |rR|}|S)Nrzneed to call stepzQtried to get lr value before scheduler/optimizer started stepping, returning lr=0lr)is_deepspeed_enabled lr_scheduler get_last_lrrr0rrr$r%optimReduceLROnPlateau optimizer param_groups is_tensorr)rlast_lrer r r!_get_learning_rates&     rScCs4tt|t|d}tjt|dd|dS)zN Convert seconds to hh:mm:ss.msec, msecs rounded to 2 decimal places. d)secondsr?02d)rabsdatetime timedelta)secsmsecr r r!_secs2timedeltasr\metricscCs|}|D];\}}d|vr|d?d||<qd|vr%t|||<q|dkr5t|d?d||<qt||trCt|d||<q|S) z Reformat Trainer metrics values to a human-readable format. Args: metrics (`Dict[str, float]`): The metrics returned from train/evaluate/predict Returns: metrics (`Dict[str, float]`): The reformatted metrics _mem_MB_runtime total_flosGFr )rBrQr\rr$r round)rr] metrics_copyrIvr r r!metrics_formats rhcCs|sdStd|d||}tdd|D}tdd|D}t|D]}td|d|d ||d |q/dS) a? Log metrics in a specially formatted way. Under distributed environment this is done only for a process with rank 0. Args: split (`str`): Mode/split name: one of `train`, `eval`, `test` metrics (`Dict[str, float]`): The metrics returned from train/evaluate/predictmetrics: metrics dict Notes on memory reports: In order to get memory usage report you need to install `psutil`. You can do that with `pip install psutil`. Now when this method is run, you will see a report that will include: ``` init_mem_cpu_alloc_delta = 1301MB init_mem_cpu_peaked_delta = 154MB init_mem_gpu_alloc_delta = 230MB init_mem_gpu_peaked_delta = 0MB train_mem_cpu_alloc_delta = 1345MB train_mem_cpu_peaked_delta = 0MB train_mem_gpu_alloc_delta = 693MB train_mem_gpu_peaked_delta = 7MB ``` **Understanding the reports:** - the first segment, e.g., `train__`, tells you which stage the metrics are for. Reports starting with `init_` will be added to the first stage that gets run. So that if only evaluation is run, the memory usage for the `__init__` will be reported along with the `eval_` metrics. - the third segment, is either `cpu` or `gpu`, tells you whether it's the general RAM or the gpu0 memory metric. - `*_alloc_delta` - is the difference in the used/allocated memory counter between the end and the start of the stage - it can be negative if a function released more memory than it allocated. - `*_peaked_delta` - is any extra memory that was consumed and then freed - relative to the current allocated memory counter - it is never negative. When you look at the metrics of any stage you add up `alloc_delta` + `peaked_delta` and you know how much memory was needed to complete that stage. The reporting happens only for process of rank 0 and gpu 0 (if there is a gpu). Typically this is enough since the main process does the bulk of work, but it could be not quite so if model parallel is used and then other GPUs may use a different amount of gpu memory. This is also not the same under DataParallel where gpu0 may require much more memory than the rest since it stores the gradient and optimizer states for all participating GPUs. Perhaps in the future these reports will evolve to measure those too. The CPU RAM metric measures RSS (Resident Set Size) includes both the memory which is unique to the process and the memory shared with other processes. It is important to note that it does not include swapped out memory, so the reports could be imprecise. The CPU peak memory is measured using a sampling thread. Due to python's GIL it may miss some of the peak memory if that thread didn't get a chance to run when the highest memory was used. Therefore this report can be less than reality. Using `tracemalloc` would have reported the exact peak memory, but it doesn't report memory allocations outside of python. So if some C++ CUDA extension allocated its own memory it won't be reported. And therefore it was dropped in favor of the memory sampling approach, which reads the current process memory usage. The GPU allocated and peak memory reporting is done with `torch.cuda.memory_allocated()` and `torch.cuda.max_memory_allocated()`. This metric reports only "deltas" for pytorch-specific allocations, as `torch.cuda` memory management system doesn't track any memory allocated outside of pytorch. For example, the very first cuda call typically loads CUDA kernels, which may take from 0.5 to 2GB of GPU memory. Note that this tracker doesn't account for memory allocations outside of [`Trainer`]'s `__init__`, `train`, `evaluate` and `predict` calls. Because `evaluation` calls may happen during `train`, we can't handle nested invocations because `torch.cuda.max_memory_allocated` is a single counter, so if it gets reset by a nested eval call, `train`'s tracker will report incorrect info. If this [pytorch issue](https://github.com/pytorch/pytorch/issues/16266) gets resolved it will be possible to change this class to be re-entrant. Until then we will only track the outer level of `train`, `evaluate` and `predict` methods. Which means that if `eval` is called during `train`, it's the latter that will account for its memory usage and that of the former. This also means that if any other tool that is used along the [`Trainer`] calls `torch.cuda.reset_peak_memory_stats`, the gpu peak memory stats could be invalid. And the [`Trainer`] will disrupt the normal behavior of any such tools that rely on calling `torch.cuda.reset_peak_memory_stats` themselves. For best performance you may want to consider turning the memory profiling off for production runs. Nz***** z metrics *****cs|] }tt|VqdSrYr-r0rr r r!rFzlog_metrics..csrirYrjrr r r!rFrkz z )is_world_process_zeroprintrhr0keysvaluesr)rsplitr]metrics_formattedk_widthv_widthrVr r r! log_metricssO *ruTcCs |sdStj|jj|d}t|d}tj||dddWdn1s+wY|rtj|jjd}tj |r[t| }t |}Wdn1sUwYni}| |t|d}tj||dddWddS1s|wYdSdS)a Save metrics into a json file for that split, e.g. `train_results.json`. Under distributed environment this is done only for a process with rank 0. Args: split (`str`): Mode/split name: one of `train`, `eval`, `test`, `all` metrics (`Dict[str, float]`): The metrics returned from train/evaluate/predict combined (`bool`, *optional*, defaults to `True`): Creates combined metrics by updating `all_results.json` with metrics of this call To understand the metrics please read the docstring of [`~Trainer.log_metrics`]. The only difference is that raw unformatted numbers are saved in the current method. Nz _results.jsonrr T)indent sort_keyszall_results.json) rmospathjoinargs output_diropenjsondumpexistsloadupdate)rrqr]combinedryf all_metricsr r r! save_metricss&      "rcCs.|sdStj|jjd}|j|dS)z Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model. Under distributed environment this is done only for a process with rank 0. Nztrainer_state.json)rmrxryrzr{r|state save_to_json)rryr r r! save_stateDsrFcs4trddnddtfdd|DS)zo Calculate model's total param count. If trainable_only is True then count only those requiring grads. cSst|dr|jS|S)Nds_numel)rrrpr r r!rWsz$get_model_param_count..numelcSs|SrY)rrr r r!r\sc3s"|] }r |jr|VqdSrY) requires_grad)rCrrtrainable_onlyr r!rF_rGz(get_model_param_count..)rr parameters)modelrr rr!get_model_param_countQs rcsjdurgg}|D]\t}|fdd|D7}q |fdd|jD7}|S)zZ Returns the names of the model parameters that are not inside a forbidden layer. Ncs@g|]ttstfddDsdqS)c3s&|]}|dvVqdS)r?NlowerrC forbidden)rErnr r!rFos$1get_parameter_names...r?)r$rOanyrCchildforbidden_layer_namesforbidden_layer_typesrn)rEr!rks  z'get_parameter_names..cs&g|]tfddDsqS)c3s|] }|vVqdSrYrrrIr r!rFsrkr)rr)rrr!rrs )named_childrenget_parameter_names _parametersro)rrrr6 child_paramsr rr!rbs  rcCsVt|}|jj|kr|jSt|dkrdS|D]}t||}|dur(|SqdS)z Gets a class from a module by its name. Args: module (`torch.nn.Module`): The module to get the class from. name (`str`): The name of the class. rN)rNchildrenrrr-get_module_class_from_name)modulernmodules_children child_module module_classr r r!rxs    rcCs:|r|D]}tj||}tj|rt|qdSdSrY)rxryrzisfileremove)is_main_processr| filenamesfilenamefiler r r!remove_dummy_checkpoints  rcCs>|di|}t|tr|dn|d}||}|||S)Nlossrr )r$rbackward)rinputsgradient_accumulation_stepsoutputsrr r r!smp_forward_backwards  rcCs|di|S)Nr r )rrr r r!smp_forward_onlysrcCst|ttfrt|dd|DSt|tr$t|dd|DSt|tjs4tdt|dt |t j j }dd|D}tj d d|Dd d S) NcsrXrY smp_gatherr\r r r!rFr]zsmp_gather..cSr^r rrCrIrgr r r!rLr_zsmp_gather..z Can't gather the values of type z-, only of nested list/tuple/dicts of tensors.cSsg|]}t|qSr )r#r\r r r!rrzsmp_gather..cSsg|]}|qSr )r`r\r r r!rrrr*)r$rNrOrMrrQr%r&rSsmp allgather CommGroupDP_GROUPr/)rv all_tensorsr r r!rs  rcCsXt|ttfrt|dd|DSt|tr$t|dd|DS|S)NcsrXrYsmp_nested_concatr\r r r!rFr]z$smp_nested_concat..cSr^r rrr r r!rLr_z%smp_nested_concat..) r$rNrOrMrrQrhrr`rr r r!rs  rc@seZdZUdZedddidZeed<edddidZe eed <ed dd idZ eed <ed dd idZ eed<edddidZ e eed<edddidZ e eed<edddidZeed<eddZddZdddZdS)AcceleratorConfigaI A subset of arguments relating to the underlying [`accelerate.Accelerator`] implementation utilized in the `Trainer` that can be customized. Mostly relating to data. Parameters: split_batches (`bool`, *optional*, defaults to `False`): Whether or not the accelerator should split the batches yielded by the dataloaders across the devices. If `True` the actual batch size used will be the same on any kind of distributed processes, but it must be a round multiple of the `num_processes` you are using. If `False`, actual batch size used will be the one set in your script multiplied by the number of processes. dispatch_batches (`bool`, *optional*): If set to `True`, the dataloader prepared by the Accelerator is only iterated through on the main process and then the batches are split and broadcast to each process. Will default to `True` for `DataLoader` whose underlying dataset is an `IterableDataset`, `False` otherwise. even_batches (`bool`, *optional*, defaults to `True`): If set to `True`, in cases where the total batch size across all processes does not exactly divide the dataset, samples at the start of the dataset will be duplicated so the batch can be divided equally among all workers. use_seedable_sampler (`bool`, *optional*, defaults to `True`): Whether or not use a fully seedable random sampler ([`accelerate.data_loader.SeedableRandomSampler`]). Ensures training results are fully reproducible using a different sampling technique. While seed-to-seed results may differ, on average the differences are negligible when using multiple different seeds to compare. Should also be ran with [`~utils.set_seed`] for the best results. gradient_accumulation_kwargs (`dict`, *optional*): Additional kwargs to configure gradient accumulation, see [`accelerate.utils.GradientAccumulationPlugin`]. Any of the following (optional) keys are acceptable: num_steps (`int`): Will take precedence over [`~.TrainingArguments.gradient_accumulation_steps`] if the latter is set to 1, otherwise an exception will be raised. adjust_scheduler (`bool`): Whether to adjust the scheduler steps to account for [`~.TrainingArguments.gradient_accumulation_steps`]. The [`accelerate.utils.GradientAccumulationPlugin`] default is `True`. sync_each_batch (`bool`): Whether to synchronize the gradients at each data batch. The [`accelerate.utils.GradientAccumulationPlugin`] default is `False`. non_blocking (`bool`, *optional*, defaults to `False`): Whether to use non-blocking CUDA calls to help minimize synchronization during distributed training with prepared `DataLoader` inputs being moved to device. Best if used with `pin_memory=True` in the `TrainingArguments`. use_configured_state (`bool*, *optional*, defaults to `False`): Whether or not to use a pre-configured `AcceleratorState` or `PartialState` defined before calling `TrainingArguments`. If `True`, an `Accelerator` or `PartialState` must be initialized. May lead to issues using sweeps or hyperparameter tuning. FhelpauWhether or not the accelerator should split the batches yielded by the dataloaders across the devices. If `True` the actual batch size used will be the same on any kind of distributed processes, but it must be a round multiple of the `num_processes` you are using. If `False`, actual batch size used will be the one set in your script multiplied by the number of processes.)defaultmetadata split_batchesNaIf set to `True`, the dataloader prepared by the Accelerator is only iterated through on the main process and then the batches are split and broadcast to each process. Will default to `True` for `DataLoader` whose underlying dataset is an `IterableDataslet`, `False` otherwise.dispatch_batchesTzIf set to `True`, in cases where the total batch size across all processes does not exactly divide the dataset, samples at the start of the dataset will be duplicated so the batch can be divided equally among all workers. even_batchesa}Whether or not use a fully seedable random sampler ([`accelerate.data_loader.SeedableRandomSampler`]).Ensures training results are fully reproducible using a different sampling technique. While seed-to-seed results may differ, on average the differences are negligible when usingmultiple different seeds to compare. Should also be ran with [`~utils.set_seed`] for the best results.use_seedable_samplerzWhether to use non-blocking CUDA calls to help minimize synchronization during distributed training with prepared `DataLoader` inputs being moved to device. Best if used with `pin_memory=True` in the `TrainingArguments`. Requires accelerate v0.30.0. non_blockingaAdditional kwargs to configure gradient accumulation, see [`accelerate.utils.GradientAccumulationPlugin`]. Any of the following (optional) keys are acceptable: num_steps (`int`): Will take precedence over [`~.TrainingArguments.gradient_accumulation_steps`] if the latter is set to 1, otherwise an exception will be raised. adjust_scheduler (`bool`): Whether to adjust the scheduler steps to account for [`~.TrainingArguments.gradient_accumulation_steps`]. The [`accelerate.utils.GradientAccumulationPlugin`] default is `True`. sync_each_batch (`bool`): Whether to synchronize the gradients at each data batch. The [`accelerate.utils.GradientAccumulationPlugin`] default is `False`.gradient_accumulation_kwargszWhether or not to use a pre-configured `AcceleratorState` or `PartialState` defined before calling `TrainingArguments`.If `True`, an `Accelerator` or `PartialState` must be initialized. May lead to issues using sweeps or hyperparameter tuning.use_configured_statecstj|r tjnt}||ddd }t|}Wdn1s"wYtfdd|D}t |dkrEt d|d|d d i|S) Nrzutf-8)encodingc3s"|] }|jvr|VqdSrY)__dataclass_fields__ro)rCrVclsr r!rF2rGz3AcceleratorConfig.from_json_file..rzThe config file at z had unknown keys (zu), please try upgrading your `transformers` version or fix (and potentially remove these keys) from your config file.r ) rxryrior}r~rrror-r-)r json_file open_filer config_dict extra_keysr rr!from_json_file+s  z AcceleratorConfig.from_json_filecCs t|jSrY)rBdeepcopy__dict__rr r r!to_dict:s zAcceleratorConfig.to_dictcCs|j||SrY)rpop)rrVrr r r!r=szAcceleratorConfig.poprY)rrrrrrrr rr rrrrrr classmethodrrrr r r r!rsR -     rcsJeZdZdZd fdd Zd deddfdd Zd deefd d Z Z S)LayerWiseDummyOptimizera For Layer-wise optimizers such as GaLoRE optimizer, the optimization step is already done through the post gradient hooks. Therefore the trick is to create a dummy optimizer that can take arbitrary args and kwargs and return a no-op during training. Initial idea from @hiyouga in LLaMA-Factory: https://github.com/hiyouga/LLaMA-Factory/commit/8664262cde3919e10eaecbd66e8c5d356856362e#diff-ebe08ab14496dfb9e06075f0fdd36799ef6d1535cc4dd4715b74c4e3e06fe3ba Ncs2tdd}||_t|gd|ddidS)NrrHgMbP?)r%randnoptimizer_dictrrget)rrr{r dummy_tensorrr r!rLs  z LayerWiseDummyOptimizer.__init__T set_to_nonerxcCdSrYr )rrr r r! zero_gradQz!LayerWiseDummyOptimizer.zero_gradcCrrYr )rclosurer r r!stepTrzLayerWiseDummyOptimizer.steprYT) rrrrrrrr r rrr r rr!rAs  rcs0eZdZdZfddZddZddZZS)LayerWiseDummySchedulera For Layer-wise optimizers such as GaLoRE optimizer, the optimization and scheduling step are already done through the post gradient hooks. Therefore the trick is to create a dummy scheduler that can take arbitrary args and kwargs and return a no-op during training. cs4|d|_tdi|}d}d}t|||dS)NrHrFr ) default_lrrrr)rr{rrN last_epochverboserr r!r`s z LayerWiseDummyScheduler.__init__cCs8|jg}|jdurdd|jjD}tt|}|S)NcSsg|] }dd|jDqS)cSsg|]}|dqS)rHr )rCgroupr r r!rnrz=LayerWiseDummyScheduler.get_lr...)rO)rCrLr r r!rmsz2LayerWiseDummyScheduler.get_lr..)rrNrrprNr)rlrsparam_wise_lrsr r r!get_lrgs   zLayerWiseDummyScheduler.get_lrcCrrY)base_lrsrr r r!_get_closed_form_lrtrz+LayerWiseDummyScheduler._get_closed_form_lr)rrrrrrrrr r rr!rXs   rc Csx|}d}z|r|j||WdS|j||WdSty;}zt|j||dWYd}~dSd}~ww)zIHelper to set RNG state for a specific device type (CUDA, NPU, MLU, MUSA)zDidn't manage to set back the RNG states of the {backend} because of the following error: {exception} This won't yield the same results as if the training had not been interrupted.)backend exceptionN)rrandomset_rng_state_all set_rng_state Exceptionrerrorformat) device_name device_modulecheckpoint_rng_stateis_distributeddevice_state_key err_templaterRr r r!set_rng_state_for_devicexs"r)r)rY)NNrr )r)wrrBrXrr~rrxsysrcollections.abcrr contextlibr dataclassesrr itertoolsrrrtypingr r r rer(r%torch.distributed distributedrr torch.utils.datar rrrtorch.utils.data.distributedrintegrations.deepspeedrtokenization_utils_baserutilsrrrr add_handlerstdoutrorprqrutorch.optim.lr_schedulerr get_loggerrrrr&rRr#r7r>rBrUr[rgrkrr{rrNr rrrrrrdatarrrrrrr!r"r1r8r?rSr\rr0rhrurrrrrr!smdistributed.modelparallel.torch modelparallelrrrrrrrrL Optimizerrrrr r r r!s                48   l *#.X/m"  Z&