o
hK @ s d Z ddlZddlZddlZddlmZ ddlmZmZ ddlZ ddl
m Z ddl m
Z
mZmZmZ ddlmZ dd lmZ eeZeG d
d d ZG d d
d
ZeG dd deZG dd dZG dd deZ G dd deZ G dd deZ G dd deZ G dd deeZ dS ) zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)OptionalUnion)tqdm )HPSearchBackendIntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc @ sv e Zd ZU dZdZee ed< dZe ed< dZ
e ed< dZ e ed< dZ e ed < dZ
e ed
< dZee ed < dZe ed < dZe ed
< dZeed< dZeeeef ed< dZee ed< dZee ed< dZee ed< dZeed< dZeed< dZ eed< dZ ee ed< dZ eee eee ef f ed< dZ ed ed< d d Z!d efd d Z"e#d efd!d"Z$d#d$ Z%d%d& Z&dS )' TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`List[Dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`List[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc C s | j d u rg | _ | jd u ri | _d S t| jtrd S i }| jD ]6}t|ts/tdt| |jj}||v rOt|| t sE|| g||< ||
| q | ||< q || _d S )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r"
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr" callbackname r0 q/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/trainer_callback.py
__post_init__u s&
zTrainerState.__post_init__ json_pathc C sX t jt| dddd }t|ddd}|| W d dS 1 s%w Y dS ) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r- r3 json_stringfr0 r0 r1 save_to_json s "zTrainerState.save_to_jsonc C sH t |dd }| }W d n1 sw Y | di t|S )z3Create an instance from the content of `json_path`.r9 r: Nr0 )r@ readr< loads)clsr3 rC textr0 r0 r1 load_from_json s
zTrainerState.load_from_jsonc C sN dD ]"}t || d}|dur$|dk rt|| }t| | d| qdS )z
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r evalsave_stepsNr )getattrmathceilsetattr)r- argsr step_kind num_stepsr0 r0 r1
compute_steps s zTrainerState.compute_stepsc C s |j dur|jdur| |j| _d| _|dur.ddlm} |jtjkr'|j n|}||| _|| _
|| _ | | _ |
| _
dS )zI
Stores the initial training references needed in `self`
Nr ) hp_params)hp_name_trialr r transformers.integrationsrU hp_search_backendr SIGOPT assignmentsr r r r )r- trainerr r trialrU r[ r0 r0 r1 init_training_references s
z%TrainerState.init_training_references)'r)
__module__ __qualname____doc__r r float__annotations__r intr r r r r r r r r r* r$ strr r r r boolr r r r r r" r2 rD classmethodrI rT r^ r0 r0 r0 r1 r
# s8
;
r
c @ s* e Zd ZdZdefddZedd ZdS )r% aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc C s t d)NzInternal class that just calls the list of callbacks in order.c C sj g | _ |D ]}| | q|| _|| _|| _|| _d | _d | _tdd | j D s3t
d| j d S d S )Nc s s | ]}t |tV qd S N)r# DefaultFlowCallback.0cbr0 r0 r1 s z+CallbackHandler.__init__..zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r- r r r r r r r0 r0 r1 __init__ s zCallbackHandler.__init__c C sh t |tr| n|}t |tr|n|j}|dd | jD v r,td| dd | j | j| d S )Nc S s g | ]}|j qS r0 )r( )r cr0 r0 r1
s z0CallbackHandler.add_callback..zYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r# r' r( r r r r r+ )r- r. r cb_classr0 r0 r1 r s
z CallbackHandler.add_callbackc C sd t |tr | jD ]}t ||r| j| | S qd S | jD ]}||kr/| j| | S q d S r r# r' r remover- r. r r0 r0 r1 pop_callback s
z CallbackHandler.pop_callbackc C sF t |tr| jD ]}t ||r| j| d S qd S | j| d S r r r r0 r0 r1 remove_callback s
z CallbackHandler.remove_callbackc C s d dd | jD S )Nr7 c s s | ]}|j jV qd S r )r( r) r r0 r0 r1 r s z0CallbackHandler.callback_list..)joinr rj r0 r0 r1 r s z CallbackHandler.callback_listrQ r, r~ c C | d|||S )Nr
call_eventr- rQ r, r~ r0 r0 r1 r zCallbackHandler.on_init_endc C d|_ | d|||S )NFr )rr r r r0 r0 r1 r z CallbackHandler.on_train_beginc C r )Nr r r r0 r0 r1 r r z CallbackHandler.on_train_endc C r )NFr )rs r r r0 r0 r1 r r z CallbackHandler.on_epoch_beginc C r )Nr r r r0 r0 r1 r r z CallbackHandler.on_epoch_endc C s" d|_ d|_d|_| d|||S )NFr )rv ru rt r r r0 r0 r1 r s z CallbackHandler.on_step_beginc C r )Nr r r r0 r0 r1 r r z%CallbackHandler.on_pre_optimizer_stepc C r )Nr r r r0 r0 r1 r r z!CallbackHandler.on_optimizer_stepc C r )Nr r r r0 r0 r1 r r z CallbackHandler.on_substep_endc C r )Nr r r r0 r0 r1 r r zCallbackHandler.on_step_endc C d|_ | jd||||dS )NFr r )ru r r- rQ r, r~ r r0 r0 r1 r zCallbackHandler.on_evaluatec C s | j d||||dS )Nr r r r r0 r0 r1 r s zCallbackHandler.on_predictc C r )NFr )rt r r r0 r0 r1 r r zCallbackHandler.on_savec C r )NFr )logs)rv r )r- rQ r, r~ r r0 r0 r1 r # r zCallbackHandler.on_logc C r )Nr r r r0 r0 r1 r ' r z"CallbackHandler.on_prediction_stepc
K sP | j D ]"}t|||||f| j| j| j| j| j| jd|}|d ur%|}q|S )N)r r r r r r )r rM r r r r r r )r- eventrQ r, r~ r r. resultr0 r0 r1 r * s&
zCallbackHandler.call_eventN) r) r_ r` ra r r r r propertyr r r
rq r r r r r r r r r r r r r r r r r0 r0 r0 r1 r s0
r c @ s< e Zd ZdZdededefddZdededefddZd S )
r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rQ r, r~ c K s |j dkr |jr d|_|jtjkr |j |j dkr d|_|jtjkr3|j |j dkr3|j |j kr3d|_
|j t jkrI|j
dkrI|j |j
dkrId|_|j |jkr[d|_|j t jkr[d|_|S )Nr Tr )r logging_first_steprv logging_strategyr STEPSr
eval_strategyr
eval_delayru
save_strategyr r rt r rr r r0 r0 r1 r C s"
z DefaultFlowCallback.on_step_endc K sF |j tjkr d|_|jtjkr|j|jkrd|_|jt jkr!d|_
|S )NT) r r EPOCHrv r r r ru r r rt r r0 r0 r1 r c s z DefaultFlowCallback.on_epoch_endN) r) r_ r` ra r r
rq r r r0 r0 r0 r1 r > s r c @ s\ e Zd ZdZddefddZdd Zdd Zdd d Zd
d Z dd Z
dddZ dd Z d
S )ProgressCallbackz
A [`TrainerCallback`] that displays the progress of training or evaluation.
You can modify `max_str_len` to control how long strings are truncated when logging.
d max_str_lenc C s d| _ d| _|| _dS )a!
Initialize the callback with optional max_str_len parameter to control string truncation length.
Args:
max_str_len (`int`):
Maximum length of strings to display in logs.
Longer strings will be truncated with a message.
N) training_barprediction_barr )r- r r0 r0 r1 r y s
zProgressCallback.__init__c K s |j r t|jdd| _d| _d S )NT)total
dynamic_ncolsr )r r r r current_stepr r0 r0 r1 r s
z ProgressCallback.on_train_beginc K s* |j r| j|j| j |j| _d S d S r )r r updater r r r0 r0 r1 r s z ProgressCallback.on_step_endNc K sJ |j r!t|r#| jd u rtt|| jd u dd| _| jd d S d S d S )NT)r leaver r )r r
r r lenr r )r- rQ r, r~ r r r0 r0 r1 r s
z#ProgressCallback.on_prediction_stepc K ( |j r| jd ur
| j d | _d S d S r r r closer r0 r0 r1 r
z ProgressCallback.on_evaluatec K r r r r r0 r0 r1 r r zProgressCallback.on_predictc
K s |j rO| jd urQi }| D ]#\}}t|tr-t|| jkr-dt| d| j d||< q|||< q|dd } d|v rEt|d d|d< | j t| d S d S d S )Nz%[String too long to display, length: z > z/. Consider increasing `max_str_len` if needed.]r r r4 )
r r rl r# re r r poproundrA )
r- rQ r, r~ r r shallow_logsrn ro _r0 r0 r1 r s
zProgressCallback.on_logc K s |j r
| j d | _d S d S r )r r r r r0 r0 r1 r s
z ProgressCallback.on_train_end)r r )
r) r_ r` ra rd r r r r r r r r r0 r0 r0 r1 r s s
r c @ s e Zd ZdZdddZdS )PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc K s" | dd }|jrt| d S d S )Nr )r r print)r- rQ r, r~ r r r r0 r0 r1 r s zPrinterCallback.on_logr )r) r_ r` ra r r0 r0 r0 r1 r s r c @ sN e Zd ZdZddedee fddZdd Zd
d Z d d
Z
de fddZ dS )EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
r early_stopping_patienceearly_stopping_thresholdc C s || _ || _d| _d S )Nr r r early_stopping_patience_counter)r- r r r0 r0 r1 r s
z EarlyStoppingCallback.__init__c C sX |j rtjntj}|jd u s |||jr#t||j | jkr#d| _d S | jd7 _d S )Nr r )greater_is_betternpgreaterlessr absr r )r- rQ r, r~ metric_valueoperatorr0 r0 r1 check_metric_value s
z(EarlyStoppingCallback.check_metric_valuec K s: |j std |jd usJ d|jtjksJ dd S )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr r NOr r0 r0 r1 r s z$EarlyStoppingCallback.on_train_beginc K sl |j }|ds
d| }||}|d u r!td| d d S | |||| | j| jkr4d|_d S d S )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) r
startswithgetr r r r r rr )r- rQ r, r~ r r metric_to_checkr r0 r0 r1 r s
z!EarlyStoppingCallback.on_evaluaterh c C s | j | jdd| jidS )N)r r r r} r rj r0 r0 r1 r, s zEarlyStoppingCallback.stateN)r r )
r) r_ r` ra rd r rb r r r r r$ r, r0 r0 r0 r1 r s
r )!ra r> r< rN r typingr r numpyr tqdm.autor
trainer_utilsr r r r
training_argsr utilsr
get_loggerr) r r
r% rq r! r r r r r r0 r0 r0 r1 s4
,= 5J