Skip to content

Callbacks, metric animation, and auto-saving the model

I just reviewed the code and saw a few problems: Auto-saving the model should occur on three occasions:

  • at the end of the training,
  • whenever a new best model has been found, and
  • when the save_model_freq is met, as specified in main.yaml

The first is implemented in the early_stop callback. This should not be the case, the model should always be saved at the end of the training and not only when early stopping. So a better place for this is probably in main.yaml, after the training has happened. I have now fixed this in the branch 2d-envs and will merge it over to devel.

The second is implemented in EvalCallback2.on_step(). EvalCallback2 is somehow used for the metric animation if I see correctly. This should also not be the case. Instead, there should be a single EvalCallback class that works for both cases, with and without metric animation. In that EvalCallback, there should be the saving of the best model. It may be an option to implement an additional wrapper for the metric animation and an additional wrapper for saving the best model.

It is very important to fix this soon, because it is really quite ugly. Nevertheless, your efforts are well appreciated, @Matin

The third is working fine!