Implement hyperparameter optimization mechanism
Implement a script to start hyperparameter optimization. This should have the following features:
-
Parallelization, i.e., run multiple processes at once. - Done. We can now use the joblib launcher, specified in
config.yaml
, launchingconfig.n_jobs
processes synchronously. A disadvantage of the synchronous execution is that it waits until all processes are finished before starting the next batch. This wastes processing time. A workaround is pruning by learning to early stop.
- Done. We can now use the joblib launcher, specified in
-
Evaluate average score over n runs, do not rely on a single run. - Done, each run is now executed at least
sweeper.min_trials_per_param
times, as specified in theconfig.yaml
- Done, each run is now executed at least
-
Use GPU computing (this is not yet working in hyperopt.py
) -
Prune unpromisng runs. E.g. by learning to early stop, i.e., add a parameter that says how many more steps than the so far fastest run may have. I.e, specify a factor x that says stop after x*n_steps_of_fastest_run. Or by using optuna's built-in pruning feature. -
Apply to our RL framework -
Implement mlflow custom logger -
Check how we can define search spaces for parameter lists. - Either this can already be done with list_name.0 and list_name.1, etc. or list_name[0], list_name[1].
- Or we have to get rid of the lists.
-
Enable asyncronous multiprocessing. Currently, the joblib launcher waits until all trials in a batch are executed, so that it wastes time waiting for the longest runs. These longest runs should either be pruned early (see above) or the framework should not wait for them to finish if resources from previously finished trials are free. -
Implement run_hyperopt.sh
shell script. NOTE: Parallelism with a shell script also does not work, it completely messes up the parameter generation because the sweeper does not seem to make new suggestions after a result in a previous process has been obtained. Also, as a consequence, it generates the firstn_jobs
jobs with the same parameters.-
Make sure that each process is run at least sweeper.min_trials_per_param
times. -
Make sure that the trials of the same run are stopped in the same subdirectory, and not like now in a subdirectory named after the starting time of the run.
-