Set up and connect a good data visualization system like comet.ml or weightsnbiases.
For the data visualization, MLFlow works well but it is not powerful enough. We need the following requirements:
- Should be cloud-based
- Should be possible to aggregate plots with the same hyperparameters within one graph that displays the average or median and standard deviation or quartiles.
- Zoom in and out in the browser.
- supports multiprocessing
- Can easily be connected to our overall logging framework. That means, if someone wants to add a new methric and / or hyperparameter to log in a custom algorithm, this should be almost no work.
- Has a comfy user interface
Previously we used comet.ml, but there are several reasons to not use it.
- We already have MLFlow to track and visualize experiments.
- It has never worked properly until now
- It requires registration and an API key
- It is one more thing users have to learn to use The good thing about comet.ml is, that it connects directly to mlflow. Therefore, whenever we have the logging with mlflow solved, we also have it solved with comet.ml. There was just this problem with the multiprocessing. This led us to uploading manually with a system call in the custom optuna sweeper implementation or by hand.
Alternatively we may want to use weightsnbiases. Frank uses it a lot successfully.
This issue is the general issue about the decision process which framework to use and the implementation of it.
Edited by Manfred Eppe