Generic detector tuning

[Tveten]

All detectors in skchange have either a penalty_scale or threshold_scale parameter. They mostly come with decent default values given that the data is standardised with respect to the within-segment mean and standard deviation, and that these quantities are stationary.
Thus, finding good mean and standard deviation estimates, or, correspondingly, good values for the penalty or threshold scales, can often be challenging.

Generic ways of tuning penalty_scale, threshold_scale and possibly other hyperparameters automatically from training data should therefore be developed. There should be both supervised and unsupervised/semi-supervised tuners.

Supervised tuning

In this setting, there are labels for changepoint or anomaly locations available for training/tuning.

Requires:

• Metrics to measure the performance of detections vs. labeled detections. There are different metrics for change and anomaly detection.
• A search method for testing different hyperparameters. Could optuna be useful?

Open questions:

• Can/should some form of cross-validation be used for this?

Unsupervised or semi-supervised tuning

In this setting, there are no labels for changepoint or anomaly locations available for training/tuning. This is often the case in practice.

A possible way of tuning the methods nonetheless is to specify the desired number of detections in the training data, and search for parameters that result in this number of detections.

A common example: There is a training set that is assumed change- or anomaly-free. I.e., there should be no detections. It should be possible to find hyperparameters that give no detections on the training data in a "tight" way. By "tight", I mean that the resulting penalty or threshold is only as conservative as it needs to give no detections, but not more.