aurora: Hyperparameter Tuning

Hyperparameter Tuning

This example will fine-tune a Neural Network model.

Every model implementing Trainable can be fine-tuned using Aurora's core framework, (NeuralNetwork, LinearRegression, etc.)

Building the Hyperparameter Tuning

The building process goes through the following steps:

1. Generate all possible tunes using the MinMaxParameters
2. Parallelize the tunes if the parallel() method is called, this may speed up the tuning process
3. Evaluate the tunes using the evaluator, in this case, we use a cross validator with 4 folds for the best accuracy
4. Apply the processor function, in this case, we use a neural network applying the tune for the learning rate, epochs, and hidden layers

NOTE: While parallel processing can speed up the hyperparameter tuning process, its effectiveness may be influenced by factors such as the number of available processors, the size of the dataset, and the complexity of the model. Additionally, parallel processing may not always result in better performance and could even introduce new errors or inefficiencies. Therefore, it's important to carefully consider whether parallel processing is appropriate for your specific tuning task and to test its impact on performance before implementing it.

HyperparameterTuning tuning = new HyperparameterTuningBuilder()
  .options(options -> options
    .generateTunes(MinMaxParameters.builder()
      .learningRate(0.01, 0.1, 0.02) // min --> max --> step
      .epochs(1, 1_000) // min --> max
      .layers(1, 2) // min --> max
      .build())
    .parallel() // optional - parallelizes the tuning process on multiple threads
    .evaluator(new CrossValidator(4, new BestAccuracyEvaluator()))) // Choose your evaluator, in this case, we use a cross-validator with 4 folds for the best accuracy
  .processor(tune -> new NeuralNetworkBuilder() // This can be any trainable model, in this case, we use a neural network
    // The tune represents the current processing tune, so the learningRate could be any value from 0.01 to 0.1
    .learningRate(tune.learningRate()) // use the tune for the learning rate
    .epochs(tune.epochs()) // use the tune for the epochs
    .layers(mapper -> mapper
      .inputLayers(inputLayerSize)
      .hiddenLayers(tune.layers()) // use the tune for the hidden layers
      .outputLayers(outputLayerSize))
      .disableStatsPrint()
      .build())
    .inputs(inputs)
    .targets(outputs)
    .build();

After the HyperparameterTuning is built, we can fine-tune for the best results. Our target here is to find the most suitable accuracy.

HyperparameterTuningResult result = tuning.find();

Printing the result:

HyperparameterTuningResult{tune=StandardTune{learningRate=0.01, epochs=253, layers=1}, accuracy=97.40121606840435, score=97.40121606840435}