Skip to content

Commit

Permalink
fix: grid hp samping ignored empty nests
Browse files Browse the repository at this point in the history
Previously, when a hyperparameter contained an emtpy "nested"
hyperparameter:

    hyperparameters:
        empty: {}

It was lost during grid sampling, and was never passed to the trial.
This is because the nested hyperparameter isn't considered its own axis
for the grid search, and is only implicitly populated by being on the
path of some inner hyperparameter axis.  However, when there is no inner
hyperparameter axis, we need to detect that and emit a one-point axis
that preserves the empty nested hyperparameter.

Another possible solution would be to convert the empty nested
hyperparameter as a const hyperparameter with an empty map for a value
when unmarshaling the hyperparameters yaml object.  I think no user code
would be affected by which solution is used, since user code will see
the same hyperparameter sample either way.
  • Loading branch information
rb-determined-ai committed Sep 23, 2024
1 parent baf451f commit 1b99512
Show file tree
Hide file tree
Showing 3 changed files with 27 additions and 1 deletion.
7 changes: 7 additions & 0 deletions docs/release-notes/9966-fix-grid.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
:orphan:

**Fixes**

- Previously, during a grid search, if a hyperparameter contained an empty nested hyperparameter
(that is, just an empty map), that hyperparameter would not appear in the hparams passed to the
trial.
7 changes: 6 additions & 1 deletion master/pkg/searcher/grid.go
Original file line number Diff line number Diff line change
Expand Up @@ -260,8 +260,13 @@ func getGridAxes(route []string, h expconf.Hyperparameter) []gridAxis {
return []gridAxis{axis}
case h.RawNestedHyperparameter != nil:
axes := []gridAxis{}
// Use h.Each for deterministic ordering.
nested := expconf.Hyperparameters(*h.RawNestedHyperparameter)
// Make sure empty maps don't disappear after sampling.
if len(nested) == 0 {
axes = append(axes, gridAxis{axisValue{route, map[string]interface{}{}}})
return axes
}
// Use h.Each for deterministic ordering.
nested.Each(func(name string, subparam expconf.HyperparameterV0) {
// make a completely clean copy of route
var subroute []string
Expand Down
14 changes: 14 additions & 0 deletions master/pkg/searcher/grid_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,20 @@ func TestHyperparameterGridMethod(t *testing.T) {
len(getGridAxes([]string{"x"}, expconf.Hyperparameter{RawConstHyperparameter: &constParam})[0]),
1,
)
// Regression test: make sure empty nested hyperparameters don't disappear during sampling.
nestedParam := map[string]expconf.Hyperparameter{
"empty": {RawNestedHyperparameter: &map[string]expconf.Hyperparameter{}},
"full": {RawCategoricalHyperparameter: &catParam},
}
result := getGridAxes([]string{"x"}, expconf.Hyperparameter{RawNestedHyperparameter: &nestedParam})
assert.DeepEqual(t, result, []gridAxis{
[]axisValue{{Route: []string{"x", "empty"}, Value: map[string]interface{}{}}},
[]axisValue{
{Route: []string{"x", "full"}, Value: 1},
{Route: []string{"x", "full"}, Value: 2},
{Route: []string{"x", "full"}, Value: 3},
},
})
}

func TestGrid(t *testing.T) {
Expand Down

0 comments on commit 1b99512

Please sign in to comment.