Deprecation policy

[wjbenfold]

Background

Iris' current deprecation policy is quite conservative:

If you need to make a backwards-incompatible change to a public API that has been included in a release (e.g. deleting a method), then you must first deprecate the old behaviour in at least one release, before removing/updating it in the next major release.

It can be quite a long time between major releases, and as such a deprecation can be around for quite a while before it's actually removed.

For contrast, Numpy's deprecation policy is:

Removal of deprecated functionality:

• shall be done after at least 2 releases assuming the current 6-monthly release cycle; if that changes, there shall be at least 1 year between deprecation and removal.
• shall be listed in the release notes of the release where the removal happened.
• can be done in any minor, but not bugfix, release.

It's worth noting that Numpy consider their own policy to be quite conservative.

We could move to a deprecation policy closer to Numpy's, specifically one that allows us to remove deprecated functionality at minor releases. This would speed up the deprecation process, which keeps the codebase cleaner and thus easier to maintain and use. It also has the consequence of forcing users to update their code to keep working with Iris (which may or may not be desirable).

Questions

• What speed of deprecation policy is right for Iris?
• How does this speed compare to our current stated (and actual) speeds?
• Do we want to align to another libraries' policy, or have our own?

How to implement changes

• Check with community / stakeholders
• Update the Deprecation Policy and Code Maintenance pages in our documentation

[rcomer]

Seems reasonable to me in general: users should probably aim to address deprecation warnings when they first appear, rather than waiting till the break happens.

I would ask that we could partly tie the speed to the Met Office's parallel suite cycle, i.e. functionality should be deprecated for at least one production environment before it is removed. In our group we have code that is imported into more than one project, and those projects don't all get updated to the next environment at the same time. So it's important that the underlying modules work with consecutive environments (preferably without having to write if-loops on the Iris version!)

[jamesp]

Above all, I value the principle of least suprise. As a user I should look at the new version number and have a good sense of what to expect. If I have to make tons of changes every time there is a minor version bump because the API has changed I'm going to get a bit grumpy and start pinning things.

If you believe in semver (I know we don't use semver, we use PEP440, but I believe the spirit is the same).

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Deprecation is (by definition) not backwards compatible. So would suggest that it must occur at a major release according to semver. If we choose a different model but keep the same version numbering scheme we introduce opportunity for user astonishment.

@wjbenfold:

It can be quite a long time between major releases, and as such a deprecation can be around for quite a while before it's actually removed.

If we relax the assumption that it is a long time between major releases then we have more opportunity to make breaking changes, but keep the expected version number contract. We're not numpy in many ways, but notably:

• iris has already made it past v1!
• I would argue our API is not as mature as numpy

EDIT: correct PEP440 reference

[jamesp]

@pp-mo

I think it would mean considering major releases as more frequent and less totally transformative than they have been in the past. It would also mean using more compatibility controls, like e.g. the Future options idea -- which I think we have basically moved away from + there isn't so appetite for this approach in the team anymore, as it gets quite complicated.
But more frequent (and less transformative) 'major' releases isn't necessarily a bad thing.

Yes, this is what I meant above:

If we relax the assumption that it is a long time between major releases then we have more opportunity to make breaking changes, but keep the expected version number contract.

Anyway, pretty sure we're thinking along the same lines here!

[rcomer]

The full "backwards compatibility" of those ideas was the thing that fell apart (and was quietly dropped) when we moved to Dask in Iris 2.0 : The key reason for that was : it would have been hideously complicated to make an Iris version that could work with either biggus or Dask at the users option, which these "rules" required.

Wasn't the Biggus --> Dask move a pretty unusual situation? I think it's fair enough that there can be exceptions but, where practicable there should still be a deprecation pathway. I'm particularly thinking of #4414, because I very much hope that merge, extract, etc would be available as standalone functions for a while before the CubeList disappears.

[pp-mo]

Wasn't the Biggus --> Dask move a pretty unusual situation?

Maybe, but it showed that we simply can't promise to always smooth over the big changes.
Of course we are still doing deprecations where possible. We have just added several for Iris3.2, and we could have removed some things already deprecated, if we had followed the Numpy approach -- but we have not done so at this time.

To be frank, I think we are just less bothered now about avoiding all possible breaking change.
Our previous policy has been heavily driven by MO-internal considerations, and (internal) use of conda is finally relieving a lot of the concerns about that.

[trexfeathers]

I would like to see more frequent major releases, but they might not be well received by a sizeable fraction of our user base - those that use Iris without having consciously embraced scientific Python's fast paced development. Not a blocker, but we should be vigilant to the need for helping people transition when we make a major release.

[jamesp]

The problem is not the major release, the problem is the breaking changes!