USDE for Progressive

[juliev0]

Summary

For Progressive, there are 3 things to consider:

1. : Some fields are only safe to change with Progressive and not PPND - this means that even if the user prefers PPND, it should be done with Progressive (or there will be completely breakage otherwise) - the alternative is to delete and re-create.
2. We still care about all of the data loss fields, which would indicate that we should do "Progressive" rather than "Direct Apply".
3. if the Image changes, that's a good time to do Progressive, since this is when we can most benefit from the Blue/Green aspect of Progressive.

Examples of Case 1:
Apparently, some changes to e.g. a Pipeline's Reduce Vertex through use of spec.vertices.udf.groupBy, or changing isbsvc from one provider to another, could result in broken Pipeline(s) unless:

1. the resource(s) are deleted and then recreated, or
2. Progressive delivery is done

Restructuring of the USDE:

I'm thinking something like this...

dataLoss:                      # case 2: these are the fields we have now
   pipeline:
   interstepbufferservice:
progressiveRequired:  # case 1
   pipeline:
   interstepbufferservice:
progressiveDesired:    # case 3
   pipeline:
   monovertex:

Configurability in numaplane-controller-config.yaml:
If user has selected PPND as their preferred strategy, perhaps they need to indicate if progressive should be used as a backup strategy? (If so, they would use the fields from Case 1 to determine if they needed to do progressive)

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.