Dacha cache poisoning

[rickybasra]

Describe the bug
Experiencing a cache poisoining issue with the data returned from Edge. We are getting different versions of features returned every few GET requests. It should always show the correct and latest version of the feature but intermittently toggles between an old version.

Which area does this issue belong to?

• FeatureHub Admin Web app
• SDK
• SDK examples
• Documentation
• Other

To Reproduce
Steps to reproduce the behavior:

1. Run 3 pods for each service; edge, mr, dacha2, nats
2. Go to 'FeatureHub Admin Web app'
3. Kill the NATs pods and check they have been recreated
4. Click on 'Toggle a feature'
5. Use the rest api to get the feature numerous times and version and value should always match that in the mr console.

Expected behavior
Each time you retrieve the feature it should give back the correct value and version.

Screenshots
it either returns this payload
{
"id": "2f55cdfd-e3cc-4ff4-9326-5c959d7404e2",
"key": "rb-test-20240726-feature-01",
"l": false,
"version": 16,
"type": "BOOLEAN",
"value": true,
"strategies": []
},
or this
{
"id": "2f55cdfd-e3cc-4ff4-9326-5c959d7404e2",
"key": "rb-test-20240726-feature-01",
"l": false,
"version": 17,
"type": "BOOLEAN",
"value": false,
"strategies": []
},
keeps rotating every few refreshes

Versions

• FeatureHub version [e.g. 1.8.0-RC]
• Browser [chrome]
• Bruno

You can get the version of the FeatureHub container by running docker ps command.
Please include the OS and what version of the OS and Docker you're running.

Additional context
Slack thread discussing the issue
https://anyways-labs.slack.com/archives/C0150T7AF25/p1722508085063969

[rvowles]

So we have ascertained this is likely because the service isn't being destroyed because of a health check failure but there is enough of a lapse so that the dacha instance is losing events. The desire is to add an option which will cause Dacha to dump its cache if it detects a loss of connection to NATs and not reinstate caching until the connection has been re-established.

[rickybasra]

Sounds good @rvowles . Do you have any ETA for when this feature would ready for us to use?

[rvowles]

FWIW, we have some folks with big clusters running on NATs on k8s and they haven't reported this issue, so I am wondering if there is a configuration issue with your setup. Part of the change for this is that it will log the discovered servers on a connection event. Please check when this happens that all of your servers are getting discovered by the jnats client. There may be an interconnection issue in your NATs deployment.

[rvowles]

This has been pushed into the pre-release 1.8.2-rc of FeatureHub now available on DockerHub - could you try it out please? If it is fit for purpose, we'll cut the release and make it generally available. Check the PR for the docs on variants you may wish to try.

[gavinwoolley]

I've pulled 1.8.2-rc and added dacha2.streaming.disconnect-behaviour: use-passthrough to application.properties via the HELM env var setting.
I've confirmed those values are mounted in the running container.
It does not seem as though any of the extra logging information is exposed that you've added