Unexplained rise in storage activity causing higher costs

[laveryn]

We have multiple deployments of a Functions App across several tenants. Since the 1st November we have seen a sudden rise in transactions across all storage accounts for these apps, without any clear cause, including on deployments which are idle. This has cause a large increase in costs. The bug has affected different versions of the app, with different runtime versions, in similar ways

The fall in transactions shows where I have stopped the relevant function app.

The issue seems similar to those described in Azure/Azure-Functions#2231 and in Azure/Azure-Functions#2417, although in our case the majority of transactions are queue transactions.

The apps shown above are v4, .Net 6 (LTS) In-Process, running on Windows, with individual Consumption plans. The problem also occurred with a version of the app in v3. The apps do not use the AzureWebJobsDashboard setting.

[SpyroxQ]

@nzthiago

Dear Azure-Functions team, we really need some help with this issue. We have multiple deployments of a Functions App across several tenants. Two of those tenants are productive in a customer environment. Is their any way to escalate this ticket or do we need to give you more information?

Thank you very much.

[Sajith-TalentzQ]

We face similar issue from Nov 1st - and have logged a support request with Microsoft support

In our case , we have this issue with durable functions app deployment (on its storage queues) - Other function app deployments is fine.

Our Durable Function App deployment is in consumption plan - isolated mode.

[simon-tooley-t4a]

We're experiencing the same issue. It appears to be the Queue Peek Message and GetQueueMetadata calls.

[simon-tooley-t4a]

We've performed some more investigations as well as having a support ticket open with Microsoft. It appears that the increase in queue transactions correlates with the number of durable activities and orchestrators within the function app, something that wasn't happening previously.

If there's a total of 10 activities and orchestrators we are seeing 10 times the number of transactions, compared to only one, or compared to the prior Oct 31st Azure update.

[Sajith-TalentzQ]

Our issue is resolved by Microsoft by running a patch on the servers till they permanently fix the issue.

So for all those who are facing this issue - It is not really your problem. Just contact Microsoft and they will resolve it.

[SpyroxQ]

@Sajith-TalentzQ so actually we can´t open a support ticket because our subscription plan is not high enough.
You can write everything down and continue the process until this error message appears.

Do you got any reference or contact to microsoft for us? I can´t see any way to contact Microsoft directly?

Thank you very much for your help, our problem cant be resolved at the moment

[Sajith-TalentzQ]

Mostly its a generic email is - [email protected] which is of no use to you.

What we did is that - we took a min support ticket - something like a 25$ and then logged it.

There is a workaround though - You can downgrade the storage to V1 from V2 - This will reduce the cost by 1/10th

However this will not reduce the spike in transactions - but will only reduce the cost associated with a cheaper storage.

Otherwise the only option is patch update and then you can file for a refund. ( I am told we are also eligible for a refund which we have filed now )

[pragnagopa]

Thanks for reporting the issue. We have identified that this issue is affecting Function Apps with Durable triggers.

Will transfer the issue to https://github.com/Azure/azure-functions-durable-extension

We are actively working on the fix and plan to roll it out starting this week

[pragnagopa]

Addressed in #2967

Pending roll out