Neos Backend Services Latency and Rate Limit Issues

[stiefeljackal]

Describe the bug?

Based on my logs from June 7th to June 11th as well as the rise of syncing issues, it appears that the backend services from api.neos.com (as well as from cloudx.azurewebsites.net) are experiencing issues where it will respond with 429 Too Many Requests errors. As of now, it appears the following areas within the service are being affected:

• Record Syncing
• Session Listings and Joining Sessions (/api/sessions)
• Calls to Fetch Contact Statuses (/api/users/U-MyAccount/friends)

Relevant issues

None reported.

To Reproduce

Sessions and Friends 429 Errors

Startup your client, wait, and open your logs for the current session. You should see Server Error messages for the above mentioned endpoints (/api/sessions and /api/users/U-MyAccount/friends). Within the log, open any of the URLs in your browser and perform a regression testing of performing a handful of GET requests against those URLs. From the testing, you should notice latency of the response and the occasional 429 error.

Record Sync

Hold a world build session and start saving imported assets in your inventory. From this, a stuck sync should occur. This happened during CJ 211.

---

Unfortunately, these are the best steps to reproduce the issue since this is mainly a backend issue and not a client issue. Logs from the backend services will need to be reviewed by the development team to see what is going on.

Expected behavior

• Records should be mostly syncing correctly assuming that there are no other issues such as conflicting versions.
• The /api/sessions endpoint should not be failing at client startup and throughout the duration of the session.
• The /api/users/U-MyAccount/friends endpoint should not be failing at client startup and through the duration of the session.

Log Files

JACKAL_MK2 - 2022.1.28.1310 - 2023-06-10 23_49_32.zip

Pay attention to Server Error running GET request to... messages in logs.

Screenshots

No response

How often does it happen?

Always

Does the bug persist after restarting Neos?

Yes

Neos Version Number

2022.1.28.13XX

What Platforms does this occur on?

Windows, Linux

Link to Reproduction Item/World

No response

Did this work before?

Yes

If it worked before, on which build?

2022.1.28.13XX

Additional context

Others may add to this report as this problem is slowly starting to become a real issue.

Reporters

U-StiefelJackal (Discord: Stiefel Jackal | stifeljackal)

Will include others when more findings are submitted.

[ginjake]

I had a similar situation myself.
I tried the following without success and was having trouble
Delete the mod
Restart Neos (steam version)
Start Neos (StandAlone version)
Reboot PC
Delete Cache and database.

Finally, I rebooted my Wifi router at home and it worked!

I don't understand English, so I use deepl to translate!

[stiefeljackal]

I had a similar situation myself. I tried the following without success and was having trouble Delete the mod Restart Neos (steam version) Start Neos (StandAlone version) Reboot PC Delete Cache and database.

Finally, I rebooted my Wifi router at home and it worked!

I don't understand English, so I use deepl to translate!

This is something that is on the backend services side (api.neos.com and others) and not a local issue. Mainly, the issue is with CosmosDB when the time this issue was submitted. @bontebok from QC is monitoring the situation, but it appears the issue is not too bad at the moment. You still get the occasional 429 or 404 from CosmosDB still.

[kazu0617]

I think this problem based RU/s setting in azure, it written in Microsoft Document.
https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-request-rate-too-large?tabs=resource-specific

[stiefeljackal]

I think this problem based RU/s setting in azure, it written in Microsoft Document.
https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-request-rate-too-large?tabs=resource-specific

That’s correct. Please see the QC channel in the Neos Volunteer Discord server for the full history as @bontebok and I are monitoring this to the best of our abilities.