Bug: GRPC sometimes errors

[nplasterer]

Describe the bug

Some users are seeing this error:

Failed to load Inbox: Call to function 'XMTP.listConversations' has been rejected.→ Caused by: io.grpc.StatusException: UNAVAILABLE: Unable to resolve host production.xmtp.network 

Expected behavior

No response

Steps to reproduce the bug

No response

[neekolas]

That's starting to look more like an issue on the nodes. Are you able to reproduce the same error when using grpc.production.xmtp.network?

[neekolas]

I did a little digging in to the metrics from AWS, and there are some spikes of errors coming from the load balancer. These are the ELB_5XXs metric over the past week.

Amazon describes that metric like this, which rules out issues on our nodes. It seems like the Amazon Managed LB is failing on its own (maybe through slow autoscaling or an Amazon incident)

The number of HTTP 5XX server error codes that originate from the load balancer. This count does not include any response codes generated by the targets.

[neekolas]

The metrics from the node are healthy, but that would be consistent with a load balancer error since the requests never even would make it to the node.

[nplasterer]

The android repo does use grpc.production.xmtp.network there is nothing in the android repo that actually chomps the grpc portion of the host.... I looked a little into the grpc android library and I think it's trying to treat grpc as the scheme? When I add https it fails with a no scheme error for https.

Fwiw I was able to reproduce this once. But after restarting my computer and updating android studio I haven't been able to. We do know atleast 3 others are experiencing this on there actual device though.