Potential issue with GCP Global HTTPS load balancers

[mjj209]

Issue

On GCP, if you use a global HTTPS load balancer, the reliability isn't as high as most customers require. This can potentially be mitigated by using a Regional TCP Load Balancer. The Toolsmiths team is currently testing this out, and they should have a good data set in another 2-3 weeks.

This applies to all versions of PCF

Context

Global HTTPS load balancers are really only a thing on GCP; AWS doesn't have them.

Expected result

99.999% reliability when making requests to the cf API

Current result

Less than 99.999% reliability, with the majority of issues being of failed_to_connect_to_backend

Steps to Reproduce

Install PCF on GCP, run smoke tests continuously.

Possible Fix

Copy the work we did here: pivotal-cf/toolsmiths-shared-scripts@5282d70

Logs and Screenshots

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[mjj209]

After several weeks of testing, the overall R&D recommendation is to switch to Regional TCP load balancers for the time being. I'm not sure if you want to go ahead up update the PCF-Pipelines, given that R&D may change the recommendation in the future. I'll leave this decision up to you. Would be happy to help you either way.

R&D may update the recommendation pending a test with modified version of the global load balancers, the test is going to happen in the next month, but rollout to the rest of the GCP infrastructure may be Q3 2018 or later.