FEATURE: Using a CDN to scale RPDE

[nickevansuk]

Rough description below (will be reformatted in time)

You can put a CDN such as cloudflare in front of your endpoint and then scale to millions of requests inbound - the spec is fully compatible with that. We've had queries about this already so will write this up in a wiki. It works provided you set your CDN to "pass through and respect cache TTL headers" instead of overriding, and have the cache headers for RPDE pages return e.g 60 minutes, except the cache headers for the last page, which should return 1 minute.

Further notes on how this works:

RPDE is compatible with edge-caching and CDNs, as e.g. 200 data consumers polling at the end of the list will request the same next page, so the first can be cached, and the other 199 served via CDN. When this page contains content, it will have a new next UuuRL, so all 200 data consumers will follow the same next URL, and again the first request can be cached and the other 199 given the same page. Hence because data consumers are kept in step, the polling should scale to a large number of consumers, assuming that a CDN is used, and the cache expiry is set correctly (both on content pages, and on empty pages). This can be added to the next version of the RPDE spec after we have experience of this in action. This should be more efficient and simpler than any webhook based solution, as the CDN is doing the simplest form of HTTP request-based caching, and much simpler to implement than any UDP or broadcast-based solution.

[nickevansuk]

@ldodds for 1.1 keen to include this too

[ldodds]

I don't think this is a thing to go in the specification. It's implementation advice. Might be better to just write a tutorial on it: "How to put a CDN in front of your RPDE endpoint".

If there's specific things that implementers should do, e.g. serve caching headers, then this is worth going into the specfication.