The Polly.Contrib.DuplicateRequestCollapser nuget package provides CacheStampedeResilienceStrategy.
CacheStampedeResilienceStrategy is a Polly policy which prevents duplicate requests executing concurrently.
When duplicate requests (detected by matching key) are placed concurrently through the policy, the policy executes the underlying request delegate only once, and returns the same result value to all concurrent callers.
A primary use case is to prevent request storms when repopulating cached expensive data.
Imagine a high throughput system which caches frequently-used data. The query to create this data at the underlying system is expensive.
In a high-frequency system with 100s or 1000s of requests per second, when a popular cache key expires, it is possible that multiple external requests may arrive simultaneously which would invoke the underlying call to repopulate that cache key.
If the underlying data is expensive/time-consuming to create, multiple parallel requests could potentially mount up, placing additional unnecessary stress on the downstream system.
CacheStampedeResilienceStrategy solves this by collapsing concurrent duplicate executions into a single downstream call. Requests are detected to be duplicates by matching key.
When a request arrives, the policy checks if it has a record of a pending downstream request for the same key. If so, second and subsequent duplicate calls simply block on the same original request. When the answer to that single request arrives, all blocking callers for that key are handed the answer. The cycle of concurrent-blocking for that key is then complete.
Requests are detected to be duplicates by matching key, making the policy is easy to combine with key-based caches.
Bear in mind, the collapser is not re-entrant safe. Do not call the collapser inside itself.
var pipeline = new ResiliencePipelineBuilder()
.AddCacheStampedeResilience(options)
.Build();
// Now, convert it to a v7 policy. Note that it can be converted to both sync and async policies.
ISyncPolicy syncPolicy = pipeline.AsSyncPolicy();
IAsyncPolicy asyncPolicy = pipeline.AsAsyncPolicy();
// Finally, use it in a policy wrap.
ISyncPolicy wrappedPolicy = Policy.Wrap(
syncPolicy,
cachePolicy);
var result = await syncPolicy.WrapAsync(cachePolicy)
.ExecuteAsync(context => GetExpensiveFoo(), new Context("SomeKey"));
Note: A ResilienceContext must be passed to the execution which, when combined with the key strategy (see below) specifies a key. If no key is specified, the policy is a no-op (no request collapsing occurs).
The example above demonstrates a correct pattern when using the default key strategy.
var options = new CacheStampedeResilienceStrategyOptions();
options.KeyStrategy = keyStrategy// optional
options.LockProvider = lockProvider // optional
var pipeline = new ResiliencePipelineBuilder()
.AddCacheStampedeResilience(options)
.Build()
Similar configuration overloads exist for strongly-typed forms CacheStampedeResilienceStrategy<TResult>.
An optional IKeyStrategy can be supplied to control how the collapser key is obtained or generated from the Polly execution ResilienceContext. The IKeyStrategy interface defines a single method string? GetKey(Context context).
The default when IKeyStrategy is not supplied takes ResilienceContext.OperationKey as the key to identify duplicates. This is the same default key as for Polly's CachePolicy.
ILockProvider provides an optional mechanism to vary the locking mechanism used by CacheStampedeResilienceStrategy from the default.
CacheStampedeResilienceStrategy uses a lock internally to accurately identify concurrent duplicate calls.
Note: The lock is not held over slow calls to the underlying system; only for order-of-nanosecond-to-microsecond timings to atomically check or update the internal store of pending downstream calls.
See separate page on locking for more details.
Use PolicyWrap to combine policies. For more on the Polly CachePolicy, see the Polly Cache wiki.
var pipeline = new ResiliencePipelineBuilder()
.AddCacheStampedeResilience(options)
.Build();
// Now, convert it to a v7 policy. Note that it can be converted to both sync and async policies.
ISyncPolicy syncPolicy = pipeline.AsSyncPolicy();
IAsyncPolicy asyncPolicy = pipeline.AsAsyncPolicy();
// Finally, use it in a policy wrap.
ISyncPolicy wrappedPolicy = Policy.Wrap(
syncPolicy,
cachePolicy);
var result = wrappedPolicy
.Execute(context => GetExpensiveFoo(), new Context("SomeKey"));
is the recommended pattern with distributed caches. On cache expiry, both making the underlying expensive call and storing in the distributed cache will occur only once.
var pipeline = new ResiliencePipelineBuilder()
.AddCacheStampedeResilience(options)
.Build();
// Now, convert it to a v7 policy. Note that it can be converted to both sync and async policies.
ISyncPolicy syncPolicy = pipeline.AsSyncPolicy();
IAsyncPolicy asyncPolicy = pipeline.AsAsyncPolicy();
// Finally, use it in a policy wrap.
ISyncPolicy wrappedPolicy = Policy.Wrap(
cachePolicy
syncPolicy);
var result = wrappedPolicy
.Execute(context => GetExpensiveFoo(), new Context("SomeKey"));
is a viable alternative for memory caches.
On cache expiry:
- The collapser policy guarantees the expensive underlying call will only execute once.
- Multiple calls collapsed by the collapser policy will all return quasi-simultaneously, and the result will be placed in the memory cache multiple times. As this is a memory cache, this is not as expensive.
If using both a memory cache and distributed cache:
var doubleCacheWithCollapser = Policy.WrapAsync(memoryCachePolicy, collapserPolicy, distributedCachePolicy);
var result = await doubleCacheWithCollapser.ExecuteAsync(context => GetExpensiveFoo(), new Context("SomeKey"));
When double-caching, take care to consider how both cache policies' TTLs interact in the overall call.
CacheStampedeResilienceStrategy is stateful: it maintains records of current downstream pending calls.
For the policy to function, you must use the same single stateful instance across calls and call sites for which you wish to collapse duplicate concurrent requests. Do not create a new policy instance per call or call site.
CacheStampedeResilienceStrategy is available in the usual four forms for Polly policies:
| Policy | use for |
|---|---|
CacheStampedeResilienceStrategy |
asynchronous Func<ValueTask> calls; or functions returningFunc<ValueTask<TResult>> |
CacheStampedeResilienceStrategy<TResult> |
strongly-typed for synchronous calls returning Func<ValueTask<TResult>> |
A mechanism for collapsing duplicate requests which are not concurrent is otherwise known as a cache ;~). Use CachePolicy.
The code examples here demonstrate typical usage in conjunction with a cache policy. However, CacheStampedeResilienceStrategy could be used in any ResilienceContext. For example, you may have expensive initialization on startup of an app - perhaps to configure some expensive in-memory singleton - which must be performed only once however many incoming requests arrive. CacheStampedeResilienceStrategy could be used in this context too.
CacheStampedeResilienceStrategy supports .Net Standard 2.0 upwards. The package also offers direct tfms for .NET 4.6.1 and .NET 4.7.2
Thanks to @mrmartan for the original idea, and to @jjxtra and @phatcher for deep contributions to thinking. The v0.1.0 implementation is by @reisenberger of the Polly team.