From e9c82d18f39da989b82a317340a4132a89df8ae3 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Sun, 9 Aug 2020 23:29:01 +0200 Subject: [PATCH] Fix incorrect rounding in tdigest_compute_quantiles(_of) The functions used int64 counter to track the number of values in the centroids, but right before approximating the two centroids we either add or subtract (prev->count / 2.0); and assign the value back to the counter, forcing rounding. That may do nothing (if there's just a single value, as in the centroids on the tails) or add/subtract a bit more than expected. Both cases may produce bogus results. Reported by Matt Watson (sporty81) on tdigest_percentile(), but AFAIC the inverse has roughly the same problem, so fix that too. --- tdigest.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tdigest.c b/tdigest.c index e5d0b70..1190290 100644 --- a/tdigest.c +++ b/tdigest.c @@ -445,7 +445,7 @@ tdigest_compute_quantiles(tdigest_aggstate_t *state, double *result) for (i = 0; i < state->npercentiles; i++) { - int64 count; + double count; double delta; double goal = (state->percentiles[i] * state->count); bool on_the_right; @@ -552,7 +552,7 @@ tdigest_compute_quantiles_of(tdigest_aggstate_t *state, double *result) for (i = 0; i < state->nvalues; i++) { int j; - int64 count; + double count; centroid_t *c = NULL; centroid_t *prev; double value = state->values[i];