Skip to content

Commit

Permalink
Fix incorrect rounding in tdigest_compute_quantiles(_of)
Browse files Browse the repository at this point in the history
The functions used int64 counter to track the number of values in the
centroids, but right before approximating the two centroids we either
add or subtract

    (prev->count / 2.0);

and assign the value back to the counter, forcing rounding. That may
do nothing (if there's just a single value, as in the centroids on the
tails) or add/subtract a bit more than expected. Both cases may produce
bogus results.

Reported by Matt Watson (sporty81) on tdigest_percentile(), but AFAIC
the inverse has roughly the same problem, so fix that too.
  • Loading branch information
tvondra committed Aug 9, 2020
1 parent 5d666b8 commit e9c82d1
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions tdigest.c
Original file line number Diff line number Diff line change
Expand Up @@ -445,7 +445,7 @@ tdigest_compute_quantiles(tdigest_aggstate_t *state, double *result)

for (i = 0; i < state->npercentiles; i++)
{
int64 count;
double count;
double delta;
double goal = (state->percentiles[i] * state->count);
bool on_the_right;
Expand Down Expand Up @@ -552,7 +552,7 @@ tdigest_compute_quantiles_of(tdigest_aggstate_t *state, double *result)
for (i = 0; i < state->nvalues; i++)
{
int j;
int64 count;
double count;
centroid_t *c = NULL;
centroid_t *prev;
double value = state->values[i];
Expand Down

0 comments on commit e9c82d1

Please sign in to comment.