Downsampling for graphing

[davidkohn88]

Starting a discussion on some of the elements of #29
Original text copied here:
Is your feature request related to a problem? Please describe.
Sending thousands of points to the client to be graphed causes slowdowns throughout the process, sometimes clients can downsample themselves, but even if they can sending all of those points is a lot of extra work. Using sums/averages/other typical downsampling approaches isn't always a great way of getting fidelity to the visual shape of a graph or expressing the right kind of information.

Describe the solution you'd like
Implementing the Largest Triangle Three Buckets (LTTB) algorithm (and potentially some other related algorithms could be a great place to start: https://github.com/sveinn-steinarsson/flot-downsample, this seems to be the leading downsampling algorithm.

Open questions for discussion
What should the interface for such a function be?
What sort of API would be best for being able to downsample at different rates when zooming into a graph?
Are there examples of good integrations with other libraries that we could provide? Does anyone have experience that would help design a good interface?
Is there a way to create a partial form that can be stored in a continuous aggregate? Are these combinable? What would best practices around using that be?

[davidkohn88]

Is LTTB the best / only algorithm we should be considering? What other algos have people used?

[JLockerman]

For instance, ASAP smoothing seems popular right now, though it might be solving a slightly different problem.

[JLockerman]

A preliminary version was of ASAP was released in 0.2.0 and should be available on Forge at the beginning of next week.

[JLockerman]

TODO: should we call asap return a regularized timeseries instead of normalized?

[davidkohn88]

Interface discussion:
It seems like any interface needs to take a timeseries in, a set of parameters to the downsampling algorithm, ie number of buckets etc. and produce a timeseries out.

1. the timeseries api elements discussed in Timevector Pipeline API Discussion #10 could be a very good way of integrating this.
2. it seems like we might also want some sort of aggregate that produces a downsampled timeseries or some sort of intermediate form that could be converted to a downsampled timeseries at a later point. In this case the WHERE and GROUP BY clauses would define the timeseries that is being downsampled.
   Does anyone have experience with other apis that worked well for this sort of thing?

It seems like the API elements here have a lot in common with interpolation type APIs they're trying to solve basically opposite problems (lack vs overabundance of data) but they might both need to take the same inputs and outputs? Should these discussions be combined?

[davidkohn88]

I'm not sure whether the LTTB algorithm has a good partializable/parallelizable/combinable form, I think it would be good to explore that though. This can serve as a place to put some logic for what that might look like when we're ready to consider it.

[davidkohn88]

What graphing libraries / integrations would be good to think about? (Add replies and also add plus ones if you use a library/tool etc.)

[davidkohn88]

Grafana

[davidkohn88]

Plotly.js

[davidkohn88]

matplotlib

[davidkohn88]

flot

[JLockerman]

In theory if a graphing tool can understand and graph data from a Timescale instance, it should be able to graph downsampled data just as a graph with fewer datapoints. It doesn't necessarily help for more interactive use cases, but does suggest that we can handle simple cases by creating a view on the database that outputs the downsampled or smoothed data, and pointing the graph at that.

[JLockerman]

We now have a PR open for LTTB in #101.