Hybrid date time index - Initial Proposal

[ahmed-mahran]

In this PR, a number of time series transformations are (would be) contributed.
Those kind of transformations are basically transformations of the underlying date time index such that:

new_ts = ts.transform is equivalent to new_ts = ts.rebase(ts.index.transform)

new_ts = transform(ts1, ts2) is equivalent to

new_index = transform(ts1.index, ts2.index)
new_ts = ts1.rebase(new_index) + ts2.rebase(new_index)

[ahmed-mahran]

Additions to the public API:

• HybridDateTimeIndex is an implementation of DateTimeIndex that holds a hybrid collection of different types of DateTimeIndex. Indices are assumed to be sorted and disjoint such that for any two consecutive indices i and j: i.last < j.first. This is helpful for transformations on multiple indices. Factory methods for constructing HybridDateTimeIndex:
  • DateTimeIndex.hybrid(indices: Array[DateTimeIndex])
  • DateTimeIndex.hybrid(indices: Array[DateTimeIndex], zone: DateTimeZone)
  • DateTimeIndex.fromString
• DateTimeIndex helper methods
  • millisIterator(): Iterator[Long]
  • zonedDateTimeIterator(): Iterator[ZonedDateTime]
  • insertionLoc methods to find the location at which the given date-time could be inserted. It is the location of the first date-time that is greater than the given date-time. If the given date-time is greater than or equal to the last date-time in the index, the index size is returned. Used in transformations on multiple indices.
  • atZone(zone: ZoneId) adjusts the time zone of the index. Used in transformations on multiple indices.
  • equals uses match ... case instead of casting to avoid class cast exceptions
• Transformations on indices added to DateTimeIndex object:
  • union
  • intersect
  • except
• Transformations added to TimeSeries
  • union multiple multivariate time series of disjoint keys into one multivariate time series by applying union on all time indices and rebasing all univariate time series using the union index.
  • intersect multiple multivariate time series of disjoint keys into one multivariate time series if possible
  • leftJoin merging two multivariate time series of disjoint keys into one multivariate time series keeping instants of the left index only
  • rightJoin merging two multivariate time series of disjoint keys into one multivariate time series keeping instants of the right index only
  • withIndex rebases a time series using a new index

Additions to the private API:

• Generic DateTimeIndex rebaser at TimeSeriesUtils.rebaserGeneric(sourceIndex: DateTimeIndex, targetIndex: DateTimeIndex, defaultValue: Double). Helpful for transformations on multiple indices of different types.
• DateTimeIndexUtils object holds utilities methods of the DateTimeIndex
  • dateTimeIndexOrdering defines an ordering on DateTimeIndex s.t. for two DateTimeIndex x and y, x < y iff x.first < y.first || (x.first == y.first && x.size < y.size)
  • simplify(indices: Array[DateTimeIndex]): Array[DateTimeIndex] merges contiguous indices as possible
  • union unions a list of indices into one DateTimeIndex
  • intersect intersects a list of indices and returns a new index if possible
  • except(index1, index2) creates an index with instants of index1 that are not instants of index2
• TimeSeriesUtils
  • rebaseAndMerge(tss: Array[TimeSeries[K]], newIndex: DateTimeIndex, defaultValue: Double): TimeSeries[K] a utility for rebasing a collection of multivariate time series of disjoint keys and merging them into one multivariate time series

[sryza]

Apologies for the delay in getting to this, Ahmed. I've been traveling and on vacation the last couple weeks, but I will make sure to look at this this week.

[sryza]

I'm wondering if we can break this out into a few smaller changes? E.g. can HybridDateTimeIndex be its own change, the DateTimeIndex operations like union etc. be its own change, and the TimeSeries API additions be their own change? How difficult would it be to separate these out?

Also, should TimeSeriesRDD get the same new methods that are being added to TimeSeries?

[ahmed-mahran]

If it is for readability, I've tried to organize commits so that it is easy to understand the changes going though commits one by one. If it is not doing, I'll break it up then. Please, let me know.

I think yes TimeSeriesRDD should get the same methods as TimeSeries. The Java API as well. I'll submit those, as you recommend, either on the same PR or through a separate one.

[sryza]

Ah, yes, the commit history does look nice. My concern is that it'll take me a non-negligible amount of time to get through the whole patch, and if I have suggested revisions for minor changes, it'll hold the whole thing up. Where, if alternatively, it's broken up into a couple different changes, we can start merging parts of it sooner. Which will save you work rebasing if other things need to get merged in the mean time. Would you be open to submitting all the commits up to 066efe8 as a PR?

[sryza]

Nit: add space after curly brace

[sryza]

I think this could probably be written a little more clearly with scanLeft

[ahmed-mahran]

Great idea 👍

[ahmed-mahran]

I got your point. I've opened a new PR #96 for the HybridDateTimeIndex only.

[ahmed-mahran]

Suggested splits:

• Up to: Add a generic DateTimeIndex rebaser mashin-io@9cb83b1
• Up to: Support union of multiple TimeSeries instances mashin-io@80e5b05
• Up to: Support intersection of multiple TimeSeries instances mashin-io@b137913
• Up to: Add 'withIndex' method to TimeSeries 80c8862

What do you think?

[sryza]

How about the third - i.e. "Support intersection"?

[ahmed-mahran]

I've issued a new PR up to "Support intersection" #100