Allow for sets, filters or collections

[nickevansuk]

Proposal

There are some use-cases where a data consumer only wishes to access a subset of the data for a very large dataset.

Example use-cases include:

• Only sessions starting after a specific date
• Only specific activities
• Only specific sites / venues

The most common case is "after a specific date" URL.

Suggested principles

• The seed URL used to initiate the paging should contain the filter
• Subsiquent next URLs should contain the same filter data
• The approach should be stateless

Example approaches

• Seed URL: /getSessions?filter={startTime: "now"}
• Next URL: /getSessions?afterTimestamp=1453931101&afterId=c15814e5&filterSet={timestampAtTimeOfFirstFilter="1453931101", filterStartTime=1453931101}

   var query;
   if (queryParams.from) {
        query = Session.query().filter(
            (
                (Session.modified == queryParams.afterTimestamp && Session.id > queryParams.afterId) || Session.modified > queryParams.afterTimestamp
            ) && (
                Session.startTime > filterStartTime || Session.modified >= timestampAtTimeOfFirstFilter
            )
        );
   } else {
       query = Session.query();
   }
   return query.sort([Session.modified, Session.id])

Note that timestampAtTimeOfFirstFilter is calculated from max(timestamp) from the source table. filterStartTime is calculated from the current date of the system.