Pausing an experiment

[jfilipe]

Just wanted to get your thoughts on how to pause a running experiment/namespace. For me, pausing would mean that if the primary_unit has already been exposed, subsequent calls would continue to return the same exposure. However if this is the first time we've seen this primary_unit, they would get the default experiment while we're paused.

This would need a database in order to record the primary_units allocation for a given experiment/namespace.

Here is some sample code to illustrate what I mean:

from planout.experiment import SimpleExperiment
from operator import itemgetter

class PauseableNamespace(SimpleNamespace):
  paused = False

  def get_segment(self):
    allocated_segment = super().get_segment()

    # this value can also be hashed to make persistence easier
    primary_unit = itemgetter(*self.primary_unit)(self.inputs))
 
    # assume we have a `find_by` method that will query our db
    recorded_segment = find_by(primary_unit=primary_unit, namespace=self, segment=allocated_segment)
 
    if recorded_segment is None and self.paused:
      # First time we've seen this `primary_unit`. Since namespace is paused ensure default experiment is assigned.
      return -1

    # assume we have a `create` method that will insert a record into our db
    # record allocation
    create(primary_unit=primary_unit, namespace=self, segment=allocated_segment)

    return allocated_segment

Then my concrete namespace classes would inherit from this PauseableNamespace class.

Thoughts?

[deaneckles]

PlanOut is designed to have online, stateless assignments. Diverging from that can make things pretty complicates, especially in analysis, but you can certainly do something like that.